Fork me on GitHub
#core-async
<
2022-09-05
>
frankitox16:09:17

I'm using pipeline-blocking to upload some files, I would like to stop the pipeline if any file fails upload. How should I do this?

frankitox16:09:54

I'm thinking of closing the from channel from within xf. Does that make sense?

Ben Sless17:09:34

Do you want it to abort ongoing uploads, too?

frankitox17:09:16

Well, ideally, yes. But I'm happy with just skipping new uploads

Ben Sless17:09:57

you can close the from channel in the xform but it would be cleaner to do it at the process which consumes from to

frankitox17:09:11

oh, nice, yeah it may be a bit cleaner :thinking_face:

frankitox17:09:56

thanks for the reassurance on the approach, I was not sure if I was doing something horrific

Ben Sless18:09:31

You could just use a thread pool, by the way

frankitox20:09:49

yeah, I think I'll switch to those, core.async seems overkill for this

Ben Sless04:09:06

I like core async, it's useful, but unless I need CSP I don't reach for it. You could implement a worker pool with cancelation and other mechanisms, but that would just be reinventing the wheel. Thread pools exist already

frankitox13:09:49

well, those thread pools aren't as friendly as I thought. throttling to a max of n uploader threads is not a straightforward task. If I submit more than n files, submit throws. This is where things get ugly

Ben Sless13:09:06

What about using a blocking queue in the executor service constructor?

frankitox13:09:32

:thinking_face: will try later, I used SynchronousQueue as that's ultimately what core.async uses