Fork me on GitHub

I'm using pipeline-blocking to upload some files, I would like to stop the pipeline if any file fails upload. How should I do this?


I'm thinking of closing the from channel from within xf. Does that make sense?

Ben Sless17:09:34

Do you want it to abort ongoing uploads, too?


Well, ideally, yes. But I'm happy with just skipping new uploads

Ben Sless17:09:57

you can close the from channel in the xform but it would be cleaner to do it at the process which consumes from to


oh, nice, yeah it may be a bit cleaner :thinking_face:


thanks for the reassurance on the approach, I was not sure if I was doing something horrific

Ben Sless18:09:31

You could just use a thread pool, by the way


yeah, I think I'll switch to those, core.async seems overkill for this

Ben Sless04:09:06

I like core async, it's useful, but unless I need CSP I don't reach for it. You could implement a worker pool with cancelation and other mechanisms, but that would just be reinventing the wheel. Thread pools exist already


well, those thread pools aren't as friendly as I thought. throttling to a max of n uploader threads is not a straightforward task. If I submit more than n files, submit throws. This is where things get ugly

Ben Sless13:09:06

What about using a blocking queue in the executor service constructor?


:thinking_face: will try later, I used SynchronousQueue as that's ultimately what core.async uses