I thought pmap was ood enough for long(ish) running taks. but no evidence as such

I love pmap - it's smarter than it might seem! Since it caps threads to relative CPU count, and those threads are reused via an internal pool. I use a dedicated thread pool when I want a pool that isn't global and that is exclusive to some sort of 'module' (e.g. stuartsierra Component). If you have n Components each with their own pools, stopping those components will also stop pools in dependency order. So the tasks will also conclude in an order that makes sense. This is a lot simpler than creating some sort of shutdown priority order by hand

my use case is processing multiple tech.ml.datasets in a batch, so I'm less worried about component-y things

I've used claypoole for more control

Clojure Applied by @U064X3EF3 has a great chapter about parallelism

Thx. My pmap chunks seem large enough

May I introduce you to, Sir Biscuit The First.

Lovely <3 And creative, descriptive name :D

Yes - please post frequently!!

The “Samwer Brothers” in Germany did that with some startups and made a ton of money

afaik they tried that with Zalando, too, but the copied startup, Zappos, refused to buy, so they had to become successful on their own

https://www.forbes.com/sites/ryanmac/2014/07/31/samwer-brothers-billionaires-rocket-internet-ipo/?sh=6b8d29c58ac2

Early Just-Eat employees did this, setup competitors around the world (some while working at JE) and the made £££ when JE acquired them all.

next level domain parking 🙂

you dont HAVE to use core.async to get parallelism on transducers though

core.async is brilliant for halt-the-line style backpressure

but that's about it (imho obvs)

core.async always used to be a nightmare to troubleshoot: is that still the case?

I suppose it depends. I don't think I've written a go block myself, so when I get something that blows up I'm just able to move it into something like into/transduce/sequence and play with it there.

I'm clearly missing something if transducers give me parallelism though. I'd love something as easy to use as pipeline

(ahem) https://www.juxt.pro/blog/multithreading-transducers

actually I think theres a bug in that code around the end-case for the lagging transducer fixed in https://github.com/hammonba/clojure-attic/blob/master/src/clj/attic/parallelising_map.clj

so that feels more like the kind of thing that claypoole/map does (but fitting in with transducers better) than being able to pass in a (comp (map ,,,) (map ,,,) (mapcat ,,,) (filter ,,,)) which async/pipeline allows me to do, which is useful, but different

swings and roundabouts really

I remember the painful experiences of core.async

more than the others

I think I found the subset that worked for me. Just feels weird setting up the machinery, then pushing the data through, then getting the result (from a channel you set up ages before) rather than the usual add a bit, add a bit, add a bit that ->> and (comp ...) allow

so it tends to be a thing I do last when I want to light up all the cores

wormholes in space

we do similar concurrency stuff with manifold @U0525KG62 - it's not hard to debug, but only because we added a layer over raw streams which always propagates errors when a reduce finally happens. prior to that debugging was painful

@U0524B4UW looks like I'd need to use the manifold.stream versions of map/filter/mapcat et al?

yeah, exactly - but, like core.async iirc, raw manifold streams don't propagate errors in any sane way, so we use this layer on top, which catches any errors in map/filter etc and propagates them back to you when you reduce or take! - https://gist.github.com/mccraigmccraig/e5b9a85441db782f6debd1e8dc2697f9

Making sure that errors in core.async don't get lost is quite feasible, with little discipline https://blog.jakubholy.net/2019/core-async-error-handling/ What worries me more is making sure that all channels that need to be closed get closed...