Fork me on GitHub

today's question - is pmap good enough and when isn't it?


I thought pmap was ood enough for long(ish) running taks. but no evidence as such


I love pmap - it's smarter than it might seem! Since it caps threads to relative CPU count, and those threads are reused via an internal pool. I use a dedicated thread pool when I want a pool that isn't global and that is exclusive to some sort of 'module' (e.g. stuartsierra Component). If you have n Components each with their own pools, stopping those components will also stop pools in dependency order. So the tasks will also conclude in an order that makes sense. This is a lot simpler than creating some sort of shutdown priority order by hand

❤️ 1

my use case is processing multiple in a batch, so I'm less worried about component-y things


I've used claypoole for more control

Jakub Holý (HolyJak)20:02:12

Clojure Applied by @U064X3EF3 has a great chapter about parallelism


Thx. My pmap chunks seem large enough


Good Morning!


Good morning

🐕 2

Shall I start taking picture of my puppy too? He's only 3 days into his new family...

🐶 4

May I introduce you to, Sir Biscuit The First.


Lovely <3 And creative, descriptive name :D


Yes - please post frequently!!


yes, yes you should


I was told about a “startup-idea” thingy by a colleague yesterday. So the deal is for europeans to keep an eye on what’s happening startup/scaleup-wise in the US, and launch companies in Europe doing the exact same thing. So when the US startup/scaleup decides it wants to move into Europe, the bet is that they’ll just buy your copy. Easy peasy


The “Samwer Brothers” in Germany did that with some startups and made a ton of money

💯 1

afaik they tried that with Zalando, too, but the copied startup, Zappos, refused to buy, so they had to become successful on their own


Early Just-Eat employees did this, setup competitors around the world (some while working at JE) and the made £££ when JE acquired them all.


next level domain parking 🙂

😂 1



I realise now why I keep coming back to core.async. Being able to go from into or transduce with a xf chain to pipeline with the same chain and get massive parallelism is just so powerful

Ben Hammond16:02:31

you dont HAVE to use core.async to get parallelism on transducers though

Ben Hammond16:02:09

core.async is brilliant for halt-the-line style backpressure

Ben Hammond16:02:23

but that's about it (imho obvs)

Ben Hammond16:02:08

core.async always used to be a nightmare to troubleshoot: is that still the case?


I suppose it depends. I don't think I've written a go block myself, so when I get something that blows up I'm just able to move it into something like into/transduce/sequence and play with it there.


I'm clearly missing something if transducers give me parallelism though. I'd love something as easy to use as pipeline

Ben Hammond16:02:02

actually I think theres a bug in that code around the end-case for the lagging transducer fixed in


so that feels more like the kind of thing that claypoole/map does (but fitting in with transducers better) than being able to pass in a (comp (map ,,,) (map ,,,) (mapcat ,,,) (filter ,,,)) which async/pipeline allows me to do, which is useful, but different

Ben Hammond16:02:12

swings and roundabouts really

Ben Hammond16:02:30

I remember the painful experiences of core.async

Ben Hammond16:02:41

more than the others


I think I found the subset that worked for me. Just feels weird setting up the machinery, then pushing the data through, then getting the result (from a channel you set up ages before) rather than the usual add a bit, add a bit, add a bit that ->> and (comp ...) allow


so it tends to be a thing I do last when I want to light up all the cores

Ben Hammond16:02:12

wormholes in space


we do similar concurrency stuff with manifold @U0525KG62 - it's not hard to debug, but only because we added a layer over raw streams which always propagates errors when a reduce finally happens. prior to that debugging was painful


@U0524B4UW looks like I'd need to use the versions of map/filter/mapcat et al?


yeah, exactly - but, like core.async iirc, raw manifold streams don't propagate errors in any sane way, so we use this layer on top, which catches any errors in map/filter etc and propagates them back to you when you reduce or take! -

Jakub Holý (HolyJak)20:02:45

Making sure that errors in core.async don't get lost is quite feasible, with little discipline What worries me more is making sure that all channels that need to be closed get closed...

👍 1
👀 1

Good morning!