Fork me on GitHub
#clojure-europe
<
2022-02-10
>
otfrom08:02:20

today's question - is pmap good enough and when isn't it?

thomas08:02:29

I thought pmap was ood enough for long(ish) running taks. but no evidence as such

vemv10:02:19

I love pmap - it's smarter than it might seem! Since it caps threads to relative CPU count, and those threads are reused via an internal pool. I use a dedicated thread pool when I want a pool that isn't global and that is exclusive to some sort of 'module' (e.g. stuartsierra Component). If you have n Components each with their own pools, stopping those components will also stop pools in dependency order. So the tasks will also conclude in an order that makes sense. This is a lot simpler than creating some sort of shutdown priority order by hand

❤️ 1
otfrom10:02:33

my use case is processing multiple tech.ml.datasets in a batch, so I'm less worried about component-y things

orestis11:02:16

I've used claypoole for more control

Jakub Holý (HolyJak)20:02:12

Clojure Applied by @U064X3EF3 has a great chapter about parallelism

otfrom07:02:45

Thx. My pmap chunks seem large enough

dharrigan08:02:31

Good Morning!

genRaiy08:02:41

Good morning

🐕 2
dharrigan09:02:44

Shall I start taking picture of my puppy too? He's only 3 days into his new family...

🐶 4
dharrigan10:02:10

May I introduce you to, Sir Biscuit The First.

agigao13:02:48

Lovely <3 And creative, descriptive name :D

genRaiy15:02:28

Yes - please post frequently!!

otfrom09:02:56

yes, yes you should

slipset10:02:54

I was told about a “startup-idea” thingy by a colleague yesterday. So the deal is for europeans to keep an eye on what’s happening startup/scaleup-wise in the US, and launch companies in Europe doing the exact same thing. So when the US startup/scaleup decides it wants to move into Europe, the bet is that they’ll just buy your copy. Easy peasy

javahippie10:02:39

The “Samwer Brothers” in Germany did that with some startups and made a ton of money

💯 1
javahippie10:02:48

afaik they tried that with Zalando, too, but the copied startup, Zappos, refused to buy, so they had to become successful on their own

jasonbell10:02:46

Early Just-Eat employees did this, setup competitors around the world (some while working at JE) and the made £££ when JE acquired them all.

genRaiy15:02:31

next level domain parking 🙂

😂 1
Mno12:02:04

morning!

otfrom14:02:57

I realise now why I keep coming back to core.async. Being able to go from into or transduce with a xf chain to pipeline with the same chain and get massive parallelism is just so powerful

Ben Hammond16:02:31

you dont HAVE to use core.async to get parallelism on transducers though

Ben Hammond16:02:09

core.async is brilliant for halt-the-line style backpressure

Ben Hammond16:02:23

but that's about it (imho obvs)

Ben Hammond16:02:08

core.async always used to be a nightmare to troubleshoot: is that still the case?

1
otfrom16:02:45

I suppose it depends. I don't think I've written a go block myself, so when I get something that blows up I'm just able to move it into something like into/transduce/sequence and play with it there.

otfrom16:02:15

I'm clearly missing something if transducers give me parallelism though. I'd love something as easy to use as pipeline

Ben Hammond16:02:02

actually I think theres a bug in that code around the end-case for the lagging transducer fixed in https://github.com/hammonba/clojure-attic/blob/master/src/clj/attic/parallelising_map.clj

otfrom16:02:37

so that feels more like the kind of thing that claypoole/map does (but fitting in with transducers better) than being able to pass in a (comp (map ,,,) (map ,,,) (mapcat ,,,) (filter ,,,)) which async/pipeline allows me to do, which is useful, but different

Ben Hammond16:02:12

swings and roundabouts really

Ben Hammond16:02:30

I remember the painful experiences of core.async

Ben Hammond16:02:41

more than the others

otfrom16:02:01

I think I found the subset that worked for me. Just feels weird setting up the machinery, then pushing the data through, then getting the result (from a channel you set up ages before) rather than the usual add a bit, add a bit, add a bit that ->> and (comp ...) allow

otfrom16:02:18

so it tends to be a thing I do last when I want to light up all the cores

Ben Hammond16:02:12

wormholes in space

mccraigmccraig16:02:18

we do similar concurrency stuff with manifold @U0525KG62 - it's not hard to debug, but only because we added a layer over raw streams which always propagates errors when a reduce finally happens. prior to that debugging was painful

otfrom17:02:11

@U0524B4UW looks like I'd need to use the manifold.stream versions of map/filter/mapcat et al?

mccraigmccraig17:02:26

yeah, exactly - but, like core.async iirc, raw manifold streams don't propagate errors in any sane way, so we use this layer on top, which catches any errors in map/filter etc and propagates them back to you when you reduce or take! - https://gist.github.com/mccraigmccraig/e5b9a85441db782f6debd1e8dc2697f9

Jakub Holý (HolyJak)20:02:45

Making sure that errors in core.async don't get lost is quite feasible, with little discipline https://blog.jakubholy.net/2019/core-async-error-handling/ What worries me more is making sure that all channels that need to be closed get closed...

👍 1
👀 1
lread14:02:56

Good morning!