This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2018-01-12
Channels
- # arachne (1)
- # aws (2)
- # beginners (123)
- # boot (22)
- # boot-dev (8)
- # chestnut (3)
- # cider (38)
- # clara (36)
- # cljs-dev (148)
- # clojars (2)
- # clojure (76)
- # clojure-austin (2)
- # clojure-greece (1)
- # clojure-italy (6)
- # clojure-russia (5)
- # clojure-spec (8)
- # clojure-uk (65)
- # clojurescript (45)
- # core-async (38)
- # cursive (9)
- # data-science (5)
- # datomic (28)
- # docs (1)
- # emacs (2)
- # fulcro (34)
- # hoplon (18)
- # jobs-discuss (7)
- # keechma (8)
- # lumo (5)
- # om (3)
- # onyx (31)
- # parinfer (1)
- # pedestal (1)
- # re-frame (20)
- # reagent (5)
- # ring-swagger (16)
- # shadow-cljs (56)
- # spacemacs (11)
- # specter (8)
- # sql (5)
- # unrepl (29)
- # yada (6)
Pseudo-Random core function of the day:
-------------------------
clojure.core/pmap
([f coll] [f coll & colls])
Like map, except f is applied in parallel. Semi-lazy in that the
parallel computation stays ahead of the consumption, but doesn't
realize the entire result unless required. Only useful for
computationally intensive functions where the time of f dominates
the coordination overhead.
I’m not sure I’ve ever used pmap
. I think I’ve only ever seen it used in examples of how not to do concurrency.
I have read several times that pmap
isn't that useful. Reducers(?) is much better I think.
Lazy sequence in, we need to do some transforms on each item, maintain the sequence as a whole and the order of it, but there's no state so multiple transforms can be done in parallel as long as input order == output order
@reborg I don’t have an opinion on whether it is good or bad, or if I should use it more or not. I just haven’t used it, and haven’t felt a need for it.
I can well believe that our BDFL knew what he was doing when he thought it was worth adding. He has a track history of making good decisions after all.
@peterwestmacott I don't use it that often either, I have a couple of examples that I'm digging. I don't think pmap
is particularly evil tho (like the examples you mentioned seem to imply)
yes, from memory the examples I’ve seen are more using pmap
to demonstrate that you have to be careful with mutable bindings
rather than that there’s anything wrong with pmap
itself
Friday Puzzle:
(sequence (partition-by keyword) ["1" "none" "2" "clojure.core/none" "3" "4"])
;(["1"] ["none"] ["2"] ["clojure.core/none" "3"] ["4"])
Would they not be better using (Object.)
as a sentinel? That’s what I try normally to use when I need one.
It seems that the ::c.c/none
pattern could potentially cause security problems etc.
https://dev.clojure.org/jira/browse/CLJ-2312 I screwed up the formatting of the code snippet, is it possible to edit a post?
also didn’t mean to post it as major
doh
Legend. Thank you.
pmap
actually "solves" this dynamic binding issue:
(def ^:dynamic *ctx* {})
(binding [*ctx* {:a 1}] (map #(update *ctx* :a (partial + %)) [1 2]))
; NullPointerException
(binding [*ctx* {:a 1}] (pmap #(update *ctx* :a (partial + %)) [1 2]))
; ({:a 2} {:a 3})
user=> (doall (binding [*ctx* {:a 1}] (pmap #(update *ctx* :a (partial + %)) (vec (range 33)))))
NullPointerException clojure.lang.Numbers.ops (Numbers.java:1018)
it’s a by product of pmap
being implemented with future
which does binding conveyance, and the fact that the first numproc+2 elements are eagerly produced (and with a vector, that means eagerly producing max(numproc+2, chunkSize)
elements)
ah thanks, I was trying to remember what problem I had in the past related to this, didn't dig deeper enough
Can we put there the following rule of thumb? Prefer pmap
for non-trivial jobs of predictable and uniform size and when 32 parallel threads are ok. Prefer reducers for more unpredictable jobs and proc+2 threads parallelism.
@reborg: Is prefer really the right word? It seems a bit strong, and I suspect it’s more subtle than that. Why should you prefer pmap
? Presumably because it’s just a one character change from map
.
I think there’s some subtlety to “when 32 parallel threads are ok”. i.e. they’re ok on a dual core machine… but you’ll probably see a lot of context switching compared to fewer threads… dunno, would really need to benchmark.
I didn't want to imply one is better than the other, or use pmap first if you can, it just happens to be first in the sentence 🙂
yeah. I think that’s the main problem I have with the above sentence, though you articulated it much better than me.
What about: use r/fold
when you are not concerned by laziness or you have unpredictable size tasks or want parallelism to be driven by your amount of cores. Use pmap
when your input is a lazy sequence that you prefer to consume on demand, you're not concerned by 32 parallel threads and your tasks are of uniform size.
“not concerned with laziness” is a bit ambiguous I think, as it also could mean you should use r/fold
if you don’t mind that your collection is lazy.
I think you’re trying to say the opposite to that though. i.e.
Use `r/fold` when you have a vector or a map (or the collection is `CollFold`able) and you have unpredictable size tasks and want parallelism to be driven by your amount of cores.
But I think r/fold
is perhaps more complicating too… as you also need to have an associative combinef
operation.
e.g. you couldn’t reliably r/fold
-
but you could +
.
So you’d probably need to add the caveat to the r/fold
that the function you want to reduce with is a monoid
😕
I’d suggest also the word “try” instead of “use” or “prefer”. That way at least you pass some of the burden/context/caveats back to the reader.
Maybe something like:
Try pmap
because it’s easiest to try, though be careful as it can create a lot of overhead due to it spawning 32 threads and consuming & emitting a lazy sequence. It’s best also that pmap
does a large amount of work per item to reduce this overhead.
Try r/fold
if you have less work to do for each item you want to process, and your input collection is a map?
a vector?
or it satisfies?
CollFold
able.
Thanks @U06HHF230 for the suggestions. I agree for the "try" and laziness implications. It's definitely complicated to come up with a complete rule in a few sentences but this is already good
It's my attempt to remember the gist of discussions we are having here and other channels, and possibly put it on the book :)
I love the platform - and think it has much more popularity to come - but Clojure support could be better.
nanobox is interesting. It seems like they're going for / are HIPAA & PCI compliant. Which makes it quite interesting.
Can anyone tell me what ^
is for in Clojure? E.g. https://github.com/ring-clojure/ring/blob/master/ring-core/src/ring/core/protocols.clj
I'm guessing it's related to metadata? https://clojure.org/reference/metadata
More specifically, the metadata reader macro?
Ooo I can see myself using macroexpand
a lot to dig deeper into succinct code examples
In vim (which you're of course using 😉 ) you can hit c1mm to run macroexpand-1 on code under the cursor.
In vim (which you're of course using 😉 ) you can hit c1mm to run macroexpand-1 on code under the cursor.