This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2020-11-13
Channels
- # aleph (1)
- # announcements (18)
- # babashka (11)
- # beginners (112)
- # business (1)
- # calva (19)
- # cider (8)
- # clj-kondo (63)
- # cljsrn (10)
- # clojure (188)
- # clojure-australia (1)
- # clojure-dev (38)
- # clojure-europe (112)
- # clojure-nl (3)
- # clojure-provo (1)
- # clojure-spec (22)
- # clojure-uk (108)
- # clojurescript (37)
- # cryogen (4)
- # cursive (8)
- # data-science (1)
- # datomic (13)
- # emacs (9)
- # events (1)
- # fulcro (26)
- # funcool (3)
- # graalvm (2)
- # graphql (11)
- # helix (8)
- # jobs (1)
- # jobs-discuss (7)
- # nrepl (3)
- # off-topic (72)
- # pathom (10)
- # pedestal (1)
- # reagent (6)
- # reitit (7)
- # remote-jobs (1)
- # shadow-cljs (28)
- # xtdb (12)
Out of curiosity I poked around clojure.walk/walk
and it looks like its performance can be improved, with one change requiring almost no care, and another plenty of care:
The quick win is changing the coll?
case to use a transducer:
(coll? form) (outer (into (empty form) (map inner form)))
(coll? form) (outer (into (empty form) (map inner) form))
Another more elaborate change which will require lots of care is replacing the entire cond
with a protocol then extending that protocol to all of Clojure's collections.
I ran a few benchmarks and it seems like most of the gain is from switching to a transducer.
Should I post it on ask.clojure?that would reverse lists
which I don't think any consumer of walk actually wants
oh, OK
I see now, it was already using into / empty
you can post it on ask, please make sure to include the tests you're using
The performance difference is very noticeable with maps and vectors. The protocol saves about 5% from that (give or take)
in general, a single transducer is not necessarily faster and can even be slower vs chunked seqs for small seqs.
Won't chunked seqs be caught before the coll?
case in
(seq? form) (outer (doall (map inner form)))
(map f [...])
operates on a chunked seq of the vector
sorry, not looking at the code, just talking generally of things to consider
the next step of analysis is to get a sense of what people use walk for and a rough frequency of that
basically toward answering the question of whether this change is usually a win
if you're interested in actually providing a patch, feel free to become a https://clojure.org/dev/contributor_agreement and https://clojure.atlassian.net/servicedesk/customer/portal/1
Done, I think another patch for https://clojure.atlassian.net/browse/CLJ-1239 would be nice
Asked https://ask.clojure.org/index.php/9801/clojure-walk-walk-can-use-transducers Looks like protocols won't give any performance gains there, focused only on transducers
There, found the mistake. Protocols do offer a speedup after all https://ask.clojure.org/index.php/9801/clojure-walk-walk-can-use-transducers-and-protocols
@ben.sless How do you explain speedups with protocols? How can they be faster than instance checks?
A summary of the benchmark would help, it's a bit difficult to read imo. I don't see significant differences?
1. The speedup with protocols isn't dramatic
2. They may be faster in a REPL environment but won't be in a direct-linked AOTed jar
3. It isn't just one instance check. To get to coll?
you need 5 instance checks and non-taken branches. In a heterogeneous collection does that mean branch prediction goes out the window?
protocols cache at the call-site
I think there is actually an existing ticket about clojure.walk and protocols btw
A summary of the benchmarks:
tested a few use cases:
ā¢ (1 2 [3 4 5] {:a 6 7 8} [9 [10]] #{:b 7})
a heterogeneous collection
ā¢ (defn walk* ,,,)
a big collection with lots of lists and symbols
ā¢ {1 {2 {3 {4 {5 {6 {7 {8 {9 {10 11}}}}}}}}}}
a deeply nested map
ā¢ [1 [2 [3 [4 [5 [6 [7 [8 [9 [10 11]]]]]]]]]]
a deeply nested vector
@ben.sless if you need a "walk but fast and impl via protocols" take a look at #specter
Now that I have a new patch prepared, do I just upload it to the ticket and comment, or is any other action required? Already made sure to follow the https://clojure.org/dev/developing_patches and run tests.
Upload and comment is great
https://github.com/stuartsierra/clojure.walk2 is an old impl of that
prob predates transducers though
Now that I have a new patch prepared, do I just upload it to the ticket and comment, or is any other action required? Already made sure to follow the https://clojure.org/dev/developing_patches and run tests.