This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
- # beginners (92)
- # boot (2)
- # cljsrn (6)
- # clojure (33)
- # clojure-austin (1)
- # clojure-dev (18)
- # clojure-spec (7)
- # clojure-uk (2)
- # clojurescript (35)
- # cursive (13)
- # data-science (3)
- # datomic (1)
- # defnpodcast (1)
- # figwheel (1)
- # fulcro (27)
- # instaparse (1)
- # java (2)
- # leiningen (8)
- # off-topic (5)
- # onyx (1)
- # portkey (2)
- # re-frame (9)
- # reagent (2)
- # ring-swagger (1)
- # shadow-cljs (136)
- # test-check (3)
- # tools-deps (29)
.reduce is actually speeding it up much more. But you don't get very accurate benchmark numbers since the
doall dominates most of the runtime (naive first/next). You can get more accurate benchmark numbers on your code changes by avoiding
doall. E.g. I'd do:
(defn count-chunked [s] (let [s (seq s)] (if (chunked-seq? s) (+ (count (chunk-first s)) (count-chunked (chunk-rest s))) (count s))))
count-chunked to walk all the chunkedCons. That way you're in for ~50-100% speedup for your code change
@rauh agree. I guess my benchmark emulates a scenario in which the chunked sequence is fully consumed by some sequential processing (eg with
last etc) that is unaware of chunkiness. Your
count-chunked sounds more a specialized consumer that knows what is dealing with. I guess it should optimize for the more general scenario?
@reborg But now you're benchmarking the consumption and not your code change (which efficiently creates the chunks). Ideally you want to only bench the creation but that's not possibly due to the lazy seq. So you need some way to walk the lazy seq (efficiently).
But there is no
count-chunked in the stdlib. If I use that as a benchmark then I'm showing improvements that no one will get in real file unless they roll their own chunked processing. Perhaps those should be discussed as a separate issue (i.e. take advantage of chunked processing where this is not already done). Actually, I'm not sure why other sequential functions do not have the
if (chunked-seq?) scenario. Only
from my understanding it was not done pervasively because it’s a pain in the code and so was focused on some of the most common functions.
for is chunked.
range’s seq impl is chunked.
@reborg Well then I'd rename the above fn as "walk lazy chunked cons" and make it return nothing. Just calling
chunk-rest basically. My guess is that eventually most people will end up in a
reduce on their data structures in one way or the other.
But then you're benchmarking the reduce more. When you want to benchmark the actual
map code that you want to improve upon
maybe something like
that’s basically just next’ing through the full sequence
I can add that to the bench table for a comparison @alexmiller
@reborg I guess this would bench the chunking code inside
map (etc) best:
(defn doall-fast [s] (when-some [s (seq s)] (if (chunked-seq? s) (doall-fast (chunk-rest s)) (doall-fast (next s)))))
(defn dochunk [xs] (when xs (recur (chunk-next xs)))) if only for the benchmark