Clojurians Log v2

And just in case - neither of those handle the situation where you want multiple calls to (f) overlap if some (f) takes longer than 100 ms.

by persistent I mean... the app starts then splits into many threads along with the mainthread. All threads are holding and waiting for data to arrive so that these threads can pick up that data from their input channels and process the data; possibly put the results on to their individual output channels.

I'll try your solutions out and see how they work

thanks again @p-himik

your go loop recipe is working for my use-case quite well @p-himik! Thanks again!

If you want cancellation, a common approach is to have a separate channel and alt! on it. That channel should be returned from the function that starts the whole thing and, when you want to cancel it, just deliver any value to it.

It's not super clear what you want. But say you want some kind of asynchronous command process executor, you'd want something like:

(defn make-process
  [input-chan]
  (let [output-chan (chan)]
    (thread
     (loop []
       (if-let [command (<!! input-chan)]
         (try
          (let [{:keys [op args]} command
                result (case op
                         :foo (apply foo args)
                         :bar (apply bar args)
                         ::unknown-op)]
            (cond
             (= result ::unknown-op)
             (>!! output-chan {:error :unknown-op
                               :command command})
             (nil? result)
             (>!! output-chan {:error :nil-result
                               :command command})
             :else
             (>!! output-chan result)))
          (recur)
          (catch Exception e
            (>!! output-chan {:error :exception
                              :command command
                              :ex e})
            (recur)))
         (close! output-chan))))
    output-chan))

Say for example you always wanted only foo done on input. And foo is a possibly blocking or heavy computation. You can just do:

(defn make-foo-process
  [input-chan n]
  (let [output-chan (chan)
        xf (map (fn [msg]
                  (try
                   (let [result (foo msg)]
                     (if (nil? result)
                       {:error :nil-result :command msg}
                       result))
                   (catch Exception e
                     {:error :exception :command msg :ex e}))))] 
    (pipeline-blocking n output-chan xf input-chan true)
    output-chan))

Actually, even the first version, if you need "dispatch" is probably cleaner with using pipeline-blocking as well, just like this:

(defn make-process
  [input-chan n]
  (let [output-chan (chan)
        xf (map (fn [command]
                  (try
                   (let [{:keys [op args]} command]
                     (case op
                       :foo (if-let [r (apply foo args)]
                              r
                              {:error :nil-result :command command})
                       :bar (if-let [r (apply bar args)]
                              r
                              {:error :nil-result :command command})
                       {:error :unknown-op :command command}))
                   (catch Exception e
                     {:error :exception :command command :ex e}))))] 
    (pipeline-blocking n output-chan xf input-chan true)
    output-chan))

@didibus I'll try out your suggested approaches. Many thanks!

Do a thread dump, look for threads with names starting with async-.

Note that some of those threads could be there simply because they're cached. But they could still be unused. I don't know for sure, but I would expect for the threads that are actually doing something to be in the RUNNABLE and maybe BLOCKED state.

range takes only what the doc describes

add more constraints by transforming the collection that range returns (e.g. using filter or remove)

what’s more, you can see in the clojure https://github.com/clojure/clojure/blob/clojure-1.11.1/src/clj/clojure/core.clj#L3036 that range pivots off of whether the step param is a Long or not, and you can see in the source code java doc that both only take int/long: • https://www.javadoc.io/static/org.clojure/clojure/1.5.0-RC6/clojure/lang/Range.html#Range(clojure.lang.IPersistentMap,%20int,%20int) takes an int step • https://www.javadoc.io/static/org.clojure/clojure/1.8.0/clojure/lang/LongRange.html#create(long,%20long,%20long) takes a Long step

My first attempt was the following:

(defn sum-n [n]
  (->> (range n)
       (filter #(or (zero? (mod % 3))
                    (zero? (mod % 5))))
       (reduce +)))

in theory “in theory is better” is always better. in practice, “in theory is better” is not always better. but rough numbers

(time (let [n 500]
        (->> (range n)
             (filter #(or (zero? (mod % 3))
                          (zero? (mod % 5))))
             (reduce +))))
"Elapsed time: 4.167208 msecs"
57918
(time (letfn [(multiple-of [n] #(zero? (mod % n)))]
        (transduce (filter (some-fn (multiple-of 3) (multiple-of 5)))
                   +
                   0
                   (range 500))))
"Elapsed time: 1.737542 msecs"
57918

Here is the code I wrote to generate the multiple of 3 and 5:

(defn multiples-of-3-and-5
  ([limit]
   (lazy-seq
    (cons 0 (multiples-of-3-and-5 limit 0 0))))

  ([limit cur idx]
   (let [pattern [3 2 1 3 1 2 3]
         cur-idx (if (>= idx (count pattern)) 0 idx)
         cur-sum (+ cur (get pattern cur-idx))]
     (when (< cur-sum limit)
       (lazy-seq
        (cons cur-sum (multiples-of-3-and-5 limit cur-sum (inc cur-idx))))))))

(defn sum-n3 [n]
  (->> (multiples-of-3-and-5 n)
       (reduce +)))

Is something wrong with my implementation?

Your approach is 100% faster than mine @dpsutton, does transduce may be impacting?

absolutely. You are messing with cons and lazy sequences, creating thunks to lazily realize the sequence. Transduce turns this on its head. range is optimized to call a function on each value in its range. especially with how dense multiples of 3 and 5 are in the integers, calling a function that accepts more than 1/3 of the time is faster than the construction of sequences containing the values

if this were multiples of much larger numbers i suspect your version becomes more efficient at some point. also more fun. kinda writing an infite and lazy merge sort on infinite streams of integers But your way is more fun. Kinda write a lazy merge sort on the infinite sequences of

For some reason, My approach is faster for a very specific case: when the number limit is 10000000 (ten millions), but I definitely could not find a reason for that.

that is surprising to me for intervals of 3 and 5

(time (letfn [(multiple-of [n] #(zero? (mod % n)))]
        (transduce (filter (some-fn (multiple-of 3) (multiple-of 5)))
                   +
                   0
                   (range 10000000))))
"Elapsed time: 465.324084 msecs"
23333331666668

Your approach still faster than mine for 10 million, but my approach is faster than a simple range - reduce

if you’re benchmarking, make sure you’re trying to start ‘fresh’ often - maybe not in this case, but it can often be easy to ‘warm up’ the JVM and your runtime for one test just to be wowed by speed increases on a subsequent test 🙂

user=> (time (sum-n3 10000000))
"Elapsed time: 919.840041 msecs"
23333331666668

i find mine twice as fast as yours on 10000000

It is, your approach is the fastest so far

ah i thought you said yours was faster on a large number

and @ebernard is correct. profiling is very tricky. we’re using a very coarse instrument with time and the jvm might be doing more work for us in some cases so its not apples to apples

@ebernard How can I "start fresh"? I'm not used to the JVM world, so I would really appreciate if you suggest anything that I can study.

in short: maybe in this case, try with a ‘fresh’ JVM (e.g. if you’re running this at the REPL, consider creating a new REPL for each test case and only run a test once). like @dpsutton said, the JVM has been developed over many, many years (as well as our CPUs, and our operating systems’ control over memory, etc. etc. etc…) to try to make educated guesses as to what to return based on a given instruction. during execution, the various layers at play are building up their education to make those guesses. ergo, a benchmark run against a system that just benchmarked something else may very well perform differently (better? worse? who’s to say!) than if this were the first benchmark you’d run

there’s only so much one can reasonably do, and if you’re just comparing speeds on a couple of implementations like this, a fresh JVM is probably a good bang for your buck. and if you don’t notice a difference, then it may very well not even apply in this case. like i said - it’s a complex problem!

Got it, thanks for the information @ebernard! I think this small experience (The first function was basically the solution to the first problem from Project Euler) made me learn new things today. Also, thanks @dpsutton for the transducer example, I'll try to apply it more frequently on my daily studies. Have a great day!

sure thing - I’m not well-versed enough in benchmarking to really be able to say if that’s at play here - I just know I’ve been somewhat bit in the past by re-using processes and not otherwise accounting for the system components running the thing I was benchmarking, and it’s easy to miss 🙂

if they don’t quite click for you yet the reference page (https://clojure.org/reference/transducers) is really, really good

Kind of tangential... if the end goal is to sum them, it might not be necessary to even create the range - the answer is always going to be a multiple of the number, and with a bit of analysis, I think it's possible to get the answer with some division and some multiplication. It's not a direct "how to deal with the ranges" answer, but it could make the ranges unnecessary for this particular problem, and may be instructive when doing math-y things like this (maybe Project Euler?)

btw, transduce is pretty powerful! you may occasionally bump into other occasions where something like (-> m foo bar baz) may be better served by, instead, using comp to compose those functions together and pass it as a transducing function. it can often be a computational speed boost

Now that you mentioned, it totally makes sense to use comp, I have seen this function in the past, but could not really understand why use it instead of thread operators. Thanks a lot for this example Evan!!!

a user friendly language should have a built in fizz-buzz in core

Even shorter:

(reductions + (cycle [3 2 1 3 1 2 3]))

(for [i (range)
      d [3 5 6 9 10 12 15]]
   (+ (* i 15) d))

btw: you can use criterium to get more accurate benchmarks.

(require 'clojure.repl.deps)
(clojure.repl.deps/add-lib 'criterium/criterium {:mvn/version "0.4.6"})
(require '[criterium.core :as cc])

(cc/quick-bench (sum-n 1000))

(->> (mapcat vector
             (iterate (partial *' 3) 1)
             (iterate (partial *' 5) 1))
     (take 100)
     (run! println))

No idea if performant or not 👆

I suspect the presence of duplicated numbers and the fact that the numbers are out of order are violating unstated requirements that said, I compared to transducing version of the same thing and the perf is pretty close

(defn v1 []
  (->> (mapcat vector
               (iterate (partial *' 3) 1)
               (iterate (partial *' 5) 1))
       (take 100)
       (into [])))

(defn v2 []
  (into []
        (take 100)
        (mapcat vector
                (iterate (partial *' 3) 1)
                (iterate (partial *' 5) 1))))

(assert (= (v1) (v2)))

(defn measure-time
  [f]
  (let [start (System/nanoTime)
        _ (f)
        end (System/nanoTime)]
    (- end start)))

(defn dumb-crit
  [f n]
  (/ (apply + (repeatedly n #(measure-time f)))
     (* n (Math/pow 10 9))))

(println 'v1 (dumb-crit v1 1000))
(println 'v2 (dumb-crit v2 1000))

v1 4.0298073E-5
v2 2.623332E-5