This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2019-07-11
Channels
- # announcements (1)
- # aws (3)
- # beginners (48)
- # calva (2)
- # cider (47)
- # clj-kondo (1)
- # cljs-dev (23)
- # cljsrn (10)
- # clojure (81)
- # clojure-chicago (4)
- # clojure-europe (3)
- # clojure-greece (4)
- # clojure-italy (8)
- # clojure-losangeles (1)
- # clojure-nl (6)
- # clojure-sanfrancisco (1)
- # clojure-seattle (1)
- # clojure-uk (21)
- # clojurescript (40)
- # core-async (82)
- # cursive (18)
- # datomic (6)
- # duct (11)
- # figwheel-main (4)
- # fulcro (26)
- # jobs-discuss (22)
- # leiningen (18)
- # off-topic (10)
- # pathom (3)
- # re-frame (5)
- # reagent (16)
- # reitit (4)
- # shadow-cljs (8)
- # specter (7)
- # sql (16)
- # tools-deps (58)
- # xtdb (30)
There is some docs/reference about how to "asyncfy" a library like clojure.jdbc
?
I know that I need to create a worker pool, inset in a queue and wait in a channel. But I'm not sure about the wait part (who will give me a channel?)
a decent pattern is to use one channel, and N threads consuming, running a function that takes two items: something describing the work, and a channel on which to put the result
the N decides your maximum parallelization
you don't need a separate queue - a channel is already a queue
any consumer that wants a result would write a message tuple [params, c], then wait for a result on c
You can also use async/thread or pipeline-blocking for all your io, you don't need to setup your own threadpool
Hi there, A question related to the performances of go processes. I am executing a similar code in thread vs. go processes and it seems that go processes takes more time. Here is my code:
(do
(let [c (chan)
_ (put! c 2)
t (. System (nanoTime))]
(thread
(let [val (<!! c)
elapsed (- (. System (nanoTime)) t)]
(println (str "taken value, delivery time with thread -- " val " time: " (float (/ elapsed 1e6)) "ms" "\n")))))
(let [c (chan)
_ (put! c 1)
t (. System (nanoTime))]
(go (let [val (<! c)
elapsed (- (. System (nanoTime)) t)]
(println (str "taken value, delivery time with go: -- " val " time: " (float (/ elapsed 1e6)) "ms" "\n")))))))
The output is:
taken value, delivery time with thread -- 2 time: 0.228456ms
taken value, delivery time with go: -- 1 time: 1.516011ms
I was expecting go processes to run faster than threads.a go block is nothing more than a "coordination block", in the end it all runs in threads, it's not a lightweight thread per say
thread
has no bookkeeping to do about parking put/take, it just runs the body in a thread, returns a chan with the result and bye.
@viebel the benchmark is flawed to the point of not telling you anything. > expecting go processes to run faster than threads. Why?
because go processes are kind og lightweight threads
it's just a coordination primitive that can run blocking stuff (wrapped in chan put/takes) on a fixed threadpool, like in go
Within threads, yes but no need to create a new thread each time
Pounds are not a unit of speed :)
your benchmark isn't actually doing much work, and it's incorrectly calling blocking I/O (`println`) within the go block
blocking that isn't channel operations is not recommended because it can cause deadlocks
and things that have more operations per process: put, take, especially with unbuffered channels where they might not be available to proceed
something is not clear to me: Does a go block creates a new thread or does it use a thread that is already ready to run?
Let’s say I create 10 go blocks sequentially, will the thread be reused or will it be recreated?
core async dispatches its work on a thread pool that defaults to 8 threads.
if you created 10 go blocks sequentially it would likely create all 8 threads and would then begin reusing them as needed
And between two go blocks execution, the thread from the thread pool is not kept alive. Right?
it's in a waiting state where the vm won't try to execute it between running tasks
it's still allocated, it just doesn't try to run
(it might be more a question of whether the underlying OS tries to wake it up at that point, but regardless the execution dispatcher knows it's not ready to do anything)
Is running a thread from a waiting state cheaper than creating a new thread or is it the same?
it's cheaper, this is why we have thread pools - allocation and initialization aren't free
the threads in the core async dispatch pool are never destroyed once they've been created
unless there's an exception
in which case a new one will be created if it's needed
Now I am back to my original question: if go blocks are cheaper to create than thread, why running sequentially go blocks is not faster that running threads sequentially?
because creating the thread isn't the expensive part of running a task on a thread
(or at least it shouldn't be)
what core.async improves is the time it takes to context switch (thanks to reusing small callbacks on a small thread pool), and the correctness of code that coordinates light weight channel ops
but if all you are doing is a sequence of actions (IO or CPU) and you aren't juggling async results, core.async can only match regular perf at best, and likely slows things down
@viebel in your example that you pasted earlier, what is the thread
that is used? the core async thread macro?
I have improved my example. Now it’s
(let [iterations 10000]
[(let [t (. System (nanoTime))]
(dotimes [_ iterations]
(let [c (chan)
_ (put! c 1)]
(deref (future (<!! c)))))
(let [elapsed (- (. System (nanoTime)) t)]
(str (float (/ elapsed 1e6)) "ms")))
(let [t (. System (nanoTime))]
(dotimes [_ iterations]
(let [c (chan)
_ (put! c 1)]
(<!! (go (<! c)))))
(let [elapsed (- (. System (nanoTime)) t)]
(str (float (/ elapsed 1e6)) "ms")))])
It’s go
vs future
On my MacBook Pro, the result is:
["191.04631ms" "211.95337ms"]
future is faster than go
why is it misconceived @ghadi ?
it's clear if you macroexpand the go
block that there are (small) costs to the go machinery itself
But, if a channel blocks in a thread or future, that thread is useless until the channel yields
@ghadi I understand that the real value of go blocks, is when you run them in parallel. I brought this sequential example in order to help understand how things work
When I say future is faster than go, I don’t mean to “blame” go
I was assuming that creating a thread was an expensive OS operation and that’s why I thought go blocks would be futures even in the sequential case
I still don’t get it 100%
future also uses a thread pool
What about core.async thread? Does it also use a thread pool?
and the thread pools that future and core.async thread use are actually completely unbounded
the threadpool used by the go blocks are by default fixed at 8. the threadpool used by the thread
macro is unbounded.
though, I guess them being unbounded doesn't matter as much in your example since you wait for the future before you make the next one
What do you mean by unbounded?
there's no limit to how many threads they can start up
besides the obvious memory limits and whatnot
I added a core.async/thread
scenario
(let [iterations 100]
[(let [t (. System (nanoTime))]
(dotimes [_ iterations]
(let [c (chan)
_ (put! c 1)]
(deref (future (<!! c)))))
(let [elapsed (- (. System (nanoTime)) t)]
(str (float (/ elapsed 1e6)) "ms")))
(let [t (. System (nanoTime))]
(dotimes [_ iterations]
(let [c (chan)
_ (put! c 1)]
(<!! (thread (<! c)))))
(let [elapsed (- (. System (nanoTime)) t)]
(str (float (/ elapsed 1e6)) "ms")))
(let [t (. System (nanoTime))]
(dotimes [_ iterations]
(let [c (chan)
_ (put! c 1)]
(<!! (go (<! c)))))
(let [elapsed (- (. System (nanoTime)) t)]
(str (float (/ elapsed 1e6)) "ms")))])
The results are: ["3.231052ms" "20.063465ms" "5.740195ms"]
thread is much slower!
And when I raise the number of iterations to 10,000 the thread
code triggers an out of memory exception