This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2019-12-17
Channels
- # adventofcode (23)
- # announcements (2)
- # aws (11)
- # babashka (181)
- # beginners (59)
- # chestnut (2)
- # clj-kondo (9)
- # clojure (90)
- # clojure-brasil (2)
- # clojure-europe (18)
- # clojure-italy (24)
- # clojure-nl (9)
- # clojure-spec (3)
- # clojure-uk (28)
- # clojured (4)
- # clojuredesign-podcast (3)
- # clojurescript (12)
- # community-development (49)
- # core-async (49)
- # cryogen (5)
- # cursive (16)
- # data-science (1)
- # datascript (7)
- # datomic (54)
- # defnpodcast (4)
- # events (2)
- # figwheel-main (14)
- # fulcro (139)
- # graphql (1)
- # jobs-discuss (6)
- # kaocha (1)
- # luminus (2)
- # malli (3)
- # music (1)
- # off-topic (34)
- # pathom (24)
- # re-frame (13)
- # reitit (5)
- # shadow-cljs (8)
- # test-check (6)
where are the best posts/articles/books/chapters to help understand how to use core.async to build data processing pipelines? I've found this one: https://adambard.com/blog/stream-processing-core-async/ but would like to find some more (and/or hear opinions about this one)
Clojure Applied has a chapter on that
@alexmiller cool. I'll review that.
If I understand correctly, using IO
in a go block
is bad because it would block the thread that the go block
is using. Meaning that if you would want to read from a database (for example) you could use core.async/thread
to spawn a new thread. Then park the go block
while listening to the spawned thread using <!
. Is this correct?
async/thread just gets the work off the go block threadpool, so you can use something else to do that as well
If you just spawn thread unmanaged, you could use up all your resources I would assume?
it depends, but yes it is often useful to limit that in someway, which is one reason you might using something other than core.async/thread (at work we have something similar that uses a threadpool we control)
but we have a few places where a singleton go loop sometimes calls async/thread and waits on the result, so by construction it can't create many threads
there's nothing special about the threads created by thread
Ok, but in the case of where for example traffic dictates the amount of spawned threads, you’d definitely need a pool
use any thread pool you like, just communicate via channels
thread
is just a helper and does the extra return channel thing - there's almost nothing there
it's mostly the binding conveyance by loc
(I might be misunderstanding some concepts) Let’s say you limit your thread pool to X amount of threads. If all X threads are busy you will have to wait for a thread to free up. How is this different from bumping up the go loop thread pool to X amount of go loop threads?
if you did that, you'd easily have code that succeeds in local / staging / tests and fails under real load
for one thing
I assume because you can’t replicate the load, but how is that different from testing a regular thread pool?
because you can starve the thread pool for go blocks, if it happens faster / at lower usage, you can catch it easier
go blocks can do coordination faster and cheaper than a thread pool if used correctly - because they context switch without system calls
(or at least can)
other libraries, other parts of your code, etc may want to use it, so if you are are blocking it that is a problem
But doesn’t that mean that if no libraries are using core.async, and you manage the go loop pool, then it would work the same as managing your own thread pool?
sort of, it is a complicated threadpool where tasks run for a bit, then get put on the back of the queue, which is more complicated to manage then a threadpool that pulls a task and runs it to completion
I dont' have proof but my personal theory is that the go block thread pool was intentionally shrunk as an anti-foot-gun measure, to lead core.async users toward the kind of constructions that actually benefit in any way from core.async
when a go block reads from a channel, the continuation of the block is added as a callback to the channel, and the go block stops running so some other go block can run on that thread, and once something is written to that channel the callback is put back on the queue