This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2020-09-03
Channels
- # announcements (11)
- # atom-editor (8)
- # aws (16)
- # babashka (34)
- # beginners (59)
- # calva (32)
- # cider (8)
- # clj-kondo (43)
- # cljs-dev (52)
- # clojure (26)
- # clojure-europe (11)
- # clojure-italy (2)
- # clojure-nl (5)
- # clojure-spec (16)
- # clojure-uk (44)
- # clojurescript (5)
- # core-async (21)
- # cursive (14)
- # datomic (53)
- # figwheel-main (4)
- # fulcro (5)
- # graphql (6)
- # java (3)
- # kaocha (5)
- # leiningen (6)
- # local-first-clojure (1)
- # malli (25)
- # off-topic (40)
- # other-languages (1)
- # pathom (5)
- # pedestal (3)
- # re-frame (4)
- # reitit (2)
- # reveal (8)
- # rum (21)
- # sci (16)
- # shadow-cljs (90)
- # spacemacs (8)
- # tools-deps (10)
- # vrac (6)
- # xtdb (12)
Hey, I'm (still) trying to understand the pipeline-blocking
and it's uses:
I did a REPL experiment:
(let [stdin (async/chan)
stdout (async/chan)
prn2 #(locking prn (prn %))]
(async/pipeline-blocking 2
stdout
(map #(doto % prn2))
stdin)
(async/go-loop []
(when-let [x (async/<! stdout)]
(prn2 [:stdout x]))
(prn2 [:close]))
(dotimes [i 10]
(prn2 [:put i (async/put! stdin i)]))
(async/close! stdin))
It behaves:
- All 10 put!
returns true
- Only 5 (the first 5) elements get processed
- The stdout only receive 1 processed result, then receive a close
- On VisualVM, I see a lot of of "zombie threads"
None of these behaviors make sense for me.
Where I can read more about it? There is use examples?put!
is async so will basically queue up 10 pending puts (returns true if the channel isn't closed and it's not)
the "zombie" threads are probably not "zombies", they are mostly daemon threads in pools waiting for work and will go away if they don't get any. the go-loop is backed by a pool, and pipeline blocking is backed by a pool, so I'd expect threads from both of those
you're using a transducer but unbuffered in/out channels, that may be an issue, could be an unhandled exception happening there
if you change stdin / stdout to have a buffer (async/chan 10)
does that change what you see?
@alexmiller I'm trying to implement a "clojure idiomatic" http server, from java SocketServer (for learning/fun)
ATM I'm using java.util.concurrent/newFixedThreadPool
as thread pool. Does make sense use pipeline-blocking in this case?
pipeline-blocking and pipeline will act like their own additional threadpool, they won't use threads from your threadpool
Or they are not "equivalent"? I'm still not understand well the use-case of pipeline-blocking
so you could use that to implement processing an http request, but there are a lot of decisions to be made on how to do that
if you are using SocketServer, the example servers for that kind of thing usually spin up a new thread (or put a task on a threadpool) for each request, the analogue of that would be running a go block, not using a pipeline
a pipeline is more like a combination of an executor and a ExecutorCompletionService, with some more bits (combining an executor and an exceutorcompletionservice doesn't get you the ordering you get from a pipeline)
Tnks @U0NCTKEV8 I will keep at "regular" threadpools One signal that it's not a pipeline blocking problem is that i don't care about "output" I just need to "submit" and get the work done in another thread
A pipeline might work well if you implement http request pipelining or whatever it is called where you re-use the same tcp connection for multiple http requests
After some interations with @U9ABG0ERZ I see that this use-case is more about agent
and less about pipeline
😉