This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2020-04-13
Channels
- # announcements (3)
- # babashka (130)
- # beginners (73)
- # calva (22)
- # cider (46)
- # cljdoc (18)
- # cljs-dev (196)
- # cljsrn (18)
- # clojure (255)
- # clojure-europe (2)
- # clojure-finland (8)
- # clojure-gamedev (1)
- # clojure-germany (2)
- # clojure-losangeles (6)
- # clojure-nl (1)
- # clojure-spec (16)
- # clojure-uk (33)
- # clojurescript (32)
- # community-development (1)
- # conjure (40)
- # core-logic (11)
- # cursive (4)
- # datascript (8)
- # devcards (17)
- # emacs (21)
- # exercism (2)
- # fulcro (29)
- # funcool (15)
- # graalvm (18)
- # jobs (17)
- # jobs-rus (1)
- # lambdaisland (1)
- # lumo (1)
- # malli (19)
- # off-topic (15)
- # pathom (22)
- # quil (7)
- # re-frame (3)
- # reagent (3)
- # shadow-cljs (14)
- # spacemacs (41)
- # specter (2)
- # sql (5)
- # tree-sitter (1)
- # unrepl (16)
- # vscode (3)
- # xtdb (11)
- # yada (1)
I have a question regarding propagating backpressure with promesa I have created this event function:
(fn [_ rec]
(->> (p/promise rec)
(p/map exec deserialize)
(p/mapcat exec http-request)
(p/map exec serialize)
(p/map exec produce)))
And I invoke it in a loop
(loop [recs (poll)]
(reduce event-fn nil recs)
(recur (poll)))
(using reduce because recs
are iterable and I want to go fast)
this implementation incorrectly propagates backpressure to the poll
function, which just returns an iterable of data.
What did I miss here and how do I fix it?
Thanks, promesa is coolif you execute event-fn in a loop without waiting it result, it can cause that event-fn can start again before the previous execution ends
Makes sense. But I want to get to 100% cpu usage, so would want to start execution before the previous execution ends, and only block on execution start when all the threads in the exec
pool are busy
What I ended up doing was
(loop [recs (poll)]
@(p/all (reduce event-fn [] recs))
(recur (poll)))
But it didn't get close to eating all the 100% utilizationWhat would happen if I provided the executor at the beginning? (p/promise rec exec)
It depends, not using 100% can have different causes. In any case, in normal situation you dont need to pass a custom executor service, and in your case it can cause this nrgative effect. The CompletableFuture, by default resolves all chained computations synchronously in the same thread where the first promise is resolved. Providing an specific executor service will cause additional task spwaning, adding more latency and being less efficient.
Spawning a task for each small "partial" computation is usefull when you have 100000+ small tasks and you dont want to block a thread with a single chain of compuation
In any case, promesa here is just a sugar syntax on top of completable future, if you have problems with backpressure or not using 100% cpu is probably a design problem than promesa problem.