This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2022-10-17
Channels
- # announcements (3)
- # babashka (3)
- # beginners (53)
- # biff (2)
- # calva (16)
- # cider (1)
- # clj-commons (1)
- # clj-kondo (97)
- # clj-on-windows (137)
- # clojure (49)
- # clojure-europe (63)
- # clojure-gamedev (1)
- # clojure-nl (2)
- # clojure-norway (50)
- # clojure-uk (4)
- # clojurescript (36)
- # core-async (28)
- # datomic (32)
- # emacs (22)
- # events (1)
- # graalvm (8)
- # honeysql (6)
- # jobs (2)
- # lambdaisland (5)
- # malli (6)
- # nbb (31)
- # off-topic (171)
- # pathom (14)
- # rdf (2)
- # reitit (4)
- # releases (2)
- # scittle (19)
- # shadow-cljs (46)
- # sql (6)
- # squint (6)
What’s the difference between these impls of onto-chan!!
?
1. Impl in core.async:
(defn onto-chan!!
([ch coll] (onto-chan!! ch coll true))
([ch coll close?]
(thread
(loop [vs (seq coll)]
(if (and vs (>!! ch (first vs)))
(recur (next vs))
(when close?
(close! ch)))))))
2. Non-blocking impl:
(defn onto-chan!!
([ch coll] (onto-chan!! ch coll true))
([ch coll close?]
(go-loop [vs (!< (thread (seq coll)))]
(if (and vs (>! ch (<! (thread (first vs)))))
(recur (<! (thread (next vs))))
(when close? (close! ch))))))
My understanding is that in the case of core.async’s impl, we have a thread that might be locked forever if ch’s buffer is full and we never consume from ch again (e.g. because we decided we are no longer interested in that data source and enqueued a close of it). Is it a memory leak? Is the non-blocking impl better because a parked go block will be garbage collected if we are no longer interested in the ch and stopped consuming from/closed it?I think if you dropped it on http://ask.clojure.org it might turn into some kind of ticket and might eventually get patched
but I don't really care about it. we have one use of onto-chan at work, and I think it is an old code path that is likely not used anymore
it also seems at least somewhat unlikely that you would take the trouble to pour a collection into a channel and then just drop the channel
our main usage of core.async may be a little idiosyncratic, a chat service, which is all very single message processing at a time, so not a lot of collection processing (using onto-chan or even using transducers). There do seem to be people that care about mixing collection processing and core.async, and they would likely care more
I also have 3 different patches in jira to address memory issues (leaks depending on your point of view) in core.async, 2 of the patches are coming up on 2 years old. the most recent patch is only 4 months old, but literally started with alex asking for a hand with the issue, and I've heard no feedback about it in the four months since
so with my luck it is likely that an issue I don't care about like onto-chan will get a priority fix and a new release for it cut immediately
onto-chan!!
is an illustrative example, I’m just trying to get a more whole understanding of how to use core-async, what to use and what not to use. So far it seems any use of <!!
or >!!
is a potential memory leak since it might never return if no one takes from a chan, and thread will be stuck forever.
sure, just like any deref of a promise, or tryLock or call to https://docs.oracle.com/javase/7/docs/api/java/lang/Object.html#wait() or take from a blocking queue or ...
but channel ops (I assume you mean <!
, >!
, alt!
etc.) will be garbage collected if they are waiting for event that never arrives and no one else has references to the chan?
but there is still a difference between <!!
and <!
that are forever stuck — the former will lock a thread, i.e. a “global resource”, and the latter will be quietly parked and garbage collected if there are no more references to the go block with <!
?
in particular if start using alt! with timeout a lot, timeouts sit a in a global queue with a strong reference, and even if the timeout branch of the alt! is not taken, the callback will sit in the queue until it times out
actually we just switched to a forked core.async at work with the 0001 patch from that ticket applied, which will go out in the next deploy (we recently changed how a lot of things are deployed, and as a result are trying to cut down on memory usage in the service which is our biggest use of core.async)
the timeout in alts case is illustrative of how easy it is to turn a callback into a strong global reference
the alts case is particularly problematic since you would want to reach for alts+timeout to solve other kinds of stalling/hanging issues
Yeah I saw ASYNC-234, strange that such a critical issue with existing patches has no activity for a year
I'm not sure the core team still maintains core.async really. Feel it might be one of those that could make sense to fork too clj-commons if a shepherd showed up.
This were some recent discussion related to this in #clojure-dev, https://clojurians.slack.com/archives/C06E3HYPR/p1666114279375419