Fork me on GitHub
#core-async
<
2022-10-17
>
vlaaad08:10:04

What’s the difference between these impls of onto-chan!! ? 1. Impl in core.async:

(defn onto-chan!!
  ([ch coll] (onto-chan!! ch coll true))
  ([ch coll close?]
   (thread
     (loop [vs (seq coll)]
       (if (and vs (>!! ch (first vs)))
         (recur (next vs))
         (when close?
           (close! ch)))))))
2. Non-blocking impl:
(defn onto-chan!!
  ([ch coll] (onto-chan!! ch coll true))
  ([ch coll close?]
   (go-loop [vs (!< (thread (seq coll)))]
     (if (and vs (>! ch (<! (thread (first vs)))))
       (recur (<! (thread (next vs))))
       (when close? (close! ch))))))
My understanding is that in the case of core.async’s impl, we have a thread that might be locked forever if ch’s buffer is full and we never consume from ch again (e.g. because we decided we are no longer interested in that data source and enqueued a close of it). Is it a memory leak? Is the non-blocking impl better because a parked go block will be garbage collected if we are no longer interested in the ch and stopped consuming from/closed it?

vlaaad15:10:42

Anyone? Isn’t it important? Maybe my comments are inappropriate for this channel?

hiredman16:10:50

I think if you dropped it on http://ask.clojure.org it might turn into some kind of ticket and might eventually get patched

hiredman16:10:55

but I don't really care about it. we have one use of onto-chan at work, and I think it is an old code path that is likely not used anymore

hiredman16:10:34

it also seems at least somewhat unlikely that you would take the trouble to pour a collection into a channel and then just drop the channel

hiredman16:10:36

our main usage of core.async may be a little idiosyncratic, a chat service, which is all very single message processing at a time, so not a lot of collection processing (using onto-chan or even using transducers). There do seem to be people that care about mixing collection processing and core.async, and they would likely care more

hiredman16:10:16

I also have 3 different patches in jira to address memory issues (leaks depending on your point of view) in core.async, 2 of the patches are coming up on 2 years old. the most recent patch is only 4 months old, but literally started with alex asking for a hand with the issue, and I've heard no feedback about it in the four months since

hiredman16:10:05

so with my luck it is likely that an issue I don't care about like onto-chan will get a priority fix and a new release for it cut immediately

vlaaad17:10:38

onto-chan!! is an illustrative example, I’m just trying to get a more whole understanding of how to use core-async, what to use and what not to use. So far it seems any use of <!! or >!! is a potential memory leak since it might never return if no one takes from a chan, and thread will be stuck forever.

hiredman17:10:09

sure, just like any deref of a promise, or tryLock or call to https://docs.oracle.com/javase/7/docs/api/java/lang/Object.html#wait() or take from a blocking queue or ...

hiredman17:10:43

(in fact the way <!! and >!! block a real thread is by derefing a promise)

ghadi17:10:12

channel ops (of any variety) might wait for an event that will never occur

hiredman17:10:03

I hear that one lonely whale found a friend

vlaaad17:10:23

but channel ops (I assume you mean <! , >! , alt! etc.) will be garbage collected if they are waiting for event that never arrives and no one else has references to the chan?

ghadi17:10:35

i mean any op <!! <!, etc

ghadi17:10:56

this is why alts + timeout/cancel exists

hiredman17:10:23

(fact check: it looks like blue 52 finding a friend is likely not true)

1
hiredman17:10:19

the inversion of control does all kinds of fun stuff to resource usage

vlaaad17:10:30

but there is still a difference between <!! and <! that are forever stuck — the former will lock a thread, i.e. a “global resource”, and the latter will be quietly parked and garbage collected if there are no more references to the go block with <! ?

hiredman17:10:35

in particular if start using alt! with timeout a lot, timeouts sit a in a global queue with a strong reference, and even if the timeout branch of the alt! is not taken, the callback will sit in the queue until it times out

hiredman17:10:57

which is one of the memory usage related patches I have sitting in jira

hiredman17:10:37

actually we just switched to a forked core.async at work with the 0001 patch from that ticket applied, which will go out in the next deploy (we recently changed how a lot of things are deployed, and as a result are trying to cut down on memory usage in the service which is our biggest use of core.async)

hiredman17:10:22

the timeout in alts case is illustrative of how easy it is to turn a callback into a strong global reference

hiredman17:10:23

the alts case is particularly problematic since you would want to reach for alts+timeout to solve other kinds of stalling/hanging issues

vlaaad17:10:39

Yeah I saw ASYNC-234, strange that such a critical issue with existing patches has no activity for a year

didibus01:10:07

I'm not sure the core team still maintains core.async really. Feel it might be one of those that could make sense to fork too clj-commons if a shepherd showed up.

phronmophobic01:10:43

This were some recent discussion related to this in #clojure-dev, https://clojurians.slack.com/archives/C06E3HYPR/p1666114279375419

👍 1