• Hey folks, I'm new to the slack, so please forgive me, if I don't follow any CoC... I was wondering if I stumbled over something weird. I currently introduced core.async to one of our services, myself being new to core.async. Basically 4 CSPs (go-loops, alts!, timeout, pipeline-blocking, ...), that do requesting of big data structures via UUID from a db, then processing it, and transforming it into a file. Unfortunately when working on large data (long E-Mail with attachment, "metadata", ...) we stumble across some Heapdumps in the logs. The service recovers (after a gc, I guess?), but still I find it quite puzzling. I managed to load it into the Eclipse-MAT and noobily found out, that apparently there were a bunch of clojure.core.async.impl.timers.TimeoutQueueEntry with the biggest item showing 65,893,400 (4.57%) bytes. In total that 'problem suspect' seems to be a quite large (1.3Gi) object array inside of core.async. Is there a known issue or something, that the channels keep track of content somehow, although items are <! out of the channel?

when you put a value to a channel that can't immediately be completed a handler wrapping the value goes into the put queue

there was a bug fixed recently that could allow alt handlers to get stranded in channel queues so that could be a match to your description

https://clojure.atlassian.net/browse/ASYNC-204 is the bug and it was fixed in core.async 1.5.640 (latest is 1.5.644)

interesting! I'm gonna try the 1.5.644 version, and if the OOM still occurs

Thanks for the lightspeed response btw. Really appreciate it 🙂

there is also https://clojure.atlassian.net/browse/ASYNC-234 pertaining to alts and timeouts and might be worth thinking about architecturally. I am still on the fence about whether that even is a "bug" or whether there is something that should be done.

mmh... The issue still seems to persist after 1.5.644.

I was reading through 234 and thought that it may be related. Interestingly the use case seems to be quite similar. So a quick overview about what we do here: (A) puts UUIDs on an ids-chan, that will be consumed by (B) pipeline-blocking (10 Threads / 20 Cores). B RPCs to a db-wrapper and puts the return value onto the file-chan. The file-chan is then consumed by a loop (C) until the file-chan is closed. C then returns to the main thread. So what I'm curious about is, if the pipeline may persist items (i.e. keep them as a reference) on the to-channel. The big data-structures in MAT would support this (65Mi being a large mail), as (C) takes these big items via alts!, but then immediately writes them into the file. However, alts! usually selects the file-chan. The timeout on the other hand is the same all across.

(pipeline-blocking (processes-count env)
    file-chan
    (map (partial get-x-rpc env tenant ex-chan))
    ids-chan
    true
    #(>!! ex-chan %))

 (loop [[item port] (alts! [file-chan timeout-chan ex-chan])]
        (cond
          (= port data-chan)    (when item
                                  (item-fn item)
                                  (recur (alts! [file-chan timeout-chan ex-chan])))
          (= port timeout-chan) (throw (TimeoutException.))
          (= port ex-chan)      (throw (AbortException.))

It very well could be, the issue in 234 is subtle, the way a mutable array is used for locals and slots in it tend to be overwritten in loops often mitigates it

Any easy check/mitigation is to instead of sending item send (atom item) and nil out the atom when you get it on that branch of the cond

My first patch on 234 basically does the atom thing automatically

I don't consider that a complete solution, because it still potential leaks the atoms

But an atom is potentially much smaller then a big chunk of data