Fork me on GitHub
#core-async
<
2016-03-24
>
stijn14:03:10

hi, question on memory consumption by core.async processes.

(def in (chan 1 (map (fn [_] (apply str (repeatedly 1000 #(rand-int 10)))))))
=> #'user/in
(def done
  (async/go-loop []
    (when (<! in)
      (recur))))
=> #'user/done
(async/<!! (async/onto-chan in (range 10000000)))

stijn14:03:29

this gives the following heap consumption (jvisualvm)

stijn14:03:58

it ever increases, GC'ing doesn't have an impact.

stijn14:03:13

any ideas? am I missing something obvious here?

stijn14:03:39

how can you otherwise consume contents from a channel without accumulating memory?

stijn14:03:19

this is

[org.clojure/core.async "0.2.374"]

stijn14:03:32

and clojure 1.7.0

stijn14:03:58

and once it reaches max heap size limits, GC activity and CPU usage will increase significantly

stijn15:03:39

yes, that might be the case

ghadi19:03:30

you definitely are hitting that

ghadi19:03:52

@alexmiller: That bug should be severity critical, needs some attention

Alex Miller (Clojure team)19:03:09

the patch looked non-trivial

ghadi19:03:19

yeah, it's a code analysis issue

ghadi19:03:37

the fix in the ticket is a ^:once workaround IIRC

ghadi19:03:42

wrap it in a thunk

Alex Miller (Clojure team)19:03:02

if someone with context wanted to walk me through more of it, that would be helpful - I can prob do it tomorrow

ghadi19:03:11

Thankfully we have one of the best code analysis people in here, hi @bronsa

Alex Miller (Clojure team)19:03:46

I have not looked at async tickets in a while and had missed some of the recent activity there

bronsa19:03:35

I'll have to re-dig a bit into the core.async emission to be sure, but I remember that that workaround was the only solution I could find -- basically it makes go close over a thunk wrapping the local rather than closing over the local itself -- then binds that local again from inside the go block, so that clojure's locals clearing analysis can work, manualy bridging fn boundaries

bronsa19:03:05

so the transformation is (conceptually) doing: (let [x 1] (fn [] x)) -> (let [x 1 x (^:once fn* [] x)] (fn [] (let [x (x)] x)))

bronsa19:03:32

the extra Compiler$LocalBinding stuff is to preserve type hints

ghadi20:03:22

we could use the information available in the AST to do locals clearing in the macro emission

ghadi20:03:36

It's essentially a locals clearing issue, right?

bronsa20:03:17

I don't think we can solve it in the general case -- definitely not without additional support from Compiler.java

bronsa21:03:36

@ghadi: right, now I remember why I went with this workaround. The problem is not crossing fn boundaries while doing locals clearing, it's crossing loop boundaries

bronsa21:03:15

Compiler.java would have to be able to understand that one branch inside a loop is only ever visited once in order to clear locals closed over by go blocks