core-async

Ben Sless 2022-10-14T08:24:34.927309Z

I have a service with a huge memory leak. After analyzing the heap dump I see it all ties back to timeouts in alts. Is it fixed in the latest version of core async? Any workarounds or mitigations? I remember seeing a few Jiras about it

Alex Miller (Clojure team) 2022-10-14T12:14:15.361779Z

There is an outstanding issue about head holding of closed over values, but that’s not directly tied to timeouts

Alex Miller (Clojure team) 2022-10-14T12:14:44.373679Z

Can you explain more about what you’re seeing?

Alex Miller (Clojure team) 2022-10-14T12:17:18.201229Z

And are you using the latest version or something older? 1.5.640 fixed an issue with clearing alt handlers

2022-10-14T12:53:09.011569Z

In systems with callbacks in java, timers can often be a source of leaks, because the inversion of control plus timers being a global resource means a timer is a strong reference to whatever data until it expires, and core.async doesn't remove timers when they are not chosen by an alt, it just marks them as cancelled

Ben Sless 2022-10-14T12:55:12.109289Z

That may be it. We'll rerun with latest version and check again

Ben Sless 2022-10-14T12:55:41.799409Z

And async-234 is the one I remembered

Ben Sless 2022-10-14T12:58:52.292719Z

I can't explain it succinctly and I don't have the heap dump at hand, if you'd like I could probably share it with you. When you analyze it with Eclipse MAT it points to the synchronized queue behind timers at the dominating object which holds all the references in the heap

2022-10-14T13:03:17.397379Z

I am perfectly happy to blame the issue laid out in async-234 for any leaks in a program that uses core.async without any more evidence (I reported it and wrote the patches)

2022-10-14T13:09:53.334469Z

At work we had a lot of trouble with memory leaks in our project that heavily uses core.async and I ended up reducing our direct usage of timeouts. This was all a year or two before I figured out async-234(reducing timer usage was a wild guess "what is global and could be holding a ref?!"), so hard to definitively say.

Ben Sless 2022-10-14T13:12:18.608839Z

And yes, all our leaks seem to point to timers in alts

vlaaad 2022-10-14T14:25:01.227729Z

Why is there no drain in core.async? I can create my own impl for sure, but it feels so essential it should be in the library as a basic primitive…

Ben Sless 2022-10-14T16:01:34.528389Z

Doesn't reduce accomplish the same thing?

vlaaad 2022-10-14T16:37:49.834139Z

reduce’s fn is 2 args, I only need one

vlaaad 2022-10-14T16:38:16.069909Z

90% of the time it will be something like tap> or println

Ben Sless 2022-10-14T16:46:57.962119Z

Reduce and ignore the accumulator? Basically copy run!'s definition

vlaaad 2022-10-14T16:53:26.706039Z

Yes, my question is why this doesn't exist in core.async?

Ben Sless 2022-10-14T17:33:44.908199Z

Oh. Uhh, reasons? 🙃