Fork me on GitHub
#aleph
<
2021-09-02
>
Ben Sless07:09:56

Is there an issue relating to memory leaks with aleph? When profiling with wrk2 I noticed terrible results which eventually got me to investigate gc and memory usage. Looks like ~50% of the heap is consumed by LinkedList, Deferred and ReentrantLock. Also looks like old gen keeps growing indefinitely. I can share more details and help with reproduction

Ben Sless10:09:30

Anyway, reproduction is easy run Aleph server with non blocking ring handler for 10 minutes at a high rate (I tried 10k qps), attach visual vm, behold the mess

Ben Sless10:09:15

The garbage can be collected once the benchmark finishes, could be perhaps some link between open connections and deferred-s not being collected?

nha20:09:07

I am not sure if this can be related: https://github.com/juxt/yada/issues/324

Ben Sless20:09:51

I'll give it a shot tomorrow, thanks for the hint!

Ben Sless11:09:53

Still leaks memory

😵 2
Matthew Davidson (kingmob)12:09:18

Zach just cut dirigiste 1.0.0, based off 0.1.6-alpha3. However, I don’t know who besides Zach can upload it to clojars. It’s not one of his repos that we added to clj-commons. Not sure if it fixes your issue, but fyi

Ben Sless12:09:43

You can ping me when he releases it an I'll give it another try. In the meanwhile, I tried passing an executor both to the server and the wrapper which creates the original deferred value. No luck.

Matthew Davidson (kingmob)16:09:40

@ben.sless My apologies, it’s available now. He released it to Maven Central, not clojars: https://search.maven.org/artifact/io.aleph/dirigiste/1.0.0/jar

Ben Sless17:09:15

Nice. I'll probably be able to squeeze in a test in the next couple of hours

Ben Sless18:09:01

ugh, still broken. I'm not going to hunt the cause down but if you need help reproducing it feel free to ping me

Ben Sless10:09:05

I'm sorry if it sounds like I'm abandoning it. It's just something I accidentally came across while benchmarking different servers. I don't use aleph and I don't think any of my colleagues do, either.

Ben Sless06:09:51

update: Made some changes which should not be related and now the leak is gone

2
Ben Sless10:09:07

I changed two things: • return byte array instead of input stream in muuntaja • did not inject match in reitit

Matthew Davidson (kingmob)04:03:12

@ben.sless If you have the time/interest to check it out again, I think I have a fix for the memory leak with ByteArrayInputStreams: https://github.com/clj-commons/aleph/issues/523#issuecomment-1075429913

Ben Sless04:03:36

Thank you for bringing this up, I intend to do another run this weekend 👍

Matthew Davidson (kingmob)04:03:33

I need to push the branch up then…

Ben Sless04:03:29

No pressure, worst case I can use a local install

mccraigmccraig08:09:26

@ben.sless i've seen a leak issue with manifold and stream/transform - https://github.com/clj-commons/manifold/issues/197

Ben Sless08:09:19

not even sure if it's the same issue :thinking_face:

Ben Sless08:09:52

how can we find the cause?

mccraigmccraig08:09:41

yeah, it might not be the same issue - this was quite specific to stream/transform. i played around in yourkit for a while - i could reliably duplicate the issue on our prod system, but never managed to get a minimal test-case, and in the end i ran out of time