hey @lucasbradstreet , we had what looks like a leak situation again. i don’t have the time now to dig into it, but sharing it with you as we discussed

Thanks. Still no retries, but latency is spiking?

Do you track "complete latency" anywhere in there?

the yellow line in INPUT is 99.9 complete latency

a restart fixed it

Wow. So 40K being 40 second complete latency

So basically it got quite close to being retried

yes

what’s weird is that the complete latency recovered but all the individual tasks were spiking up and down like that

which suggests some sort of GC churn again

Hmm. Flight recorder would be able to tell you whether it's GC related. If all the tasks are spiking together you might be right

i’ll definitely get JMX stuff going next week

have a great weekend

Cool. Let us know when you know more. Really want to get this one solved for you

You too