Fork me on GitHub
#onyx
<
2015-11-20
>
robert-stuttaford08:11:19

hey @lucasbradstreet , we had what looks like a leak situation again. i don’t have the time now to dig into it, but sharing it with you as we discussed simple_smile

lucasbradstreet08:11:14

Thanks. Still no retries, but latency is spiking?

lucasbradstreet08:11:54

Do you track "complete latency" anywhere in there?

robert-stuttaford08:11:07

the yellow line in INPUT is 99.9 complete latency

robert-stuttaford08:11:22

a restart fixed it

lucasbradstreet08:11:08

Wow. So 40K being 40 second complete latency

lucasbradstreet08:11:36

So basically it got quite close to being retried

robert-stuttaford12:11:37

what’s weird is that the complete latency recovered but all the individual tasks were spiking up and down like that

robert-stuttaford12:11:45

which suggests some sort of GC churn again

lucasbradstreet12:11:28

Hmm. Flight recorder would be able to tell you whether it's GC related. If all the tasks are spiking together you might be right

robert-stuttaford14:11:29

i’ll definitely get JMX stuff going next week

robert-stuttaford14:11:35

have a great weekend simple_smile

lucasbradstreet14:11:36

Cool. Let us know when you know more. Really want to get this one solved for you