@lucasbradstreet: hi should the riemann logstorm be gone from 0.8.2?

locally, i’ve got it configured to connect but have no riemann running, and i’m getting lots of those errors still

Hmm. It should've been but maybe I did it wrong. I'm only doing it for the timed out messages - maybe the unable to connect ones are still filling up the logs

ah, ok, that’s likely

I'll have a look at it when I get home in 30. I may push up a new version

i think it might be worth doing the same thing - it wrote 42mb in under a minute

also, the CPU was poked

busy testing our stuff for 0.8.2 now

testing without trying to connect to riemann now

That one is going to be trickier because I definitely want to log exceptions. Guess I'll have to rate limit it

yup. without riemann config switched on, no CPU craziness

in other news, 0.8.2 is all good!

Great!

Hmm, when my test can’t connect to riemann it just hangs at creating the connection. Does yours take a while and then time out?

Nevermind, config was bad and we don’t validate well 😛

@robert-stuttaford: I’ve release metrics 0.8.2.2 which exponentially backs off connection failures until a maximum of 1s

so you’ll get a log entry every second which I think is semi acceptable

ok, great, thank you Lucas!

@lucasbradstreet: Possibly of use in the future - https://github.com/brunoV/throttler

Ooh, that’s pretty cool looking. It has to be adaptive though

In what sense?

Well, we’re only throttling the log messages / backing off on writes on failures

Ohh - I see what you mean.

@robert-stuttaford: complete latencies aren’t showing up for me in riemann/grafana. I think it’s due to the period in the names: e.g. 99.9th_percentile_complete_latency

@robert-stuttaford: do they work with you with datadog? I might change the service to use 99_9th or something similar

yep we see them in datadog

Hrm. I wonder if it’s just a grafana thing, or whether making the service [task-name]_peer-id will affect things for you

If it’s just Grafana then I may just leave it the way it is