observability

timo 2022-01-12T14:22:45.001600Z

Hi, do you guys use ulog or ken in conjunction with metrics from iapetos or do you don't need that anymore? How do you see a timeline of performance issues in something like Grafana with ulog/ken?

timo 2022-01-12T15:05:14.001700Z

Looking at mulog samplers right now

greglook 2022-01-12T18:00:21.001900Z

We use ken (primarily with http://honeycomb.io) but we also send events marked with certain keys to our metrics system, where they get recorded to InfluxDB and from there surfaced in Grafana dashboards. Main thing you'd do is write a function to subscribe to the ken tap and then record your events to your backend of choice.

timo 2022-01-12T20:10:40.002100Z

so you would then probably compute something like counters/gauges/histograms within the backend of choice? (sorry, i only know prometheus where you get it properly served on /metrics)

timo 2022-01-12T20:11:08.002300Z

actually watching your talk right now 🙂

👍 1
greglook 2022-01-12T23:14:08.002700Z

Ah, I understand what you're asking better now - yeah, in our case you can do aggregates within the InfluxDB query itself, so basic stuff like sum/count/mean/p99 etc are directly computable on the raw metrics. We actually run http://riemann.io/ as the destination for all our metrics, which is then responsible for writing to influx - but it also gives us a place to do more sophisticated logic to power alerts and decide when to post to slack or page someone.

👍 1