observability

lukasz 2022-09-13T21:18:24.164539Z

My google-fu is failing me - is there are reliable and maintained Java (or Clojure) client for Prometheus' remote write endpoint? I've seen some examples, but they were far from finished and it looks like it involves messing around with protocol buffers

Ivar Refsdal 2022-09-14T10:22:19.765409Z

https://github.com/BrunoBonacci/mulog looks solid, and has a prometheus publisher. I have not tried it myself though. CC @bruno.bonacci

2022-09-14T10:50:33.395779Z

Hi @lukaszkorecki if you are looking just for the standard Prometheus monitoring you best option is: https://github.com/clj-commons/iapetos Mulog is a event-based monitoring library, while Prometheus is a metrics-based. the difference is huge! If you want to try an event-based monitoring system then something like mulog + Elasticsearch (or Cloudwatch) is your best starting point. You can add Prometheus to support existing alerting systems if you have the need but mulog + Elasticsearch generally will be a much more powerful combo.

lukasz 2022-09-14T14:26:07.916509Z

@bruno.bonacci thanks! I'll check out iapetos - looks like it requires using the push gateway instead, rather than remote write (I think they're different).

lukasz 2022-09-14T14:26:30.401559Z

@ivar.refsdal that looks like a logging solution - we have that covered via cambium + logback

2022-09-14T14:31:36.019209Z

mulog is a logging and tracing solution, it logs data-points as "events". Logback just logs messages. with mulog you can do everything you do with your traditional logging + everything you do with your metering system (like Prometheus) + everything you do with a tracing system (Zipkin and co) This talk explain the motivations and how it works https://www.youtube.com/watch?v=P1149dWnl3k&ab_channel=LondonClojurians

lukasz 2022-09-14T15:52:41.103369Z

interesting, I didn't realize I can use it for both - I'll check it out