This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2023-11-27
Channels
- # announcements (4)
- # beginners (41)
- # biff (8)
- # cider (14)
- # clj-kondo (5)
- # clojure (45)
- # clojure-brasil (1)
- # clojure-europe (20)
- # clojure-nl (1)
- # clojure-norway (30)
- # clojure-uk (10)
- # clojurescript (8)
- # cursive (25)
- # datomic (20)
- # emacs (11)
- # events (1)
- # hoplon (9)
- # humbleui (7)
- # hyperfiddle (6)
- # lsp (63)
- # matrix (1)
- # observability (20)
- # off-topic (36)
- # polylith (11)
- # re-frame (2)
- # releases (1)
- # rewrite-clj (6)
- # scittle (42)
- # sql (6)
- # squint (86)
- # tools-deps (9)
What is the preferred observability stack now-a-days? I am hosting my Clojure app on https://fly.io (using a Dockerfile), and http://fly.io provides infra metrics on a Grafana dashboard. Alongside that, they also let me push https://fly.io/docs/reference/metrics/#custom-metrics , if I expose a Prometheus end-point. I’m currently in the process of publishing metrics from my app using https://github.com/clj-commons/iapetos. For logs, I am writing the logs to disk at the moment, but it would be great if I could visualise the logs across all my apps at a single place. I am thinking of creating a https://grafana.com/products/cloud/ account, which will at least get me started for free. I’m using http://pedestal.io/pedestal/0.7-pre/reference/logging.html to write logs to the console, and am planning to deploy https://github.com/superfly/fly-log-shipper for my org and write the logs to https://grafana.com/docs/loki/latest/. I would also love to publish traces from my app, perhaps to Grafana Tempo, but I haven’t investigated this much. So my questions: 1. Do you have any feedback on my observability stack? I do not have any scale right now, so cost is my primary motivation for choosing this stack, at least to start with, but I feel that this stack is promising for the long term as well. 2. Do you have experience with above Clojure libraries? Please warn me of gotchas, if any. 3. What do you use and/or recommend for tracing? (Both Clojure library as well as sink/visualisation). Any gotchas for Tempo that I should be aware of?
I used Grafana Cloud extensively until just few months ago and I can't say anything about about it - Loki is really good, if you grok PromQL then it's pretty similar. I used http://Vector.dev to funnel both metrics and logs to Grafana Cloud, as we had somewhat mixed stack (90% Clojure, 10% Ruby). I didn't stay long enough to implement tracing and try Tempo. In terms of the stack... if it works it works ;-) Although, OpenTelemetry seems to be gaining a lot of traction and it gives you a bit more bang for your buck: one library for both metrics + tracing + logs - it's something I need to look at some point.
not sure if it helps, just some loose thoughts - in my experience investing observability really depends on the stage and use of the product/service you're working on - right now what I'm working on is in private beta, so I'm getting by with structured logs that get turned into metrics, (all funneled to LogTail) + alerting via Sentry. At some point this will not be enough, but I'd rather keep it as simple as possible and re-evaluate once I start identifying gaps in the data I'm collecting
I am also in Private Beta, and I don’t really care about Traces at the moment. I do absolutely need logging and metrics though, and if that stack can be easily extended to tracing in the future, even better. My stack is Clojure and Next.js on the frontend, and I’d like logging and metrics from both to go to Loki. The “shipper” I have linked above is Vector based, so the approach will be similar to what you’ve specified
I can't say anything about about it
<- I’m guessing you meant to say “I can’t say anything bad about it” 😛
on a broader note: I'm still toying with the idea of just using logs rather than logs + metrics, the interesting thing about LogTail (and you can do it with Loki too) is that if logging is set up correctly, you can visualize them just like metrics - it's all events at the end of the day - you just have to figure out how to count them ;-)
It's kinda the whole idea behind https://www.honeycomb.io/
Hmm… I’ll look into the structured logs that get turned into metrics + alerting via sentry approach. Can you explain it in a bit more detail? How are you emitting the logs? Can you share some example log usage code? How does this get converted to metrics and how is it hooked to sentry?
I’ve looked at otel previously, but the ecosystem was nascent in Clojure at the time. Interested in knowing what your setup looks like right now.
I tend to use Java libraries directly for this sort of things, so unless there's a very compelling reason I don't use Clojure wrappers, but admittedly looking into clj-otel is still on my list.
As for logs + Sentry, I'm using Logback as the logging backend - this in turn does two things:
• uses logstash encoder to produce JSON formatted logs that are sent to STDOUT, and these get shipped to LogTail
• Sentry Logback appender to send all logs with error level
I wrote a tiny wrapper around clojure.tools.logging
to inject the MDC as well, so that I don't have to log in JSON directly
I'll just note that we use New Relic at work -- infra agents running on the servers to monitor health; Java agent running in all Clojure apps to monitor health/performance; plus NR's metrics library for explicitly reporting app-specific stuff. All logs are auto-forwarded to New Relic (c.t.l. + log4j2 on our side), including MDC data. The only caveat for all of that was using a "standard" web server (in our case Jetty, rather than something like http-kit which New Relic doesn't support). We're ingesting about 40GB of data per data into New Relic -- mostly metrics and tracing data.
That's a good point - OpenTelemetry comes with a lot of auto-instrumentation, (including JDBC and Jetty too I believe), but Clojure-specific functionality will require manual setup
Understood, thank you @U0JEFEZH6. io.pedestal.log
supports injecting the MDC, so I’ll look into leveraging it. Re: logstash encoder to produce JSON formatted logs
<- this is a new idea for me, I was just planning on relying on vector to turn the log-line to a structured event. Sentry Logback appender to send all logs with error level
<- thanks, I’ll implement this as well.
@U04V70XH6: re: logs are auto-forwarded
, I’m assuming this is also through an agent like Vector, right? My only concern with NR is about whether we will get locked in to it, though admittedly, at this point it shouldn’t really matter to us.
@U051S5XR3 Ideally you avoid parsing your logs in your log collector if you can help it, Vector is pretty ok at this, as it's processing language is quite reasonable (at least compared to the gymnastics I had to do when using Fluentd) but again, in my experience if your application can produce something structured from the get go, and the collector is only responsible for moving data around it is easier to manage down the line. I'll check out pedestal log library, I didn't know about it (or forgot it exists :-))
I’m still not clear on how only logging is generating metrics for you @U0JEFEZH6, can you share an example of say counting the number of times a function was invoked, or estimating the time it takes to run a function? I understand that tracing / instrumentation might auto-publish this in some format that the ingestor understands (how otel works), but are you doing this manually at the moment? (ps: sorry, I haven’t looked at logtail yet, so if the answer is on their docs just say so)
If you're using OpenTelemetry, take a look at https://cljdoc.org/d/com.github.steffan-westcott/clj-otel-api/0.2.4.1/doc/introduction
We use https://github.com/RutledgePaulV/newrelic-clj for New Relic. Makes it easy to defntraced
a function and get metrics about calls and timings etc.
@U051S5XR3 LogTail uses ClickHouse as storage engine, so you can write SQL queries to aggregate logs and visualize them and derive metrics that way. as simple as select count, ... group by ..
query. Loki can do similar things, just with it's own language
if you want the simplest and cheapest setup I'd look into openobserve. I've started using it with https://github.com/BrunoBonacci/mulog right now but can not really tell any stories yet. But all the cruft is in one app and mulog can easily push the events into openobserve via the elastic-compatible api. openobserve is still young and might not be the best option but in the small company I am we don't want to pay big dollars and don't want to setup a stack for each of logging, metrics and tracing. It's all in one
@U051S5XR3 take a look into https://github.com/BrunoBonacci/mulog or https://github.com/amperity/ken for events instead of logging, metrics, tracing. it's a compelling story.