What are people's approaches these days to propagating diagnostic/telemetry context (mdc, ndc, otel context etc.) in asynchronous Clojure applications? With clojure core's future etc, core.async, manifold, promesa, thread pools, and other possibilities available, are there established best practices in the community? Example: the app handles a request with a unique id, and all sorts of code pieces, using one or the other of the previously mentioned libs, work together to serve the request, while emitting logs at key points. When you look at the logs you want to gather all the log entries that were produced during the handling of request 111, which means whatever code writes the actual log line, it has to have request id 111 at hand.
There's #C010TGGL02X channel :-) That said, at $dayjob we use do the following: • Otel Agent for automatic instrumentation of our HTTP "framework" (Jetty-based web server and a Hato wrapper which uses JDK's built-in HTTP client), most Java libraries that we use are auto-instrumented (Logback, Jedis, Mongo client etc) • clj-otel for injecting span data to Ring routes • Otel SDK enabled in GCP's PubSub (two lines of code if I remember correctly) The point is that your stack will really determine how you have to instrument all of this, there's no one size fits all approach here.
Thanks for the channel link, I tried telemetry and stuff like that, but not observability 🙂
do dynvars cross async boundaries?
they also don't work great with early return stuff, right?
@nbtheduke they cross some: https://clojure.org/reference/vars#conveyance but I'm unsure of some others
and yes they cross async boundaries. what early return?
yes, that's what i meant
Well the way they cross async boundaries will be bespoke per the kind of boundary
And yeah that usually means threading bound-fn in some places
clj-otel author here 👋🏼 For instrumenting asynchronous applications, clj-otel originally required explicit context to be passed as an argument from the application. This works fine but can be inconvenient for the application to manage. Later, clj-otel added a dynamic var *bound-context* to be used as a default. The application needs to pass context to futures and the like, which can be a nuisance because it's easy to forget. When using dynamic vars, bound-fn is needed in some cases. clj-otel has examples on this for various async libraries. clj-otel allows either explicit or bound context usage for flexibility.
Thanks for all the input folks. @steffan have you ever compared the performance of explicit vs bound context when there's a lot of processes moving between threads, like in an app that uses core.async heavily?
@imre To be honest, I have not done any performance analysis on clj-otel. My guess is that using dynamic vars may increase GC pressure and thus negatively impact long-running applications. Incidentally, when making the example core.aync.flow app, explicit context was the natural fit.
FWIW, we use async closures. We wrote https://github.com/multiplyco/quiescent (no, not that one), which uses https://github.com/multiplyco/scoped under the hood to convey contexts using ScopedValue.
My measurements of ScopedValue vs. ThreadLocal was considerably less overhead per closure, favouring ScopedValue.