clj-otel

devurandom 2024-08-05T09:49:35.240269Z

Hi! Do you have suggestions how to use get-text-map-propagator to extract a context that I want to link to? (As opposed to becoming a child of.)

(let [creation-context (.extract otel-propagator WHAT_GOES_HERE job-data otel-getter)
  (span/with-span! {:span-kind  :consumer
                    :links      [[creation-context]]}
    ,,,))
Should WHAT_GOES_HERE be (context/dyn), (context/current) or (context/root) or something else? From what I understood, I want to link pass with-span! a context that is empty apart from the trace ID and span ID I want to link to (the ones in the job-data). Thus using (context/current) or (context/dyn) would be wrong? extract would, as I understand, in the worst case return the context fed to it unaltered, which (if passed the current context) would mean I would link to the trace ID and span ID of my current context. (Not sure about the attributes. Could feeding context/current to extract also result in wrong attributes on the creation-context?)

steffan 2024-08-05T10:56:19.935119Z

The :links option works with SpanContext (an object that contains span ID, trace ID and flags), but also has conveniences to extract the SpanContext from a span or context. Here is a minimal example that creates two spans, where the ::bar span contains a single link to the ::foo span:

(defn do-foo []
  (span/with-span! ::foo
    (span/with-span! {:name   ::bar
                      :parent nil
                      :links  [[(context/current)]]})))
Viewing the ::bar span in Jaeger shows a References section containing the link.

devurandom 2024-08-05T10:57:09.736929Z

Thanks!

devurandom 2024-08-05T10:58:31.145229Z

So context/current should work. Here I sometimes don't get the "back links" in Aspire.Dashboard, despite the code definitely being executed. I'm trying to figure out what's going on. Thought maybe I am using the wrong context -- maybe it has bits of the wrong data... But I also get System.InvalidOperationException: Multiple traces found with trace id '31...'. in the console logs of Aspire.Dashboard. Maybe it's related...

devurandom 2024-08-05T11:00:43.038439Z

The TextMapPropagator works with Context, not SpanContext. So I cannot just pass that instead, right?

steffan 2024-08-05T11:29:28.244679Z

TextMapPropagator are for reading and writing context information, usually as HTTP headers. I don't see the connection here with what you are trying to achieve.

devurandom 2024-08-05T11:33:11.753069Z

I have a (custom) job queue. My goal is to apply https://opentelemetry.io/docs/specs/semconv/messaging/messaging-spans/ to it, which suggests to link to the message (job) creation context from the message (job) processing context. To do that I transport the creation context through the queue using TextMapPropagator. At the receiving end (when job processing starts) I want to extract the creation context from the job and set :links on the processing context. This already works in principle when using injecting context/current into my job in the "create" span and extracting it into context/current on the other side in the "process" span. But for some reason it sometimes does not work and the "process" span never shows up in Aspire.Dashboard. I wasn't able to figure out the pattern yet, so I am trying to figure out whether I am using all the tools correctly / within their spec.

devurandom 2024-08-05T11:26:06.266789Z

I'm a bit confused about context/current vs context/dyn. In the context of auto instrumentation from the OTel java agent, I will never get into a situation where context/dyn contains anything other than context/current (unless I set it myself), right? Even with-span! does not call bind-context! / touch the bound context, right?

steffan 2024-08-05T11:37:13.429179Z

The value of (context/dyn) will always be the same as (context/current) unless you are working with clj-otel bound context in async code, where you will set it yourself through various clj-otel functions. Think of bound context as an "override" of the current context for sections of async Clojure code.

devurandom 2024-08-05T11:43:24.442599Z

But there are no automatisms using it, right?

devurandom 2024-08-05T11:45:44.535239Z

I think the OTel java agent also does some magic to follow traces through asynchronous processing. (I think I saw some java agent code dealing with Executor or something like that.) Do you know whether this also applies to Clojure? Will automatic OTel traces also follow futures, agents, core.async and the likes?

steffan 2024-08-05T11:51:11.508839Z

I don't think there is any support for general Java/Clojure constructs provided by OpenTelemetry. Instead, its support is for specific Java libraries and frameworks.

devurandom 2024-08-05T12:09:59.912359Z

From issues like https://github.com/open-telemetry/opentelemetry-java-instrumentation/issues/9324 and bits of code like: • https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/instrumentation/kotlinx-coroutineshttps://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/instrumentation/executors it seemed that there is some capability of instrumentation of asynchronous execution built into the java agent. But this is never automatic, and I always have to manually specify which thread pool / executor / ... to instrument?

steffan 2024-08-05T12:20:37.218869Z

Yes, this is what I mean by supporting specific libraries and frameworks. The design of the instrumentation requires knowledge of the program to be instrumented.

steffan 2024-08-05T12:21:26.991359Z

It does not work at a general level to work with all Java programs.

devurandom 2024-08-05T12:23:55.437319Z

Ok, so if I wanted a trace to follow a certain future (e.g.) I would wrap that specific call to future with bind-context!? Or if I wanted no parent/child relationship, but only a link from within the future to the span where the future was originally spawned, I would use code like in your example in https://clojurians.slack.com/archives/C034UN5273N/p1722855379935119?thread_ts=1722851375.240269&cid=C034UN5273N ?

devurandom 2024-08-05T12:27:19.564349Z

Or alternatively I could create some Context.taskWrapping(mySpecificExecutor) (or ExecutorService) and run my functions on that, but just for the functions where I want the context to follow them onto the executor's thread, not for all functions anywhere on my Clojure program that run on some thread pool (I assume a thread pool is an example for an executor)?

steffan 2024-08-05T12:28:40.621639Z

To work with asynchronous Clojure code, you can use bound or explicit context passing. A future will convey the bound context as you suggest.

steffan 2024-08-05T12:30:57.339419Z

I haven't added support for Executor in clj-otel, I have not looked at them in detail. From what I understand there are some facilities in OpenTelemetry Java to set the current context in the executed tasks.

devurandom 2024-08-05T12:31:58.880139Z

> there are some facilities in OpenTelemetry Java to set the current context in the executed tasks Yes, that's how I read the code, too: https://github.com/open-telemetry/opentelemetry-java/blob/6fc1d216ca0f199670a1cc5b081ad764d4a043fb/context/src/main/java/io/opentelemetry/context/Context.java#L119-L121