Fork me on GitHub
#clj-otel
<
2024-08-05
devurandom09:08:35

Hi! Do you have suggestions how to use get-text-map-propagator to extract a context that I want to link to? (As opposed to becoming a child of.)

(let [creation-context (.extract otel-propagator WHAT_GOES_HERE job-data otel-getter)
  (span/with-span! {:span-kind  :consumer
                    :links      [[creation-context]]}
    ,,,))
Should WHAT_GOES_HERE be (context/dyn), (context/current) or (context/root) or something else? From what I understood, I want to link pass with-span! a context that is empty apart from the trace ID and span ID I want to link to (the ones in the job-data). Thus using (context/current) or (context/dyn) would be wrong? extract would, as I understand, in the worst case return the context fed to it unaltered, which (if passed the current context) would mean I would link to the trace ID and span ID of my current context. (Not sure about the attributes. Could feeding context/current to extract also result in wrong attributes on the creation-context?)

steffan10:08:19

The :links option works with SpanContext (an object that contains span ID, trace ID and flags), but also has conveniences to extract the SpanContext from a span or context. Here is a minimal example that creates two spans, where the ::bar span contains a single link to the ::foo span:

(defn do-foo []
  (span/with-span! ::foo
    (span/with-span! {:name   ::bar
                      :parent nil
                      :links  [[(context/current)]]})))
Viewing the ::bar span in Jaeger shows a References section containing the link.

devurandom10:08:31

So context/current should work. Here I sometimes don't get the "back links" in Aspire.Dashboard, despite the code definitely being executed. I'm trying to figure out what's going on. Thought maybe I am using the wrong context -- maybe it has bits of the wrong data... But I also get System.InvalidOperationException: Multiple traces found with trace id '31...'. in the console logs of Aspire.Dashboard. Maybe it's related...

devurandom11:08:43

The TextMapPropagator works with Context, not SpanContext. So I cannot just pass that instead, right?

steffan11:08:28

TextMapPropagator are for reading and writing context information, usually as HTTP headers. I don't see the connection here with what you are trying to achieve.

devurandom11:08:11

I have a (custom) job queue. My goal is to apply https://opentelemetry.io/docs/specs/semconv/messaging/messaging-spans/ to it, which suggests to link to the message (job) creation context from the message (job) processing context. To do that I transport the creation context through the queue using TextMapPropagator. At the receiving end (when job processing starts) I want to extract the creation context from the job and set :links on the processing context. This already works in principle when using injecting context/current into my job in the "create" span and extracting it into context/current on the other side in the "process" span. But for some reason it sometimes does not work and the "process" span never shows up in Aspire.Dashboard. I wasn't able to figure out the pattern yet, so I am trying to figure out whether I am using all the tools correctly / within their spec.

devurandom11:08:06

I'm a bit confused about context/current vs context/dyn. In the context of auto instrumentation from the OTel java agent, I will never get into a situation where context/dyn contains anything other than context/current (unless I set it myself), right? Even with-span! does not call bind-context! / touch the bound context, right?

steffan11:08:13

The value of (context/dyn) will always be the same as (context/current) unless you are working with clj-otel bound context in async code, where you will set it yourself through various clj-otel functions. Think of bound context as an "override" of the current context for sections of async Clojure code.

devurandom11:08:24

But there are no automatisms using it, right?

devurandom11:08:44

I think the OTel java agent also does some magic to follow traces through asynchronous processing. (I think I saw some java agent code dealing with Executor or something like that.) Do you know whether this also applies to Clojure? Will automatic OTel traces also follow futures, agents, core.async and the likes?

steffan11:08:11

I don't think there is any support for general Java/Clojure constructs provided by OpenTelemetry. Instead, its support is for specific Java libraries and frameworks.

devurandom12:08:59

From issues like https://github.com/open-telemetry/opentelemetry-java-instrumentation/issues/9324 and bits of code like: • https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/instrumentation/kotlinx-coroutineshttps://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/instrumentation/executors it seemed that there is some capability of instrumentation of asynchronous execution built into the java agent. But this is never automatic, and I always have to manually specify which thread pool / executor / ... to instrument?

steffan12:08:37

Yes, this is what I mean by supporting specific libraries and frameworks. The design of the instrumentation requires knowledge of the program to be instrumented.

steffan12:08:26

It does not work at a general level to work with all Java programs.

devurandom12:08:55

Ok, so if I wanted a trace to follow a certain future (e.g.) I would wrap that specific call to future with bind-context!? Or if I wanted no parent/child relationship, but only a link from within the future to the span where the future was originally spawned, I would use code like in your example in https://clojurians.slack.com/archives/C034UN5273N/p1722855379935119?thread_ts=1722851375.240269&amp;cid=C034UN5273N ?

devurandom12:08:19

Or alternatively I could create some Context.taskWrapping(mySpecificExecutor) (or ExecutorService) and run my functions on that, but just for the functions where I want the context to follow them onto the executor's thread, not for all functions anywhere on my Clojure program that run on some thread pool (I assume a thread pool is an example for an executor)?

steffan12:08:40

To work with asynchronous Clojure code, you can use bound or explicit context passing. A future will convey the bound context as you suggest.

steffan12:08:57

I haven't added support for Executor in clj-otel, I have not looked at them in detail. From what I understand there are some facilities in OpenTelemetry Java to set the current context in the executed tasks.

devurandom12:08:58

> there are some facilities in OpenTelemetry Java to set the current context in the executed tasks Yes, that's how I read the code, too: https://github.com/open-telemetry/opentelemetry-java/blob/6fc1d216ca0f199670a1cc5b081ad764d4a043fb/context/src/main/java/io/opentelemetry/context/Context.java#L119-L121