I’m adding tracing in a concurrent app where parent tasks spawn child tasks. However, I’m facing an issue where child task spans aren’t properly attached to their parent spans. My initial approach looks like this:
(defn start-task!
[]
(span/with-span! :start-task
(let [do-sub-task (fn [sub-task]
(span/with-span! :sub-task
;; do work
))
sub-tasks []]
(doseq [sub-task sub-tasks] (submit! executor do-sub-task sub-task)))))
This results in disconnected traces - the sub-task spans aren’t linked to the parent span in trace visualizations.
I found that explicitly passing the context to child tasks resolves the issue:
(defn start-task!
[]
(span/with-span! :start-task
(let [ctx (context/dyn)
do-sub-task (fn [sub-task]
(span/with-span! {:name :sub-task :parent ctx}
;; do work
))
sub-tasks []]
(doseq [sub-task sub-tasks] (submit! executor do-sub-task sub-task)))))
Question
Is explicitly capturing and passing the parent context the recommended pattern for preserving trace relationships across concurrent boundaries? Are there alternative approaches or best practices I should consider? Is this best done by using the steffan-westcott.clj-otel.context API?thx!
Another approach I tried was using (span/get-span-context).
(defn start-task!
[]
(span/with-span! :start-task
(let [ctx (span/get-span-context)
do-sub-task (fn [sub-task]
(span/with-span! {:name :sub-task :parent ctx}
;; do work
))
sub-tasks []]
(doseq [sub-task sub-tasks] (submit! executor do-sub-task sub-task)))))
However, this throws an exception.
class io.opentelemetry.api.internal.AutoValue_ImmutableSpanContext cannot be cast to class io.opentelemetry.context.Context (io.opentelemetry.api.internal.AutoValue_ImmutableSpanContextAs https://cljdoc.org/d/com.github.steffan-westcott/clj-otel-api/0.2.7/doc/concepts#_instrumenting_asynchronous_clojure_code, when manually instrumenting asynchronous code, use either explicit context passing or the bound context value. The second code sample you gave is a good example of the explicit context passing approach.
To compare the two approaches, look for the bound-async and explicit-async directories in each of the microservices examples. Here is one example : https://github.com/steffan-westcott/clj-otel/tree/master/examples/microservices/auto-instrument/middleware/sentence-summary-service/src/example/sentence_summary_service
For further description of context, current context and bound context please see https://cljdoc.org/d/com.github.steffan-westcott/clj-otel-api/0.2.7/doc/concepts#_context.