Clojurians Log v2
clj-otel
Hi! I'm getting starting using otel, and starting with the javaagent. I really like how we get so much instrumentation out of the box. For a http service, I'm using the Prometheus exporter and visualizing http.server.duration and http.client.duration metrics in grafana. Those metrics have a maximum bucket boundary of 10s (https://opentelemetry.io/docs/specs/otel/metrics/semantic_conventions/http-metrics/#metric-httpserverduration), so nearly all of our requests get bucketed as Inf+ . My understanding is that I'll need to create a new Metric and View with histogram buckets that are more appropriate for our use case. Is there a way for me to do this and have these metrics exported using the javaagent's Prometheus exporter?
https://opentelemetry.io/docs/specs/otel/metrics/api/#instrument-advisory-parameters are used to indicate preferred histogram bucket boundaries. Advisory parameters are currently experimental and available only with programmatic configuration of the OpenTelemetry SDK. They are not yet supported by the agent or autoconfiguration. Once advisory parameters become part of the OpenTelemetry API, I will be sure to add support for them in clj-otel. Until then, all I can suggest is taking a look at ExtendedDoubleHistogramBuilder that is returned when getting a DoubleHistogramBuilder
https://github.com/open-telemetry/opentelemetry-java/blob/main/extensions/incubator/src/main/java/io/opentelemetry/extension/incubator/metrics/ExtendedDoubleHistogramBuilder.java
Cheers. To summarize and confirm my understanding: if I want to add histograms with different bucket boundaries, I need to use the sdk rather than the javaagent (at least right now). Correct?
Yes, that is a good summary. This https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/instrumentation-api-semconv/src/main/java/io/opentelemetry/instrumentation/api/instrumenter/http/HttpMetricsUtil.java#L25 in the OpenTelemetry instrumentation code may be a good starting point.
Cheers. Next on my list is to figure out whether instrumentation is available for the libraries we're suing. Looking at https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/docs/supported-libraries.md#libraries--frameworks it's not clear: • jetty: the application server section doesn't include info as to whether there's standalone instrumentation available • apache httpclient says it's available: https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/instrumentation/apache-httpclient/apache-httpclient-4.3/library but no instructions on how to use it — looks like it's here: https://central.sonatype.com/artifact/io.opentelemetry.instrumentation/opentelemetry-apache-httpclient-4.3 • java-net-http client looks pretty straightforward: https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/instrumentation/java-http-client/library • looks like jvm runtime metrics are here: https://central.sonatype.com/artifact/io.opentelemetry.instrumentation/opentelemetry-runtime-telemetry-java17
Do you happen to have any insight into adding the instrumentation for jetty server? We're using the ring-jetty-adapter.
Looking at https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/instrumentation/jetty, it looks like it might be agent-only 😞
As https://cljdoc.org/d/com.github.steffan-westcott/clj-otel-api/0.2.4.1/doc/concepts#_instrumenting_libraries_and_applications in the documentation, you should use the agent if possible. If you can't, clj-otel offers https://cljdoc.org/d/com.github.steffan-westcott/clj-otel-api/0.2.4.1/doc/guides#_use_ring_middleware_for_server_span_support and https://cljdoc.org/d/com.github.steffan-westcott/clj-otel-api/0.2.4.1/doc/guides#_use_pedestal_interceptors_for_server_span_supportwhich create HTTP server spans and metrics. There are also https://cljdoc.org/d/com.github.steffan-westcott/clj-otel-api/0.2.4.1/doc/guides#_manually_add_http_response_data_to_a_client_span. The microservices examples demonstrate using the agent (`auto-instrument`) and without the agent (`manual-instrument`).
All the examples use Jetty server.
> As https://cljdoc.org/d/com.github.steffan-westcott/clj-otel-api/0.2.4.1/doc/concepts#_instrumenting_libraries_and_applications in the documentation, you should use the agent if possible. Okay, I'm a bit confused here. This whole thread is about me not being able to use the agent because the histogram buckets don't capture our latency range.
I definitely want to use the agent, but I also need metrics with histograms with different bucket boundaries.
Sorry, my answer was more general and confusing. If you require programmatic configuration of the SDK, you will not be using the agent.
Gotcha. Thanks!
And thanks for the answers above. I haven't yet gotten an answer to this question when I asked in the CNCF slack: https://cloud-native.slack.com/archives/C014L2KCTE3/p1697408608877929
My experience is that you should receive an expert answer within a day or so in #otel-java in the CNCF Slack. It is a very good resource for questions like yours.
There is a lot of benefit to using the agent. clj-otel makes a reasonable effort to support semantic conventions for HTTP, though it will never be as good as the agent. Support for configurable bucket boundaries will surely land in the agent, though I don't know when.
Definitely. If I could use the agent in this case, I would.