2024-06-17 clj-otel | Clojure Slack Archive

clj-otel

lepistane 2024-06-17T15:50:08.973649Z

I am having little trouble with getting service.name exported with prometheus. These are my JVM opts

:jvm-opts ["-javaagent:grafana-opentelemetry-java.jar"
             "-Dotel.resource.attributes=service.name=dev-webserver"
             "-Dotel.metrics.exporter=prometheus"
             "-Dotel.metric.export.interval=5000"
             "-Dotel.logs.exporter=none"
             "-Dotel.traces.exporter=none"
             ]

agent is started and i do have metrics: http://localhost:9464/metrics problem is that they are missing important label like service name

jvm_class_count{otel_scope_name="io.opentelemetry.runtime-telemetry-java8",otel_scope_version="2.4.0-alpha"} 37413.0

If i do collector setup - autoconfigure + default exporter (basically removed everything except service.name from jvm opts) i get this exported

jvm_class_count{exported_instance="adbb4085-ef54-4823-be3c-3747ba027012", exported_job="cube-app", instance="otel-collector:9464", job="otel-collector"}

Having exported_jobs is a must edit this seems to be prometheus config and related to relabeling. resource_to_telemetry_conversion might also be helpful

steffan 2024-06-17T17:14:21.032489Z

The OpenTelemetry Prometheus exporters map telemetry data to Prometheus format. This (among other changes) means that service.name (an OpenTelemetry semantic attribute) is mapped to job or exported_job label depending on which exporter is used.

👍 1

steffan 2024-06-17T17:19:58.516319Z

Right at this moment I am designing Grafana dashboards for the clj-otel microservice examples. The PromQL queries I'm using include selecting by job label:

histogram_quantile(0.9, sum(rate(http_server_request_duration_seconds_bucket{job="sentence-summary-service"}[$__rate_interval])) by (le))

👍 1

lepistane 2024-06-17T17:23:56.540119Z

Is job label visible when you do do GET on localhost:9464/metrics?

steffan 2024-06-17T17:23:56.775419Z

I haven't found any need to remap attributes or map resource attributes to additional labels on the exported telemetry.

👍 1

steffan 2024-06-17T17:25:39.307819Z

I'm using the prometheusremotewrite (push) exporter, and job label is present on the exported telemetry.

lepistane 2024-06-17T17:26:52.615729Z

Alright thanks for clarifications. I was looking at this wrong then expecting that /metrics should have that label or all of the agent's metrics should have the service-name if i add it. My local prometheus isn't including the job to the agent metrics

steffan 2024-06-17T17:27:36.161459Z

For the prometheus (pull) exporter, you may need to add a label containing resource information, as you say.

👍 1

steffan 2024-06-17T17:37:43.177399Z

resource_to_telemetry_conversion adds all the resource attributes as labels to the exported telemetry. For more fine-tuned control, you could consider using the contrib OpenTelemetry ~~instrumentation agent~~ collector (this is a super-charged version of the ~~agent~~ collector with extra knobs) to add only the labels you need : https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/prometheusexporter/README.md (the transform processor is in contrib, but not the core distribution of the agent).

👍 1

steffan 2024-06-17T17:38:14.675519Z

I haven't tried this myself, so do let us know if you go down this route!

👍 1

Clojurians Log v2

clj-otel