Clojurians Log v2
clj-otel
Hi @steffan! Do you have a date planned to cut a v 2.7 tag? Looking forward to using the synchronous gauge :)
Greetings @steffan - do you think a 0.2.7 might be available sometime next week? There's no rush btw. 🙂 - just checking in. Cheers!
Hello! I know I promised a release which is running a little late. Be assured I am working on it and should get it out soon, perhaps by early next week. I am working (right now!) on Grafana dashboards for all the microservice examples. I think it's worth making the effort now to showcase what one can get "for free" with the metrics provided by OpenTelemetry Java out-of-the-box instrumentation. For example, with a small tweak in Grafana's span metrics configuration, one can get a breakdown of HTTP server metrics by route, which I think is very useful. I've configured a Grafana dashboard that automatically breaks down the metrics by route for any instrumented HTTP service.
We'll take it whenever you get to it. No worries! Thanks again for all you diligence and hard work. Have a great and restful weekend!
We are enjoying using your library at work.
I'm glad to hear you are enjoying clj-otel 😄 I should be releasing 0.2.7 within the next 7 days. I am working on adding Grafana dashboards to the microservices examples. They will feature traces and metrics. The JVM metrics provided by Java OpenTelemetry out-of-the-box are illuminating and deserve to be displayed clearly and coherently.
The following release may be 0.3.0, as that will introduce some long-planned work for the logs signal. There will be some small breakage (sorry!) as changes are made to clj-otel to make entire applications "bound context" aware for correct logging of context throughout, not just the Clojure parts.
Playing around with attributes i got
[otel.javaagent] WARN io.opentelemetry.sdk.metrics.internal.state.DefaultSynchronousMetricStorage - Instrument count has exceeded the maximum allowed cardinality (1999).edit maybe histogram is a way to go... and i am overthinking with the expiration.. i dont know 😕
wanted to add attribute growth is basically entity + bunch of event names.
Why am i tracking it like this? I just wanted to compare different entities on different environments and maybe visualize differences in events.Attributes need to be of low cardinality, as there is a limit of 2000 distinct sets per instrument. This is quite a generous limit. If you are getting anywhere close to this limit, you need to reconsider the attributes you are using.
Say you are attempting to count the number of events by event type across a number of application instances. The number of event types might be too high, but perhaps they can be grouped in some way that still give insight.
Alternatively, consider if raw metrics is an appropriate tool for the task at hand. You may want to consider deriving metrics from logs instead.
If you absolutely must have high cardinality, then Honeycomb might be a good telemetry backend to consider.
If you were using OTLP rather than Prometheus to export metrics, you could try setting system property otel.exporter.otlp.metrics.temporality.preference to LowMemory to make the OTLP exporter use delta temporality (i.e. send changes) rather than cumulative temporality (i.e. send all values since application start) See https://opentelemetry.io/docs/specs/otel/metrics/sdk_exporters/otlp/
I have no idea if this would help, so maybe not a sound idea.