Fork me on GitHub
#datomic
<
2022-04-01
>
Daniel Jomphe13:04:22

I'm wondering if Cognitect now uses OpenTelemetry to track each one of our Datomic Cloud deployments... I suppose and hope not (hmm, yeah that's safe to assume), because I've begun testing setting this OverrideSettings for the Datomic compute group CloudFormation parameter:

export OTEL_EXPORTER_OTLP_HEADERS="x-honeycomb-team=<API_KEY>,x-honeycomb-dataset=<DATASET>"; export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=; export OTEL_RESOURCE_ATTRIBUTES="service.name=<NAME>"
Just checking in, in case someone has tips or warnings to share. 🙂

jaret13:04:16

Hi @U0514DPR7, > I'm wondering if Cognitect now uses OpenTelemetry to track each one of our Datomic Cloud deployments... No we do not track customers or perform any customer monitoring. In fact, Marketplace specifically requires sellers not to monitor their customers:

Daniel Jomphe14:04:44

Thanks Jaret. I realize what I wrote might seem sensitive. My focus was more on the technical aspect and consequences of using OverrideSettings the way we did.

Joe Lane18:04:31

Hey @U0514DPR7! Telemetry insights (via OTEL or vanilla HoneyComb) is an exciting addition to any project. The use of OverrideSettings is only intended for support scenarios with guidance from Datomic Support and using Cloud like this is considered an "unsupported" configuration. My recommendation would be to keep things in userspace by: • Depending on the OTEL library of your choice explicitly in your deps.edn • Store these values in SSM and fetch them using https://docs.datomic.com/cloud/ions/ions-reference.html#get-params Having worked on a project with HoneyComb before, I'm very excited for you.

Daniel Jomphe18:04:38

Hi @U0CJ19XAM, thanks a lot for your response! So there's 3 ways to configure OTel, and I think you suggest we use the 3rd one. Can you confirm? 1. https://opentelemetry.io/docs/instrumentation/java/automatic/: I think we'd like to use this mode. But I suppose Datomic Cloud will "never" allow it or offer an option to boot with it? 2. https://github.com/open-telemetry/opentelemetry-java/tree/main/sdk-extensions/autoconfigure: For this mode, we must depend explicitly on OTel's libs in our deps.edn. And upon loading, it needs to find some config envars. That's what we set up with for now. This seemed like the best next thing to try. 3. https://opentelemetry.io/docs/instrumentation/java/manual/: For this mode too, we must depend explicitly on OTel's libs in our deps.edn. And then we must expend the most energy manually instantiating and configuring every desired component, as soon as possible during the app startup. We might see this as 3 levels of easiness and control. I wonder if there are practical reasons why you seem to steer us towards #3 and block us from #1 and #2. Probably a bunch of reasons. After all, Datomic Cloud Ion is a highly opinionated PAAS. (I've also developed for Google App Engine, I'm used to have to go back to the drawing board to adapt to such environments.) Also, I wonder why Datomic Cloud doesn't expose some control knobs for us to define envars. It might be a good practice in cloud architecture to not depend on those, and instead strive to use SSM in all possible cases... Still, any library that we might want to use that depends on the presence of a regular-style envar will require adaptation to such an environment. Thinking about it, I'm not sure I should complain. 🙂 • On one hand, it forces us to slow down and spend much more time on this kind of things than we'd think we'd like to do. • On the other hand, this all might be forcing good habits on us that (it seems) I'm not yet entirely equipped to foresee the value. And it might save us from loading many useless java packages... Overall I believe we'll continue with solution #2 for now, and depending on your answers, we might invest later in migrating to #3 once we've sufficiently installed the observability we want to achieve quickly.

Joe Lane18:04:30

@U0514DPR7 (untested advice) Have you investigated invoking (System/setProperty "otel.exporter.otlp.headers" "x-honeycomb-team=<API_KEY>") before "loading" (instantiating?) your autoconfigure option (#2)?

Joe Lane18:04:10

^^ This should support #2 and #3

Daniel Jomphe18:04:54

@U0CJ19XAM I feared I might set the property too late for the lib to pick it up, and expose us to config race conditions. But I realize if I loaded a clojure namespace first in our ion manifest namespace, I'd be able to guarantee the property is set before our first use of OTel. This might be a valid way to set it up. (Although 10 years ago when I was still doing Java, I remember not being satisfied by setProperty.)