Clojurians Log v2

While it is "stuck", can you get a list of all your running java processes with their arguments? px ax | fgrep java if you're on something Linux-y.

One sec

20361 main    marko    74%     17s  3% java -classpath /home/marko/clojure/helper/mini-projects/test-agent/test:/home/marko/clojure/helper/mini-projects/test-agent/src:/home/marko/clojure/helper/mini-projects/test-agent/dev-resources:/home/marko/clojure/helper/mini-projects/test-agent/resources:/home/marko/clojure/helper/mini-projects/test-agent/target/classes:/home/marko/.m2/repository/org/jetbrains/kotlin/kotlin-scripting-jvm/1.7.10/kotlin-scripting-jvm-1.7.10.jar:/home/marko/.m2/repository/com/squareup/okio/okio-fakefilesystem/3.2.0/okio-fakefilesystem-3.2.0.jar:/home/marko/.m2/repository/org/nrepl/incomplete/0.1.0/incomplete-0.1.0.jar:/home/marko/.m2/repository/software/amazon/awssdk/profiles/2.18.4/profiles-2.18.4.jar:/home/marko/.m2/repository/org/jetbrains/kotlin/kotlin-stdlib-jdk8/1.7.10/kotlin-stdlib-jdk8-1.7.10.jar:/home/marko/.m2/repository/software/amazon/glue/schema-registry-common/1.1.17/schema-registry-common-1.1.17.jar:/home/marko/.m2/repository/io/netty/netty-codec-http/4.1.77.Final/netty-codec-http-4.1.77.Final.jar:/home/marko/.m2/repository/com/fasterxml/jackson/core/jackson-databind/2.10.1/jackson-databind-2.10.1.jar:/home/marko/.m2/repository/software/amazon/ion/ion-java/1.0.2/ion-java-1.0.2.jar:/home/marko/.m2/repository/io/netty/netty-resolver/4.1.77.Final/netty-resolver-4.1.77.Final.jar:/home/marko/.m2/repository/org/apache/commons/commons-compress/1.21/commons-compress-1.21.jar:/home/marko/.m2/repository/org/jetbrains/kotlin/kotlin-scripting-compiler-embeddable/1.7.10/kotlin-scripting-compiler-embeddable-1.7.10.jar:/home/marko/.m2/repository/com/squareup/okio/okio-fakefilesystem-jvm/3.2.0/okio-fakefilesystem-jvm-3.2.0.jar:/home/marko/.m2/repository/com/squareup/wire/wire-schema/4.3.0/wire-schema-4.3.0.jar:/home/marko/.m2/repository/org/jetbrains/kotlinx/kotlinx-serialization-core-jvm/1.4.0/kotlinx-serialization-core-jvm-1.4.0.jar:/home/marko/.m2/repository/software/amazon/eventstream/eventstream/1.0.1/eventstream-1.0.1.jar:/home/marko/.m2/repository/io/github/classgraph/classgraph/4.8.120/classgraph-4.8.120.jar:/home/marko/.m2/repository/software/amazon/glue/schema-registry-build-tools/1.1.17/schema-registry-build-tools-1.1.17.jar:/home/marko/.m2/repository/com/google/protobuf/protobuf-java/3.19.6/protobuf-java-3.19.6.jar:/home/marko/.m2/repository/org/clojure/clojure/1.11.1/clojure-1.11.1.jar:/home/marko/.m2/repository/com/google/guava/failureaccess/1.0.1/failureaccess-1.0.1.jar:/home/marko/.m2/repository/com/damnhandy/handy-uri-templates/2.1.8/handy-uri-templates-2.1.8.jar:/home/marko/.m2/repository/org/jetbrains/kotlin/kotlin-reflect/1.7.10/kotlin-reflect-1.7.10.jar:/home/marko/.m2/repository/software/amazon/awssdk/apache-client/2.18.4/apache-client-2.18.4.jar:/home/marko/.m2/repository/com/google/re2j/re2j/1.6/re2j-1.6.jar:/home/marko/.m2/repository/com/squareup/wire/wire-java-generator/4.3.0/wire-java-generator-4.3.0.jar:/home/marko/.m2/repository/com/squareup/wire/wire-runtime-jvm/4.3.0/wire-runtime-jvm-4.3.0.jar:/home/marko/.m2/repository/commons-validator/commons-validator/1.7/commons-validator-1.7.jar:/home/marko/.m2/repository/org/slf4j/slf4j-api/1.7.30/slf4j-api-1.7.30.jar:/home/marko/.m2/repository/commons-digester/commons-digester/2.1/commons-digester-2.1.jar:/home/marko/.m2/repository/software/amazon/awssdk/json-utils/2.18.4/json-utils-2.18.4.jar:/home/marko/.m2/repository/com/squareup/wire/wire-profiles/4.3.0/wire-profiles-4.3.0.jar:/home/marko/.m2/repository/org/apache/commons/commons-lang3/3.8.1/commons-lang3-3.8.1.jar:/home/marko/.m2/repository/software/amazon/awssdk/auth/2.18.4/auth-2.18.4.jar:/home/marko/.m2/repository/software/amazon/awssdk/sts/2.18.4/sts-2.18.4.jar:/home/marko/.m2/repository/software/amazon/awssdk/annotations/2.18.4/annotations-2.18.4.jar:/home/marko/.m2/repository/software/amazon/awssdk/url-connection-client/2.18.4/url-connection-client-2.18.4.jar:/home/marko/.m2/repository/com/google/j2objc/j2objc-annotations/2.8/j2objc-annotations-2.8.jar:/home/marko/.m2/repository/software/amazon/awssdk/aws-json-protocol/2.18.4/aws-json-protocol-2.18.4.jar:/home/marko/.m2/repository/software/amazon/awssdk/glue/2.18.4/glue-2.18.4.jar:/home/marko/.m2/repository/io/netty/netty-transport-native-unix-common/4.1.77.Final/netty-transport-native-unix-common-4.1.77.Final.jar:/home/marko/.m2/repository/org/checkerframework/checker-qual/3.33.0/checker-qual-3.33.0.jar:/home/marko/.m2/repository/software/amazon/awssdk/aws-query-protocol/2.18.4/aws-query-protocol-2.18.4.jar:/home/marko/.m2/repository/software/amazon/awssdk/aws-core/2.18.4/aws-core-2.18.4.jar:/home/marko/.m2/repository/software/amazon/awssdk/utils/2.18.4/utils-2.18.4.jar:/home/marko/.m2/repository/joda-time/joda-time/2.8.1/joda-time-2.8.1.jar:/home/marko/.m2/repository/com/amazonaws/aws-java-sdk-sts/1.12.151/aws-java-sdk-sts-1.12.151.jar:/home/marko/.m2/repository/io/netty/netty-common/4.1.77.Final/netty-common-4.1.77.Final.jar:/home/marko/.m2/repository/software/amazon/awssdk/netty-nio-client/2.18.4/netty-nio-client-2.18.4.jar:/home/marko/.m2/repository/org/jetbrains/kotlin/kotlin-stdlib-jdk7/1.7.10/kotlin-stdlib-jdk7-1.7.10.jar:/home/marko/.m2/repository/com/squareup/wire/wire-swift-generator/4.3.0/wire-swift-generator-4.3.0.jar:/home/marko/.m2/repository/org/apache/httpcomponents/httpclient/4.5.13/httpclient-4.5.13.jar:/home/marko/.m2/repository/org/jetbrains/kotlinx/kotlinx-coroutines-core-jvm/1.5.2/kotlinx-coroutines-core-jvm-1.5.2.jar:/home/marko/.m2/repository/com/squareup/wire/wire-schema-jvm/4.3.0/wire-schema-jvm-4.3.0.jar:/home/marko/.m2/repository/com/fasterxml/jackson/core/jackson-core/2.12.7/jackson-core-2.12.7.jar:/home/marko/.m2/repository/javax/validation/validation-api/2.0.1.Final/validation-api-2.0.1.Final.jar:/home/marko/.m2/repository/org/json/json/20230227/json-20230227.jar:/home/marko/.m2/repository/io/netty/netty-handler/4.1.77.Final/netty-handler-4.1.77.Final.jar:/home/marko/.m2/repository/org/lz4/lz4-java/1.8.0/lz4-java-1.8.0.jar:/home/marko/.m2/repository/commons-logging/commons-logging/1.1.3/commons-logging-1.1.3.jar:/home/marko/.m2/repository/nrepl/nrepl/1.0.0/nrepl-1.0.0.jar:/home/marko/.m2/repository/org/xerial/snappy/snappy-java/1.1.10.4/snappy-java-1.1.10.4.jar:/home/marko/.m2/repository/com/google/api/grpc/proto-google-common-protos/2.7.4/proto-google-common-protos-2.7.4.jar:/home/marko/.m2/repository/com/squareup/wire/wire-grpc-server-generator/4.3.0/wire-grpc-server-generator-4.3.0.jar:/home/marko/.m2/repository/io/netty/netty-transport/4.1.77.Final/netty-transport-4.1.77.Final.jar:/home/marko/.m2/repository/software/amazon/awssdk/endpoints-spi/2.18.4/endpoints-spi-2.18.4.jar:/home/marko/.m2/repository/commons-codec/commons-codec/1.15/commons-codec-1.15.jar:/home/marko/.m2/repository/com/google/guava/listenablefuture/9999.0-empty-to-avoid-conflict-with-guava/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/home/marko/.m2/repository/io/netty/netty-buffer/4.1.77.Final/netty-buffer-4.1.77.Final.jar:/home/marko/.m2/repository/com/squareup/okhttp3/okhttp/4.9.3/okhttp-4.9.3.jar:/home/marko/.m2/repository/org/reactivestreams/reactive-streams/1.0.3/reactive-streams-1.0.3.jar:/home/marko/.m2/repository/software/amazon/awssdk/http-client-spi/2.18.4/http-client-spi-2.18.4.jar:/home/marko/.m2/repository/com/github/luben/zstd-jni/1.5.5-1/zstd-jni-1.5.5-1.jar:/home/marko/.m2/repository/org/clojure/core.specs.alpha/0.2.62/core.specs.alpha-0.2.62.jar:/home/marko/.m2/repository/software/amazon/awssdk/arns/2.18.4/arns-2.18.4.jar:/home/marko/.m2/repository/com/fasterxml/jackson/dataformat/jackson-dataformat-cbor/2.12.3/jackson-dataformat-cbor-2.12.3.jar:/home/marko/.m2/repository/software/amazon/awssdk/sdk-core/2.18.4/sdk-core-2.18.4.jar:/home/marko/.m2/repository/org/apache/kafka/kafka-clients/3.6.0/kafka-clients-3.6.0.jar:/home/marko/.m2/repository/com/squareup/javapoet/1.13.0/javapoet-1.13.0.jar:/home/marko/.m2/repository/com/amazonaws/aws-java-sdk-core/1.12.151/aws-java-sdk-core-1.12.151.jar:/home/marko/.m2/repository/org/clojure/spec.alpha/0.3.218/spec.alpha-0.3.218.jar:/home/marko/.m2/repository/software/amazon/awssdk/protocol-core/2.18.4/protocol-core-2.18.4.jar:/home/marko/.m2/repository/software/amazon/awssdk/regions/2.18.4/regions-2.18.4.jar:/home/marko/.m2/repository/com/google/jimfs/jimfs/1.1/jimfs-1.1.jar:/home/marko/.m2/repository/org/jetbrains/kotlin/kotlin-scripting-common/1.7.10/kotlin-scripting-common-1.7.10.jar:/home/marko/.m2/repository/com/google/errorprone/error_prone_annotations/2.18.0/error_prone_annotations-2.18.0.jar:/home/marko/.m2/repository/com/google/guava/guava/32.0.0-jre/guava-32.0.0-jre.jar:/home/marko/.m2/repository/software/amazon/glue/schema-registry-serde/1.1.17/schema-registry-serde-1.1.17.jar:/home/marko/.m2/repository/io/outfoxx/swiftpoet/1.3.1/swiftpoet-1.3.1.jar:/home/marko/.m2/repository/software/amazon/awssdk/metrics-spi/2.18.4/metrics-spi-2.18.4.jar:/home/marko/.m2/repository/io/netty/netty-codec-http2/4.1.77.Final/netty-codec-http2-4.1.77.Final.jar:/home/marko/.m2/repository/org/jetbrains/kotlin/kotlin-script-runtime/1.7.10/kotlin-script-runtime-1.7.10.jar:/home/marko/.m2/repository/com/squareup/wire/wire-kotlin-generator/4.3.0/wire-kotlin-generator-4.3.0.jar:/home/marko/.m2/repository/com/kjetland/mbknor-jackson-jsonschema_2.12/1.0.39/mbknor-jackson-jsonschema_2.12-1.0.39.jar:/home/marko/.m2/repository/io/netty/netty-transport-classes-epoll/4.1.77.Final/netty-transport-classes-epoll-4.1.77.Final.jar:/home/marko/.m2/repository/org/jetbrains/annotations/13.0/annotations-13.0.jar:/home/marko/.m2/repository/org/apache/commons/commons-collections4/4.4/commons-collections4-4.4.jar:/home/marko/.m2/repository/com/amazonaws/jmespath-java/1.12.151/jmespath-java-1.12.151.jar:/home/marko/.m2/repository/org/apache/avro/avro/1.11.1/avro-1.11.1.jar:/home/marko/.m2/repository/io/netty/netty-codec/4.1.77.Final/netty-codec-4.1.77.Final.jar:/home/marko/.m2/repository/com/fasterxml/jackson/core/jackson-annotations/2.10.1/jackson-annotations-2.10.1.jar:/home/marko/.m2/repository/com/squareup/kotlinpoet/1.10.2/kotlinpoet-1.10.2.jar:/home/marko/.m2/repository/com/google/code/findbugs/jsr305/3.0.2/jsr305-3.0.2.jar:/home/marko/.m2/repository/software/amazon/awssdk/third-party-jackson-core/2.18.4/third-party-jackson-core-2.18.4.jar:/home/marko/.m2/repository/org/scala-lang/scala-library/2.12.10/scala-library-2.12.10.jar:/home/marko/.m2/repository/com/squareup/wire/wire-runtime/4.3.0/wire-runtime-4.3.0.jar:/home/marko/.m2/repository/com/squareup/okio/okio/3.4.0/okio-3.4.0.jar:/home/marko/.m2/repository/com/squareup/okio/okio-jvm/3.4.0/okio-jvm-3.4.0.jar:/home/marko/.m2/repository/org/jetbrains/kotlinx/kotlinx-datetime-jvm/0.3.2/kotlinx-datetime-jvm-0.3.2.jar:/home/marko/.m2/repository/com/squareup/wire/wire-grpc-client-jvm/4.3.0/wire-grpc-client-jvm-4.3.0.jar:/home/marko/.m2/repository/org/jetbrains/kotlin/kotlin-scripting-compiler-impl-embeddable/1.7.10/kotlin-scripting-compiler-impl-embeddable-1.7.10.jar:/home/marko/.m2/repository/com/github/erosb/everit-json-schema/1.14.2/everit-json-schema-1.14.2.jar:/home/marko/.m2/repository/org/jetbrains/kotlin/kotlin-stdlib/1.7.10/kotlin-stdlib-1.7.10.jar:/home/marko/.m2/repository/com/squareup/wire/wire-compiler/4.3.0/wire-compiler-4.3.0.jar:/home/marko/.m2/repository/org/apache/httpcomponents/httpcore/4.4.13/httpcore-4.4.13.jar:/home/marko/.m2/repository/org/jetbrains/kotlin/kotlin-stdlib-common/1.7.10/kotlin-stdlib-common-1.7.10.jar -Dfile.encoding=UTF-8 -javaagent:grafana-opentelemetry-java.jar -Dotel.resource.attributes=service.name=dev-webserver -Dotel.metrics.exporter=prometheus -Dotel.metric.export.interval=5000 -Dotel.logs.exporter=none -Dotel.traces.exporter=none -Dclojure.compile.path=/home/marko/clojure/helper/mini-projects/test-agent/target/classes -Dtest-agent.version=0.1.0-SNAPSHOT -Dclojure.debug=false clojure.main -i /tmp/form-init10449979469055252960.clj

Just one java process? And you have Prometheus running?

Yes. the open telemetry jar (grafana-opentelemetry-java.jar) in this case starts it by default

Hmm, maybe ask in #observability or #clj-otel? I've only used OTEL via the Clojure CLI (with Digma). At work we use New Relic instead.

I will but this seemed like lein problem since it's the java-source-paths + jvm opts that causes this stuck behavior. Off topic. Why did you switch? Are you happier with new relic? How do you use it?

We started with New Relic many years ago and we love the service.

It monitors our servers and our processes, and our JS frontend, plus all our logs are forwarded to NR, so we can look at cross-application traces, errors, heap/cpu usage, throughput, custom metrics -- and have alerts based on pretty much anything, piped into Slack or email.

The plan we're on has two full platform users and unlimited free users, and with our data ingest rate, it only costs about $550/month for the fully hosted service with, I think, 90 days of full data retention?

We also track deployments via NR so we can see deltas in performance or error rates across releases.

Going back to your problem, how long did you leave it running before you decided it was "stuck"? You have a lot of dependencies and the agent has to instrument all that code at startup so I would expect it to take a while -- I'd be tempted to leave it running until CPU drops before declaring it stuck...

Here is the minimum reproducible example: https://github.com/StankovicMarko/test-agent removing java-source-paths lein repl starts fast adding it - it's just stuck

be tempted to leave it running until CPU drops before declaring it stuck...

Don't want to be this person, but, have you considered setting up an equivalent deps.edn project and seeing if the issue exists there? Maybe there is some magic lein is doing that doesn't play nice with otel. I can look into it in a bit. You are also more likely to get support for deps.edn based projects since they are the most common project format (according to the 2023 Clojure survey)

No, not really. We've been happy with lein and it's unlikely that we will switch.

Maybe there is some magic lein is doing that doesn't play nice with otel. I can look into it in a bit.

Not a repro without the JAR file -- how/where do we get that?

uh... forgot to commit it. One sec

I committed the jar. If you don't trust it you can download either open telemetry java jar here: https://github.com/open-telemetry/opentelemetry-java/releases/tag/v1.39.0 or use grafana one that i am using from : https://github.com/grafana/grafana-opentelemetry-java/releases

I could repro (and figured one needs to add the jar, yes) With intentful use of Lein profiles, you could run the agent only in production, and in isolated local experiments. Isn't that the supposed usage anyway? I'd see little point in running telemetry for day-to-day dev activities

...There's a :repl profile, specific to lein repl besides from user and dev

Odd that you can comment out :java-source-paths and it starts the REPL just fine, or comment out the -javaagent option and it compiles the source and then it starts just fine, but hangs when both are present...

Yourkit points in this direction, sometimes it's not actually a deadlock although it seems a good lead Edit: most likely not

With agent we get free spans and metrics for different libraries we use (c3p0, netty, etc...) I need agent locally in order to see what is happening and what we get I could find a way around it and create docker image and have everything setup - true but isn't this a bug?

Ah, if you comment out the -javaagent option, so it compiles the source and then quit the REPL and uncomment the -javaagent, it all starts up. So it is something about lein compiling the Java source and then something hanging before moving on to start the REPL...

(~/clojure/test-agent)-(!2016)-> lein repl
OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended
[otel.javaagent 2024-06-14 14:48:57:602 -0700] [main] INFO io.opentelemetry.javaagent.tooling.VersionLogger - opentelemetry-javaagent - version: 2.4.0
[otel.javaagent 2024-06-14 14:48:57:638 -0700] [main] INFO com.grafana.extensions.instrumentations.TestedInstrumentationsContext - Grafana OpenTelemetry Javaagent: version=2.4.0-beta.1, includeAllInstrumentations=true, useTestedInstrumentations=false, includedUntestedInstrumentations=[], excludedInstrumentations=[]
[otel.javaagent 2024-06-14 14:48:57:644 -0700] [main] INFO com.grafana.extensions.cloud.GrafanaCloudConfigCustomizer - will not attempt to send data to Grafana Cloud: zone is not set
[otel.javaagent 2024-06-14 14:48:57:710 -0700] [main] INFO io.opentelemetry.sdk.resources.Resource - Attempting to merge Resources with different schemaUrls. The resulting Resource will have no schemaUrl assigned. Schema 1:  Schema 2: 
nREPL server started on port 39611 on host 127.0.0.1 - 
REPL-y 0.5.1, nREPL 1.0.0
Clojure 1.11.1
OpenJDK 64-Bit Server VM 21.0.1+12-LTS
    Docs: (doc function-name-here)
          (find-doc "part-of-name-here")
  Source: (source function-name-here)
 Javadoc: (javadoc java-object-or-class-here)
    Exit: Control+D or (exit) or (quit)
 Results: Stored in vars *1, *2, *3, an exception in *e

test-agent.core=>

That's where profiles could be useful, you could have a profile with java-source-paths set but no agent set, and a :repl profile with the agent but no source paths Personally I always found Lein's approach to compilation highly complected - it should happen in discrete composable steps. You can achieve that with profiles, or a Makefile, or tools.deps. To me it's not super surprising when edge cases are hit with the default Lein approach

Yeah, even if you just do lein javac, with -javaagent, it just hangs after compilation (and is using no cpu).

Juicest thing I could extract from Yourkit is that the agent leaves a non-daemon http thread running Any non-daemonized thread prevents the JVM from completion

hm... well the config is to start prometheus server locally and scrape every 5sec. But why would repl start without java-source-paths ? Just adding agent with jvm-opts should have same behavior right? I also created an issue: https://github.com/technomancy/leiningen/issues/2818 Maybe something good comes out of it

It's some interaction between the agent and the Java compiler that Leiningen invokes.

In case it wasn't obvious, Lein allocates two JVMs in succession - one for computing the project, running javac , etc and one for running the actual repl. The former JVM has no real reason to have the agent in, so I'd consider removing the agent less of a workaround and more of a "run as little software as possible for the task at hand" (which is probably an uncontroversial practice) Even if you didn't face this bug, it would still be a good idea as it could easily shave some ms in startup.

Yeah, I would definitely move the agent opts to a profile so that you can control when it is active. The fact that you can run javac just fine without the agent, but it hangs after compilation if you try to run javac with the agent -- and you can definitely start the REPL with the agent, is what points at some interaction between the agent and the (Java) compiler process and that's what you need to avoid.

I tried putting it

:profiles {:dev {:jvm-opts ["-javaagent:opentelemetry-javaagent.jar"
                              "-Dotel.resource.attributes=service.name=dev-webserver"
                              "-Dotel.metrics.exporter=prometheus"
                              "-Dotel.metric.export.interval=5000"
                              "-Dotel.logs.exporter=none"
                              "-Dotel.traces.exporter=none"
                              "-Duser.timezone=UTC"]

:dev is automatically included for certain operations -- use a distinct profile instead.

(I can never remember the "magic" behind how lein combines implied and explicit profiles for which operations 😞 )

tried it under dev-test-try same behavior. I think jvm opts just get applied anyway and same behavior happens

:profiles {:javac {:java-source-paths ["java_src"]}
           :repl {:jvm-opts ["-javaagent:grafana-opentelemetry-java.jar"
                             "-Dotel.resource.attributes=service.name=dev-webserver"
                             "-Dotel.metrics.exporter=prometheus"
                             "-Dotel.metric.export.interval=5000"
                             "-Dotel.logs.exporter=none"
                             "-Dotel.traces.exporter=none"]}}

lein with-profile javac javac && lein repl

AAAAh... ok got it now. Thanks this works. At least i am unblocked for local dev now will use cider-connect It's more neat than docker container setup i had used before

thank you both for the time and effort 🙇

My pleasure! Last year I gave some retouches to Docker stuff in CIDER - for many years there would be misc quirks. Probably it remains quite usable overall (as it is with other remote connections like Tramp)