Clojurians Log v2

I am using this Dockerfile to generate the container image to execute the application:

While building the image, we have this output:

While executing the container I am getting this exception:

Exception in thread "main" java.lang.ExceptionInInitializerError
2024-11-07T08:28:30.145812583Z Caused by: java.io.FileNotFoundException: Could not locate rango/components__init.class, rango/components.clj or rango/components.cljc on classpath.
2024-11-07T08:28:30.145858083Z 	at clojure.lang.RT.load(RT.java:482)
2024-11-07T08:28:30.145860583Z 	at clojure.lang.RT.load(RT.java:444)
2024-11-07T08:28:30.145861833Z 	at clojure.core$load$fn__6931.invoke(core.clj:6189)
2024-11-07T08:28:30.145902750Z 	at clojure.core$load.invokeStatic(core.clj:6188)
2024-11-07T08:28:30.145904791Z 	at clojure.core$load.doInvoke(core.clj:6172)
2024-11-07T08:28:30.145905666Z 	at clojure.lang.RestFn.invoke(RestFn.java:411)
2024-11-07T08:28:30.145906750Z 	at clojure.lang.Var.invoke(Var.java:386)
2024-11-07T08:28:30.145907625Z 	at clojure.lang.Util.loadWithClass(Util.java:251)
2024-11-07T08:28:30.145908458Z 	at rango.components.<clinit>(Unknown Source)

rango/components__init.class this is the namespace that holds my main function. I am trying to figure out why it is not being found, but without success.

I have to initialize it at run time --initialize-at-run-time=rango.components

If I remove the --initialize-at-run-time=rango.components I am still able to build the docker image but while executing it I am greeted with:

Error: Could not find or load main class rango.components
2024-11-07T08:37:08.575149635Z Caused by: java.lang.ClassNotFoundException: rango.components

This is my project.clj file, just to provide more context.

Check out https://github.com/clj-easy/graal-build-time and use it in your project

I am trying to use graal-build-time . But I am only able to build the image with --initialize-at-run-time=rango

why?

I get this error:

Error: Classes that should be initialized at run time got initialized during image building:
#13 70.73  org.joda.time.DateTimeZone was unintentionally initialized at build time. To see why org.joda.time.DateTimeZone got initialized use --trace-class-initialization=org.joda.time.DateTimeZone

Then also add:

--initialize-at-build-time=org.joda.time.DateTimeZone

if Java classes are used at the top level in Clojure namespaces, you also need to include those

Adding all the reported classes worked for the image build.

RUN native-image \
      --report-unsupported-elements-at-runtime \
      --initialize-at-build-time=io.opentracing.util.GlobalTracer \
      --initialize-at-build-time=org.slf4j.helpers.Reporter,org.slf4j.LoggerFactory \
      --initialize-at-build-time=org.pg.Pool \
      --initialize-at-build-time=io.opentelemetry.api.common.ArrayBackedAttributes,io.opentelemetry.api.DefaultOpenTelemetry,io.opentelemetry.api.GlobalOpenTelemetry,io.opentelemetry.context.propagation.DefaultContextPropagators \
      --initialize-at-build-time=org.eclipse.jetty.http.HttpVersion,org.eclipse.jetty.http2.hpack.HpackContext,org.eclipse.jetty.util.BufferUtil,org.eclipse.jetty.http.HttpMethod,org.eclipse.jetty.server.Response,org.eclipse.jetty.http.DateGenerator,org.eclipse.jetty.http.HttpHeader,org.eclipse.jetty.http2.hpack.HpackEncoder,org.eclipse.jetty.util.TypeUtil,org.eclipse.jetty.http.compression.Huffman,org.eclipse.jetty.util.StringUtil,org.eclipse.jetty.http.PreEncodedHttpField,org.eclipse.jetty.http.HttpScheme,org.eclipse.jetty.http.HttpTokens \
      --initialize-at-build-time=org.joda.time.chrono.ISOChronology,org.joda.time.Period,org.joda.time.DateTimeFieldType,org.joda.time.DateTimeZone,org.joda.time.chrono.ISOYearOfEraDateTimeField,org.joda.time.chrono.BasicGJChronology,org.joda.time.chrono.GregorianChronology \
      --initialize-at-build-time=org.bouncycastle.asn1.pkcs.PKCSObjectIdentifiers,org.bouncycastle.asn1.nist.NISTObjectIdentifiers,org.bouncycastle.util.Strings,org.bouncycastle.crypto.CryptoServicesRegistrar,org.bouncycastle.asn1.x9.X9ObjectIdentifiers,org.bouncycastle.util.Properties,org.bouncycastle.crypto.engines.Salsa20Engine,org.bouncycastle.asn1.cryptopro.CryptoProObjectIdentifiers,org.bouncycastle.asn1.x509.X509ObjectIdentifiers \
      --features=clj_easy.graal_build_time.InitClojureClasses \
      --enable-url-protocols=http,https \
      -Dio.pedestal.log.defaultMetricsRecorder=nil \
      -jar ./target/rango-0.1.0-SNAPSHOT-standalone.jar \
      -H:Name=rango-native \
      -H:+ReportExceptionStackTraces \
      -H:+StaticExecutableWithDynamicLibC \
      -H:CCompilerOption=-pipe

But while I execute the image, I am still getting

Error: Could not find or load main class rango.components
2024-11-07T11:33:49.257499173Z Caused by: java.lang.ClassNotFoundException: rango.components
2024-11-07T11:33:50.289990937Z Error: Could not find or load main class rango.components
2024-11-07T11:33:50.290015563Z Caused by: java.lang.ClassNotFoundException: rango.components

From the logs I see that rango is being loaded by clj-easy/graal-build-time

[clj-easy/graal-build-time] Registering packages for build time initialization: clojure, buddy, camel_snake_kebab, cheshire, clj_commons, clj_easy.graal_build_time, clojurewerkz.scrypt, cognitect, common_clj.error, common_clj.integrant_components, , common_clj.traceability, crypto, humanize, integrant, io.pedestal, java_time, less.awful, medley, pg, porteiro_component, postgresql_component, rango, ring.middleware, ring.util, schema, taoensso, weavejester

Is range.components dynamically loaded using requiring-resolve or so?

no, it is the namespace that holds the -main function and project.clj points to it: :main rango.components

Perhaps any of your other namespaces are still requiring rango.components dynamically or so? I would need to see the entire trace (perhaps better to put it in a gist)

while executing the image this is the entire trace, there is no more information:

Error: Could not find or load main class rango.components
2024-11-07T11:33:49.257499173Z Caused by: java.lang.ClassNotFoundException: rango.components

rango.components is not required anywhere in the source code

it looks like this is logged by timbre. perhaps you can tweak things to show the entire exception or so.

I am going to try

could it be that integrant is loading namespaces dynamically

https://github.com/weavejester/integrant/blob/ff7f1c34e375d3ee0129e0021ae903b318d57c50/src/integrant/core.cljc#L250

I'm not sure wether this is called or not, but could be

is there a documented way to make the native image build process to recognize it properly?

but I will try o see the complete trace first

> make the native image build process to recognize it properly? not sure what you mean by this

If there is a expected workaround when the code loads namespaces dynamically.

I don't want to stop using integrant to make it work sadge

I'm not sure if integrant is the cause, let's first dig into the exception

I was able to get this stack trace while building the native image: https://gist.github.com/macielti/1509469b760baf8143dc12310806c5d2

interesting:

#15 130.5       at clojure.core$require.invokeStatic(core.clj:6066)
#15 130.5       at io.pedestal.interceptor.chain$loading__6812__auto____8196.invoke(chain.clj:14)
#15 130.5       at io.pedestal.interceptor.chain__init.load(Unknown Source)
#15 130.5       at io.pedestal.interceptor.chain__init.<clinit>(Unknown Source)

ah:

#15 130.5       at common_clj.integrant_components.service$loading__6812__auto____6648.invoke(service.clj:1)
#15 130.5       at common_clj.integrant_components.service__init.load(Unknown Source)
#15 130.5       at common_clj.integrant_components.service__init.<clinit>(Unknown Source)

this is the source code for the https://github.com/macielti/common-clj/blob/main/src/common_clj/integrant_components/service.clj

but I think that this still related with how rango.components loads this, that makes sense?

I saw that integrant is compatible with graalvm out of the box https://github.com/clj-easy/graalvm-clojure/tree/master/integrant

perhaps something is trying to reload your code somehow at runtime in the native image?

do you use any tools like clj-reload or tools.namespace.refresh?

no, I am not intentionally using any of those

also replacing this:

:aot :all

:aot [rango.components]

:aot :all is known to cause problems with native-image. this is probably not the issue here

but still good to exclude this source of error

can you also set this on the top level after your ns form:

(alter-var-root #'clojure.core/*loading-verbosely* (constantly true))

this will print if any other namespaces get loaded beyond that

e.g.:

user=> (alter-var-root #'clojure.core/*loading-verbosely* (constantly true))
true
user=> (require '[clojure.set])
(clojure.core/load "/clojure/set")

wait, loading the namespace at REPL it printed a lot of stuff

https://gist.github.com/macielti/72b375f2dcc2997b5bbca45f64f1e3b4

Manually requiring the ns, nothing is printed:

(require '[rango.components])
=> nil

that's not what i meant

I mean, compile the native image and then see if it prints anything

you could also try this by starting your main function from the repl

I have a question, it is better to use --no-fallback ? when it is added the image build fails with the exception that I shared previously. When I remove it, the build process is completed with success, but fails while executing the image.

yes, use --no-fallback. a fallback image doesn't make sense, it's basically just a JVM

There was nothing printed related with clojure.core/load https://gist.github.com/macielti/5805a7ac778c7b518708fd407f7c5903

I made the repository public: https://github.com/macielti/rango I think that this use case will help others with the same problem while trying to use GraalVM to run webapps

It stopped working again, but i upgraded it to the latest version and it worked.

I am suspicious about the pedestal.interceptor I will start to add them to see what happpens.

@borkdude could you review this PR? https://github.com/clj-easy/graalvm-clojure/pull/64 I updated Pedestal adding more use cases and bump the dependencies.

Looks great. I don't know if the markdown table changes are intentional.

and the native-image invocation contains a specific path of your system

✅

✅

Awesome of you to share your findings @brunodonascimentomaci! Thanks so much!

I will put a lot of effort on it. I like to run my applications on isolated docker containers, and I liked the way that GraalVM native images deal with memory resources handling if compared with the JVM.

My objective is to have a fully functional CRUD backend.

Yeah, very cool! If you have any numbers, I'd love to hear about them: How is server startup time after moving to native-image? And how does memory usage compare?

Yep, I already have the JVM version. After finishing the GraalVM one, we will be able to run it and compare side by side.

Another thing that I observed was that the GraallVM 23 runs more smoothly while generating the native image and with more stable results while executing the binary.

Do you know how we can get access to the latest docker image with the version 23? https://github.com/graalvm/container/pkgs/container/native-image

Best place to ask would probably be that repo (or GraalVM Slack community)

I think that one is a good candidate:

docker pull container-registry.oracle.com/graalvm/native-image:23

👍

The last one for today https://github.com/clj-easy/graalvm-clojure/pull/65, now adding PG2 reviewplease

borkdude@m1-3 /tmp/rango (main) $ rg native-image

maybe you can add some explicit script whichs builds the native image, else it's still hard to reproduce what you're doing

I ended up reverting everything related to the GraalVM. I will create a new repo focused on the GraalVM version adding everything incrementally and testing it until it breaks.

👍

good approach

One of the problems was the pedestal version, downgrading to:

[io.pedestal/pedestal.service "0.5.10"]
[io.pedestal/pedestal.jetty "0.5.10"]

Although... back when all this was new we were taking time to record libraries that worked out-of-the-box here: https://github.com/clj-easy/graal-docs/blob/master/doc/external-resources.adoc#libraries-compatible-with-graalvm

But we haven't maintained that list in eons. So... dunno.

I think that it would be great to have a easy way to check if the lib works out of the box before testing it.

yeah, but I don't think it makes sense to keep doing that since we already have enough examples of how to make a generic graalvm native-image project

well yeah, that does make sense

It is done! It still missing the integration with New Relic for logs, but I will work on that later. https://github.com/macielti/rango-graalvm

The JVM and the GraalVM side by side, on idle memory consumption.

I will study more about GraalVM. I think we can optimize it for even better results. I discovered about GraalVM last week.

amazing :)

Wowzie!

@borkdude https://github.com/clj-easy/graalvm-clojure/pull/66 reviewplease

I wonder, does it make sense to include examples that don't require any configuration, as in "works out of the box"? This increases maintenance overhead a bit for stuff people likely won't look at because they won't run into any problems in the first place? What do you think @lee?

I agree with you, but this one does not requires any configuration before executing lein do clean, uberjar, native, run-native

wasn't that my point -- doesn't require any configuration?

now I understand the point. It makes sense.

I wasn't able to make the latest version of org.xerial/sqlite-jdbc work. We can still left the PR open and focus on seeking the configuration needed for the latest version.

Yeah, I agree with you both. We don't need to document when nothing special is required.

Use delay (def *auth-session-db (delay (duratom ....)) And, for sure, update all usages from auth-session-db to @*auth-session-db

@souenzzo thanks I will look into that, duatom itself is an atom so does everything become @@`auth-session-db` I assume there is no reason you can't double deref 😛

It really looks really ugly It is probably possible to fix it at duratom, avoiding the creation of threads before usage

yeah that was my thought as well, also less obvious whats going on, in this case I probably only have to do it in a couple of places, I will give it a go good to know about using delay as well 🙂