@U6VPZS1EK cheshire isn't related to transit as far as I know. and I've never had a problem with cheshire, but it does need some configuration

@U6VPZS1EK There are a few things that you need to do. First of all, use --initialize-at-runtime

o no wait, you're using graal-build-time already, this is good

So this is the hint. > com.fasterxml.jackson.core.io.SerializedString

You need to also initialize that package at build time.

--initialize-at-build-time=com.fasterxml.jackson

> No instances of com.fasterxml.jackson.core.io.SerializedString are allowed in the image heap as this class should be initialized at image runtime.

Then there are these warnings: > > clj-easy/graal-build-time] WARN: Single segment package found for class: java_time__init.class. This class has no package and it will not be added to the result packages. > [clj-easy/graal-build-time] WARN: Single segment package found for class: clj_tuple__init.class. This class has no package and it will not be added to the result packages. > [clj-easy/graal-build-time] WARN: Single segment package found for class: potemkin__init.class. This class has no package and it will not be added to the result packages.

Whew. Do you have a repo that shows how configuration required with cheshire? Ya, I saw those, but don’t know what to do with them…. Ohhhhh…. Trying it now. Roger — I think I’m starting to understand. Can I submit a PR to add jackson explanation somwhere, say, https://github.com/clj-easy/graal-docs?

There is no workaround for this other than to just do --initialize-at-build-time=. as those Clojure namespaces do not generate a Clojure package.

Actually we have packaged configs for a few libraries here already: https://github.com/clj-easy/graal-config

Including cheshire

https://github.com/clj-easy/graal-config/tree/master/config/cheshire/cheshire

It is also possible to contribute this config to cheshire so it will work automatically

Similar to what we've done here: https://github.com/http-kit/http-kit/blob/master/resources/META-INF/native-image/http-kit/http-kit/native-image.properties

I think another classic is that you're initializing components in top level defs.

env: telepresence:  nil
dbhost:  127.0.0.1
env: telepresence:  nil
dbhost:  127.0.0.1

But as I can't see the code, I'm not sure about that ;)

When you initialize everything at build time, those vars will get initialized in the image at compile time. You will need to delay those using delay or a normal function and then initialize in your -main function.

This is starting to make sense… 1. What do I do with the single segment packages? Something like: --initialize-at-build-time=com.fasterxml.jackson,java_time,clj_tupe,potemkin? 2. configs: I saw you mention that above… ah, so worst case, just append those to --initialize-at-run-time. 3. Ah yes, running in top-level def. That was something I wrote 6 years ago when I didn’t know any better…. I know how to fix that one! 🙂 Thx for the help here, @borkdude — I’m finally experiencing that strange experience of “@borkdude just materializes out of nowhere and helps you.” I appreciate it!

1. as the Clojure namespace java-time doesn't generate packages, but just top level classes, you have no other option than to initialize everything at runtime (or to enumerate all the generated classes by Clojure for this namespace, which is painful). You can do this by --initialize-at-build-time=.. But in general it's better to avoid this and ask those libs to don't use single segment namespaces.

2. yes

3. cool :)

I've also experienced problems with the java-time library and GraalVM. Imo it's better to get rid of it (for this project) and use java.time interop directly, if you can. This library uses eval which doesn't play nicely with GraalVM, even if you do it at compile time.

Oh, I see that got fixed. https://github.com/dm3/clojure.java-time/commit/08a700a2583ada0908066dbc8d6684e3a032ce96#diff-25a6634263c1b1f6fc4697a04e2b9904ea4b042a89af59dc93ec1f5d44848a26 Yeah, perhaps upgrading to 0.3.3 is best

Am I doing this right? I’m using these flags, and getting a ton of new errors about classes being requested to be initialized at run time…

--initialize-at-build-time=com.fasterxml.jackson \
             --initialize-at-build-time=clojure,cheshire \
             --initialize-at-run-time=java_time,clj_tupe,potemkin \ 
 
java_time.temporal$fn__13341 the class was requested to be initialized at run time (from the command line with 'java_time'). To see why java_time.temporal$fn__13341 got initialized use --trace-class-initialization=java_time.temporal$fn__13341
potemkin.types$deftype__GT_deftype_STAR_ the class was requested to be initialized at run time (from the command line with 'potemkin'). To see why potemkin.types$deftype__GT_deftype_STAR_ got initialized use --trace-class-initialization=potemkin.types$deftype__GT_deftype_STAR_
potemkin.macros$gensym_QMARK_ the class was requested to be initialized at run time (from the command line with 'potemkin'). To see why potemkin.macros$gensym_QMARK_ got initialized use --trace-class-initialization=potemkin.macros$gensym_QMARK_
potemkin.collections$def_map_type the class was requested to be initialized at run time (from the command line with 'potemkin'). To see why potemkin.collections$def_map_type got initialized use --trace-class-initialization=potemkin.collections$def_map_type

yeah, no, read what I wrote at the answer to 1. :)

just use --initialize-at-build-time=. . there is no other way

(notice the dot)

the problem is the single segment namespace

Ah. Giving that a try. (And I just learned how the configs work, by including them in deps.edn. Will eventually try that, too.). Okay, pull slot machine lever again! 🙂

This is what happens if you compile a single segment namespace:

$ echo '(ns foo)' > src/foo.clj
$ clojure -M -e "(compile 'foo)"
foo
$ ls classes
clojure                              foo$fn__128.class                    foo$loading__6721__auto____126.class foo__init.class

There is no foo package, just top level classes starting with foo

and if you try --initialize-at-build-time=foo, native image looks for a package foo

1. …continued. To confirm: --initialize-at-build-time=. inits all packages and classes, just as in the docs would be done with an emtpy string? (I don’t think I’ve ever had this many Chrome tabs open. I was hacking way on this late into the night last night. 🙂

yes, you can also omit the =. part but then it will warn you about that this will become deprecated in GraalVM 22

This was what we did before this warning. But then this warning made us create graal-build-time and graal-config .

4. Holy cow. I’m down to a couples lines of errors! Can I fix this by doing --initialize-at-run-time=“.google.auth.oauth2.OAuth2Credentials,com.google.auth.oauth2.ServiceAccountCredentials,com.google.auth.oauth2.GoogleCredentials,com.google.cloud.NoCredentials,com.google.auth.oauth2.OAuth2Utils ?

To see how the classes got initialized, use --trace-class-initialization=com.google.auth.oauth2.OAuth2Credentials,com.google.auth.oauth2.ServiceAccountCredentials,com.google.auth.oauth2.GoogleCredentials,com.google.cloud.NoCredentials,com.google.auth.oauth2.OAuth2Utils
[pubsub-web-standalone:67102]     analysis: 135,298.37 ms,  6.53 GB
Error: Classes that should be initialized at run time got initialized during image building:
 com.google.auth.oauth2.OAuth2Credentials the class was requested to be initialized at run time (from the command line with 'com.google.auth.oauth2'). To see why com.google.auth.oauth2.OAuth2Credentials got initialized use --trace-class-initialization=com.google.auth.oauth2.OAuth2Credentials
com.google.auth.oauth2.ServiceAccountCredentials the class was requested to be initialized at run time (from the command line with 'com.google.auth.oauth2'). To see why com.google.auth.oauth2.ServiceAccountCredentials got initialized use --trace-class-initialization=com.google.auth.oauth2.ServiceAccountCredentials
com.google.auth.oauth2.GoogleCredentials the class was requested to be initialized at run time (from the command line with 'com.google.auth.oauth2'). To see why com.google.auth.oauth2.GoogleCredentials got initialized use --trace-class-initialization=com.google.auth.oauth2.GoogleCredentials
com.google.cloud.NoCredentials the class was requested to be initialized at run time (). To see why com.google.cloud.NoCredentials got initialized use --trace-class-initialization=com.google.cloud.NoCredentials
com.google.auth.oauth2.OAuth2Utils the class was requested to be initialized at run time (from the command line with 'com.google.auth.oauth2'). To see why com.google.auth.oauth2.OAuth2Utils got initialized use --trace-class-initialization=com.google.auth.oauth2.OAuth2Utils

4. This may be triggered by your top level defs

If you solve those the error may go away

top level defs are ok if you're starting a server, etc, but do it in a delay

so the server doesn't start during the image build

4. Roger. But I’m pretty sure this is in a completely different library. Uses a Google Cloud jar. Is usage of --initialize-at-run-time correct?

Can you maybe post a new 0bin with the entire output?

Then we'll start a new thread from there

(in the database library, I’m using korma, which is another can of worms… which I’ve been procrastinating on getting rid of… today might be a good day for that. 🙂 Roger on pasting out. Seriously, thanks for the help.

if you're looking for a korma replacement: honeysql is what I can recommend

https://0bin.net/paste/7bT2MFyA#gUmriG027+K4HhTjntLRGpf6UfHRgPsemEjDEgxFn96

ok, why did you write: > --initialize-at-run-time=com.google.auth.oauth2

(Yes, it’s currently a frankenstein of korma, clojure.jdbc, and maybe even some honeysql in there. 🙂

I was hoping that it would pickup all the things that oauth2 hierarchy. I’ll delete that.

but why do you want to avoid build time init of oauth2, do you have more info on that?

Doh. I was just trying to make the error message go away. Let me take all those lines out and try again.

OK, the explanation of this output: You have requested to initialize everything at build time now. Except oauth2. But that also got initialized at build time, probably because one of your top level defs which execute something that needs that class (indirectly)

As the output is saying, you can trace what initialized it by using the suggested option

but you can also just delete it and try again

Here’s the forked library that I created that is using the Google Oauth stuff. I don’t see any top level defs except a definition of an atom… so even that (time/now) causes a problem, and should be deferred? https://github.com/realgenekim/google-cloud/blob/master/src/google_cloud/oauth.clj

@U6VPZS1EK It doesn't only matter if the library itself doesn't use any top level initializations of java classes, it also matters what your app is doing.

e.g. if you lib has (defn foo [] (Initialize.)) but your app has (def x (foo)) then Initialize still gets initialized.

Okay, you were right about the oAuth stuff. Totally different set of errors, but I remember getting some these last night. https://0bin.net/paste/LqV3bnXj#+HTAtbMocuVu+HZzwUTc4Kw3FgHIdvEwhSBJvRtz3NI

@U6VPZS1EK use clojure 1.10.3 or newer

(see topic of this channel: the number 1 rule here ;))

Drats. I must have missed that somewhere. (I’m using 5 local/root components.).

Checking.

I see your app is pulling in a bunch of clojure.spec stuff. I don't know if you use any run time instrumentation or checking, but probably ghostwheel and orchestra you may want to turn off and leave outside of the deployed jar, this will give you a smaller binary

but this can be a later optimization

Roger that. ^^. PS: I promise to write up all the bumps along the way, and put it someplace, and submit a PR for docs someplace. Okay, trying again with updated deps.

clojure 1.10.3 is specifically important since this has a fix for the locking macro. if you use spec in your app, you will likely hit that unbalanced monitor "bug" with earlier versions of clojure

Yes, I saw that traffic way back when — delighted (and super impressed) at how quickly your changes are making it into clojure.core/etc. You’re in rarified air!

oh it wasn't my fix but I was super happy that issue got solved - the workaround was pretty gnarly

It’s been years since I’ve dealt with compile times in my daily work. 🙂 And I’m afraid to close any of these tabs…

hehe

PS: I watched this video yesterday, and thought of posting this here: https://www.youtube.com/watch?v=lqKTPnWUmvs It was what made me decide to invest some time to make native image work. The notion that these long build times are what would normally get amortized during multi-day JIT runs. Super compelling case for Graal AOT.

It's often best to start with a trivial amount of code and then start adding stuff.

both to check if things work and also to monitor the image size (if you care about that)

Yes, the wisdom of that is quite clear right now. 🙂 Sunk cost fallacy killing me right now. 🙂 I’m getting same unbalanced monitor error. clj -Stree shows only clojure 1.11.0-alpha…. hmm… https://0bin.net/paste/QMCZRZuW#dJr0a0z7ljtw9tJY3ykONWJaSjVam87tKcJCsxWNGR8

did you rebuild your uberjar?

to be sure, do (println **clojure-version**) on the top level of your program

and see what it outputs during compile

yep! Okay, will do that…

and then please also paste the entire output in http://0bin.net of the uberjar process + the compile output

{:major 1, :minor 11, :incremental 0, :qualifier alpha3}

Super cool that it’s a map, not a string.

There's also a function for that, which uses the map:

user=> (clojure-version)
"1.10.1"

Will paste output when run is done in 3m. Hey, I really appreciate the help here. I can’t tell you how much it means to me!! Getting this running has been a multi-year aspiration.

I surely hope that this isn't a regression in clojure wrt native image ;)

oh god, I hope they didn't compile spec 0.3.214 using an older version of clojure

oh crap. https://github.com/clojure/spec.alpha/blob/a924efab53b0465f7d2a589d4ae3c79a3826403b/pom.xml#L39

I can’t imagine such a thing! But I’ll try using 1.10.3, just in case. Here’s the output. https://0bin.net/paste/co8gBhTX#sHKTeGkD0uSYzhhm-3VLOg6jhS4wbufY3hF5D1nLL5f

(That’s so… novel and unexpected. 🙂). Rebuilding with older versions of clojure and spec.

I can “downgrade” to spec 0.2.194, and that should/could work, yes?

That won't help in this case. You're just hitting these instrument/unstrument functions, which I think you shouldn't even be using in a native image.

I mean, they are supposed to work, but they are still using the old locking macro from 1.10.1 so that just won't work in graalvm native image.

So it's better to get rid of the spec instrumentation orchestra/ghostwheel stuff

Roger that. Giving that a try!

nice, I made some money tonight, thank you :-D

Wow, that was fast — I seriously thought that it would take a couple of days to process. Amazing! (I’m looking to find the way to elide ghostwheel / guardrails in production… that would be sufficient so that native-image wouldn’t see / process it, right?)

yeah

usually you can do this by moving all of that stuff to an isolated namespace which you don't load in production

Hopefully last question of the day — would getting rid spec instrumentation fix this error?

Error: Detected a MBean server in the image heap. This is currently not supported, but could be changed in the future. Management beans are registered in many global caches that would need to be cleared and properly re-built at image build time. Class of disallowed object: com.sun.jmx.mbeanserver.JmxMBeanServer  To see how this object got instantiated use --trace-object-instantiation=com.sun.jmx.mbeanserver.JmxMBeanServer. The object was probably created by a class initializer and is reachable from a static field. You can request class initialization at image runtime by using the option --initialize-at-run-time=<class-name>. Or you can write your own initialization methods and call them explicitly from your main entry point.
Trace: Object was reached by 
	reading field com.mchange.v2.c3p0.management.ActiveManagementCoordinator.mbs of
		constant [email protected]e91 reached by 
	reading field 
Error: Detected a MBean server in the image heap. This is currently not supported, but could be changed in the future. Management beans are registered in many global caches that would need to be cleared and properly re-built at image build time. Class of disallowed object: com.sun.jmx.mbeanserver.JmxMBeanServer  To see how this object got instantiated use --trace-object-instantiation=com.sun.jmx.mbeanserver.JmxMBeanServer. The object was probably created by a class initializer and is reachable from a static field. You can request class initialization at image runtime by using the option --initialize-at-run-time=<class-name>. Or you can write your own initialization methods and call them explicitly from your main entry point.

ok. if you don't have a clue why this might get initialized at build time, then try the suggested option:

--trace-object-instantiation=com.sun.jmx.mbeanserver.JmxMBeanServer

Ah. Got it. Thank you!!!

Oh btw, since they mention that this class may not be initialized at build time, you have to write:

--initialize-at-runtime=com.sun.jmx.mbeanserver.JmxMBeanServer