Fork me on GitHub
#graalvm
<
2021-02-12
>
ericdallo12:02:09

graal-vm New GraalVM release: https://www.graalvm.org/release-notes/21_0/#21002, with just bug fixes on the updater tool

ericdallo13:02:53

Did anyone try this https://upx.github.io/ to decrease the native image size?

borkdude13:02:09

yes. it works, but it will inccrease startup somewhat (like it adds 100-200ms or so)

borkdude13:02:48

which for babashka / clj-kondo I did not find acceptable, but for clojure-lsp I can see that being acceptable since it's a long running server

ericdallo13:02:45

I see, do you know why it increase the startup? it's kind of curious :thinking_face:

borkdude13:02:01

because it has to unzip stuff at startup

ericdallo13:02:10

hum, make sense

ericdallo13:02:46

nice, I'll give a try and try some benchmarks to see if it works nice for clojure-lsp

borkdude13:02:49

most people have indicated in the babashka survey 2020/11 that binary size wasn't a huge priority for them

ericdallo13:02:20

yeah, babashka is really small, that's so cool

ericdallo13:02:31

but clojure-lsp is with 118 mb 😞

borkdude13:02:36

I think what matters most is network size, so if the release file is a zip, you save network bandwidth, but on the machine the unzipped size doesn't matter much

borkdude13:02:57

babashka is now 70-80mb, but zipped 20mb or so

ericdallo13:02:02

I agree, I do the same with clojure-lsp, the zip is about 30 mb

borkdude13:02:52

I have sometimes found that having a runtime require or resolve in your code can bloat the binary with 30 mb or so. so I usually hunt that spot down and get rid of it using alter-var-root or whatnot

borkdude13:02:25

build-time (top-level or macro-expansion time) require/resolve is ok though

ericdallo13:02:15

oh, good to know! I think the only require clojure-lsp has at runtime is via dynaload but only require nrepl if in a debug profile (not graalvm)

borkdude13:02:18

that dynaload could trigger the bloat still though

borkdude13:02:47

this is why I've made a graalvm variation of this here: https://github.com/borkdude/dynaload

ericdallo13:02:23

you mean the borkdude.dynaload.aot variable?

borkdude13:02:40

by setting the java property borkdude.dynaload.aot=true when compiling natively, it will yield smaller binaries

ericdallo13:02:52

I think I forgot to do that 😅 thanks!

borkdude13:02:38

so in the uberjar you will need to do that already

borkdude13:02:08

and also during graalvm compilation

ericdallo13:02:00

Got it, will try this later, hope it'll decrease the size

borkdude13:02:17

to be sure, put a (println (System/getProperty "borkdude.dynaload.aot")) at the top level in your code

ericdallo13:02:47

yeah, I'll certainly do that hahah

ericdallo16:02:07

Do you have an example of a project setting that variable to true @borkdude?

ericdallo16:02:52

just tried adding to the :jvm-opts of my :native-image profile that is used during the uberjar:

:jvm-opts ["-Xmx2g"
   "-server"
   "-Dclojure.compiler.direct-linking=true"
   "-Dclojure.spec.skip-macros=true"
   "-Dborkdude.dynaload.aot=true"]

ericdallo16:02:02

and to the graalvm compilation flag:

ericdallo16:02:16

"-J-Dborkdude.dynaload.aot=true"

ericdallo16:02:10

looks the same I'm using

ericdallo16:02:16

the println prints nil

ericdallo16:02:21

hum, if I java -jar it still prints nil

ericdallo16:02:40

so the issue is the flag in the jar

borkdude16:02:40

so direct linking also doesn't work? good to check because that is also important

ericdallo16:02:19

yeah, I'll check it too, probably is not working 😨

borkdude16:02:58

direct linking will likely make your image smaller as well

🤞 3
ericdallo16:02:31

so with lein with-profile native-image run it prints true

ericdallo16:02:43

but with the generated jar not, let me debug

ericdallo16:02:35

so I changed the uberjar task to this:

:uberjar {:aot :all
                       :jvm-opts ["-Dclojure.compiler.direct-linking=true"
                                  "-Dclojure.spec.skip-macros=true"
                                  "-Dborkdude.dynaload.aot=true"]}
But even when lein uberjar && java -jar target/clojure-lsp-*-standalone.jar I still get the nil print for all those variables

ericdallo16:02:57

I tested a lot of params, even moving the jvm-opts to the root of the project.clj

ericdallo16:02:05

it only work for lein run

ericdallo16:02:30

I checked babashka and the config is the same

ericdallo00:02:45

So, after a lot of debugging I started to wonder if babashka is really being compiled with those flags, I added this to babashka main first line:

(println (System/getProperty "borkdude.dynaload.aot"))
  (println (System/getProperty "clojure.compiler.direct-linking"))
  (println (System/getProperty "clojure.spec.skip-macros"))
Then I run:
script/uberjar
script/compile
and when running ./bb I get:

ericdallo00:02:12

I'm probably missing something during the build, but could not figure it out 😔

borkdude08:02:44

These properties are not part of the runtime, only during build

borkdude08:02:18

So you should put them on the top level and watch if they are printed during compilation

borkdude08:02:36

Of both Clojure and GraalVM

ericdallo14:02:02

Oh, now it makes sense!

ericdallo16:02:31

So, the clojure.compiler.direct-linking was correctly configured, it was just missing the dynaload config that i"ll add now 🙂

ericdallo16:02:39

It indeed reduced from 117MB -> 109MB 🙂

ericdallo16:02:55

it's something indeed, with UPX this should be < 29MB

ericdallo16:02:05

thanks for the help @borkdude

ghadi21:02:01

has anyone any experience making a native image with the Kafka client lib in it?

👀 3
ericdallo21:02:11

This seems to compile a producer and consumer with clojure: https://github.com/dainiusjocas/clojure-kafka-graalvm-native-image Maybe can help you

borkdude22:02:35

There is also this: https://github.com/tzzh/pod-tzzh-kafka A pod which you can use from babashka scripts.

ghadi00:02:18

thanks y'all