graalvm

dominicm 2022-05-14T14:56:09.872979Z

I'm looking at adding native-image support to pack. I know there is already clj.native-image, but it doesn't have a clojure tools (-T) usage, nor a tools.build-like API. Are there any complications/shortcuts/clever hacks that pack could help with? @borkdude I know you do a lot of scripting voodoo with native-image compilation, and it'd be great to port those to be defaults for the whole community using pack if possible.

dominicm 2022-05-14T14:56:46.857979Z

Additionally, on the off chance anyone knows, I'd love to know the state of using svm-driver to build native-image without graalvm/native-image being installed. I've asked on the graal slack, but I'm not expecting an answer. The maven plugin briefly had support for it, but it was removed.

dominicm 2022-05-14T18:51:43.399019Z

Seemingly it's not supported right now, due to certain patches in graalvm not being upstreamed yet. That means it's unreliable between versions. That's unfortunate.

borkdude 2022-05-14T18:53:33.882509Z

Why would you want to add the svm driver?

dominicm 2022-05-14T18:54:06.588569Z

Then pack could compile a native image without needing the user to use graalvm.

👍 1
dominicm 2022-05-14T18:54:21.195559Z

It would run on, eg openjdk

borkdude 2022-05-14T18:54:51.612439Z

Is that even possible? Where is that documented?

borkdude 2022-05-14T18:55:25.494539Z

My experience with including the svm driver in a project is not good. It makes it incompatible over different version of GraalVM

dominicm 2022-05-14T18:55:53.480719Z

It is. The maven plugin used to do it. It's not documented. The maven plugin removed the functionality because it doesn't consistently work for all JDKs and svm versions because of patch upstreaming.

borkdude 2022-05-14T18:57:08.178049Z

It is not that hard to just download GraalVM to a temp folder and use it. The convention that is used is GRAALVM_HOME- I use that in all my compile scripts

dominicm 2022-05-14T18:58:13.926319Z

I am considering doing that, yeah. I prefer the version of graal being pinned to the repo, rather than whatever version the user chooses, as it's got a higher reproducibility.

borkdude 2022-05-14T18:58:54.835979Z

I didn't mean that the tool should download graalvm, I don't like that

dominicm 2022-05-14T18:59:12.364469Z

Why not?

borkdude 2022-05-14T18:59:20.837249Z

I mean, for the user it is not hard at all to grab a version of graalvm and use that for compilation

borkdude 2022-05-14T18:59:53.125689Z

Because I want to use a specific version of graalvm and not be dependent on some tool doing that for me, I'm capable of doing that myself

dominicm 2022-05-14T19:00:12.407749Z

But if you rock up at a repo, which version is that?

borkdude 2022-05-14T19:00:24.969769Z

rock up at a repo, what does that mean?

dominicm 2022-05-14T19:00:48.165669Z

If you are contributing to a new repo, which graal version does the maintainer want to make releases with?

borkdude 2022-05-14T19:01:32.205109Z

usually the newest, but older ones work too. it's the same with Clojure, you don't usually specify the JVM version in your repo, and clj doesn't magically grab a 1GB download from the internets

dominicm 2022-05-14T19:08:52.435069Z

I don't think that comparison is analogous, it's more like specifying which version of the JDK you want your docker image to run with. Releases are a little different from development in terms of increasing consistency. Native image also seems more unstable than the standard JDK. It is also far more user-dependent due to a lack of distribution packaging because of the licensing. My perspective on instability could be inaccurate because I'm watching from afar.

borkdude 2022-05-14T19:09:33.248279Z

It's been pretty stable since 21.3

borkdude 2022-05-14T19:09:41.857859Z

or 21.0 or so

dominicm 2022-05-14T19:10:33.952089Z

Didn't 22 break how initialization worked and break many clojure programs?

borkdude 2022-05-14T19:13:08.664729Z

It deprecated an option, but it didn't actually break anything

borkdude 2022-05-14T19:13:23.153419Z

and clj-easy/graal-build-time is what fixes this

borkdude 2022-05-14T19:13:30.891939Z

so we're ready for deprecation by removal

dominicm 2022-05-14T19:14:00.069089Z

Oh, so it isn't going yet. Good to know. I misunderstood. Still, people might get stuck on older versions.

borkdude 2022-05-14T19:14:43.232179Z

I usually specify the graalvm version in documentation and people can also check in CI or a Dockerfile. If something is not working, I tell them to upgrade.

borkdude 2022-05-14T19:15:18.823269Z

but I might not be the right audience for your tool, I don't think I would ever use a tool again which wraps native-image

borkdude 2022-05-14T19:15:37.917599Z

as that has bitten me in the past. just use the vanilla tool is what I recommend

borkdude 2022-05-14T19:19:32.197909Z

There's also nuances like musl, etc which pack is probably not going to solve

borkdude 2022-05-14T19:20:06.565969Z

My opinion is that having an uberjar tool is great, but from there on, just write a script to build with native image

borkdude 2022-05-14T19:20:57.668209Z

Perhaps @lee has a different opinion :)

lread 2022-05-14T19:23:14.741589Z

Hiya!

lread 2022-05-14T19:25:55.620519Z

My humble opinions? simple_smile So for the topic of svm… if I’ve understood… I recommend going with the flow and using Graal tooling. Graal is complicated enough without introducing unsupported or uncommonly used configurations.

lread 2022-05-14T19:31:45.670449Z

And… are you also talking about a build wrapper? I thinking rolling your own makes sense. It is not that complicated and also educates you, at least a bit, on the working of Graal. There are lots of levers and dials to fiddle with for Graal native-image. I personally want to be able to twiddle all that myself. https://github.com/clj-commons/rewrite-clj/blob/5ca373690efd8eb348f9a06eda7c4d32caeece3d/script/helper/graal.clj. I might reuse this on other projects, but only as a template that I would copy and paste.

👍 1
dominicm 2022-05-14T20:42:09.757259Z

It seems inefficient and exclusionary to have scripts which are copied around. At the least, a tool could handle locating the native image binary and perhaps give you a few of the boring arguments (eg -cp) to pass to native image. I'm certain there's a solution which both exposes the dials AND reduces the maintenance effort across everyone. Clojure always optimizes for power users first, but it still becomes easier over time.

dominicm 2022-05-14T20:43:38.417669Z

For example, that version assertion is dope. The perfect function to expose in a library. You could even stick a simple range on it.

lread 2022-05-14T20:56:02.936299Z

Yeah, I felt the temptation at one point too @dominicm! But Graal native-image is so early days and fiddly that I felt that it was not practical to generalize. That’s just me, though! Not at all saying you are wrong and I am right, just sharing.

borkdude 2022-05-14T21:07:08.094139Z

> At the least, a tool could handle locating the native image binary This is what the GRAALVM_HOME environment variable convention is for. The Github Action https://github.com/graalvm/setup-graalvm/ helps you with setting this up in CI, but doing it manually is trivial too (I've done it manually so far).

borkdude 2022-05-14T21:08:15.731619Z

I also don't want to call native-image from within a JVM process since graalvm is very memory hungry and wasting extra memory on calling it from a JVM isn't something I want. Just a shell script will do.

borkdude 2022-05-14T21:09:11.608639Z

If you leave more memory to native-image it will likely also be faster as it spends less time in GC.

dominicm 2022-05-14T21:38:17.458229Z

Yeah, that's why I wanted to use the svm-driver so there's only one VM.

borkdude 2022-05-14T15:35:13.193749Z

@dominicm A good default is to use https://github.com/clj-easy/graal-build-time

👍 1
dominicm 2022-05-14T15:41:22.422249Z

That's a great tip, exactly the kind of thing I'm looking for. I know native image will always be highly configured, but I'd love to make the basic hello world fairly seamless.

borkdude 2022-05-14T15:47:11.150019Z

I think it makes sense to always compile to an uberjar first and then use -jar foo.jar

borkdude 2022-05-14T15:47:24.187129Z

And always use "--no-fallback"

borkdude 2022-05-14T15:47:46.182459Z

that should be about it for Clojure I think

dominicm 2022-05-14T15:47:50.599319Z

Why an uberjar and not a classpath?

borkdude 2022-05-14T15:48:57.620799Z

It doesn't matter much, but I find it easier to reason about an uberjar. e.g you can test if java -jar foo.jar works, if that doesn't work, then native image will fail as well

borkdude 2022-05-14T15:49:13.914989Z

but classpath should work as well

borkdude 2022-05-14T15:49:37.701249Z

I've had some problems with long classpaths on Windows where an uberjar did work correctly

borkdude 2022-05-14T15:49:48.609699Z

but perhaps tools.build will fix that correctly now

dominicm 2022-05-14T15:50:02.663739Z

I wonder if that applies when using process command?

dominicm 2022-05-14T15:50:34.850109Z

I also wonder if native image can handle classpaths inside a jar manifest

borkdude 2022-05-14T15:51:04.985279Z

I also find myself wanting to publish an uberjar as an alternative to the native image anyways, so I would need both

borkdude 2022-05-14T15:35:37.909659Z

And a good default is to allow arbitrary native image arguments on top of the defaults as every projects requires its own settings

borkdude 2022-05-14T15:35:56.460359Z

You can read more info here: https://github.com/clj-easy/graal-docs