Fork me on GitHub
#graalvm
<
2021-06-17
>
borkdude20:06:07

Please upvote or join this discussion: https://github.com/oracle/graal/discussions/3476

👍 13
😨 11
chrisn23:06:52

Wow. Thanks for getting in there for the rest of us.

rickmoynihan10:06:00

Can anyone explain some of the finer details here? I know java static initializers are code blocks that execute at class instantiation; so I’m assuming for native image that --initialize-at-build-time just means those blocks are executed when the native image is compiled (hence for example any side effects etc there will happen at compile time not runtime). What I don’t fully understand is how the clojure compiler uses static initializers, and exactly what the implications of this are for clojure. I’m assuming it’s because clojure’s a lisp, and even aot’d clojure code still has to apply effects at runtime. e.g. ns initialisation presumably runs as a static initializer etc.

rickmoynihan10:06:50

so presumably that’s why this impacts all clojure code

borkdude10:06:05

I've tried several times to build without this option. It's educational, but I can't really put words to this to explain it in detail, but due to how Clojure is set up / compiles things, it's needed.

borkdude10:06:19

I would say: try without the option and perhaps come up with a better explanation, I think that would be very useful.

rickmoynihan10:06:25

so build a clojure hello world, with and without this option? And inspect the clojure generated class files with javap to see if we can explain it?

borkdude10:06:37

Clojure emits these kinds of things for functions:

static {
        const__0 = RT.var("clojure.core", "println");
    }

borkdude10:06:50

which could be related to this

rickmoynihan10:06:26

yeah I’ve seen those before… presumably static linking changes those too?

borkdude10:06:53

Try to build a graalvm hello world program without the option and see how far you get

borkdude10:06:26

If you don't initialize at build time then you will get:

Caused by: java.io.FileNotFoundException: Could not locate clojure/core__init.class, clojure/core.clj or clojure/core.cljc on classpath.
	at clojure.lang.RT.load(RT.java:462)

borkdude10:06:03

this can be "fixed" by including clojure.core (init.class) on in the resources

rickmoynihan10:06:42

Firstly it’s unsurprising that the line in RT is in a static initializer 🙂

borkdude10:06:29

but that uses the dynamicclassloader, etc, which won't work in a native-image anymore anyway

borkdude10:06:38

so I think that already explains why you need build time

borkdude10:06:55

so we can quit here already

rickmoynihan11:06:29

yeah — was going to say something similar though hopefully you can clear it up for me, if I’m inaccurate/wrong… My reasoning was essentially: 1. clojure is a single pass compiler… i.e. essentially your whole program is a flattened require tree / repl session… all deps “essentially concatenated”. 2. therefore if we’re loading clojure/core there, at some point after that we’ll be loading all of your apps dependencies in a similar initializer block.

rickmoynihan11:06:43

3. hence we can quit here

rickmoynihan11:06:52

Though I guess the dynamic class loader essentially just implements what I said

rickmoynihan11:06:12

i.e. resolving clj / class files, and compiling clj into .class etc… essentially controlling “Read (compile) Eval”

borkdude11:06:41

yeah. another way to put it: you can't "dynamically load classes" at runtime in GraalVM native-image, but Clojure does this in static initializer blocks, hence these must be initalized at build time.

rickmoynihan11:06:20

Yeah that’s a good way to put it

borkdude11:06:17

Perhaps this could be resolved if you make a Java class which does all the loading in a static initializer block and you only initialize that one at build time and the rest of your classes could be inialized at runtime, but this would probably require changes to Clojure itself

borkdude11:06:30

But interestingly these changes can be accomplished using substitutions as well perhaps.

rickmoynihan11:06:34

I was wondering why the clojure compiler needs to use the dynamic class loader for AOT’d code? Presumably it could (at least in a graal compilation context) avoid that? Or is that essentially what you’re describing?

borkdude11:06:52

yeah, that's what I was trying to describe

👍 3
borkdude11:06:47

in an AOT-ed (native-image) setting you know which namespaces you want, so you could just write that out explicitly

rickmoynihan11:06:43

Backing up for a second to the graal issue: Re thomaswue’s point: > Specifying the option for all classes in a specific jar file seems quite reasonable. Would it be OK for this to only work in such broad manner if an uber jar is created first or is that too limiting? Why do we need to bundle into an uberjar? Could we not also just give them a classpath?

borkdude11:06:37

yes, you can already do this, but the original problem in that topic is that they want to get rid of the option without explicitly specifying the classes for which you want build time initialization

borkdude11:06:08

and here he offers some kind of compromise to be able to say for which .jar you want it. so if you provide an uberjar you will get all the classes again

borkdude11:06:49

Feel free to follow up the discussion

rickmoynihan11:06:48

Yeah… I’m just trying to understand the tooing and froing of conversation. So to summarise the thread / they don’t want people to mark every class for build-time-initialization; because for some possible classes it’ll screw things up. For most idiomatic clojure code we need build-time-initialization. Though for some clojure code that also won’t work (e.g. an ns with (def data (fetch-data-from-postgres ,,,)) will need to either be rewritten or opt in to runtime initialisation). If we default a classpath into build-time-initialization we may build invalid “build time” state into the runtime (adinn’s point) for java deps etc.

rickmoynihan11:06:54

Is that the general gist of it?

borkdude11:06:57

correct. but imo taking away this option will make it harder on Clojure developers since for most CLIs I built this stuff worked fine (or I was able to work around it). Occasionally a library like httpkit would give problems: https://github.com/http-kit/http-kit#native-image

borkdude11:06:35

But perhaps listing all clojure-related namespaces through some script is possible, I was just trying to make sure Clojure projects would still be able to run

rickmoynihan11:06:39

Yeah I agree with that. The default for .clj(c) files should be build time, because of the nature for clojure

rickmoynihan11:06:15

@U04V15CAJ: Yeah I was literally typing: presumably we could use something like mranderson to move all clj code under a new top level package/ns, and then flag that to default as build time

borkdude11:06:48

blegh, I don't like that solution. I like the uberjar solution much better

borkdude11:06:07

mranderson is a hack to make multiple versions of the same library work together

rickmoynihan11:06:45

how do you avoid mixing java library / classes into the uberjar though?

borkdude11:06:28

@U06HHF230 well you don't have to, you could of course just make a jar with your project code + clojure libs and put the Java code into another jar

rickmoynihan11:06:31

(I agree the mranderson thing would be a hack)

borkdude11:06:55

but personally I would just go with all build at runtime for everything and figure out the exceptions

borkdude11:06:17

the tools I build are usually CLI tools and not huge micronaut web server things which I think the issue is more concerned about

borkdude11:06:54

Perhaps we can figure out a good pattern to build only clojure classes at build-time

rickmoynihan11:06:56

> but personally I would just go with all build at runtime for everything and figure out the exceptions well to be fair that is how any approach in clj will eventually end up working — the main difference would be starting from a point where you didn’t picking the wrong default for java libs.

rickmoynihan11:06:26

@U04V15CAJ: Yeah I was going to say the issue is that there’s no tooling that knows what a clojure lib is vs a java lib. We’d need something that knew how to biject clj files into their class files…. essentially mapping munge over the .clj(c) classpath.

borkdude11:06:41

well, that is certainly doable

borkdude11:06:12

but I was trying to avoid getting into this, it all works beautifully now

rickmoynihan11:06:35

it would be nice to avoid having to have another step

borkdude11:06:49

btw, I'm trying these flags:

"--initialize-at-build-time=clojure."
           "--initialize-at-build-time=clojure.core.server"
           
but I'm still getting errors about clojure.core.server

borkdude11:06:27

(trying this in refl)

rickmoynihan11:06:59

with graal 22?

borkdude12:06:57

ok, for refl this seems to work:

"--initialize-at-build-time=clojure,refl"

borkdude12:06:03

but it's a small project without any dependencies

rickmoynihan12:06:53

yeah that’s essentially equivalent to listing all of the top level namespaces you use there.

rickmoynihan12:06:31

it’s good to prove what they’re suggesting will work for us… it’s just a shame it’s more clunky.

borkdude12:06:42

I will try with the httpkit library

borkdude12:06:16

unfortunately there the clojure and java package overlaps

borkdude12:06:43

@U06HHF230 yeah, so this works with httpkit (2.5.3):

"--initialize-at-build-time=clojure,refl,org.httpkit"
           "--initialize-at-run-time=org.httpkit.client"
           

borkdude12:06:16

which doesn't buy you anything really

borkdude12:06:32

since you still have to make explicit because of the overlapping package name

borkdude12:06:41

but at least, it seems doable, but annoying

borkdude12:06:02

I might try for babashka, which is a way bigger project

borkdude12:06:06

later this week

borkdude12:06:08

it seems a namespace refl.main makes a package refl and a class main inside of it

borkdude12:06:25

so you have to use the package name refl to get all the related classes refl.main__init, etc.

borkdude12:06:47

so perhaps a "simple" all-ns with some munging/post-processing could be all that's needed

borkdude12:06:19

@U06HHF230 Something like this:

user=> (->> (map ns-name (all-ns)) (remove #(str/starts-with? % "clojure")) (map #(str/split (str %) #"\.")) (keep butlast) (map #(str/join "." %)) distinct (map munge) (cons "clojure"))
("clojure" "refl" "org.httpkit")

👍 3
borkdude12:06:30

which is what I used for refl + httpkit

borkdude12:06:21

for babashka:

("clojure" "sci.impl" "selmer" "babashka.nrepl" "babashka.impl.clojure.java" "babashka.impl" "rewrite_clj.node" "bencode" "rewrite_clj.parser" "babashka.impl.clojure" "org.httpkit" "rewrite_clj.custom_zipper" "rewrite_clj.zip" "borkdude.graal" "babashka.nrepl.impl" "babashka.pods" "cognitect" "babashka" "edamame.impl" "cheshire" "rewrite_clj" "hiccup" "sci" "borkdude" "flatland.ordered" "babashka.pods.impl" "clj_yaml" "babashka.impl.clojure.core" "datascript" "hf.depstar" "babashka.impl.tools" "sci.addons" "babashka.impl.clojure.test")

borkdude12:06:20

(could probably clean this up by looking at the existence of a prefix in others)

borkdude12:06:33

but you get the gist

borkdude12:06:27

ok, that leads to:

Exception raised in scope ForkJoinPool-2-worker-25.ClosedWorldAnalysis.AnalysisGraphBuilderPhase: org.graalvm.compiler.java.BytecodeParser$BytecodeParserError: com.oracle.graal.pointsto.constraints.UnsupportedFeatureException: No instances of com.fasterxml.jackson.core.io.SerializedString are allowed in the image heap as this class should be initialized at image runtime. To see how this object got instantiated use --trace-object-instantiation=com.fasterxml.jackson.core.io.SerializedString.

borkdude12:06:23

kind of demonstrating that it would be painful to have to do this exercise for every graalvm project

borkdude12:06:39

This jackson thing seems to be the only problem though

borkdude12:06:16

so here's what I ended up with:

borkdude12:06:26

"--initialize-at-build-time=clojure,sci.impl,selmer,babashka.nrepl,babashka.impl.clojure.java,babashka.impl,rewrite_clj.node,bencode,rewrite_clj.parser,babashka.impl.clojure,org.httpkit,rewrite_clj.custom_zipper,rewrite_clj.zip,borkdude.graal,babashka.nrepl.impl,babashka.pods,cognitect,babashka,edamame.impl,cheshire,rewrite_clj,hiccup,sci,borkdude,flatland.ordered,babashka.pods.impl,clj_yaml,babashka.impl.clojure.core,datascript,hf.depstar,babashka.impl.tools,sci.addons,babashka.impl.clojure.test"
       "--initialize-at-build-time=com.fasterxml.jackson"

borkdude13:06:22

so it seems it's feasible

rickmoynihan13:06:56

Sorry was afk for lunch 🙂 > unfortunately there the clojure and java package overlaps What do you mean? Clojure and java code inhabiting the same package/ns? Meaning the java classes are defaulted into build time init?

borkdude13:06:19

yes, for org.httpkit for example

rickmoynihan13:06:59

I’m guessing for babashka you just ran that at a repl and pasted the output into the shell script; but would plan to automate it at somepoint (or convince the graal folk to do something different)

borkdude13:06:13

@U06HHF230 are you on linux btw?

borkdude13:06:59

ok. in #babashka-circleci-builds there are new binaries compiled on the init-at-build-time branch. I wonder if this would impact startup time

borkdude13:06:08

I don't see a real difference on macos yet

rickmoynihan13:06:08

> if this would impact startup time In which direction were you thinking?

borkdude13:06:23

perhaps it's slower if more work has to be done at run time?

rickmoynihan13:06:44

Shouldn’t we be expecting for essentially the same coverage? i.e. all clojure code (except the few exceptions) to be initialised at build time?

borkdude13:06:09

perhaps when you're doing interop it's going to be different

borkdude13:06:21

but perhaps it's not really significant

borkdude13:06:43

so it's good to have a working solution now and be prepared for 22

👍 3
rickmoynihan13:06:44

yeah assuming both builds behave the same wrt to correctness, I’d expect there not to be a significant difference in startup time… If there were it’d probably mean we weren’t covering everything we needed to.

rickmoynihan13:06:06

Do you think any of this changes how the graal thread has been left? > Specifying the option for all classes in a specific jar file seems quite reasonable. Would it be OK for this to only work in such broad manner if an uber jar is created first or is that too limiting?

borkdude13:06:25

I already responded in that thread

borkdude13:06:44

He seems to be in favor of that

rickmoynihan13:06:47

ah thanks — just refreshed

rickmoynihan13:06:05

What are the use cases for the uberjar case thomaswue is pushing for? I’m not even sure for clojure it’s sufficient

borkdude13:06:38

I usually tend to compile and collect all the code into an uberjar first and then feed that to graalvm

borkdude13:06:59

you don't have to do this, but I find this easier, since you just know what code you're dealing with after the uberjar step

borkdude13:06:12

also I distribute the uberjars so people who want to make nixos derivations etc can use them

borkdude13:06:07

I could also say in case of an issue to a graalvm dev: here you have the uberjar, I do this to compile it, but it doesn't work

borkdude13:06:13

without him/her having to install clojure, etc

rickmoynihan13:06:44

Yeah I get that it’s useful for your other requirements (you want uberjars anyway etc). But an uberjar is just a reified/flattened classpath… so why can’t they just take a classpath?

borkdude13:06:19

You should ask this to Thomas, I don't know his reasoning

rickmoynihan13:06:22

I should probably ask them 🙂

borkdude13:06:43

His reasoning could be:

rickmoynihan13:06:54

Just want to check that I’m not arguing against what you want 🙂

borkdude13:06:55

Libraries aren't allowed to say: everything at build time

borkdude13:06:07

but if you have a fat jar, you're not a library owner saying this, you are the end user

rickmoynihan13:06:24

that makes sense

rickmoynihan13:06:16

(actually I was meaning to ask you about this for another reason… I’ll start another thread on the channel for it though as it’s a change of topic)

chrisn15:06:29

I think this also relates to clojure startup time. Perhaps we attempt a clojure-side compilation flag that solves (or makes progress towards) both issues at AOT time?

chrisn15:06:16

meaning when this flag is in effect the clojure compiler generates different byte code and this byte code both starts up faster and works with graal native without needing --initialize-at-build-time.

borkdude15:06:15

@UDRJMEFSN do you have any concrete ideas of what can be done differently?

chrisn15:06:34

Not without more careful consideration I do not.

borkdude15:06:06

what Clojure does in static initializers is resolve vars, load classes, etc.

borkdude15:06:25

delaying the class loading to run time won't work in a native image

chrisn15:06:27

I will look a lot more closely; I just know those are two related things and my profilers always show var initialization as one of the startup issues so somehow compiling that data down into something perhaps more concrete that loads faster is an interesting issue that seems related.

chrisn15:06:38

Also interesting for dalvik.

chrisn15:06:03

I know this is an area smart people have looked at before.

borkdude15:06:08

delaying var initialization to build time will make native images slower to start up right?

chrisn15:06:48

I don't want to delay anything, I want AOT to produce data as a side effect that can be quickly loaded to initialize vars during runtime initialization.

borkdude15:06:34

ok, but now these are are already initialized in the image heap, so that work has already been done when starting the image

chrisn15:06:35

Exact opposite of delaying.

borkdude15:06:10

My point is: moving work from build to run time makes things slower

chrisn15:06:31

Well, for example in your javap above:

const__0 = RT.var("clojure.core", "println");

chrisn15:06:27

Yes, I agree and that is not what I am suggesting. const_0 being initialized to a static class instance in your example above would make things faster as it would bypass the RT.var mechanism.

borkdude15:06:41

it could directly reference the AOT-ed class which represents the println var right?

chrisn15:06:14

Yes, in this case. You also have the case where something is initialized via a complex function that produces a persistent datastructure and in this case the data can be saved in resources and found via a hashtable lookup or straight array lookup in constant time eliding the generating code. I haven't looked at this in huge depth but for instance I was extremely careful with dtype-next and it still takes some time even after an AOT run to pull in, for instance, the ND system via require. This is a solvable problem.

chrisn15:06:43

My thought is more of the form move --initialize-at-build-time into the clojure compiler and allow anything that it did in the graal vm system to be done during the AOT step. Then --init-at-build-time should be a noop if done during graal vm compilation.

borkdude15:06:59

As clojure.core is AOT-ed by default anyway, I guess Compiler could be instrumented in such a way that it can reference these classes directly when generating more code. For core vars only it would already be a win

chrisn15:06:45

This is complicated by the fact that bytecode files aren't general data storage mechanisms (at least as far as I know) which means you need some level of sidecar file generated at build time for pure data.

borkdude15:06:56

The Compiler could keep track of what vars map to which classes

chrisn15:06:20

It would be nasty and error prone. Definitely a YMMV pathway but with time it could work well.

chrisn15:06:02

If it opened up both simpler Graal native and more dalvik development that would IMO be a very solid win worth real invesment.

chrisn15:06:11

Well, I guess if nubank agrees it may be worth real invesment 🙂.

borkdude15:06:33

What problems does Dalvik currently have with Clojure?

borkdude15:06:08

I'm seeing that Dalvik is now replaced with something else

borkdude15:06:14

maybe just a detail. Does ART (Android Runtime) interpret Java bytecode directly?

chrisn16:06:12

I am referring to this article:

borkdude16:06:09

Interesting article, thanks for sharing