Clojurians Log v2

Another lead - Java version mismatch? https://stackoverflow.com/questions/77807600/graalvm-in-docker-fatal-error-reported-via-jni-could-not-allocate-set-propert

More hints: https://www.graalvm.org/latest/reference-manual/native-image/native-code-interoperability/JNIInvocationAPI/#load-the-native-library Since LD_LIBRARY_PATH did not work for me, need to find how to set Java property java.library.path

I added to hello_world.clj's -main the following code at beginning:

(println "Trying to print something")
(println "java.library.path =" (java.lang.System/getProperty "java.library.path"))
(java.lang.System/setProperty "java.library.path" (str "/path/to/membrane-native-image:" (java.lang.System/getProperty "java.library.path")))
(println "java.library.path =" (java.lang.System/getProperty "java.library.path"))

I think I've made a little progress. Based on https://www.praj.in/posts/2021/compiling-swing-apps-ahead-of-time/. I think the problem is config related. Their example for generating the config does seem generate config, but I think it also generates a bunch of incorrect config for clojure based code. I ran it with a subset of the config and it crashed on a java exception instead of a heap dump, which seems like progress.

ok, I actually got it to run

but a simpler version since it's an ec2 instance and I haven't setup a way to see if a window pops up

ok, it's getting late here. just pushed a new commit that seems to work locally. currently, you have to run it and pass in java.home, but it should be possible to work around that. For now, it should be run with:

./hello-world -Djava.home=/home/ubuntu/graalvm

Do I need to update the following in deps.edn?

:git/sha "b04bbe291658d8883ef9741f517c7b6ddf756b75"
:git/url ""

the changes are • there's a new flag for native-image, -H:ConfigurationFileDirectories=config • the main function just prints the size of the app since I don't have a linux setup to see if a window opens. you'll likely need to regenerate config if you do more than that • there's now a config directory

the membrane commit hasn't changed.

so there will likely be more work to get it working in the general case, but hopefully we're on the right track

hopefully, it's just a matter of generating the right config

which can be done using the agent while exercising all of the relevant code

I see no new commit when trying to sync my fork with your membrane-native-image.

Did you push your new commit to your GitHub repository?

let me double check

you don't see https://github.com/phronmophobic/membrane-native-image/commit/fada9ea0ed2b6898230d959fa0c7ddcb383eb5db

Now I see it, and have pulled to my repository. Good night (your time).

A big progress but no dice, yet. Now hello-world complains about Exception in thread "main" java.lang.NoClassDefFoundError: sun/awt/SunToolkit rather than about Fatal error reported via JNI: Could not allocate library name. I invoked it using: ./hello-world -Djava.home=../../graal-21/graalvm-jdk-21.0.2+13.1 (it is is where my GraalVM is) and it still has the java.lang.System/setProperty "java.library.path" code (I did not remove it).

I think it's just a matter of getting the right config for graalvm. However, I tried it with liberica and it seems to just work™. You can try using liberica or try to follow the steps from https://github.com/phronmophobic/membrane-native-image?tab=readme-ov-file#resources to figure out the right config for graalvm. I updated the ./config.sh script to hopefully make that easier if you go that route.

1. I need to use GraalVM, at least in the sense that our work is not done until it works with GraalVM. 2. What is the job of config.sh? When (if at all) should I run it? 3. What can you tell me about sun/awt/SunToolkit?

> I need to use GraalVM, at least in the sense that our work is not done until it works with GraalVM. why?

3. I think this can be fixed by adding the right config 2. It helps create the right config For more info about configs, see the links I sent you.

I ran JAVA_HOME=../../graal-21/graalvm-jdk-21.0.2+13.1 ./config.sh and it failed:

+ clojure -T:build compile
WARNING: compile already refers to: #'clojure.core/compile in namespace: build, being replaced by: #'build/compile
Execution error (ClassNotFoundException) at jdk.internal.loader.BuiltinClassLoader/loadClass (BuiltinClassLoader.java:641).
org.graalvm.nativeimage.ImageInfo

for whatever reason, it's not using graalvm when running

you need to make sure which java uses graalvm

hmmm. I set JAVA_HOME, forgot to set also PATH.

I have the following in my bashrc. Maybe you'll find it useful:

function graalvm() {
     export JAVA_HOME=/Library/Java/JavaVirtualMachines/graalvm-community-openjdk-21.0.2+13.1/Contents/Home
     export GRAALVM_HOME="$JAVA_HOME"
     export PATH=$JAVA_HOME/bin:$PATH

    java -version
}
# export PATH=$GRAALVM_HOME/bin:$PATH

function liberica() {
     export JAVA_HOME=/Library/Java/JavaVirtualMachines/bellsoft-liberica-vm-core-openjdk21-23.1.2/Contents/Home
     export GRAALVM_HOME="$JAVA_HOME"
     export PATH=$JAVA_HOME/bin:$PATH

    java -version
}

I then just do:

$ graalvm

Weird. I fixed config.sh to set JAVA_HOME,`PATH` and when running it, I saw a membrane window (seems to be your todo demo app). However, when I later ran compile.sh, I got:

Error: Class initialization of clojure.core.reducers__init failed. Use the option 

    '--initialize-at-run-time=clojure.core.reducers__init'

 to explicitly request initialization of this class at run time.

yes. If you read the link I sent you, you generate the config by running the program with the agent running, which records all the runtime info needed.

Not sure if you're using the latest commit, but there should be a fix-config step which should fix the clojure.core.reducers__init error.

I am using commit 4db0ce69112347bd84f1303ba39d4bb57971f5a4 of yours (my own commits are rebased following it).

hmmm... I saw you pushed another commit, which GitHub neglected to inform me about. I am fixing now.

actually, that commit probably won't help as it is meant to test liberica

you need to have -H:ConfigurationFileDirectories=config in the compile script

Is there a particular reason you require graalvm over liberica?

Babashka uses GraalVM. On second thought, actually, Babashka pods do not need to use the same JVM, they do not even need to use a JVM at all (there is a pod which is a Python script).

Yes, I don't think it should matter as far as creating a pod goes.

or if you wanted to distribute your own bb+ui binary

it shouldn't matter for that either.

I should ask borkdude (Babashka's BDFL) if he ever tried to build Babashka with liberica.

I would worry about relying on any implementation details specific to liberica, but that doesn't seem to be the case. It seems like someone will eventually upstream awt support into graalvm.

Anyway, this time compile.sh ran to completion, but hello_world aborted with a familiar error message:

Fatal error reported via JNI: Could not allocate library name

does your compile.sh include H:ConfigurationFileDirectories=config?

My bad, the line was included but commented out. ##########################################

Now it failed with:

Error: Class initialization of clojure.core.reducers__init failed. Use the option 

    '--initialize-at-run-time=clojure.core.reducers__init'

what does you config script look like? did you update the config before compiling?

Attached please find my config.sh, and I run it with

GRAALVM_HOME=../../graal-21/graalvm-jdk-21.0.2+13.1 ./config.sh

I thought the fix-config script would fix the clojure.core.reducers__init problem

I don't have fix-config.sh neither does your repository have it (at least in root directory).

it's in build.clj

clojure -T:build fix-config

This command is at end of config.sh. So it did get execute. How long should I keep the todo demo window active for fix-config to finish its work?

I don't think it matters just for testing. Eventually, you'll need a demo that exercises all the code paths you'll be interested in the pod to get the right config.

Your hello_world uses the following:

(java2d/run
    (membrane.component/make-app
     #'td/todo-app
     td/todo-state)))

?

My version doesn't open a window since my linux doesn't have a screen.

If you do want to try that as your hello world, you'll probably need something like:

(defn -main [& _args]
  (java2d/run-sync
      (membrane.component/make-app
       #'td/todo-app
       td/todo-state))
  (shutdown-agents))

I notice that config.sh somehow runs Todo app, but grepping all files failed to yield anything pointing at it (especially not deps.edn).

so that the app exists when you close the window

So config.sh runs membrane.example.todo under Clojure and it works. Then when compile.sh is successful, membrane.example.todo tries to run under GraalVM, and it aborts. Is this what is happening?

I hope you have patience with me - it is the first time I am trying to build a Babashka pod, my first contact with native-image, and I knew no Clojure at all three months ago.

native image + clojure is kind of tricky and native image + clojure + AWT is very bleeding edge

clojure is very dynamic and native image needs runtime information to know which dynamic features are necessary. ./config.sh runs the app and collects a list of dynamic usage which is recorded as config in the config dir. ./compile.sh compiles the app and uses the config info

I see there are some *.json files in ./config/ Is it necessary to clean the directory, or does config.sh regenerate from 0 all of them?

I think ./config.sh followed by ./compile.sh should work, but there are enough differences between the working setup and your setup.

> Is it necessary to clean the directory, or does config.sh regenerate from 0 all of them? I'm not actually sure. It probably wouldn't hurt to clean before recreating the config.

The config directory in your repository has jni-config.json,reflect-config.json Are those files regenerated?

they should be regenerated when you run ./config.sh

So they should not be under version control (however this does not harm - seems they are unchanged when regenerated).

the ones in the repo were generated by running a different main function, so they won't contain the right config since your main function is different.

> So they should not be under version control (however this does not harm - seems they are unchanged when regenerated). (edited) if you were using the same main function, then it would allow you to skip the config step.

Eventually, you'll need a demo that exercises all the code paths you'll be interested in the pod to get the right config. That config can be saved for reproducible builds without having to go through the config step again.

I've found the config step isn't needed for liberica. It's likely they automatically include the right config to support AWT.

Hopefully, graalvm will also have better AWT support in the future

OK, I'll try liberica. Can I use the same compile.sh script (by using a different GRAALVM_HOME)?

Yes. It should work for either.

I've been switching back and forth for the github action.

GitHub actions are another spell, with which I am not currently familiar.

Reading the Liberica installation instructions, it mentions "language plugins". Do I need any language plugin for our undertaking?

Nope

you can see how I install it in the github action, https://github.com/phronmophobic/membrane-native-image/blob/6aba2262238dd148b0fef1e19811cc47b5c88005/linux-install.sh#L9

It just downloads the tarball, put it's somewhere, and then adds it to the path

I think there are some extra dependencies that are recommended which you may already have

I don't point to that script in the docs since it would modify files outside of the membrane-native-image repo.

but hopefully it gives you an idea of what's necessary and you can tweak for your desired setup

Thanks. By the way, wget did not work for me due to invalid certificate for http://download.bell-sw.com So I downloaded using the Google Chrome browser which apparently has root certificates that wget does not have.

is the code you're using available somewhere?

Not an answer to your question: I am now experimenting with --initialize-at-build-time, and find all kinds of sun.awt.* classes which insist that they must be initialized at run time. I override by --initialize-at-run-time=sun.awt.X11 and such and each time I try to compile, GraalVM complains about another class. An answer to your question: My code is not available, yet, but if you want to deep dive into it, I'll create a GitHub repository by cloning your repository.

You may have to find some graalvm+swing specific docs to figure out the right configuration. Generally, you want clojure classes to be initialized at build time (I recommend clj-easy/graal-build-time, https://github.com/clj-easy/graal-docs?tab=readme-ov-file#class-initialization) and you want java classes to be initialized at runtime. If your clojure uses a java class, you'll need to make sure it defers instantiating classes until runtime.

"If your clojure uses a java class" - it seems to be YOUR code that is using all those problematic Java classes. 🥳 I am now officially stuck. I'll let you know when my code is ready for your experimentation.

I haven't used graalvm with the java2d backend. It's not clear which backend(s) you're trying to use, but the java2d backend will likely require some changes to support native image (which I'm happy to help support).

Actually my script tries to access all backends (see src/pod/tddpirate/membrane.clj lookup function). You may want to comment out some of the backends. The pull request (#77) with my pod related stuff is waiting now for you. Enjoy!

it looks like it's only using the java2d backend. the other namespaces are backend independent

Please let me know when the GraalVM build succeeds, so that I can start debugging the Babashka pod code itself.

I've been in this position before where someone wants a feature that sounds reasonable. I do the work to help add the feature, and then I never hear back to find out if the feature was used or solved a real problem. I try to prioritize features that solve useful problems so it would be helpful for me to hear a bit more about the intended use case and the problem you are trying to solve.

Fair enough. Actually, it is an excellent idea to verify that a feature request is based upon a real need! I am building miscellaneous scripts, which will replace my bash scripts, and which will use the power of Clojure. Hence, Babashka. Some of those scripts have an UI, currently implemented using whiptail. I would like to get them to display Clojure-driven GUI. I looked into humble-ui and liked more the philosophy of membrane. By the way, as an alternative to creation of a membrane Babashka pod, it is possible to create a custom version of Babashka which has the membrane library already built into it. I'll look into it. It'll probably have the same problem as the membrane Babashka pod work I have been doing. But who knows?

I'm not familiar with whiptail. I'll have to check it out.

I do think integration with babashka would be neat for various use cases. One worry I had with compiling membrane with native image is that binary size might be too big for some uses. I'm actually not sure what size binary to expect, but my guess is that it should be fine for a dev tool that gets used repeatedly. Binary size would be a bigger concern if you wanted to distribute an app that only gets used once or twice.

> it is possible to create a custom version of Babashka which has the membrane library already built into it. Yes, that's another alternative, but it would still require the same changes that building a pod would require.

I suggest that for now, let's not worry about binary size of membrane with native image. Current sizes are: membrane-0.11.111-beta-standalone.jar (0.11.111 is the version which includes my Babashka pod stuff): 10MB bb (Babashka executable): 65MB So I guess that adding membrane to Babashka executable will not get the custom version to be longer than 100MB. It may be possible to reduce the size by removing libraries not likely to be needed by people who need GUI. OK, please let me know how are things going, especially if there is anything in my code which is problematic for you.

I don't think membrane's code will bloat the binary by much, it's pulling in all the swing and AWT dependencies that I'm worried about.

it's probably not too bad, but it will be something to look out for

Making progress...

The hello world was ~60MB and I tried a todo app which was 80MB. I think there are also some optimizations that help. Anyway, seems promising.

Where in GitHub can I find what you did to accomplish this? I think I can live, for now, with 60MB hello world & Co. while we work on optimizing the stuff. I would like to be able to start developing my GUI-enhanced scripts right away.

It required some minor changes to membrane to work. I'm still checking in the changes and documenting things.

I'll let you know when it's ready. Hopefully later today!

Did you need to fix anything in my script (which was not debugged before making it available to you)?

I ended up creating a fresh project so that I could test things in isolation

so I haven't tried with your code yet

OK, once membrane compiles and works under GraalVM, I'll be happy to start debugging my Babashka pod script, if still needed.

https://github.com/phronmophobic/membrane-native-image

I haven't tested running the app on linux, but the build successfully compiles, https://github.com/phronmophobic/membrane-native-image/actions/runs/8310661320/job/22743413655

Let me know if you have any questions or run into any issues.

1. For my curiosity: what use case/s are satisfied by liberica (I understand, from Google, that liberica implements yet another JVM)? 2. Do I work from your membrane's master branch or from its native-image branch?

It doesn't really make sense to include the pod inside the membrane repo. I suggest either creating a separate project based off of https://github.com/phronmophobic/membrane-native-image or just forking it. You can either use the git dep referenced in that repo or use a local repo repo from a checkout of the native-image branch of membrane.

It might be useful to start with reproducing the steps from the membrane-native-image repo.

I think liberica is just a fork of graalvm that has already added better support for AWT. I assume that AWT support will get upstreamed back to graalvm at some point.

I found liberica based on a comment on this issue https://github.com/oracle/graal/issues/4124

Good morning @smith.adriane (here it is 07:30 AM), I tried building and running your membrane-native-image. The fatal error that I got was:

Error: Main entry point class 'com.phronemophobic.membrane.hello_world' neither found on

> I used an existing GraalVM native-image binary rather than run clojure -T:build:native-image compile Skipping clojure -T:build:native-image compile sounds suspicious. It creates class files which you need during native image

(cont'd) I have some questions: 1. When you ran your compile.sh, what output did you see for java -version? 2. What is the exact effect of clojure -T:build:native-image compile (I commented out without understanding exactly what it does)?

There's almost no way you'll have a class file with main at com.phronemophobic.membrane.hello_world unless you do clojure -T:build:native-image compile

if you look at the compile function in build.clj , it compiles the clojure to a jvm class file.

It essentially the equivalent of $ clojure -M -e "(compile 'hello-world.main)" in https://github.com/clj-easy/graal-docs/blob/master/doc/hello-world.adoc#step-2-compile-project-sources-to-class-files

I tried compiling with graalvm instead of liberica on linux and it did successfully compile, but I haven't tried running the executable which should hopefully work, but might not.

not sure if you can see the logs in https://github.com/phronmophobic/membrane-native-image/actions/runs/8310704846

here's the output of java -version for that run:

+ java -version
openjdk version "21.0.2" 2024-01-16
OpenJDK Runtime Environment GraalVM CE 21.0.2+13.1 (build 21.0.2+13-jvmci-23.1-b30)
OpenJDK 64-Bit Server VM GraalVM CE 21.0.2+13.1 (build 21.0.2+13-jvmci-23.1-b30, mixed mode, sharing)

it's important that the clojure -T:build:native-image compile is run with graalvm rather than some other jvm

the class files from clojure -T:build:native-image compile end up in the target/classes folder

the basic work flow is 1. compile clojure to .class files 2. use native-image to compile .class files to native code

Thanks, the clojure -T:build:native-image looked to me like rebuilding a custom version of native-image. 😂😂😂 Anyway, after performing the above clojure -T, hello-world executable was successfully built. However, when it ran, it aborted with the error message:

Fatal error reported via JNI: Could not allocate library name

libawt_headless.so  libfontmanager.so  libjvm.so
libawt.so           libjavajpeg.so     liblcms.so
libawt_xawt.so      libjava.so         libmlib_image.so

oh interesting. I saw something about that mentioned on github. where do those files end up?

I can try to find the issue on github, but they said something about those needing to be in the same place as the executable

Those library files end up in the same directory as hello-world i.e. main directory of membrane-native-image.

oh, I think I remember, you somehow need to set a javapath because it doesn't include the current directory. one sec

I'm having trouble finding it. can you run ldd hello-world? It should print the libraries it depends on and the path it expects them to be

$ ldd hello-world 
	linux-vdso.so.1 (0x00007ffe79322000)
	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f8fd0c15000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f8fd0c0f000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f8fd0bed000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f8fcb82c000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f8fd0c76000)

My system has all five libraries (linux-vdso.so.1 is a pseudo-library or something).

yea, so my guess is that it's trying to load those other libraries at runtime from java.library.path, but I have no idea how to set it

is there something in the error dump about java.library.path?

No.

ok. I'll see if I can reproduce on an aws instance

My PC runs on Debian Linux 11.9 and my "regular" clojure is Clojure CLI version 1.11.1.1429.

I'll also be able to try with liberica

My modified compile.sh script has the following at its beginning:

if [ -z "$GRAALVM_HOME" ]; then
    echo "Please set GRAALVM_HOME"
    exit 1
fi

if [ ! -e "${GRAALVM_HOME}/bin/native-image" ]; then
    echo "Your ${GRAALVM_HOME} does not have native-image, aborting."
    exit 1
fi

export JAVA_HOME=$GRAALVM_HOME
export PATH=$GRAALVM_HOME/bin:$PATH

which java
java -version

GRAALVM_HOME=../../graal-21/graalvm-jdk-21.0.2+13.1 ./compile.sh

It's gotta be something wrong with the script

either some missing flag for native-image or some missing post step

By the way, hello-world aborts also with the above setting and when setting JAVA_HOME to the same.

hopefully, it works on linux. I thought I saw some indications that it worked on linux, but I'm having trouble finding the ticket on github that I'm thinking of

... or some extraneous flag (I added the flag H:+UnlockExperimentalVMOptions, should try also without this flag).

ok, I have a linux instance setup which is hopefully enough to try to test things out myself on linux

I've only done basic native-image compilation for linux. There's probably some linux specific options that might help.

I was able to reproduce the error.

Google search result: https://lightrun.com/answers/gmu-swe-phosphor-could-not-allocate-library-name-when-load-native-library-by-jna

I tried to set LD_LIBRARY_PATH before running hello-world. Did not help.

Another thing to try? The option addBuiltinPkgNativePrefix according to https://github.com/oracle/graal/issues/3359

well, I tried liberica and now the server isn't responding 😐