graalvm

Andrey Subbotin 2024-06-18T07:08:50.286719Z

Hi all, does anyone have a clue why a binary produced by native-image on macOS would try to load libc.so.6 when ran... and not, say, libc.dylib or something similar...? Might someone have ideas on how to fix the below?

Exception in thread "main" java.lang.ExceptionInInitializerError
	at java.base@22.0.1/jdk.internal.foreign.SystemLookup.find(SystemLookup.java:43)
	at testlib.NativeBridge.ffi(NativeBridge.java:16)
	at testlib.core$run_app.invokeStatic(core.clj:11)
	at testlib.core$run_app.invoke(core.clj:7)
	at testlib.kitchen_sink.main$_main.invokeStatic(main.clj:10)
	at testlib.kitchen_sink.main$_main.invoke(main.clj:8)
	at clojure.lang.AFn.applyToHelper(AFn.java:152)
	at clojure.lang.AFn.applyTo(AFn.java:144)
	at testlib.kitchen_sink.main.main(Unknown Source)
	at java.base@22.0.1/java.lang.invoke.LambdaForm$DMH/sa346b79c.invokeStaticInit(LambdaForm$DMH)
Caused by: java.lang.IllegalArgumentException: Can't load library: libc.so.6
	at org.graalvm.nativeimage.foreign/com.oracle.svm.core.foreign.Util_java_lang_foreign_SymbolLookup.createNativeLibraries(Target_java_lang_foreign_SymbolLookup.java:159)
	at org.graalvm.nativeimage.foreign/com.oracle.svm.core.foreign.Util_java_lang_foreign_SymbolLookup.libraryLookup(Target_java_lang_foreign_SymbolLookup.java:181)
	at org.graalvm.nativeimage.foreign/com.oracle.svm.core.foreign.RuntimeSystemLookup.makeSystemLookup(RuntimeSystemLookup.java:63)
	at org.graalvm.nativeimage.foreign/com.oracle.svm.core.foreign.RuntimeSystemLookup.<clinit>(RuntimeSystemLookup.java:39)
	... 10 more
Caused by: java.lang.UnsatisfiedLinkError: Can't load library: libc.so.6
	at org.graalvm.nativeimage.builder/com.oracle.svm.core.jdk.NativeLibraries.loadLibraryPlatformSpecific(NativeLibraries.java:90)
	at org.graalvm.nativeimage.foreign/com.oracle.svm.core.foreign.Util_java_lang_foreign_SymbolLookup.createNativeLibraries(Target_java_lang_foreign_SymbolLookup.java:156)
	... 13 more
The FFI code failing is merely this:
package testlib;

import java.lang.foreign.Arena;
import java.lang.foreign.FunctionDescriptor;
import java.lang.foreign.Linker;
import java.lang.foreign.MemorySegment;
import java.lang.foreign.SymbolLookup;
import java.lang.foreign.ValueLayout;
import java.lang.invoke.MethodHandle;

public class NativeBridge {
  public static long ffi() throws Throwable {
    Linker linker = Linker.nativeLinker();
    SymbolLookup stdlib = linker.defaultLookup();
    MethodHandle strlen = linker.downcallHandle(
      stdlib.find("strlen").orElseThrow(),
      FunctionDescriptor.of(ValueLayout.JAVA_LONG, ValueLayout.ADDRESS)
    );

    try (Arena arena = Arena.ofConfined()) {
      MemorySegment cString = arena.allocateFrom("Hello");
      long len = (long)strlen.invokeExact(cString); // 5
      return len;
    }
  }
}
And the native-image command to produce the binary was this: >
$GRAALVM_HOME/bin/native-image \
>   -jar target/testlib-kitchen-sink-0.0.1-standalone.jar \
>   -cp target/testlib-kitchen-sink-0.0.1-standalone.jar \
>   -H:+ReportExceptionStackTraces \
>   -H:+UnlockExperimentalVMOptions \
>   -H:+ForeignAPISupport \
>   -J-Dclojure.spec.skip.macros=true \
>   -J-Dclojure.compiler.direct-linking=true \
>   --initialize-at-build-time \
>   --no-fallback \
>   --report-unsupported-elements-at-runtime \
>   --native-image-info \
>   --enable-url-protocols=http,https \
>   --verbose \
>   -march=native \
>   target/testlib-kitchen-sink-0.0.1-standalone
Environment:
❯ java -version                                                                                                                               
openjdk version "22.0.1" 2024-04-16
OpenJDK Runtime Environment Zulu22.30+13-CA (build 22.0.1+8)
OpenJDK 64-Bit Server VM Zulu22.30+13-CA (build 22.0.1+8, mixed mode, sharing)

❯ $GRAALVM_HOME/bin/native-image --version                                                                                                    
native-image 22.0.1 2024-04-16
GraalVM Runtime Environment Oracle GraalVM 22.0.1+8.1 (build 22.0.1+8-jvmci-b01)
Substrate VM Oracle GraalVM 22.0.1+8.1 (build 22.0.1+8, serial gc, compressed references)
The FFI call runs fine when the jar is ran with java, but it fails when ran from the binary produced by native-image.

borkdude 2024-06-18T07:11:46.928889Z

Are you running the binary on the same system as where you compiled it?

Andrey Subbotin 2024-06-18T07:12:35.268469Z

Yep.

Andrey Subbotin 2024-06-18T07:13:10.430669Z

Someone has just replied to the same question in the graalvm slack. It seems macos is not really supported for FFI yet: https://github.com/oracle/graal/blob/81d23df042d19a359b3a978f776744fa310bf93a/substratevm/src/com.oracle.svm.core.foreign/src/com/oracle/svm/core/foreign/RuntimeSystemLookup.java#L60-L66

borkdude 2024-06-18T07:13:18.985009Z

what is the output for otool -L ./the-binary?

borkdude 2024-06-18T07:13:40.197319Z

oh ok

borkdude 2024-06-18T07:14:44.102689Z

Is this the new Panama stuff?

🎯 1
borkdude 2024-06-18T07:16:07.095139Z

You can do FFI via the bespoke graalvm C API https://yyhh.org/blog/2021/02/writing-c-code-in-javaclojure-graalvm-specific-programming/

borkdude 2024-06-18T07:16:25.474889Z

or JNI

borkdude 2024-06-18T07:16:33.863269Z

but it would be nice if Panama would be supported in the future

borkdude 2024-06-18T07:18:31.493109Z

I've got a Rust example here: https://github.com/borkdude/clojure-rust-graalvm

Andrey Subbotin 2024-06-18T07:21:36.705239Z

I have the jni version working... and was merely looking into whether I could avoid writing all the native bridge code. O:-)

Andrey Subbotin 2024-06-18T07:21:41.480939Z

A constraint I have on all of this though, making a native binary have to be optional in my case. Will the graalvm C API approach work when the code utilising it runs in a normal jvm?

borkdude 2024-06-18T07:22:29.775589Z

No, that only works in a native-image so you have to write that FFI code twice in that case, unless you use JNI (or JNA)

borkdude 2024-06-18T07:24:32.442159Z

I think datalevin (project by the same author as the article I linked) did this too I believe

Andrey Subbotin 2024-06-18T07:25:55.680699Z

Thanks heaps for the article link. I'll dig a little more into this... might as well just resort to using ffi anyway and maybe, by the time I'm finished with the implementation, graalvm would have landed a proper Panama support. 🤞

phronmophobic 2024-06-18T16:35:32.390129Z

There are a couple of options available that are in flux while Panama becomes supported more widely. You can use JNA, which can be used on the JVM and when compiled via native image. It's slower compared to JNI and Panama, which may or may not be acceptable depending on your use case.

phronmophobic 2024-06-18T16:36:53.913239Z

dtype.next has a generic ffi interface with implementations for JNA and graalvm, https://cnuernber.github.io/dtype-next/tech.v3.datatype.ffi.html. I think there is some support for an old version of Panama.

Andrey Subbotin 2024-06-19T05:49:23.474939Z

Whoa, I guess I was mistakingly under impression JNA wasn't going to work in native image after reading https://github.com/oracle/graal/issues/673, which said "Full support for JNA is still not available or planned". 😁 But after reading your suggestion, I've decided to give it a try, and as far as I'm concerned it seems to work just fine. So I guess it might be the best option for me for now. Thanks heaps!