Clojurians Log v2

@smith.adriane from my mac

% otool -L rapidyaml/native/librapidyaml.dylib.0.6.0 
rapidyaml/native/librapidyaml.dylib.0.6.0:
	@rpath/librapidyaml.dylib (compatibility version 0.0.0, current version 0.0.0)
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1600.157.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1336.61.1)

Are you loading that specific version?

> I'm assuming you did (Native/loadLibrary "rapidyaml") or similar?

when you load the library, are you just passing the name "rapidyaml" or a specific version of rapidyaml?

what is otool -L rapidyaml/native/librapidyaml.dylib?

My guess is you're loading library without specifying a version (which is typical) and librapidyaml.dylib references librapidyaml.dylib.0.6.0 which can't be found.

https://github.com/yaml/yamlscript/blob/rapidyaml/rapidyaml/src/main/java/org/rapidyaml/LibRapidyaml.java#L62 https://github.com/yaml/yamlscript/blob/rapidyaml/rapidyaml/src/main/java/org/rapidyaml/ILibRapidyaml.java is the loading code I think

note, this all works when using it as a JVM jar

just not as a native-image file

so you are loading a specific version

% YS_PARSER=rapidyaml ./ys/bin/ys -pe 42
Error: java.lang.RuntimeException: Shared library file librapidyaml.dylib.0.6.0 not found
{:eval ["42"], :debug-stage {}, :print true, :mode "code"}

that is the native-image (on the mac)

It looks like you wrote your own loading code, https://github.com/yaml/yamlscript/blob/bc3bc5a8ae7569a1465c92570146e4d25809fc87/rapidyaml/src/main/java/org/rapidyaml/LibRapidyaml.java#L36

You should be able to print out the paths it searched and figure out where it's currently looking and where it's not looking that you would expect it to look.

This is pretty interesting... https://gist.github.com/ingydotnet/9389709e80d23588e9cf6ddeeee9855c

2 failures with rapidyaml, but different when DYLD_LIBRARY_PATH set

wasn't expecting that

I'll try printing the path in the code you linked to

So it looks like you’ve written your own code for finding and loading libraries, which also uses jna’s code for finding and loading libraries.

I'm thinking we need to tell native-image about librapidyaml, like spire does here: https://github.com/epiccastle/spire/blob/3ff1a18a843f56a2a3b0aa646cc3dd8e6ca3891d/graal-configs/resource-config.json

I'm not sure about windows, but I think linux and osx require shared libs to be actual files on the file system. JNA has its own thing for extracting libs to a temp folder. It relies on resources at specific paths.

So you can probably include it as a resource, but you have to put it in the right subdirectory of the resource path.

where is that resource path?

I don't remember off the top of my head.

https://java-native-access.github.io/jna/4.5.0/javadoc/index.html?com/sun/jna/NativeLibrary.html and https://github.com/search?q=repo%3Aepiccastle%2Fspire%20library.path&type=code look promising for configuring this...

> I'm not sure about windows, but I think linux and osx require shared libs to be actual files on the file system. spire packages the shared libs in the native-image but writes them out to disk on startup

Interesting. Thanks for that.

@smith.adriane this is a bit embarrassing but I've come full circle back to your original message in the thread. Google just led me to https://github.com/phronmophobic/jna-native-image 🙄 I'm out of tuits for the night but I think I can put that to good use in the morning. TBH, I don't think I would have understood enough about all this to try it when you first posted...

I've studied and experimented enough to feel pretty comfortable with the Java and JNA. I was coming to the conclusion that native-image needed the right persuasions here and that led me back to you.

that example expects that the library can be found through JNA natural library lookup. If you want to distribute a shared library, you'll have to either have it preinstalled where JNA expects or extract it somewhere that JNA can find it.

There are a lot of cool native libraries. Once everything is setup, it's pretty easy to use them from clojure, but the setup itself is unfortunately very complicated given the various platforms and tools.

https://github.com/epiccastle/spire/blob/3ff1a18a843f56a2a3b0aa646cc3dd8e6ca3891d/graal-configs/resource-config.json#L4 looks like what I need to solve:

Native library (home/ingy/src/yamlscript/rapidyaml/native/librapidyaml.so.0.6.0) not found in resource path ()

Does that work?

I'll let you know 🙂

~/src/yamlscript rapidyaml $ LD_LIBRARY_PATH=$PWD/rapidyaml/native YS_PARSER=rapidyaml ./ys/bin/ys -e 'say: 42'
Exception in thread "main" java.lang.UnsatisfiedLinkError: Unable to load library '/home/ingy/src/yamlscript/rapidyaml/native/librapidyaml.so.0.6.0':
com.sun.jna.Native.open(Ljava/lang/String;I)J [symbol: Java_com_sun_jna_Native_open or Java_com_sun_jna_Native_open__Ljava_lang_String_2I]
com.sun.jna.Native.open(Ljava/lang/String;I)J [symbol: Java_com_sun_jna_Native_open or Java_com_sun_jna_Native_open__Ljava_lang_String_2I]
Native library (home/ingy/src/yamlscript/rapidyaml/native/librapidyaml.so.0.6.0) not found in resource path ()
        at com.sun.jna.NativeLibrary.loadLibrary(NativeLibrary.java:325)

./ys/bin/ys is native-image

for building a native executable, it seems like you would want to build a static library and statically link it.

probably.

I'm not sure about that... If I can load stuff dynamically then it's possible to build libraries using Java classes that are not backed into bb or ys...

does librapidyaml have other shared dependencies?

you can go that route, where librapidyaml is required to be installed by the user

no. for librapidyaml doing it staticly is probably better. but exploring the dynamic option right now

I can make it part of the various YS install methods

in this particular case, it seems like librapidyaml is not "installed" correctly

yeah but I think you need to tell native-image where that is exactly

No, it just needs to be on the load path.

you can see above that I set LD_LIBRARY_PATH

right and it did load librapidyaml, which probably has a dependency on /home/ingy/src/yamlscript/rapidyaml/native/librapidyaml.so.0.6.0

I can't remember the exact terminology, but it has something to with some combination of rpath and install name

hmm, am I misunderstanding the error? not sure what you mean by > right and it did load librapidyaml, which probably has a dependency on /home/ingy/src/yamlscript/rapidyaml/native/librapidyaml.so.0.6.0

@joaoppmagalhaes over here 🙂

I'm assuming you did (Native/loadLibrary "rapidyaml") or similar?

what does otool -L /home/ingy/src/yamlscript/rapidyaml/native/librapidyaml.dylib say?

is that a mac thing?

yea

although, I think there's a similar thing on linux

but with slightly different tools and names.

last message got messed up.

You should get the same result on a mac with:

git clone --branch=rapidyaml  && cd yamlscript && make -C ys build && LD_LIBRARY_PATH=$PWD/rapidyaml/native YS_PARSER=rapidyaml ./ys/bin/ys -e 'say: 42'

rapidyaml not building on mac for me

I need to meet people for dinner

will try more after

I was pushing for packaging the library in the jar archive. IIUC, that's the proper way to ship; eg https://stackoverflow.com/questions/2937406/how-to-bundle-a-native-library-and-a-jni-library-inside-a-jar?rq=1

Are you distributing a jar or an executable built with native image?

Both

https://github.com/yaml/yamlscript/releases/tag/0.1.58

https://github.com/phronmophobic/jna-native-image/pull/1

oh, I guess cos isn't available in libc on linux. you can either use another libc function or change it to load libm instead.

I'll add a check for mac and load the right lib

if you like the PR

+ ./target/jna
cosine of 42 is -0.39998531498835127

yeah, cos is frequently on libm.so, and that would require explicit linking

https://github.com/ingydotnet/jna-native-image/commit/640956eaa9aa7d0cecfec190a2558743e9a2593f

ideally, there would be a function in libc that is easy to test and is included on both mac and linux

@smith.adriane do you think you can make that code work with Native/load instead of NativeLibrary/getInstance ? That's what I'm really trying to figure out...

I like your very small example repo 🙂

probably. why do you want that?

If you've already loaded the library, you can just use NativeLibrary/getProcess instead of get instance.

I think.

Because that is what I have working fine with the JVM... I rewrote https://github.com/yaml/yamlscript/blob/rapidyaml/rapidyaml/src/main/java/org/rapidyaml/LibRapidyaml.java#L74-L96 to be much clearer and separate the code path for JVM (works) from native-image (does not work)

> f you've already loaded the library, you can just use NativeLibrary/getProcess instead of get instance. Problem is I can't load the library yet with a native image: https://gist.github.com/ingydotnet/0363ad5c0647d35169b6cbd45dede8f6

@smith.adriane We are at the point where we'd like to add print debugging near https://github.com/java-native-access/jna/blob/master/src/com/sun/jna/NativeLibrary.java#L325 Do you know how one would do that? I can't find that jar on my system 😞

And if I did, wouldn't know how to edit a jar source...

Any help would be greatly appreciated

have you tried enabling jna's builtin logging? https://github.com/java-native-access/jna/blob/master/www/FrequentlyAskedQuestions.md#calling-nativeload-causes-an-unsatisfiedlinkerror

ha! we were just starting to look into logging (like 10 secs ago)

See https://github.com/phronmophobic/jna-native-image for a JNA example.

I can't seem to find the documentation for ffi calls that I used to use. Maybe it's been deprecated. Anyway, I'm pretty sure they support JNI and JNA. See https://www.graalvm.org/latest/reference-manual/native-image/native-code-interoperability/ more info.

I've used https://cnuernber.github.io/dtype-next/tech.v3.datatype.ffi.graalvm.html with success previously.

Spire is another example