Fork me on GitHub
#graalvm
<
2021-02-27
>
borkdude14:02:43

Slides from babashka talk at GraalVM Workshop 2021

catjam 6
🎉 3
ericdallo16:02:32

I'll doing a POC on clojure-lsp to use datalevin , it worked really well, but having issues compiling the clojure-lsp native-image, should I manually compile the datalevin/native/src libs manually? I thought we could have the libs compiled in the classpath just like https://github.com/xerial/sqlite-jdbc/blob/master/src/main/resources/org/sqlite/native/Linux/x86_64/libsqlitejdbc.so

ericdallo16:02:06

Otherwise should I need to compile it like https://github.com/juji-io/datalevin/blob/master/native/script/compile#L19-L21 besides the need to install liblmdb-dev?

ericdallo16:02:08

I mean, I can copy the c files to clojure-lsp, manually gcc them during CI but it seems odd to me, it'd be great to have those libs from datalevin somehow

ericdallo17:02:51

Even manually compiling and including in my graalvm compile, I get a JNI error:

ericdallo17:02:52

com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine            PosixJavaThreads.java:  192
                              com.oracle.svm.core.thread.JavaThreads.threadStartRoutine                 JavaThreads.java:  519
                                                                   java.lang.Thread.run                      Thread.java:  834
                                     java.util.concurrent.ThreadPoolExecutor$Worker.run          ThreadPoolExecutor.java:  628
                                      java.util.concurrent.ThreadPoolExecutor.runWorker          ThreadPoolExecutor.java: 1128
                                                    java.util.concurrent.FutureTask.run                  FutureTask.java:  264
                                    java.util.concurrent.Executors$RunnableAdapter.call                   Executors.java:  515
                          org.eclipse.lsp4j.jsonrpc.json.ConcurrentMessageProcessor.run  ConcurrentMessageProcessor.java:  113
                            org.eclipse.lsp4j.jsonrpc.json.StreamMessageProducer.listen       StreamMessageProducer.java:   94
                     org.eclipse.lsp4j.jsonrpc.json.StreamMessageProducer.handleMessage       StreamMessageProducer.java:  194
                                       org.eclipse.lsp4j.jsonrpc.RemoteEndpoint.consume              RemoteEndpoint.java:  190
                                 org.eclipse.lsp4j.jsonrpc.RemoteEndpoint.handleRequest              RemoteEndpoint.java:  261
                             org.eclipse.lsp4j.jsonrpc.services.GenericEndpoint.request             GenericEndpoint.java:  120
                       org.eclipse.lsp4j.jsonrpc.services.GenericEndpoint.lambda$null$0             GenericEndpoint.java:   65
                                                                                    ...                                       
clojure_lsp.main.proxy$clojure_lsp.ClojureExtensions$LanguageServer$c8d5825a.initialize                                       
                                                                 clojure-lsp.main/fn/fn                         main.clj:  288
                                                                                    ...                                       
                                                        clojure-lsp.handlers/initialize                     handlers.clj:   39
                                                 clojure-lsp.crawler/initialize-project                      crawler.clj:  197
                                                    clojure-lsp.crawler/analyze-project                      crawler.clj:  192
                                                  clojure-lsp.crawler/analyze-classpath                      crawler.clj:  170
                                                               clojure-lsp.db/read-deps                           db.clj:   39
                                                                 clojure-lsp.db/make-db                           db.clj:   25
                                                               datalevin.lmdb/open-lmdb                         lmdb.clj:  841
                                                               datalevin.lmdb/open-lmdb                         lmdb.clj:  851
                                                          org.lmdbjava.Env$Builder.open                         Env.java:  512
                                                          org.lmdbjava.Env$Builder.open                         Env.java:  486
                                     org.lmdbjava.Library$Lmdb$jnr$ffi$0.mdb_env_create                                       
                                                        com.kenai.jffi.Invoker.invokeN1                     Invoker.java:  997
                                                      com.kenai.jffi.Foreign.invokeN1O1                      Foreign.java     
                         com.oracle.svm.jni.JNIGeneratedMethodSupport.nativeCallAddress   JNIGeneratedMethodSupport.java:   57
                         com.oracle.svm.jni.access.JNINativeLinkage.getOrFindEntryPoint            JNINativeLinkage.java:  153
java.lang.UnsatisfiedLinkError: com.kenai.jffi.Foreign.invokeN1O1(JJJLjava/lang/Object;III)J [symbol: Java_com_kenai_jffi_Foreign_invokeN1O1 or Java_com_kenai_jffi_Foreign_invokeN1O1__JJJLjava_lang_Object_2III]

Huahai04:02:12

That error indicate thats you are not compiling the native version of Datalevin, you are compiling the Java version that use JNR

Huahai04:02:57

Native Datalevin does not use JNI at all, it uses the GraalVM SDK C API

Huahai04:02:14

The decision on which version to use is made in Datalevin using a multimethod, which check whether or not it is running inside a GraalVM image. Even at the image build time, the code should be in GraalVM image so the native binding should be chosen, but somehow, yours is running in regular Java mode

Huahai05:02:52

Maybe point me to your repo so I can take a look at how you are compiling native image?

Huahai05:02:32

ok, i found your datalevin branch

Huahai05:02:32

you are depending on datalevin 0.3.17, which is the release before the work on native compilation, of course it won’t work

Huahai05:02:55

I am going to release v0.4.0 today, as I just finished the native command line shell for datalevin

Huahai05:02:21

v0.4.0 should work

Huahai06:02:13

also, you are right, i should bundle the dtlv.a in the library

Huahai06:02:34

i am new to this business of releasing native code, so allow me sometime to get up to speed

Huahai06:02:24

It compiles to native just fine.

ericdallo12:02:19

Oh that was fast haha! I'm new too don't worry ;)

ericdallo12:02:25

Thank you for the quick response

ericdallo12:02:56

I'll test it with the 0.4.0 as soon you release it, excited for that!

ericdallo13:02:41

Thanks for the help @U0A74MRCJ, with your PR merged I'm getting a

java.lang.IllegalArgumentException: No method in multimethod 'open-kv' for dispatch value: :graal

ericdallo13:02:36

Does the key-value store lmdb work with graal?

Huahai16:02:53

you mean when you run the binary?

Huahai16:02:38

it looks to me that the graal version of the LMDB binding is not compiled in, did you include datalevin’s reflect-config.json?

Huahai16:02:07

ldd ./clojure-lsp linux-vdso.so.1 (0x00007ffcc5ff9000) libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f57728c4000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f57728a1000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f577289b000) libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f577287f000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f577268d000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f577253e000) /lib64/ld-linux-x86-64.so.2 (0x00007f57790e9000) libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f5772521000)

Huahai16:02:40

so liblmdb.so.0 is missing

ericdallo16:02:41

Yes, I merged the reflect-config json with datalevin one

Huahai16:02:44

ok, had this warning, so the release jar doesn’t include graal version

Huahai16:02:45

WARNING: Could not resolve datalevin.binding.graal.LMDB for reflection configuration. Reason: java.lang.ClassNotFoundException: datalevin.binding.graal.LMDB.

ericdallo16:02:09

BTW i'm having the same issue with a JVM version, not graal one but the defmethod warn about the :java one

Huahai16:02:11

let me figure out how to include that jar

Huahai16:02:36

interesting

ericdallo16:02:55

> WARNING: Could not resolve datalevin.binding.graal.LMDB for reflection configuration. Reason: java.lang.ClassNotFoundException: datalevin.binding.graal.LMDB. I had this one too

ericdallo16:02:28

But right now the issue seems to be the defmethod not recognizing the defmulti implementations

Huahai16:02:30

i think i know what’s wrong, hold on, let me fix it

👍 3
Huahai17:02:23

please test the updated PR, i tested locally, it seems to include lmdb.so now

ericdallo17:02:43

Alright, testing it

ericdallo18:02:32

Thanks for the help @U0A74MRCJ,the issue is fixed and it seems to work properly 🙂 I just needed to add the require [datalevin.binding.graal] as well. But probably clojure-lsp will follow a simpler way persisting the db with spit as we just need to persist things and read on time when clojure-lsp initializes. Anyway, thanks for helping on this POC, datalevin seems really cool!

🥳 3
Huahai19:02:14

Thanks for trying it out. You gave me some ideas on how to improve the packaging

ericdallo19:02:22

Glad to hear 😄

borkdude17:02:02

^ @denik might also know, since he's also working on something native datalevin related

😉 3