Fork me on GitHub
#graalvm
<
2021-03-20
>
chrisn12:03:11

@huahaiy - that isn't the case. I have three 'ffi's implemented. libpython-clj, for example already works on with JNA direct mapping and JDK-16 and it just depends on what dtype-next's ffi system finds on the system. You write one https://github.com/clj-python/libpython-clj/blob/master/src/libpython_clj2/python/ffi.clj and it works across https://github.com/cnuernber/dtype-next/blob/master/src/tech/v3/datatype/ffi/jna.clj, https://github.com/cnuernber/dtype-next/blob/master/src/tech/v3/datatype/ffi/mmodel.clj, and https://github.com/cnuernber/dtype-next/blob/master/src/tech/v3/datatype/ffi/graalvm.clj. So you, as the implementor, do not have to write multiple FFI layers yourself. You write one, describe it with data, and I translate that data into the subsystem-appropriate classes and implementations.

borkdude12:03:55

That's great, no more writing JNI stuff I guess

chrisn12:03:57

I have a link just for you - https://graalvm.slack.com/archives/CN9KSFB40/p1616200527026800?thread_ts=1616189277.022900&amp;cid=CN9KSFB40. I made a bit of noise on the graalvm native-image channel and got a fantastic response.

borkdude12:03:06

@chris441 Does this also work if you want to expose Clojure stuff as a native lib? like I did here? https://github.com/borkdude/sci/blob/master/doc/libsci.md

chrisn13:03:07

I can try to expose avclj as a native lib and we can find out. My overall goal was to get some nontrivial portion of libpython-clj working both as an executable and as a native lib.

chrisn13:03:31

So people could write python extensions in Clojure but be able to talk back to the python system that loaded them.

borkdude13:03:51

My question is just out of curiosity, not that I have any plans

chrisn13:03:50

As far as JNI is concerned, the best way to do that is javacpp.

chrisn13:03:57

Nothing else is even close IMO.

chrisn13:03:14

I used that for cortex and aside from some teething issues it worked great.

chrisn13:03:44

But I don't like JNI. I like to find libraries dynamically so I can account for version changes in code without separate releases bound to specific binary versions of things.

borkdude13:03:58

I guess I could ask the question like: avclj compiles to an executable. Is it possible to compile it a shared library again (which kind of defeats the purpose of that app, but just as an example), so you can call that shared lib from python as a native lib?

chrisn13:03:32

It should be. Nothing I did precludes that.

chrisn13:03:45

Well, nothing that I know that I did precludes that.

borkdude13:03:05

cool stuff.

chrisn13:03:16

🙂. Appreciated. Agreed 🙂.

chrisn13:03:20

I do have a question. Sometimes I have an extra require or something that really explodes both the build time and the executable size. It usually takes me quite a long time to track down the pprint statement or whatever that causes it. Are there any tricks to tracking down these issues faster?

chrisn13:03:54

For instance, RT/var will cause the explosion as will pprint.

borkdude13:03:53

@chris441 I usually just do this by divide and conquer. Start as small as possible and then start adding things. Things known to explode: Look up vars at runtime. E.g. require, resolve, requiring-resolve. I have made a patch in babashka to avoid the explosion of pprint. Also I made this variation of dynaload that won't blow up the binary: https://github.com/borkdude/dynaload

borkdude13:03:26

Also I have logged a JIRA issue about pprint

chrisn13:03:33

Beautiful, dynaload is a great insurance policy.

borkdude13:03:21

Possibly you can use https://github.com/borkdude/grasp to find these likely causes

borkdude13:03:46

Grasp for non-top-level require, find-var, requiring-resolve, resolve

chrisn13:03:41

Yep, makes sense. Interestingly enough, RT/var in the static initialization code section of a class doesn't cause it. But RT/var in any instance-level code definitely well.

borkdude13:03:46

I think GraalVM static analysis accounts for static initialization, which is influenced by --initialize-at-runtime and --initialize-at-build-time. Is (var x) triggering something? This is a special form which is compiled away to the var object I think?

borkdude13:03:36

If GraalVM can't decide that an entire namespace isn't needed, it may leave all vars/functions around and everything related to it, which may blow up the binary, is my guess

borkdude13:03:46

Sometimes -J-Dclojure.compiler.direct-linking=true also helps

borkdude13:03:21

Maybe I should add a graalvm native-image linter to clj-kondo :thinking_face:

chrisn13:03:39

well, and with multimethods and protocol implementations graal often can't figure out if a namespace is OK to elide and I can't blame them.

chrisn14:03:35

It looks like we can include the truffle nfi in a native image. That would give you a cython-equivalent system I believe.