Fork me on GitHub
#data-science
<
2020-05-24
>
whilo03:05:55

Hey. I have done some simple warmup tests to talk to Julia through libpython-clj (basically using pyjulia), but could not get it to work. I am interested in having a libpython-clj style interface to Julia. Has anybody worked in this direction yet?

smith.adriane04:05:00

libjulia has support for embedding, so you could access julia using jna (the same way libpython-clj accesses python), https://docs.julialang.org/en/v1/manual/embedding/index.html

smith.adriane04:05:33

it would probably be helpful to be comfortable in C if you were going to go down this path

whilo09:05:56

Yes, I took a look at that and I am aware of the C bits. I think the tricky part is to map memory layouts between Clojure and the type signatures of Julia. @ I have heard that you have already thought about this?

smith.adriane16:05:05

that’s what jna helps you do

whilo20:05:25

yes. are you working with it?

smith.adriane20:05:28

I’ve been using JNA to integrate with c libraries, but not julia

smith.adriane20:05:44

I’ve been using jna via https://github.com/Chouser/clojure-jna/, but it looks like libpython-clj uses their own clojure wrapper over jna, https://github.com/techascent/tech.jna

chris44113:05:07

@ - I am down to work with someone to take a crack at it. I also want to see julia access from Clojure but i havent put any real time in.

chris44113:05:02

I think it is doable, I think it is a moderate amount of work, and I think Julia is really visionary with a lot to offer.

chris44113:05:24

Another tack you could take would be Clojurists together funding. But my assumption is that it will take some pretty in-depth C knowledge as did libpython-clj.

whilo19:05:47

Right, that makes sense. I think it would be cool if it would be integrated with pyjulia as well, so you can share the same objects between Python and Julia, effectively being able to mix and match libraries from Clojure. Do you think this would be feasible?

chris44100:05:07

The best way to do that is to use pyjulia. Maybe file an issue with and we can look at it. Bespoke bindings would be best in the siruation where you do not want python involved.

chris44115:05:49

I just tried pyjulia and it isn't stable when used via python-clj. That should be an issue like I said but tracking it down may be really tough. This is why bindings to julia that don't involve python would be wise if we really want have julia in the ecosystem.

chris44116:05:56

Julia also isn't stable itself when loaded via jna. It fails on the initialize call. Julia support for the JVM appears to be a research project at this point. =

chris44119:05:04

Got past the intialize code if I use the binary distribution from http://julia.org and not the system installation. Still crashes on shutdown but I can run strings. It isn't clear that you can, for instance, register a foreign function although the pathway for that may be using the julia system to 'find' the function...

whilo04:06:37

@ Hmm, interesting. Is your code for this experiment somewhere?

whilo04:06:58

That sounds already useful enough to play around and try out deeper integrations.

chris44123:06:24

https://github.com/cnuernber/julia-clj. It will always crash given time. Unless we get help from the julia team this can't go anywhere. The use stack manipulations that are incompatible with the JVM. I filed an issue on this that didn't get a response; the next step is to compile julia with some custom flags and see if that mitigates or eliminates the issue.

smith.adriane23:06:16

are you hanging onto return values referenced from the clj side?

smith.adriane23:06:48

otherwise, the jvm will think noone is using those values and garbage collect them

smith.adriane23:06:35

oh didn’t realize you had also worked on libpython-clj and probably run into a million of those issues. nvmd. homerdisappear

chris44114:06:04

Yep, and this is simple calling `jl_init__threading` which doesn't return anything and then calling System/gc a few times_._ Once `jl_init__threading` is called *for sure* the jvm will crash at some undefined time in the future. My current guess is this is because Julia does something really fancy with stack frames and so does the jvm. This may have to come in via graal native and currently our entire ml stack would need to be changed somewhat to support graalnative so I am not in a rush to go there.

smith.adriane17:06:54

the other type of crash I’ve seen when creating bindings for exotic libraries is issues with both environments adding signal handlers. see https://bitbucket.org/chromiumembedded/java-cef/issues/355/crash-on-cef-init-due-to-signal-handlers for an example. julia does have handlers for SIGBUS, SIGSEGV, but I’m not sure if this is related to the crash you’re seeing or is a problem in this case.

smith.adriane17:06:16

I found this java library that provides embedded julia bindings https://github.com/rssdev10/julia4j . if I have some time, it would be interesting see if they run into the same issues, and if not, maybe that will provide a clue as to workaround. however, just browsing through the code seems like they aren’t doing anything special.