Fork me on GitHub
#data-science
<
2018-01-13
>
Daniel Slutsky22:01:15

@sb @smith.adriane I'm currently testing the option of calling python through https://github.com/ninia/jep . After some struggle with the setup, it seems nice. I'll play with some clojure abstractions around it, and share if it seems practical.

phronmophobic19:01:08

That sounds great. I’d be interesting to hear your thoughts on the jni/java bindings vs using jna. The biggest hurdle I’ve found for java/jni wrapper libraries is that making a jni binding usually requires a bunch of boilerplate code that is annoying to write. One consequence is that jni/java libraries are usually tied to specific c library versions. They won’t necessarily support the version you want or won’t have the latest version available. Using jna instead of java/jni let’s me use which ever c library version I want. I’ve usually found it easier to write clojure abstractions on top of the c interface than on top of the jni/java interface. A major reason is that there is usually better documentation of the c api than there is of the java/jni api. There will often be examples of what you’re trying to accomplish in c and then you’re sometimes on your own for transalating that into the java/jni version. I’ve also found that the c apis are generally more data oriented than the java apis. It’s not all sunshine and roses though. Java/jni will generally have less overhead than clojure-jna and if you use jna, it’s your responsibility to not crash the jvm instead of the java/jni wrapper library author. It’d be great to hear your experience as I don’t always have time to investigate both options.

sb22:01:40

@U066L8B18 I think so, would be great one example. I tried import nania/jep to clojure and failed (build + load from local repo). I get “Unable to resolve symbol jep”. So, If you have “initial kick” to see the things, would be useful for everybody here (if that isn’t problem for you). /jna version works well, I tried (Adrian helped me).

Daniel Slutsky23:01:18

Thanks for the comments. @smith.adriane, what you wrote about jna's simpler experience and about the c APIs offering a more a data oriented experience make sense to me (tough my experience here is very little). It seems like a good direction to go in the longrun. Using jep is just a lazy shortcut, for now. @sb, I hope to write a complete example in a couple of days. In the meantime, some remarks: 1. I had to add the jep jar as a dependency (if you use leiningen, you can use the "lein-localrepo" plugin to add it as a local repository). 2. I had to set some environment variables. In my case: LD_LIBRARY_PATH=/usr/lib/python3.6/site-packages/jep/ LD_PRELOAD=/usr/lib/libpython3.6m.so but of course, the precise locations may vary. 3. To be able to import pandas, for some reason I had to set the environment variable LC_ALL to the override the localization settings. 4. The jep interpreter can be used only from the Thread where it was created. So, for stable experience under the Clojure REPL, I had to write a macro that runs a code block under some newly created Jep object.

phronmophobic23:01:51

to be fair, the jna setup has a few caveats as well. I’ve also crashed the jvm more times than I’d like to admit, but I’ve since gotten better about avoiding some of the more common pitfalls

sb08:01:49

@U066L8B18 thanks for the detailed answer & help!! (I hope, later I can help/ improve these solutions.) 👍👍

sb13:01:03

For others: plus I need to add this to project.clj :jvm-opts ["-Djava.library.path={LD_LIBRARY_PATH}"]