Fork me on GitHub
#graalvm
<
2020-02-21
>
Nico13:02:06

I'm trying to build my project (https://git.itwont.work/samples) with graalvm and it just runs for a long time (like, half an hour) then runs out of memory

borkdude13:02:10

@nihilazo Instead of starting with a big project, it's often better to start with smaller parts and then add things bit by bit to see what's causing this

borkdude13:02:43

for reference, all my own graal projects compile within 2 minutes on CircleCI. if it takes longer, I find that suspicious

Nico14:02:48

hmm, I'll try compiling a hello world project and see if that works

Nico14:02:59

should I be using the jdk8 or jdk11 version of graalvm for clojure?

borkdude14:02:29

jvm11 works, but I've ran into some issues with it (unlikely that you will encounter them with hello world)

borkdude14:02:38

I'm still using jdk8

borkdude14:02:46

@nihilazo Are you trying to graalvm compile datahike? That's slightly more advanced than hello world I would say?

borkdude14:02:56

last time I tried that, it didn't work for me

borkdude14:02:44

oh sorry, I misread. yeah, hello world should work

Nico14:02:14

yeah, I was trying to graalvm compile datahike

Nico14:02:34

which is probably a bad idea

Nico14:02:39

I'll see if the hello world works

borkdude14:02:02

I tried that before, but that didn't work for me. that's why I ended up using transit instead of datahike in clj-kondo (I wasn't even able to make a performance benchmark because I never got that far)

borkdude14:02:53

if you want to have a fast CLI solution with a database you could consider babashka + the sqlite-cli: https://github.com/borkdude/babashka/blob/master/examples/sqlite.clj

borkdude14:02:16

I wasn't able to compile sqlite itself with graalvm either

Nico14:02:24

I'm not sure I even really need a database? The queries are super simple (literally just checking if something has a tag, and checking if it doesn't) but there is potentially a reasonable amount of things being checked

Nico14:02:40

actually, that said, I wonder if datahike is actually worse performance-wise than doing something like that manually

borkdude14:02:10

@nihilazo What I did for clj-kondo is split the data up into partitions so I can choose to only read from disk what I need. you can tailor this to your problem to make it performant

borkdude14:02:40

transit is compact and fast to read/write

borkdude14:02:48

and works with graalvm

Nico14:02:13

ah, for this in every case I want to search all the data, and it's stored in a text format on disk already, I might try doing it without any DB at all and see how it goes

Nico14:02:29

might even be able to do everything I was doing with datahike using a couple of filter s

Nico14:02:47

but using a database seemed like the more "proper" way

borkdude14:02:02

@nihilazo JUXT published a post today where they plowed through some text files with babashka: https://juxt.pro/blog/posts/babashka.html

borkdude14:02:45

if you can make the search faster by pre-organizing it into an indexed way first, that can help

Nico15:02:34

compiling the new datahike-less version results in some unsupported things, but that's progress!

Nico15:02:39

also, the hello world works perfectly

Nico15:02:31

I'm getting the errors from clojure.spec.gen.alpha, is it workaround time?

borkdude15:02:04

@nihilazo Are you loading clojure.spec yourself from anywhere?

borkdude15:02:35

is it the CLJ-1472 error?

Nico15:02:11

maybe, I'm not pulling in spec myself but I am pulling in tools.cli

Nico15:02:21

actually, I don't need tools.cli, I can just remove that

borkdude15:02:26

tools.cli should work without this error, I've used it in a couple of projects

Nico15:02:48

the only thing I'm using outside of my own code is http://clojure.java.io and clojure.string

borkdude15:02:00

your own code could be triggering it. are you using things like resolve?

borkdude15:02:32

also, it could be that you have to set the macro spec checking off

Nico15:02:05

ah, maybe I need the -J-Dclojure.spec.skip-macros=true line

borkdude15:02:49

could very well be

Nico15:02:04

hmm, with that I'm getting errors due to spec as well, despite not using spec at all

Nico15:02:28

there's probably bad practice all over this thing tbh, I'm fairly new to clojure

borkdude15:02:34

I don't mean it to be offending, the reason it's "bad" is that you should not trust other tools to load http://clojure.java.io for you. If you use a linter like clj-kondo, it will warn you about such practices.

Nico15:02:44

no offence taken, just warning you that most of it is probably bad

Nico15:02:53

and thanks for trying to help me figure this out

Nico16:02:14

might be worth building the patched spec anyway?

borkdude16:02:41

I'm not sure. your project doesn't seem to be doing anything weird. does it work with clojure 1.9?

Nico16:02:09

seems like it

Nico16:02:17

just tested with 1.9 and it works fine

Nico16:02:09

when I try a graal build with 1.9, I get a different error, Error: No instances of .FilePermission are allowed in the image heap as this class should be initialized at image runtime.

borkdude16:02:34

oh are you by any chance using graal v20?

borkdude16:02:57

try 19.3.1 and report this error on the github graal project. @sfyire also got it

Nico16:02:48

I'm using whatever the graalvm in AUR is, I'll try 19.3.1

borkdude16:02:35

yeah, 20 just got out and has a couple of issues. But reporting those issues definitely helps and they will be fixed.

borkdude16:02:52

19.3.1 is the LTS release

Nico16:02:28

gotta love arch linux for having the latest everything, even when the latest doesn't work

Nico16:02:42

can graalvm not access environment variables? It builds perfectly with the older graalvm and older clojure but I use System/getenv and that doesn't seem to be working, seems to fail

Nico16:02:54

won't read in the variable

borkdude17:02:27

@nihilazo It depends if you're doing that at compile time or run time. When you execute that call at compile time, the call is made too early

borkdude17:02:38

even bb can do this:

$ bb '(System/getenv "PWD")'
"/Users/borkdude/dre/DocSearch/app"
so surely your custom graalvm clojure app can.

borkdude17:02:39

yeah, so the problem here is that you're doing it top-level: https://git.itwont.work/samples/tree/src/samples/core.clj?h=no-datahike#n15

borkdude17:02:58

that won't work, since that captures the value of that environment variable when you compile it, not when you run it

Nico17:02:38

that doesn't seem like the case when I run it though

Nico17:02:48

when I run it from the jar, it reacts to whatever the variable is at that point

Nico17:02:03

I can change it between runs and it changes

borkdude17:02:14

just make it into a variable inside -main

borkdude17:02:21

I wonder, is your jar AOT-ed?

Nico17:02:55

everything is AOT-ed

borkdude17:02:10

then I would even expect that not to work on the JVM

borkdude17:02:18

anyway, this is a problem I've had before and the solution is to make reading the environment variable a call at runtime, i.e. in the execution of -main, not top-level/before it.

Nico17:02:36

ok, thanks

Nico17:02:35

everything is now working on graalVM! Thank you for helping me out and answering my stupid questions

borkdude17:02:45

they're not stupid and you're welcome!

borkdude17:02:32

if you can make a tiny repro of the: > No instances of http://java.io.FilePermission are allowed in the image heap as this class should be initialized at image runtime that would be helpful for fixing it in v20

borkdude17:02:08

usually posting your AOT-ed uberjar + native-image build instructions is sufficient for the graalvm devs to reproduce it

borkdude17:02:18

so they won't have to bother with compiling clojure

Nico17:02:24

I'll try that, not sure exactly what the cause was though

borkdude17:02:44

wasn't it just v20 jdk8 versus v19 jdk8?

Nico17:02:08

it was v20 jdk11 vs v19 jdk11

Nico17:02:25

it worked in v19 and not v20

Nico17:02:37

but I'm not sure what the thing in the code was that caused it so I can do a tiny repro

borkdude17:02:02

well, that's up for them to find out I think. just saying: this works with graalvm 19.1.3 jdk 11 on macOS/linux/Windows but not on v20 .... is sufficient, provided you give them the uberjar + build arguments of native-image

Nico17:02:26

that makes things easier