Fork me on GitHub
#babashka
<
2020-03-23
>
penryu03:03:23

Hehe. I run out of memory trying to build a native-image of babashka using graalvm20-java8, but not graalvm19-java8. I underestimated how much RAM it takes.

penryu03:03:59

(It's only an 8GB VPS)

pez07:03:56

I got the same on a 32GB Macbook Pro.

...
  Resources have been added by ServiceLoaderFeature. Automatic registration can be disabled with -H:-UseServiceLoaderFeature
[bb:95710]   (typeflow): 256,136.04 ms,  2.67 GB
[bb:95710]    (objects): 316,187.40 ms,  2.67 GB
[bb:95710]   (features):   7,216.23 ms,  2.67 GB
[bb:95710]     analysis: 583,729.52 ms,  2.67 GB
[bb:95710]     universe:  58,864.95 ms,  2.67 GB
Fatal error: java.lang.OutOfMemoryError
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at java.util.concurrent.ForkJoinTask.getThrowableException(ForkJoinTask.java:598)
        at java.util.concurrent.ForkJoinTask.get(ForkJoinTask.java:1005)
        at com.oracle.svm.hosted.NativeImageGenerator.run(NativeImageGenerator.java:462)
        at com.oracle.svm.hosted.NativeImageGeneratorRunner.buildImage(NativeImageGeneratorRunner.java:357)
        at com.oracle.svm.hosted.NativeImageGeneratorRunner.build(NativeImageGeneratorRunner.java:501)
        at com.oracle.svm.hosted.NativeImageGeneratorRunner.main(NativeImageGeneratorRunner.java:115)
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
Error: Image build request failed with exit status 1
com.oracle.svm.driver.NativeImage$NativeImageError: Image build request failed with exit status 1
        at com.oracle.svm.driver.NativeImage.showError(NativeImage.java:1527)
        at com.oracle.svm.driver.NativeImage.build(NativeImage.java:1289)
        at com.oracle.svm.driver.NativeImage.performBuild(NativeImage.java:1250)
        at com.oracle.svm.driver.NativeImage.main(NativeImage.java:1209)

pez07:03:23

But now I will try with 19! 😃 Thanks for sharing that!

pez07:03:51

Works with 19, yes.

borkdude07:03:20

Hmm really, that’s pretty annoying. But master does build with 20 right?

pez08:03:03

Haven't tried. Will do that now.

pez08:03:37

Need I clean something out in order to test it properly?

sogaiu08:03:22

not quite sure what you're asking, but sometimes i find lein clean can help between compilations

sogaiu08:03:53

@U0ETXRFEW i had success building with 20 -- though it helped here to do: export BABASHKA_XMX="-J-Xmx25g" before running the build

sogaiu08:03:27

the build took around 90 seconds here

sogaiu08:03:41

well, at least the native-image part

pez08:03:21

I could build master with graal 20.

pez08:03:47

No, I actually tested to build babasska master with graal 20 as my java dist, not my GRAALVM_HOME... Rebuilding now.

penryu08:03:10

Ah, I should have tried the -Xmx... but dinner was calling.

sogaiu08:03:03

fwiw, there's an env var that the compile script references to tweak that

pez08:03:03

Sounds like using Xmx25g could make the build take a while on your 8gb VPS? 😃

sogaiu08:03:29

i'm going lower -- 20g worked -- trying 17 now

sogaiu08:03:42

ofc that won't be enough i guess 🙂

sogaiu08:03:22

the default value here got set at 14 and i think that might have failed

pez08:03:38

I think it uses 14g default on my machine.

sogaiu08:03:20

in one run, it took a really long time (and i don't recall if it worked in the end) and it was 14 then

borkdude08:03:24

graalvm always defaults to 14g. please test with the newest dev build: https://github.com/graalvm/graalvm-ce-dev-builds/releases if that doesn't work, we'll have to make an issue

borkdude08:03:45

I'd like to compile with 4gb as we've always done in circleci

sogaiu08:03:52

i've tested with both 20 and the latest dev: GraalVM CE 20.1.0-dev-20200321_0524

borkdude08:03:19

(although by now the memory may have been bumped to 8gb by Marc)

pez08:03:34

It's taking forever here. Think it will end the same way as it did with the nrepl branch.

sogaiu08:03:01

and it was 96 secs

pez08:03:34

but of course, I forgot to update my submodules when switching to master. I am not constructed for these kinds of exercises! 😃

sogaiu08:03:55

that happens to me too 🙂

pez08:03:32

But maybe that is for the good and can help isolate the problem. Or at least rule out one source as the problem.

sogaiu08:03:36

interesting...now master builds with 20.0.0 using 14

pez08:03:30

Out of memory buidling master. WIth sci from the nrepl branch. Updating that now and will test again. Then I need to go work for food.

sogaiu08:03:12

nrepl-server also built with 20.0.0 using -Xmx14g

sogaiu09:03:21

when i scroll back, i see that the really long compilation that ultimately failed for me was nrepl-server with graal 20.0.0 java 8. there are two -Xmx options in the invocation. the later one is -Xmx3g which i presume wins.

pez09:03:26

Which java are you using? Graal also?

sogaiu09:03:02

i'm using the script/compile script which sets JAVA_HOME to GRAALVM_HOME

sogaiu09:03:14

so, in short, i believe so

pez09:03:54

Yeah, I saw that now too.

sogaiu09:03:12

i'm trying -Xmx8g and it is looking promising

sogaiu09:03:35

yep, it worked

sogaiu09:03:31

so iiuc the default value that ends up being effective is currently 3g

sogaiu09:03:04

the script sets it to this if the env var BABASHKA_XMX is unset(?)

sogaiu09:03:58

-Xmx7g works too, so it seems possible that this might work on the VPS machine mentioned earlier

pez09:03:12

I'm trying with 4g now.

pez09:03:29

Yes, that worked. On master.

sogaiu09:03:38

:thumbsup:

sogaiu09:03:27

it does seem to take somewhat longer with less memory, but at least it appears possible

sogaiu09:03:41

ok, -Xmx4g worked here too.

pez09:03:35

And I could compile nrepl-server with 4g as well.

pez09:03:59

99 secs.

sogaiu09:03:32

i should have a result for you in a few moments for the same settings 🙂

pez09:03:11

You have a faster machine than I have! Not fair.

sogaiu09:03:31

i'm not sure that's true -- what else might be happening on our machines? 😉

sogaiu09:03:37

i do have more memory though

borkdude08:03:49

Btw if you want to test the bb nREPL without building a native image first, you can also run it via the JVM, e.g.: lein bb

borkdude08:03:28

But it’s good to know this thing is problematic with Graal 20, I’ll take a look

borkdude09:03:54

Bumping memory to 4gb didn't work on circleci: https://circleci.com/gh/borkdude/babashka/5262 😕

borkdude09:03:12

However this worked on my machine:

BABASHKA_XMX="-J-Xmx3500m" script/compile

borkdude10:03:20

@marc-omorain I though you bumped memory on my account to 6 or 7gb, at least this is what I saw for some time?

borkdude10:03:00

hmm, I think graalvm 20 may have another issue btw. the binary is around 8mb bigger on v20. -rwxr-xr-x@ 1 borkdude staff 49647716 Mar 23 11:06 bb

telekid23:03:44

Is it possible to get babashka working with Cider?

telekid23:03:56

Oh, just saw the nrepl post above! 🔥

🔥 4