This probably is because you have a different zlib than graalvm expects. Have you compiled babashka yourself?

Yes. Looks like the https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/interpreters/clojure/babashka.nix derivation is building it with the following flags:

"-H:+ReportExceptionStackTraces"
    "--no-fallback"
    "--native-image-info"

NixOS is always the most fun. 😅

I've seen this error before when the environment uses zlib other than ZLIB_VERSION="1.2.11"

Hrm, lemme see what zlib bb is using.

cc @UFDRD93RR

Yeah, I have 1.2.12.

cc @U4NGX0FHN

I think it's better if the derivation compiled a static binary with 1.2.11 in the environment

You can download the static binary yourself and that should work (although that would be against the nix spirit I guess)

https://github.com/oracle/graal/issues/4479#issuecomment-1122505727

I can whip up an overlay that does this. It's odd that the person who updated Babashka from 0.8.2 to 0.8.156 didn't run into this issue, but as that commit is only 7 days old, maybe it just hasn't been used enough in the wild to uncover this issue.

you only hit this error when you do zip stuff

I guess all Babashka pods need to be unzipped though, right?

Or is this just something clj-kondo is doing?

no, it's bb

but pods aren't touched when building + testing in nix I guess

Exactly.

Is there an example anywhere of which flags to pass to compile a static binary?

Static will not work in GraalVM from nixpkgs as-is

yes, check out the linux-static job on circleci in the bb repo. but note that this only works for linux amd64

ah I see. then forcing zlib 1.2.11 should be the solution for now

And it wouldn't solve the issue anyway since you would still use zlib 1.2.12

until 1.2.13 or so comes out

Yeah, 1.2.13 would be best. I could also apply the Arch patch when building zlib, I suppose.

I can probably backport the same patch Arch is using if this does fix the issue though, at least until 1.2.13 is released

In fact, that might be the easiest thing to do, though it would probably rebuild everything on my system. 😅

No, it doesn't

Haha, great minds think alike.

You should only backport this patch to GraalVM derivation

Not globally

Right, that's smart!

I need to repro this issue first though. Do the babashka test suite includes a test for this case?

yes, the CI will definitely fail

So I need to overlay a zlib derivation that applies the Arch patch, then overlay a graal derivation that uses that zlib.

no, I think only having the patched zlib in your environment will be enough, since it's dynamically linked

@UFDRD93RR Are you running NixOS? I can give you a snippet that will definitely blow up.

Perhaps using fs/unzip will also trigger this

Yes

But I still want to check this automatically

If possible, of course

Why does nix want to create 32 users on my mac?

build users:
    Username:	UID
     _nixbld1:	301
     _nixbld2:	302
     _nixbld3:	303
     _nixbld4:	304
     _nixbld5:	305
     _nixbld6:	306
     _nixbld7:	307
     _nixbld8:	308
     _nixbld9:	309
     _nixbld10:	310

For building

Nix build things in isolation

The approach for this is to create separate users, so each package is build in isolation from one another

and why 32 and not 16 or 128?

You could have only 1 user, but then you would have to build one package at a time

Nix also has a single user mode, but that's a PITA to get working on a Mac, AFAIK.

I guess it assumes that most CPUs have max 16 hyperthreaded cores?

I think you can increase or decrease the number of users, but 32 looks like a good default anyway

Hence running nix build with more than -j 32 is something no reasonable person would ever do. 😉

This is a different thing

You can have one derivation using more than 32 cores

I'm deep in a rabbit hole that started when I just wanted to write a short blog entry. 😅

But you can't have more than 32 different derivations building simultaneously

> I'm deep in a rabbit hole This has been the case every time I tried to use nix ;)

Sure, you can do. I'm talking about the flag that tells nix build how many parallel builds to attempt.

I'm going to install nix once again though

Or maybe that flag is just passed straight to compilers?

But you don't need to pass any flags btw

Nix automatically gets the number of CPUs and pass to the derivation

The derivation itself needs to declare that it is multicores safe though

enableParallelBuilding = true;

ok, installed now: https://nix.dev/tutorials/install-nix

And why a derivation wouldn't be multicores safe? Well, reproducibility

Some derivations doesn't reproduce a consistent result in a multicores environment

And yeah, this is fun

But anyway, going back to the issue

Please send me a repro

But I will probably try to get the test suite of babashka working in nixpkgs

ok, on macos I also see zlib 1.2.12:

$ otool -L $(which bb)
/nix/store/msgq0r8v9kma3fqxmsr0mdylrxgv1y8x-babashka-0.8.156/bin/bb:
	/System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices (compatibility version 1.0.0, current version 1122.11.0)
	/nix/store/kzba6lbw9sgzmvip6jjf35nz3nnmri8y-libcxx-11.1.0/lib/libc++.1.0.dylib (compatibility version 1.0.0, current version 1.0.0)
	/nix/store/7zfc5xjjigzh5i33rv7ph1hnma5x4rzz-zlib-1.2.12/lib/libz.dylib (compatibility version 1.0.0, current version 1.2.12)
	/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation (compatibility version 300.0.0, current version 1770.255.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1292.60.1)
	/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1770.255.0)

Hmm, on macOS I can't reproduce the issue though

This should explode:

$(nix-build ~/Downloads/bb.nix)/bin/bb <something that unzips stuff>

[nix-shell:~]$ rm -rf ~/.babashka/pods/repository

[nix-shell:~]$ bb -e "(babashka.pods/load-pod 'clj-kondo/clj-kondo \"2022.02.09\")"
Downloading pod clj-kondo/clj-kondo (2022.02.09)
Successfully installed pod clj-kondo/clj-kondo (2022.02.09)
#:pod{:id "pod.borkdude.clj-kondo"}

I wonder if it's just a Linux thing?

@U04V15CAJ Which bb version is that?

$ bb --version
babashka v0.8.156

And which JDK?

The bb version is irrelevant

Or the JDK

Mmm, OK.

the JDK version doesn't matter, bb does not rely on a JDK except for shelling out to download some deps

GraalVM doesn't use the JDK from your system

Makes sense.

And this is especially true for nixpkgs

I'm brand new to GraalVM, so most of my assumptions about how Java stuff works are gonna be really wrong. 😂

GraalVM is its own JDK

How do I download a pod?

@UFDRD93RR

[nix-shell:~]$ rm -rf ~/.babashka/pods/repository

[nix-shell:~]$ bb -e "(babashka.pods/load-pod 'clj-kondo/clj-kondo \"2022.02.09\")"
Downloading pod clj-kondo/clj-kondo (2022.02.09)
Successfully installed pod clj-kondo/clj-kondo (2022.02.09)
#:pod{:id "pod.borkdude.clj-kondo"}

https://clojurians.slack.com/archives/CLX41ASCS/p1655367028248699?thread_ts=1655365001.427619&cid=CLX41ASCS

lol, you're fast!

bb -e "(babashka.pods/load-pod 'clj-kondo/clj-kondo \"2022.02.09\")"
----- Error --------------------------------------------------------------------
Type:     java.io.IOException
Message:  Cannot run program "/home/thiagoko/.babashka/pods/repository/borkdude/clj-kondo/2022.02.09/clj-kondo": error=2, No such file or directory
Location: <expr>:1:1

----- Context ------------------------------------------------------------------
1: (babashka.pods/load-pod 'clj-kondo/clj-kondo "2022.02.09")
   ^--- Cannot run program "/home/thiagoko/.babashka/pods/repository/borkdude/clj-kondo/2022.02.09/clj-kondo": error=2, No such file or directory

----- Stack trace --------------------------------------------------------------
babashka.pods.impl/run-pod           - <built-in>
babashka.pods.impl/load-pod          - <built-in>
babashka.pods.sci/load-pod/fn--30887 - <built-in>
babashka.pods.sci/load-pod           - <built-in>
clojure.core/apply                   - <built-in>
babashka.impl.pods/load-pod          - <built-in>
user

This is exactly what I tried

I'm going to make some breakfast. Thanks so much for all of your help, folks!

@UFDRD93RR Try the rm -rf first though

bb -e "(babashka.pods/load-pod 'clj-kondo/clj-kondo \"2022.02.09\")"
Downloading pod clj-kondo/clj-kondo (2022.02.09)
Successfully installed pod clj-kondo/clj-kondo (2022.02.09)
----- Error --------------------------------------------------------------------
Type:     java.io.IOException
Message:  Cannot run program "/home/thiagoko/.babashka/pods/repository/borkdude/clj-kondo/2022.02.09/clj-kondo": error=2, No such file or directory
Location: <expr>:1:1

----- Context ------------------------------------------------------------------
1: (babashka.pods/load-pod 'clj-kondo/clj-kondo "2022.02.09")
   ^--- Cannot run program "/home/thiagoko/.babashka/pods/repository/borkdude/clj-kondo/2022.02.09/clj-kondo": error=2, No such file or directory

----- Stack trace --------------------------------------------------------------
babashka.pods.impl/run-pod           - <built-in>
babashka.pods.impl/load-pod          - <built-in>
babashka.pods.sci/load-pod/fn--30887 - <built-in>
babashka.pods.sci/load-pod           - <built-in>
clojure.core/apply                   - <built-in>
babashka.impl.pods/load-pod          - <built-in>
user                                 - <expr>:1:1

Ok, I think I know what it is

clj-kondo probaby is not a static binary

Yep

But it seems to download and extracts fine

So I don't think this is related to the zlib 1.2.12 issue at all

Is there any other pod I can test that is compiled static :thinking_face: ?

let me update the clj-kondo pod to use a static one

@UFDRD93RR Try 2022.05.31 - should work now

> So I don't think this is related to the zlib 1.2.12 issue at all I do think it's related: I've seen this happen before several times

I think they fixed the zlib issue on the latest release of GraalVM (22.1.0)

bb -e "(babashka.pods/load-pod 'clj-kondo/clj-kondo \"2022.05.31\")"
Downloading pod clj-kondo/clj-kondo (2022.05.31)
----- Error --------------------------------------------------------------------
Type:     java.util.zip.ZipException
Message:  invalid entry CRC (expected 0x269cdf2c but got 0x13b86fd8)
Location: <expr>:1:1

Ok, I will take a look at this issue later them

Is there a way to force zlib to be on 1.2.11 in nix?

Yeah

The nice thing is that we can probably force just for GraalVM

:thumbsup:

Because otherwise we would reintroduce a very big and serious security issue

(It was quite nasty one)

But even better, maybe we can backport the same patch Arch is using (that is from zlib master branch)

On a positive note, I've now got nix installed again and this time I don't have any issues. Last time it broke my native zsh install

The nix installer improved a lot for macOS

It was kinda of a pain before, for sure

I'm gonna hack up the GraalVM derivation now. 🙂

Oh my goodness, it looks like one can't override zlib in an overlay! https://github.com/NixOS/nixpkgs/issues/61682

This is gonna get yucky. 😂

It is easier to fix this issue in nixpkgs itself by overriding the zlib

Yeah, that's what I'm working on now.

At least I think we're talking about the same approach.

OK... the way zlib makes its way into the GraalVM derivation is through mkGraal: https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/compilers/graalvm/community-edition/default.nix#L8

And since I can't overlay zlib, I really have no clue how I can inject it all the way down there without overlaying graalvm and doing some creative copy and paste.

@UFDRD93RR Do you have any better ideas?

maybe use a flake? I got something here with the help of @UFDRD93RR https://github.com/borkdude/graalvm-nix

(not sure why I forked this again, it's a while ago)

oh it was only because of musl

you should be using @UFDRD93RR’s version

Oh my, I forgot that I still haven't learned flakes. I think I'm rapidly wading out of my depth here. 😅

I'll wait to see if kokada has any ideas before continuing to bash my head against this wall. 😉

https://github.com/NixOS/nixpkgs/pull/177865

until you can patch zlib, you could just manually download the clj-kondo pod (v2022.05.31) and use an absolute path with load-pod

@UFDRD93RR I guess @U054AT6KT can try your: https://github.com/thiagokokada/nixpkgs/tree/fix-graalvm-zlib fork as his nixpkgs now right, to test this locally?

Yeah, if he want

@UFDRD93RR You’re an absolute legend! I’ll try this out in an hour or two. Today's blog entry is actually all about my fun rabbit hole trying to blog. 😂

@UFDRD93RR This worked, of course! 🏆

Hell yeah!

Now the next mole to whack: I guess I need to run patchelf on the pod after downloading it, otherwise this happens:

[jmglov@laurana:~/Documents/code/jmglov.net/blog]$ bb render
Downloading pod clj-kondo/clj-kondo (2021.10.19)
Successfully installed pod clj-kondo/clj-kondo (2021.10.19)
----- Error --------------------------------------------------------------------
Type:     java.io.IOException
Message:  Cannot run program "/home/jmglov/.babashka/pods/repository/borkdude/clj-kondo/2021.10.19/clj-kondo": error=2, No such file or directory

@U054AT6KT Please update to the newest clj-kondo pod (2022.05.31), that one is fully static and should not need any patching

You've bloody well thought of everything, haven't you!?

Well it's a good practice to make pods as static as possible because of these problems

Yikes! highlighter.clj loads clj-kondo with an explicit version rather than using the one specified in deps.edn:

(pods/load-pod 'clj-kondo/clj-kondo "2021.10.19")

@U054AT6KT Are you using a fork of my blog? There's a better way of doing that now

I have forked your blog, yes. 😉

I can update my blog to show the better approach

just a moment

I wanted a low-friction way to get started. 😉

Michiel, you are so incredibly generous with your time. Thank you so much! 💜

@U054AT6KT You're welcome :) https://github.com/borkdude/blog/commit/edfdcfcb9c4d0b09e27fcb4ab75d24cecda69cb8

It works! http://jmglov.net/blog/

This is seriously seriously cool!

And now I'll replace your posts with my own so as not to be naughty. 😉

🎉

Thanks again! This was a really fun adventure.

btw clj-kondo pod is only necessary for highlighting, I did it as a home-made solution, but if you don't like this, you can simply use highlight.js which I did before

the idea is you own your own blog code and you can simplify or make it more complex at your convenience

This coming week, I'm going to try to make a custom Lambda runtime for Babashka so I can edit my Babashka stuff in the Lambda console, then port my email over from JDK Clojure to Babashka.

Same for the CSS: I'm not a CSS expert so if you make it more fancy, I might steal some of your ideas

I am not great at CSS either, but I'll see what I can do. 🙂

Why don't you use :out :inherit as well?

And then just use @(process ...) to block until it's done

That… does seem much better yes

The equivalent of that would be p/shell: (p/shell "some-long-command")

Thanks, I knew I was missing something very basic 😅

Oh right, I did want to pre-process the output before printing it, but I think I removed that at some point

In that case, I do need the whole loop right? (like adding a \t before outputting each line)

in that case you need to do the loop yourself yes

maybe we could have some function in process which helps with that since it seems fairly common

Got it, thanks for the help. I think for now redirecting :out is enough for me