Hey @rahul080327 @thiagokokada what is the reason to prefer musl static linking in native images over standard libc?

@retrogradeorbit Oh I found some docs here: https://github.com/clj-easy/graal-docs#static-linking-with-musl

Hey @retrogradeorbit essentially glibc cannot be fully static linked in a way musl can be, see this https://stackoverflow.com/a/57478728 pretty much the reason why musl was built. its a general problem for any native binaries, not specific to GraalVM native images

Rust for instance only does static linked binaries when used with musl+linux targets.

Or the other way to go is Go 😛 where none of the system libs are used in the same way other binaries link with

The SO link posted by lispyclouds pretty much nails it

glibc static is something that kinda works most of the time, however may results in segfaults if you for example use a non-glibc distro or uses a glibc distro with an older version of glibc

Or an older version of the kernel, or using on the wrong phase of the moon

Etc etc

yeah one of the main reasons for the segfaults being a mismatch of the ABI expectations the binary has with the kernel

You can have trully static binaries with glibc, but AFAIK this needs a specially compiled version of glibc with some flags that are completely unsupported by upstream

yeah... i dunno. I get that glibc static linking is a bit of a hack, and doesnt even work with all toolchains...

Also, musl has some other advantages

I've had quite some problems with the glibc-ed static versions of babashka and clj-kondo and will never go back doing that: those segfaults are a major headache.

It results in smaller binaries

but the graal static glibc image doesn't seem to match what that SO answer is talking about at all

Like, I may be wrong, but the static babashka with musl is smaller than the dynamically babashka with glibc

that is true

I don't see any lib deps, no dlopens, no nss, no iconv

works on alpine

I'll go have a compare of sizes

I recommend either don't pretend to be static, or use musl.

SO post may be talking in a more general sense of why static linking glibc is problematic?

Yeah, but for example

The old babashka static version (with glibc) didn't work in NixOS at all

I dont think its pretending to be static

The new one does

> I don't see any lib deps, no dlopens, no nss, no iconv well thats some of the possible calls the binary may make, the graalvm one doesnt i guess?

> well thats some of the possible calls the binary may make, the graalvm one doesnt i guess? Yeah, Java itself may end up calling nss for example

For aarch64, musl is not supported. There I use the mostly static option: --static with dynamic glibc. that works fine.

So Java -> GraalVM can call it too

yeah. and I experimented with musl versions and graal. And the new musl versions don't seem to work with graal at all

> For aarch64, musl is not supported. There I use the mostly static option: --static with dynamic glibc. that works fine. This is also a good point, static building glibc is not supported at all with GraalVM Native Image either

It works but it is not documented

hmm but I just built a static glibc with graal

not mostly static. fully static

The documentation explicitly says: • Dynamically compilation • Static compilation with musl • Static compilation with glibc dynamically linked

> not mostly static. fully static Like I said, it works. It is NOT supported though

https://www.graalvm.org/22.0/reference-manual/native-image/StaticImages/

is it my graal version?

No

Im using 21.3.0

It was never supported

It works

But was never documented

You can see in the documentation I posted above

It either recommends you to build with musl OR dynamically loaded glibc

yeah it does

This musl or dynamic documentation showed up around a year or so ago, before then the only option was --static but this was in hindsight a mistake on their end.

yeah thats how we initially built the bb and kondo images with quite the suffering later on

its pretty much undefined behaviour when the image is "mostly static"

ok. i think I get it

its always been a hack until musl

just curiously do you remember how the problem of static glibc manifested on other platforms?

segfault?

@rahul080327 I refer to "mostly static" as a supported option explictly by the GraalVM docs. This is supported and gives defined behavior. https://www.graalvm.org/22.0/reference-manual/native-image/StaticImages/#build-mostly-static-native-image

@retrogradeorbit yes

do you remember what platform?

don't remember, but I think it happened on systems that had a different (old?) glibc than the ubuntu I compiled it on

maybe search in babashka's issue

https://github.com/oracle/graal/issues/571#issuecomment-618508346

awesome. this is great info. thanks

if i rack my brains it was debian 9 i think ?

reading that ticket... wow that is horrific

yeah when it hit us it was very nasty head scratcher for us 😕

Another fun one: https://github.com/oracle/graal/issues/3737

I now use the -H:+StaticExecutableWithDynamicLibC option for aarch64

Yeah, I took this issue exactly from Babashka source code (describing why we are using -H:+StaticExecutableWithDynamicLibC) 😅

BTW, does those binaries work on Android?

I assume no?

yes, they do on android tablets

Weird, Android shouldn't have glibc

Does it work in Alpine?

Several mentions of android here: https://github.com/babashka/babashka/issues/241

Dunno, try it ;)

I expect no, since it requires glibc

With termux I can understand

Termux may bring glibc

https://github.com/termux/termux-packages/issues/2798#issuecomment-632134264 Ok, my mind is blown