Fork me on GitHub
#babashka
<
2021-05-07
>
solf02:05:42

I’m using a M1 Air and installed bb through homebrew. I didn’t even notice it wasn’t native 😅

plexus10:05:34

Any idea what this error means exactly?

(require 'lambdaisland.uri)
java.lang.IllegalArgumentException: No implementation of method: :getName of protocol: #'sci.impl.vars/HasName found for class: nil [at /home/arne/github/lambdaisland/uri/src/lambdaisland/uri.cljc:12:1]

borkdude10:05:33

@plexus Yes, defrecord in sci currently has the limitation that you cannot implement Java interfaces on it, only Clojure protocols

borkdude10:05:48

The error message could be more informative though

plexus10:05:55

I see, I guess that makes sense

plexus10:05:29

what a shame, I was hoping lambdaisland/uri would "just work" since it's largely pure clojure

plexus10:05:53

looking at that code I do wonder why I'm implementing invoke 😅 , that should be default record behavior... and I guess I can also just define a print handler instead of implementing toString

borkdude10:05:25

@plexus you could use :bb reader conditionals to support the subset of Clojure that sci supports

plexus10:05:58

yeah that's what I was thinking, I think there's a way to do the same stuff that would be simpler and more interoperable anyway

borkdude10:05:15

The rules for this: the :clj branch will be taken, unless there is a :bb branch preceding it

borkdude10:05:46

I was running into the same issue with a parser combinator library which implemented Comparable on a defrecord. That was the only change I had to make to make it compatible with bb. I can in theory support this, but you can only implement one Java interface at a time, so if you try to do two, like in your example, you're already out again.

plexus10:05:10

the second one is overriding a method on a base class, I guess the same stuff applies?

borkdude10:05:43

yeah, it's tricky when it comes to implementing Java stuff. I can "emulate" most stuff but I can't really trick the Java type system.

borkdude10:05:59

I had a hack for reify before which went like this: at compile time, I created multiple reified objects that implemented all subsets of all supported interfaces. At runtime I would choose the most fitting type which would then dispatch to the interpreted implementations. But this didn't work out since you can get a combinatorial explosion of types, e.g. when you have 100 interfaces to support, you can't really do it anymore, since it generates such a huge list. The other approach I tried was: generate one reify object which implements all supported interfaces. But this didn't work out with instance? checks, since it would return true for a lot of stuff you would not expect it to.

borkdude10:05:04

So right now you can use reify with at most one Java interface + any combination of protocols, for this reason. I could extend that approach to defrecord.

borkdude11:05:01

Cool library to show "status lines". You can hook this up to your tasks. https://github.com/lread/status-line

lread12:05:24

Thanks for bb tasks @borkdude! Moving from scripts to bb tasks also swayed my brain to focus more on usability. My tasks are better named than my scripts were and the cmd line args my tasks accept are a bit more consistent.

🎉 4
NPException13:05:31

Hi @borkdude, I hope it's okay if I tag you directly. I just started playing around with babashka and I love it so far! I'm trying to rewrite a small Java CLI tool in Clojure using babashka, but I've hit a road block for now. Part of the tool is creating a zip file. To do that I was using java.util.zip.ZipOutputStream , which is unfortunately not available in bb. I've seen that bb comes with a few other classes from the java.util.zip package, and I was wondering if you could add the ZipOutputStream to bb as well. (Or if you would consider a PR with the necessary changes)

borkdude13:05:20

yeah, a PR is welcome. We already have GZIPOutputStream so it's a bit silly that we don't have ZIP

NPException13:05:17

Awesome! 🙂

borkdude14:05:30

@d.wetzel86 New binaries are available in #babashka-circleci-builds with your change

NPException14:05:12

I'm afraid I need to wait for the next release since I'm using bb on Windows 😅

kokada23:05:20

Is there any reason that the AWS pod is not statically linked for Linux :thinking_face: ? I use NixOS and the current binaries doesn't work

kokada23:05:52

BTW, the tzzh/aws works

kokada23:05:31

Another kinda related question, can we have private pod registries :thinking_face: ?

kokada23:05:27

Well, seeing the pod code it doesn't seem to be possible to have private pods registries

borkdude07:05:00

tzzh/aws is compiled using Go and that usually works on any platform since it doesn't depend on libc++ like graalvm binaries

borkdude07:05:34

You can load pods from anywhere on the file system, you just can provide a file path

borkdude07:05:31

On nixOS you likely have to install libc++ alongside the binary to make it work. You can see this using the ldd command

borkdude08:05:17

there is likely a nixOS package for babashka or clj-kondo too, this should work similarly

kokada18:05:54

> On nixOS you likely have to install libc++ alongside the binary to make it work. You can see this using the ldd command @borkdude NixOS doesn't have the libraries in the common location (.e.g.: they're on /nix/store, not in /usr/lib), so even when they're "installed" most programs can't find them

kokada18:05:30

> there is likely a nixOS package for babashka or clj-kondo too, this should work similarly Actually, the static linux binaries from Babashka works well in NixOS, and I think if AWS pod were compiled statically it would works as well

borkdude18:05:09

@UKFSJSM38 Can you shine more light on this, since you are also using nixOS and bb?

borkdude18:05:49

The static binary doesn't work on every system, the use case for it is mostly alpine/busybox docker images

kokada18:05:58

Well, I helped @UKFSJSM38 with NixOS issues too

kokada18:05:38

> The static binary doesn't work on every system, the use case for it is mostly alpine/busybox docker images Huh... Curious, can you give an example where it doesn't :thinking_face: ?

ericdallo18:05:12

AFAIK static libs should work with any linux distribution including NixOS, also only static bb works fine on NixOS last time I tested it

borkdude18:05:32

@UFDRD93RR I don't have a specific example, but it comes up every now and then when people try the static linux binary on their linux system and run into problems with it. My default question is: did you try the static binary... try the dynamic one instead. It has come up a handful of times. @U7ERLH6JX knows more about the trade-offs between static and dynamic here

ericdallo18:05:57

the issue is that every lib should be static, for example, even if compile clojure-lsp as static, it still fails since sqlite lib is not static

borkdude18:05:32

Having said this, it's no problem to provide a static binary for the aws pod

👍 6
kokada18:05:45

Yeah, I know about this issue @UKFSJSM38, but in this case Babashka looks like a fully static library

ericdallo18:05:01

yes, it looks indeed

kokada18:05:04

It doesn't seem to try to load anything dynamically neither

kokada18:05:44

@borkdude > You can load pods from anywhere on the file system, you just can provide a file path Yeah, I know about it, but I am thinking about the best solution for having some custom pods without needing to commit them to repository, for example

kokada18:05:54

If we could have custom repositories, this would be a breeze

borkdude18:05:58

@UFDRD93RR Please make a PR to https://github.com/babashka/pod-babashka-aws. It needs an environment variable in the circleci config to say whether it should include the static flag or not. And then the script/compile script should pick up on that environment variable. I apply this in clj-kondo/clj-kondo and babashka/babashka as well, so you could use those as examples. After that there will be a static pod, but currently there is no way to retrieve a static binary from the registry pod so you will have to load it using a relative or absolute path yourself.

kokada18:05:17

But I can find other solutions if this isn't desirable

borkdude18:05:38

I haven't thought about custom repositories but that doesn't sound too hard

kokada18:05:56

> After that there will be a static pod, but currently there is no way to retrieve a static binary from the registry pod so you will have to load it using a relative or absolute path yourself. Yeah, this is also another reason why I think custom repositories would be interesting

kokada18:05:14

I could for example point the repositories to the static version of the pods

borkdude18:05:03

Maybe the base URL of the repository can be tweaked using an environment variable, or maybe we should support a list of base-urls which will be tried in order? Have to think about. AFK for groceries now :)

👍 3
kokada18:05:20

I was thinking maybe allowing pods/load-pods receive an extra argument for options (probably a map or something), or maybe have the URL in a using (def ^:dynamic ...) var that could be overwrite with binding (I would prefer to avoid environment variables though) Having a list of URLs instead of only one would be useful too

borkdude18:05:33

Yeah, that makes sense

lispyclouds07:05:40

> Huh... Curious, can you give an example where it doesn't @UFDRD93RR we have seen issues with the static binary having issues when its being run in an environment providing a different/newer versions of ABI. The linux kernel verion being different also causes it to segfault. The binary produced by the static graalvm compilation is liked for Linux 2.6.32 and whenever we see it being run in a higher kernel version it segfaults at various places. Most notably when doing network IO. I'm not really sure what exactly is the root cause, haven't gotten around to explore that deep.

lispyclouds07:05:03

its mostly something to do with the linking done by the graal native image. Go binaries which are mostly static seem to this better. Also my hunch is around the musl versions linked to the graal binary. since Go neither uses musl nor glibc, its able to produce more portable static binaries. My very uneducated guess would be the version of musl used to by graal to build native images fails when the kernel is different. Specially when the OS is glibc/dynamic linking based like Debian etc and not musl based like Alpine

lispyclouds07:05:09

@UK0810AQ2 have a look here if you wanna know the issues we have faced with static binary, if you have some ideas, they are most welcome! 🙂

Ben Sless07:05:21

> musl I don't want to condemn it but you may be SOL. A static built with musl on glibc systems even starting up is a miracle

lispyclouds07:05:10

yeah 😕 its something we need to address upstream with the GraalVM devs somehow.

Ben Sless07:05:10

Go binaries are built completely differently, even its concept of assembly is different

borkdude07:05:46

The only difference between normal bb is that it has been built with the static flag of GraalVM native image. I don’t think it has anything specific to musl, but perhaps read the docs

lispyclouds07:05:19

@borkdude we do need to specify --libc=musl right?

borkdude07:05:23

We are just using the static flag, which I can’t type on my phone

borkdude07:05:51

The other variations were introduced much later in GraalVM

lispyclouds07:05:09

right, the docs on Graal static says

To build a static native image, use:

native-image --static --libc=musl [other arguments] Class
https://www.graalvm.org/reference-manual/native-image/StaticImages/

lispyclouds07:05:42

but in order to have a static binary on linux we definitely need to have musl linkage

lispyclouds07:05:15

i guess they could be adding it by default with --static

borkdude08:05:00

@UFDRD93RR Could you perhaps try the same that bsless tried in the main channel, on nixOs with the static bb image? (shelling out)

borkdude08:05:18

Perhaps there is some issue around ProcessBuilder in the static image on some OS/kernels?

lispyclouds08:05:24

Ben's versions:

OS: Pop 20.10 groovy
Kernel: x86_64 Linux 5.11.0-7614-generic

borkdude08:05:29

This seems to work on alpine:

$ docker run --rm -i babashka/babashka:0.4.0-alpine <<< '(:out (babashka.process/sh "ls"))'
Babashka v0.4.0 REPL.
Use :repl/quit or :repl/exit to quit the REPL.
Clojure rocks, Bash reaches.

user=> "bin\ndev\netc\nhome\nlib\nlib64\nmedia\nmnt\nopt\nproc\nroot\nrun\nsbin\nsrv\nsys\ntmp\nusr\nvar\n"

lispyclouds08:05:34

@borkdude whats the kernel version here?

borkdude08:05:14

/ # uname -r
4.19.121-linuxkit

kokada17:05:06

@U7ERLH6JX Just did a static build from Babashka too: https://github.com/babashka/babashka/pull/827 My approach is different since I build musl from scratch, but I will try your binary

lispyclouds17:05:37

yeah i managed to shave off the really long musl compilation

lispyclouds17:05:22

musl-tools is installed via apt and seems to work fine

lispyclouds18:05:19

also @borkdude and i were thinking is musl works fine, we can make that the static one

kokada18:05:34

@U7ERLH6JX Yeah, it works :thumbsup:. But keep in mind I never had issues with other static binaries from Babashka, so maybe I am not the best person to test this.

lispyclouds18:05:44

Yay! well we havent compiled any other clojure pods statically and i think using musl should be much more stable

kokada18:05:33

> musl-tools is installed via apt and seems to work fine As far I understand, there is some disadvantages of using musl-tools - It is an out-dated version of musl (in my branch it uses an 1.2.0 that is also out-dated, ideally we should use musl 1.2.2 since it has a much better malloc implementation; but building from scratch is easier to update musl) - Also, I am not sure that copying libz.a from Debian as is isn't going to cause some problems.

kokada18:05:10

But I am fine either way, if it works the advantage of your approach is not building musl (that takes quite a long time)

lispyclouds18:05:00

yeah so i build the libz after building with musl-tools, that should be okay? i copied the libz build from the link you sent

kokada18:05:50

Yeah, I think building libz with musl-tools should be a better approach to avoid issues

kokada18:05:11

Should avoid e.g.: ABI incompatibilities

lispyclouds18:05:19

yeah so explicitly copied that one instead of installing. i agree on trying on using a newer musl-tools, building it from source is really long 😕 maybe we can get around it with caching?

kokada18:05:38

BTW, bb statically-build with musl is slightly smaller too:

ls -lah bb
-rwxr-xr-x 1 thiagoko users 80M mai  8 13:15 bb
ls -lah bb
-rwxr-xr-x 1 thiagoko users 76M mai  9 14:45 bb

kokada18:05:12

> maybe we can get around it with caching? Yeah, I think caching would be the way to go if we want to build musl from source

lispyclouds18:05:11

do you reckon there could be a way to install the newer musl-tools? maybe using the debian-testing mirrors?

lispyclouds18:05:55

> BTW, bb statically-build with musl is slightly smaller too this is awesome!

kokada18:05:14

> do you reckon there could be a way to install the newer musl-tools? maybe using the debian-testing mirrors? I think using musl-tools from testing in stable is risk for the same reason that using glibc for static linking is risk (there is a chance for glibc version mismatch)

borkdude18:05:38

> Yay! well we havent compiled any other clojure pods statically We have, the sql pods are also available as static compiled

3
kokada18:05:44

If it is available in debian backports it would be safer, but even debian-testing has a chance of being out-dated

kokada18:05:30

> I think using musl-tools from testing in stable is risk for the same reason that using glibc for static linking is risk (there is a chance for glibc version mismatch) Unless musl-tools is build statically, them it should be safe to use it

lispyclouds18:05:42

yeah so we can add the apt sources and just pull musl tools from it?

borkdude18:05:19

I'm fine with any approach, but maybe we should document this somewhere, possibly in the https://github.com/lread/clj-graal-docs repo (once the correct approach is figured out)

kokada18:05:21

> yeah so we can add the apt sources and just pull musl tools from it? Yeah, I think we can try Just avoid installing anything unnecessary from unstable (otherwise we may break something else)

kokada18:05:20

I think the best way is to add unstable with low priority, and explicitly install musl-tools from unstable

lispyclouds18:05:32

yeah thats what im thinking

👍 3
kokada18:05:30

> I think the best way is to add unstable with low priority, and explicitly install musl-tools from unstable http://jaqque.sbih.org/kplug/apt-pinning.html

kokada18:05:16

With musl 1.2.2 the binaries are almost the same as glibc in size:

ls -lah bb
-rwxr-xr-x 1 thiagoko users 79M mai  9 15:44 bb

lispyclouds19:05:28

if youre okay with this approach @UFDRD93RR what say we stick to this?

borkdude19:05:22

@U7ERLH6JX Maybe you can add a few comments to the setup-musl script to explain why this approach was chosen?

lispyclouds19:05:15

sure, should i add it as a comment in the file itself?

borkdude19:05:33

yeah, I think that's the best place

borkdude19:05:47

just a few comments at the start

lispyclouds19:05:43

@borkdude i think its good to merge?

kokada19:05:13

Seems to works fine :thumbsup:

3
borkdude19:05:33

@U7ERLH6JX Do you have a PR I should merge?

lispyclouds19:05:01

the musl branch, should i raise a PR?

lispyclouds19:05:30

raising one now