This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2021-05-07
Channels
- # announcements (32)
- # asami (1)
- # babashka (127)
- # beginners (135)
- # bristol-clojurians (1)
- # calva (21)
- # chlorine-clover (5)
- # cider (2)
- # clara (9)
- # clj-kondo (24)
- # cljsrn (2)
- # clojure (25)
- # clojure-australia (4)
- # clojure-europe (135)
- # clojure-nl (8)
- # clojure-russia (3)
- # clojure-spec (4)
- # clojure-uk (9)
- # clojurescript (55)
- # cursive (6)
- # datomic (62)
- # events (1)
- # fulcro (1)
- # helix (19)
- # jobs (3)
- # jobs-rus (1)
- # kaocha (8)
- # malli (6)
- # meander (3)
- # off-topic (2)
- # pathom (3)
- # podcasts-discuss (1)
- # polylith (3)
- # practicalli (3)
- # re-frame (4)
- # reitit (5)
- # remote-jobs (1)
- # reveal (1)
- # rewrite-clj (9)
- # ring-swagger (1)
- # shadow-cljs (59)
- # xtdb (4)
I’m using a M1 Air and installed bb through homebrew. I didn’t even notice it wasn’t native 😅
Any idea what this error means exactly?
(require 'lambdaisland.uri)
java.lang.IllegalArgumentException: No implementation of method: :getName of protocol: #'sci.impl.vars/HasName found for class: nil [at /home/arne/github/lambdaisland/uri/src/lambdaisland/uri.cljc:12:1]
offending code is here: https://github.com/lambdaisland/uri/blob/main/src/lambdaisland/uri.cljc#L12
@plexus Yes, defrecord in sci currently has the limitation that you cannot implement Java interfaces on it, only Clojure protocols
what a shame, I was hoping lambdaisland/uri would "just work" since it's largely pure clojure
looking at that code I do wonder why I'm implementing invoke
😅 , that should be default record behavior... and I guess I can also just define a print handler instead of implementing toString
@plexus you could use :bb
reader conditionals to support the subset of Clojure that sci supports
yeah that's what I was thinking, I think there's a way to do the same stuff that would be simpler and more interoperable anyway
The rules for this: the :clj
branch will be taken, unless there is a :bb
branch preceding it
I was running into the same issue with a parser combinator library which implemented Comparable
on a defrecord. That was the only change I had to make to make it compatible with bb. I can in theory support this, but you can only implement one Java interface at a time, so if you try to do two, like in your example, you're already out again.
the second one is overriding a method on a base class, I guess the same stuff applies?
yeah, it's tricky when it comes to implementing Java stuff. I can "emulate" most stuff but I can't really trick the Java type system.
I had a hack for reify
before which went like this: at compile time, I created multiple reified objects that implemented all subsets of all supported interfaces. At runtime I would choose the most fitting type which would then dispatch to the interpreted implementations. But this didn't work out since you can get a combinatorial explosion of types, e.g. when you have 100 interfaces to support, you can't really do it anymore, since it generates such a huge list.
The other approach I tried was: generate one reify object which implements all supported interfaces. But this didn't work out with instance?
checks, since it would return true
for a lot of stuff you would not expect it to.
So right now you can use reify
with at most one Java interface + any combination of protocols, for this reason.
I could extend that approach to defrecord
.
Another sweet bb.edn
:tasks
, this time by @lee
https://github.com/clj-commons/rewrite-clj/blob/main/bb.edn
Cool library to show "status lines". You can hook this up to your tasks. https://github.com/lread/status-line
Thanks for bb tasks @borkdude! Moving from scripts to bb tasks also swayed my brain to focus more on usability. My tasks are better named than my scripts were and the cmd line args my tasks accept are a bit more consistent.
Hi @borkdude, I hope it's okay if I tag you directly. I just started playing around with babashka and I love it so far!
I'm trying to rewrite a small Java CLI tool in Clojure using babashka, but I've hit a road block for now. Part of the tool is creating a zip file. To do that I was using java.util.zip.ZipOutputStream
, which is unfortunately not available in bb. I've seen that bb comes with a few other classes from the java.util.zip
package, and I was wondering if you could add the ZipOutputStream
to bb as well. (Or if you would consider a PR with the necessary changes)
yeah, a PR is welcome. We already have GZIPOutputStream
so it's a bit silly that we don't have ZIP
Awesome! 🙂
@d.wetzel86 New binaries are available in #babashka-circleci-builds with your change
I'm afraid I need to wait for the next release since I'm using bb on Windows 😅
Is there any reason that the AWS pod is not statically linked for Linux :thinking_face: ? I use NixOS and the current binaries doesn't work
Well, seeing the pod code it doesn't seem to be possible to have private pods registries
tzzh/aws is compiled using Go and that usually works on any platform since it doesn't depend on libc++ like graalvm binaries
You can load pods from anywhere on the file system, you just can provide a file path
On nixOS you likely have to install libc++ alongside the binary to make it work. You can see this using the ldd
command
there is likely a nixOS package for babashka or clj-kondo too, this should work similarly
> On nixOS you likely have to install libc++ alongside the binary to make it work. You can see this using the ldd command
@borkdude NixOS doesn't have the libraries in the common location (.e.g.: they're on /nix/store
, not in /usr/lib
), so even when they're "installed" most programs can't find them
> there is likely a nixOS package for babashka or clj-kondo too, this should work similarly Actually, the static linux binaries from Babashka works well in NixOS, and I think if AWS pod were compiled statically it would works as well
@UKFSJSM38 Can you shine more light on this, since you are also using nixOS and bb?
The static binary doesn't work on every system, the use case for it is mostly alpine/busybox docker images
Well, I helped @UKFSJSM38 with NixOS issues too
> The static binary doesn't work on every system, the use case for it is mostly alpine/busybox docker images Huh... Curious, can you give an example where it doesn't :thinking_face: ?
AFAIK static libs should work with any linux distribution including NixOS, also only static bb works fine on NixOS last time I tested it
@UFDRD93RR I don't have a specific example, but it comes up every now and then when people try the static linux binary on their linux system and run into problems with it. My default question is: did you try the static binary... try the dynamic one instead. It has come up a handful of times. @U7ERLH6JX knows more about the trade-offs between static and dynamic here
the issue is that every lib should be static, for example, even if compile clojure-lsp as static, it still fails since sqlite lib is not static
Yeah, I know about this issue @UKFSJSM38, but in this case Babashka looks like a fully static library
@borkdude > You can load pods from anywhere on the file system, you just can provide a file path Yeah, I know about it, but I am thinking about the best solution for having some custom pods without needing to commit them to repository, for example
@UFDRD93RR Please make a PR to https://github.com/babashka/pod-babashka-aws. It needs an environment variable in the circleci config to say whether it should include the static flag or not. And then the script/compile
script should pick up on that environment variable. I apply this in clj-kondo/clj-kondo and babashka/babashka as well, so you could use those as examples.
After that there will be a static pod, but currently there is no way to retrieve a static binary from the registry pod so you will have to load it using a relative or absolute path yourself.
> After that there will be a static pod, but currently there is no way to retrieve a static binary from the registry pod so you will have to load it using a relative or absolute path yourself. Yeah, this is also another reason why I think custom repositories would be interesting
Maybe the base URL of the repository can be tweaked using an environment variable, or maybe we should support a list of base-urls which will be tried in order? Have to think about. AFK for groceries now :)
I was thinking maybe allowing pods/load-pods
receive an extra argument for options (probably a map or something), or maybe have the URL in a using (def ^:dynamic ...)
var that could be overwrite with binding
(I would prefer to avoid environment variables though)
Having a list of URLs instead of only one would be useful too
> Huh... Curious, can you give an example where it doesn't @UFDRD93RR we have seen issues with the static binary having issues when its being run in an environment providing a different/newer versions of ABI. The linux kernel verion being different also causes it to segfault. The binary produced by the static graalvm compilation is liked for Linux 2.6.32 and whenever we see it being run in a higher kernel version it segfaults at various places. Most notably when doing network IO. I'm not really sure what exactly is the root cause, haven't gotten around to explore that deep.
its mostly something to do with the linking done by the graal native image. Go binaries which are mostly static seem to this better. Also my hunch is around the musl versions linked to the graal binary. since Go neither uses musl nor glibc, its able to produce more portable static binaries. My very uneducated guess would be the version of musl used to by graal to build native images fails when the kernel is different. Specially when the OS is glibc/dynamic linking based like Debian etc and not musl based like Alpine
@UK0810AQ2 have a look here if you wanna know the issues we have faced with static binary, if you have some ideas, they are most welcome! 🙂
> musl I don't want to condemn it but you may be SOL. A static built with musl on glibc systems even starting up is a miracle
yeah 😕 its something we need to address upstream with the GraalVM devs somehow.
Go binaries are built completely differently, even its concept of assembly is different
exactly!
The only difference between normal bb is that it has been built with the static flag of GraalVM native image. I don’t think it has anything specific to musl, but perhaps read the docs
@borkdude we do need to specify --libc=musl
right?
right, the docs on Graal static says
To build a static native image, use:
native-image --static --libc=musl [other arguments] Class
https://www.graalvm.org/reference-manual/native-image/StaticImages/but in order to have a static binary on linux we definitely need to have musl linkage
i guess they could be adding it by default with --static
@UFDRD93RR Could you perhaps try the same that bsless tried in the main channel, on nixOs with the static bb image? (shelling out)
Perhaps there is some issue around ProcessBuilder in the static image on some OS/kernels?
Ben's versions:
OS: Pop 20.10 groovy
Kernel: x86_64 Linux 5.11.0-7614-generic
This seems to work on alpine:
$ docker run --rm -i babashka/babashka:0.4.0-alpine <<< '(:out (babashka.process/sh "ls"))'
Babashka v0.4.0 REPL.
Use :repl/quit or :repl/exit to quit the REPL.
Clojure rocks, Bash reaches.
user=> "bin\ndev\netc\nhome\nlib\nlib64\nmedia\nmnt\nopt\nproc\nroot\nrun\nsbin\nsrv\nsys\ntmp\nusr\nvar\n"
@borkdude whats the kernel version here?
@UFDRD93RR could you try https://329-325868827-gh.circle-artifacts.com/0/release/pod-babashka-aws-0.0.5-linux-amd64-static.zip ? this is the freshly brewed musl linked binary!
@U7ERLH6JX Just did a static build from Babashka too: https://github.com/babashka/babashka/pull/827 My approach is different since I build musl from scratch, but I will try your binary
yeah i managed to shave off the really long musl compilation
musl-tools
is installed via apt and seems to work fine
also @borkdude and i were thinking is musl works fine, we can make that the static one
@U7ERLH6JX Yeah, it works :thumbsup:. But keep in mind I never had issues with other static binaries from Babashka, so maybe I am not the best person to test this.
Yay! well we havent compiled any other clojure pods statically and i think using musl should be much more stable
> musl-tools is installed via apt and seems to work fine
As far I understand, there is some disadvantages of using musl-tools
- It is an out-dated version of musl (in my branch it uses an 1.2.0 that is also out-dated, ideally we should use musl 1.2.2 since it has a much better malloc implementation; but building from scratch is easier to update musl)
- Also, I am not sure that copying libz.a
from Debian as is isn't going to cause some problems.
But I am fine either way, if it works the advantage of your approach is not building musl (that takes quite a long time)
yeah so i build the libz after building with musl-tools, that should be okay? i copied the libz build from the link you sent
Yeah, I think building libz with musl-tools should be a better approach to avoid issues
yeah so explicitly copied that one instead of installing. i agree on trying on using a newer musl-tools, building it from source is really long 😕 maybe we can get around it with caching?
BTW, bb
statically-build with musl
is slightly smaller too:
ls -lah bb
-rwxr-xr-x 1 thiagoko users 80M mai 8 13:15 bb
ls -lah bb
-rwxr-xr-x 1 thiagoko users 76M mai 9 14:45 bb
> maybe we can get around it with caching? Yeah, I think caching would be the way to go if we want to build musl from source
do you reckon there could be a way to install the newer musl-tools? maybe using the debian-testing mirrors?
> BTW, bb statically-build with musl is slightly smaller too this is awesome!
> do you reckon there could be a way to install the newer musl-tools? maybe using the debian-testing mirrors?
I think using musl-tools
from testing
in stable
is risk for the same reason that using glibc for static linking is risk (there is a chance for glibc version mismatch)
> Yay! well we havent compiled any other clojure pods statically We have, the sql pods are also available as static compiled
If it is available in debian backports it would be safer, but even debian-testing has a chance of being out-dated
> I think using musl-tools from testing in stable is risk for the same reason that using glibc for static linking is risk (there is a chance for glibc version mismatch)
Unless musl-tools
is build statically, them it should be safe to use it
@U7ERLH6JX https://packages.debian.org/unstable/musl-tools
In unstable
musl-tools
is up-to-date ☝️
yeah so we can add the apt sources and just pull musl tools from it?
I'm fine with any approach, but maybe we should document this somewhere, possibly in the https://github.com/lread/clj-graal-docs repo (once the correct approach is figured out)
> yeah so we can add the apt sources and just pull musl tools from it? Yeah, I think we can try Just avoid installing anything unnecessary from unstable (otherwise we may break something else)
I think the best way is to add unstable with low priority, and explicitly install musl-tools
from unstable
> I think the best way is to add unstable with low priority, and explicitly install musl-tools from unstable http://jaqque.sbih.org/kplug/apt-pinning.html
With musl 1.2.2 the binaries are almost the same as glibc in size:
ls -lah bb
-rwxr-xr-x 1 thiagoko users 79M mai 9 15:44 bb
pushed https://github.com/babashka/pod-babashka-aws/commit/e683491e8a7bd635c1af09bf0550a5098983aa88
test are passing @borkdude!
if youre okay with this approach @UFDRD93RR what say we stick to this?
@U7ERLH6JX Maybe you can add a few comments to the setup-musl script to explain why this approach was chosen?
sure, should i add it as a comment in the file itself?
on it
pushed
@borkdude i think its good to merge?
also maybe one last check with https://346-325868827-gh.circle-artifacts.com/0/release/pod-babashka-aws-0.0.5-linux-amd64-static.zip @UFDRD93RR?
@U7ERLH6JX Do you have a PR I should merge?
the musl
branch, should i raise a PR?
raising one now