Fork me on GitHub
Matthew Twomey03:02:30

I am moving a few frequently used functions into a library. I use this library with babashka scripts and it works great. I just ran into a little issue and I’m sure I’m just missing something simple. I have some functions in my library that use babashka.process , however when I call these functions it seems prevent my tests from exiting. As a simplified example:

(ns test2
    [babashka.process :refer [shell]]))

(println "* Start *")
(print (:out (shell {:out :string :err :string} "ls -las")))
(println "* Finish *")
When I run this directly with clj -M test2.clj it “hangs” and won’t exit (it does however print out everything - including the * Finish *. If I comment out that middle line, it prints start and finish, then exits successfully. What am I missing here?

Matthew Twomey06:02:56

Oh I see, I need (shutdown agents) at the end. I was completely unaware of this.


That should only be necessary in JVM Clojure

Matthew Twomey16:02:22

yes correct, that was my experience ty.


POLL babashka What do people think here about adopting GraalVM 22.3.1 JDK 19.0.2 for the next version of babashka with loom support? I have a where all tests are currently passing. We'll have to move off JDK 11 for the next GraalVM release anyway since 11 will no longer be supported in 23.0.0. All available options: 1. Move to JDK 19 now and then followed by alternative 2. 2. Move to JDK 20 in April 2023 and then move to JDK 21 LTS in Sep 2023. 3. Move to 17 LTS now and then to 21 LTS in Sep 2023. You can vote by clicking on one of the options below (and comment in thread).

1️⃣ 48
2️⃣ 2
3️⃣ 14

@UKFSJSM38 I also wonder what implications it would have for nix builds if we moved to graalvm jdk 19


I think we latest @UFDRD93RR changes to the nix graalvm derivation should be easy to support that


Why JDK19 instead of 17?


Just curious


JDK 17 is LTS while 19 is not AFAIK


> with loom support


virtual threads


From nixpkgs perspective I would prefer to not have to add JDK 19 to nixpkgs just because of babashka


We could though, and it should be hugely difficult


But if babashka builds with JDK17 would be preferred


but with flakes it would be possible right


The issue is not how to support it


It will work


Just it is too much maintanance overhead


We will need to keep GraalVM 19 until EOL if we added it to nixpkgs


I could set up bb to be able to build with both 19 and 17 so nix users would just not get the virtual thread support yet, maybe a reasonable compromise


This looks good for me


We will probably be an early adopter of GraalVM 21


If everything went well at least


I plan to stick with 21 LTS once it comes out (unless there's a compelling reason to move again)


so then we will be in sync in 6 months again


Would JDK19 preview features be enabled in bb by default or would they require some sort of --enable-preview switch?


enabled by default


It would kinda suck for the nixpkgs version to be a downgrade from the official release, and I guess it would be a problem for other distributors that don't want to build every JDK/Graal version. What about having a separate "preview" build of babashka that runs on JDK 19 with preview features enabled? That could be provided as a Nix flake with an overlay for those who want the latest JDK. And the core version could be built on the stable LTS release for everyone.

Daniel Gerson15:02:51

I voted for JDK 19 only because it doesn't really affect me 😅. However I imagine at some point something like sdkman would be preferable for users that require particular versions? It's what I use for JDK on android development etc. Problems that come from being too popular?!


> It would kinda suck for the nixpkgs version to be a downgrade from the official release This will only be for 6 months though


nix users who really want to get parity, can always download the static linux binary as well


I voted 3 because I personally don't have any pressing need for JDK19 preview features. I can wait. But I expect folks are voting 1 because they don't want to wait, they want to use these features now. Is that the main benefit? Taking advantage of preview features now?


it's also an extra maintenance burden to keep the jdk19-loom branch up to date


option 1 should not affect anyone who doesn't want to use the virtual thread stuff yet


Yah, personally fine with option 1 as well. I guess the only con is that preview features can have breaking changes. But folks using these features are very likely aware of that.


Well I think it's also notable that to enable the preview features in Java you also have to pass a command-line flag (right?) So it's a pretty explicit opt-in. But yeah I imagine people know that they're using unstable features.


Do you have a gist of using loom with babashka?


I think we have enough votes, it's about 77% for alternative 1 vs 23% for alternative 3. This means I'll go ahead and merge the jdk19-loom branch - yay, no more maintaining a branch!

🎉 25
clojure-spin 1

You should now be able to test this binary with:

bash <(curl ) --dev-build --dir /tmp
and e.g. execute this snippet:


In a way this is a remarkable thing. I'm not aware of an interpretted language which could effectively use the computational power of multiple-real threads. Now with these virtual threads, it's even more awesome since they are cheap and you can solve most if not all concurrency with them. Stunning thing, thank you @U04V15CAJ


And I think I got it working with nix:

nix run --accept-flake-config github:jlesquembre/clj-nix/bb-loom#bb-loom -- virtual.clj
The --accept-flake-config option is there to use the binary cache and avoid re-compile it


@U4NGX0FHN awesome! I wonder how much disk space this will cost me :)

[0/1475 built, 1/344/347 copied (2851.5/3264.3 MiB), 417.5/610.1 MiB DL] fetching zulu17.34.19-ca-jdk-17.0.3 from 


All of it... Now seriously, if you used the accept-flake-config option, with a recent nix version it should use the binary cache, and don't download the build dependencies. Maybe I did a mistake, I'll check tomorrow. You are on a Mac, right?


I'm on a mac yes


what is recent nix?


$ nix --version
nix (Nix) 2.9.1


sorry, forget about the "recent version", if flakes already work, it's recent enough. I'm not sure why it's downloading jdk-17, with the --accept-flake-config should the binaries from cachix. It's still a big download, because it depends on graalvm at runtime dependency, but it should not download jdk-17


The full command I executed was:

nix --extra-experimental-features nix-command --extra-experimental-features flakes run --accept-flake-config github:jlesquembre/clj-nix/bb-loom#bb-loom -- /tmp/loom.clj
but I didn't wait for it to finish because I suspected it would take hours


running that command, did you get a warning: ignoring untrusted substituter '' warning?


btw, you can put the nix options, like extra-experimental-features in the nix.conf file:


I tried, but /etc/whatever.conf wasn't writable


I also tried ~/.config/nix/nix.conf :

nix.settings.experimental-features = [ "nix-command" "flakes" ];
but this gave me error messages


I'm not a mac user, but can't you edit /etc/nix/nix.conf with sudo?


you can but it seemed a bit weird to me that I would have to do that


with that user conf file set:

$ nix run --accept-flake-config github:jlesquembre/clj-nix/bb-loom#bb-loom -- /tmp/loom.clj
warning: unknown setting 'nix.settings.experimental-features'
error: experimental Nix feature 'nix-command' is disabled; use '--extra-experimental-features nix-command' to override


ok, for reference, this is my (partial) /etc/nix/nix.conf:

allowed-users = *
trusted-users = root jlle
experimental-features = nix-command flakes
builders-use-substitutes = true


ah: experimental-features = nix-command flakes works


you installed the nix daemon, right? I think after updating it you need to restart the service, on linux is systemctl , not sure on mac


still getting:

warning: ignoring untrusted substituter ''


trusted-users = root borkdude


or your username


I did that +:

sudo launchctl stop org.nixos.nix-daemon;
sudo launchctl start org.nixos.nix-daemon;
but still getting the same message


um, maybe you need to login again? what if you put this:

substituters =  
trusted-public-keys =


then I get the message three times:

warning: ignoring untrusted substituter ''
warning: ignoring untrusted substituter ''
warning: ignoring untrusted substituter ''


I'm going to ask some one else


it's not so important to me, since I normally don't use nix a lot, I was just messing around so if this is time consuming for you, we can just leave it. but perhaps someone who is using nix on a daily basis could try, like @UKFSJSM38


@UKFSJSM38 This is the command I use to run @U4NGX0FHN’s nix version of bb + loom support:

nix run --accept-flake-config github:jlesquembre/clj-nix/bb-loom#bb-loom -- /tmp/loom.clj


@U04V15CAJ no worries, I'm curious to know what's going on


worked for me without any warnings NixOS running nix 2.12.0

👍 4

I upgraded nix now:

$ nix --version
nix (Nix) 2.13.2
Still got the same problem :)


@U04V15CAJ if you can update your nix version, you should get a better error about why is ignoring the substituter


and still the same error message?


I'm just getting:

warning: ignoring untrusted substituter ''


strange, according to this: you should get a different error message


(same issue / PR)


@U04V15CAJ sorry, I'm out of ideas, it seems that somehow you are still on an older version. At least you got flakes enabled by default

👍 2

to summarize, I confirmed that if you want to get the --accept-flake-config option working, you must have in /etc/nix/nix.conf a line with trusted-users = root my_user_name It's not possible to pass that option in the cli, must be defined in nix.conf (and you need to restart the nix daemon to pick the changes)


$ cat /etc/nix/nix.conf

build-users-group = nixbld
trusted-users = root my_user_name


sudo launchctl stop org.nixos.nix-daemon
$ nix run --accept-flake-config github:jlesquembre/clj-nix/bb-loom#bb-loom -- /tmp/loom.clj
warning: ignoring untrusted substituter ''
error: interrupted by the user


yep, that looks good, assuming my_user_name is not your real user name 😉 I don't have the build-users-group option, maybe that's causing problems? Wait, if you stopped the service, the nix command should not work at all


facepalm - changing that to borkdude worked :)

🎉 2


$ nix run --accept-flake-config github:jlesquembre/clj-nix/bb-loom#bb-loom -- /tmp/loom.clj
Blazingly Fast! {:sum 4999950000, :time-ms 1690}


How does this work... since there must be some other machine compiling for arm64 mac?


right, I got access to a mac build machine, and I uploaded both versions to the cache


$ nix run --accept-flake-config github:jlesquembre/clj-nix/bb-loom#bb-loom -- -e '(System/getProperty "os.arch")'


indeed, now I can do nix build --system aarch64-darwin ... on my linux machine 🙂

🤯 2

is this mac build machine owned by the nix foundation or something?


it's owned by my current company, but I know the nix foundation also have mac builders


Is there a way to enter a nix shell with nix run --accept-flake-config github:jlesquembre/clj-nix/bb-loom#bb-loom so the startup of bb is immediate?


what you mean? If I run that command I get a bb shell


I mean, a normal shell where I can run bb whatever


oh, sure, with nix shell --accept-flake-config github:jlesquembre/clj-nix/bb-loom#bb-loom


well, all good then :)


thanks again for the help


sure, it was fun to get babahska with loom compiled for mac 🙂

Tomas Brejla18:02:21

Just tried the Babashka v1.1.174-SNAPSHOT REPL on my Samsung Galaxy Tab S7 FE tablet, running termux & code-server. Native threads work like a charm {:sum 4999950000 :time-ms 2788} Wow. Just wow!

🎉 2
Tomas Brejla17:03:40

using naive bench method that we tried before against mina

wrk -d 15 -c 512 -t 8 --latency \
      -H 'Accept: text/plain,text/html;q=0.9,application/xhtml+xml;q=0.9,application/xml;q=0.8,*/*;q=0.7' \
      -H 'Connection: keep-alive' \
      -s pipeline.lua
virtual threads: 269711.12 Requests/sec: 268898.90 regular threads: Requests/sec: 292119.90 So virtual threads seem to be a bit slower then regular ones. I guess that where virtual threads will shine will be a situation where massive number of concurrent requests access the server. Ie. where regular threads would eat your heap and explode on OOM.

👍 2