Fork me on GitHub
Slack Archive

This page is not created by, affiliated with, or supported by Slack Technologies, Inc.

2023-02-20

Channels

#babashka
<
2023-02-20
>
Matthew Twomey03:02:30

I am moving a few frequently used functions into a library. I use this library with babashka scripts and it works great. I just ran into a little issue and I’m sure I’m just missing something simple. I have some functions in my library that use babashka.process , however when I call these functions it seems prevent my tests from exiting. As a simplified example: 

(ns test2
  (:require
    [babashka.process :refer [shell]]))

(println "* Start *")
(print (:out (shell {:out :string :err :string} "ls -las")))
(println "* Finish *")
When I run this directly with clj -M test2.clj it “hangs” and won’t exit (it does however print out everything - including the * Finish *. If I comment out that middle line, it prints start and finish, then exits successfully. What am I missing here?

Matthew Twomey06:02:56

Oh I see, I need (shutdown agents) at the end. I was completely unaware of this.

borkdude09:02:22

That should only be necessary in JVM Clojure

Matthew Twomey16:02:22

yes correct, that was my experience ty.

borkdude09:02:03

POLL babashka What do people think here about adopting GraalVM 22.3.1 JDK 19.0.2 for the next version of babashka with loom support? I have a https://github.com/babashka/babashka/tree/jdk19-loom where all tests are currently passing. We'll have to move off JDK 11 for the next GraalVM release anyway since 11 will no longer be supported in 23.0.0. All available options: 1. Move to JDK 19 now and then followed by alternative 2. 2. Move to JDK 20 in April 2023 and then move to JDK 21 LTS in Sep 2023. 3. Move to 17 LTS now and then to 21 LTS in Sep 2023. You can vote by clicking on one of the options below (and comment in thread).

1️⃣ 24
2️⃣ 1
3️⃣ 7
borkdude09:02:32

@UKFSJSM38 I also wonder what implications it would have for nix builds if we moved to graalvm jdk 19

ericdallo13:02:50

I think we latest @UFDRD93RR changes to the nix graalvm derivation should be easy to support that

kokada13:02:24

Why JDK19 instead of 17?

kokada13:02:28

Just curious

✅ 1
kokada13:02:37

JDK 17 is LTS while 19 is not AFAIK

borkdude13:02:44

> with loom support

borkdude13:02:54

virtual threads

kokada13:02:17

From nixpkgs perspective I would prefer to not have to add JDK 19 to nixpkgs just because of babashka

kokada13:02:07

We could though, and it should be hugely difficult

kokada13:02:17

But if babashka builds with JDK17 would be preferred

borkdude13:02:40

but with flakes it would be possible right

kokada13:02:34

The issue is not how to support it

kokada13:02:39

It will work

kokada13:02:51

Just it is too much maintanance overhead

kokada13:02:08

We will need to keep GraalVM 19 until EOL if we added it to nixpkgs

borkdude13:02:09

I could set up bb to be able to build with both 19 and 17 so nix users would just not get the virtual thread support yet, maybe a reasonable compromise

kokada13:02:29

This looks good for me

kokada13:02:51

We will probably be an early adopter of GraalVM 21

kokada13:02:01

If everything went well at least

borkdude13:02:30

I plan to stick with 21 LTS once it comes out (unless there's a compelling reason to move again)

borkdude13:02:40

so then we will be in sync in 6 months again

lread14:02:07

Would JDK19 preview features be enabled in bb by default or would they require some sort of --enable-preview switch?

borkdude14:02:54

enabled by default

jkxyz15:02:44

It would kinda suck for the nixpkgs version to be a downgrade from the official release, and I guess it would be a problem for other distributors that don't want to build every JDK/Graal version. What about having a separate "preview" build of babashka that runs on JDK 19 with preview features enabled? That could be provided as a Nix flake with an overlay for those who want the latest JDK. And the core version could be built on the stable LTS release for everyone.

Daniel Gerson15:02:51

I voted for JDK 19 only because it doesn't really affect me 😅. However I imagine at some point something like sdkman would be preferable for users that require particular versions? It's what I use for JDK on android development etc. Problems that come from being too popular?!

borkdude15:02:01

> It would kinda suck for the nixpkgs version to be a downgrade from the official release This will only be for 6 months though

borkdude15:02:03

nix users who really want to get parity, can always download the static linux binary as well

lread15:02:46

I voted 3 because I personally don't have any pressing need for JDK19 preview features. I can wait. But I expect folks are voting 1 because they don't want to wait, they want to use these features now. Is that the main benefit? Taking advantage of preview features now?

borkdude15:02:06

it's also an extra maintenance burden to keep the jdk19-loom branch up to date

borkdude15:02:54

option 1 should not affect anyone who doesn't want to use the virtual thread stuff yet

lread15:02:42

Yah, personally fine with option 1 as well. I guess the only con is that preview features can have breaking changes. But folks using these features are very likely aware of that.

jkxyz15:02:42

Well I think it's also notable that to enable the preview features in Java you also have to pass a command-line flag (right?) So it's a pretty explicit opt-in. But yeah I imagine people know that they're using unstable features.

jose13:02:20

Do you have a gist of using loom with babashka?

borkdude15:02:50

I think we have enough votes, it's about 77% for alternative 1 vs 23% for alternative 3. This means I'll go ahead and merge the jdk19-loom branch - yay, no more maintaining a branch!

🎉 12
borkdude15:02:56

You should now be able to test this binary with: 

bash <(curl ) --dev-build --dir /tmp
and e.g. execute this snippet: https://clojurians.slack.com/archives/CLX41ASCS/p1666722786342499

littleli17:02:25

In a way this is a remarkable thing. I'm not aware of an interpretted language which could effectively use the computational power of multiple-real threads. Now with these virtual threads, it's even more awesome since they are cheap and you can solve most if not all concurrency with them. Stunning thing, thank you @U04V15CAJ

jose17:02:02

And I think I got it working with nix: 

nix run --accept-flake-config github:jlesquembre/clj-nix/bb-loom#bb-loom -- virtual.clj
The --accept-flake-config option is there to use the binary cache and avoid re-compile it

borkdude18:02:11

@U4NGX0FHN awesome! I wonder how much disk space this will cost me :) 

[0/1475 built, 1/344/347 copied (2851.5/3264.3 MiB), 417.5/610.1 MiB DL] fetching zulu17.34.19-ca-jdk-17.0.3 from

jose19:02:37

All of it... Now seriously, if you used the accept-flake-config option, with a recent nix version it should use the binary cache, and don't download the build dependencies. Maybe I did a mistake, I'll check tomorrow. You are on a Mac, right?

borkdude21:02:47

I'm on a mac yes

borkdude21:02:51

what is recent nix?

borkdude21:02:01

$ nix --version
nix (Nix) 2.9.1

jose14:02:49

sorry, forget about the "recent version", if flakes already work, it's recent enough. I'm not sure why it's downloading jdk-17, with the --accept-flake-config should the binaries from cachix. It's still a big download, because it depends on graalvm at runtime dependency, but it should not download jdk-17

borkdude15:02:51

The full command I executed was: 

nix --extra-experimental-features nix-command --extra-experimental-features flakes run --accept-flake-config github:jlesquembre/clj-nix/bb-loom#bb-loom -- /tmp/loom.clj
but I didn't wait for it to finish because I suspected it would take hours

jose15:02:17

running that command, did you get a warning: ignoring untrusted substituter '' warning?

jose15:02:35

btw, you can put the nix options, like extra-experimental-features in the nix.conf file: https://nixos.org/manual/nix/stable/command-ref/conf-file.html

borkdude15:02:05

I tried, but /etc/whatever.conf wasn't writable

borkdude15:02:46

I also tried ~/.config/nix/nix.conf : 

nix.settings.experimental-features = [ "nix-command" "flakes" ];
but this gave me error messages

jose15:02:57

I'm not a mac user, but can't you edit /etc/nix/nix.conf with sudo?

borkdude15:02:18

you can but it seemed a bit weird to me that I would have to do that

borkdude15:02:06

with that user conf file set: 

$ nix run --accept-flake-config github:jlesquembre/clj-nix/bb-loom#bb-loom -- /tmp/loom.clj
warning: unknown setting 'nix.settings.experimental-features'
error: experimental Nix feature 'nix-command' is disabled; use '--extra-experimental-features nix-command' to override

jose15:02:43

ok, for reference, this is my (partial) /etc/nix/nix.conf: 

allowed-users = *
trusted-users = root jlle
experimental-features = nix-command flakes
builders-use-substitutes = true

borkdude15:02:37

ah: experimental-features = nix-command flakes works

jose15:02:47

you installed the nix daemon, right? I think after updating it you need to restart the service, on linux is systemctl , not sure on mac

borkdude15:02:55

still getting: 

warning: ignoring untrusted substituter ''

jose15:02:13

trusted-users = root borkdude

jose15:02:23

or your username

borkdude15:02:55

I did that +: 

sudo launchctl stop org.nixos.nix-daemon;
sudo launchctl start org.nixos.nix-daemon;
but still getting the same message

jose15:02:55

um, maybe you need to login again? what if you put this: 

substituters =  
trusted-public-keys = cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= clj-nix.cachix.org-1:ftfCDUxeGYYoD9MSQARsBj5WyViAh2K/5Dq8nIDH150=

borkdude16:02:17

then I get the message three times: 

warning: ignoring untrusted substituter ''
warning: ignoring untrusted substituter ''
warning: ignoring untrusted substituter ''
:)

jose16:02:21

I'm going to ask some one else

borkdude16:02:55

it's not so important to me, since I normally don't use nix a lot, I was just messing around so if this is time consuming for you, we can just leave it. but perhaps someone who is using nix on a daily basis could try, like @UKFSJSM38

borkdude16:02:25

@UKFSJSM38 This is the command I use to run @U4NGX0FHN’s nix version of bb + loom support: 

nix run --accept-flake-config github:jlesquembre/clj-nix/bb-loom#bb-loom -- /tmp/loom.clj

jose16:02:26

@U04V15CAJ no worries, I'm curious to know what's going on

ericdallo16:02:46

worked for me without any warnings NixOS running nix 2.12.0

👍 2
borkdude16:02:35

I upgraded nix now: 

$ nix --version
nix (Nix) 2.13.2
Still got the same problem :)

jose16:02:35

@U04V15CAJ if you can update your nix version, you should get a better error about why is ignoring the substituter

jose16:02:05

and still the same error message?

borkdude16:02:12

I'm just getting: 

warning: ignoring untrusted substituter ''

jose16:02:43

strange, according to this: https://github.com/NixOS/nix/pull/7461 you should get a different error message

borkdude16:02:30

(same issue / PR)

jose16:02:37

@U04V15CAJ sorry, I'm out of ideas, it seems that somehow you are still on an older version. At least you got flakes enabled by default

👍 1
jose16:02:14

to summarize, I confirmed that if you want to get the --accept-flake-config option working, you must have in /etc/nix/nix.conf a line with trusted-users = root my_user_name It's not possible to pass that option in the cli, must be defined in nix.conf (and you need to restart the nix daemon to pick the changes)

borkdude16:02:36

$ cat /etc/nix/nix.conf

build-users-group = nixbld
trusted-users = root my_user_name

borkdude16:02:59

sudo launchctl stop org.nixos.nix-daemon
$ nix run --accept-flake-config github:jlesquembre/clj-nix/bb-loom#bb-loom -- /tmp/loom.clj
warning: ignoring untrusted substituter ''
error: interrupted by the user

jose16:02:31

yep, that looks good, assuming my_user_name is not your real user name 😉 I don't have the build-users-group option, maybe that's causing problems? Wait, if you stopped the service, the nix command should not work at all

borkdude16:02:11

facepalm - changing that to borkdude worked :)

🎉 1
borkdude16:02:37

Works! 

$ nix run --accept-flake-config github:jlesquembre/clj-nix/bb-loom#bb-loom -- /tmp/loom.clj
Blazingly Fast! {:sum 4999950000, :time-ms 1690}

borkdude16:02:05

How does this work... since there must be some other machine compiling for arm64 mac?

jose16:02:44

right, I got access to a mac build machine, and I uploaded both versions to the cache

borkdude16:02:31

$ nix run --accept-flake-config github:jlesquembre/clj-nix/bb-loom#bb-loom -- -e '(System/getProperty "os.arch")'
"aarch64"

jose16:02:06

indeed, now I can do nix build --system aarch64-darwin ... on my linux machine 🙂

🤯 1
borkdude16:02:42

is this mac build machine owned by the nix foundation or something?

jose16:02:47

it's owned by my current company, but I know the nix foundation also have mac builders

borkdude17:02:03

Is there a way to enter a nix shell with nix run --accept-flake-config github:jlesquembre/clj-nix/bb-loom#bb-loom so the startup of bb is immediate?

jose17:02:07

what you mean? If I run that command I get a bb shell

borkdude17:02:22

I mean, a normal shell where I can run bb whatever

jose17:02:00

oh, sure, with nix shell --accept-flake-config github:jlesquembre/clj-nix/bb-loom#bb-loom

borkdude17:02:29

well, all good then :)

borkdude17:02:34

thanks again for the help

jose17:02:15

sure, it was fun to get babahska with loom compiled for mac 🙂

Tomas Brejla18:02:21

Just tried the Babashka v1.1.174-SNAPSHOT REPL on my Samsung Galaxy Tab S7 FE tablet, running termux & code-server. Native threads work like a charm {:sum 4999950000 :time-ms 2788} Wow. Just wow!

🎉 1
Tomas Brejla17:03:40

using naive bench method that we tried before against mina 

wrk -d 15 -c 512 -t 8 --latency \
      -H 'Accept: text/plain,text/html;q=0.9,application/xhtml+xml;q=0.9,application/xml;q=0.8,*/*;q=0.7' \
      -H 'Connection: keep-alive' \
       \
      -s pipeline.lua
virtual threads: 269711.12 Requests/sec: 268898.90 regular threads: Requests/sec: 292119.90 So virtual threads seem to be a bit slower then regular ones. I guess that where virtual threads will shine will be a situation where massive number of concurrent requests access the server. Ie. where regular threads would eat your heap and explode on OOM.

👍 1