Clojurians Log v2

Not sure if anyone has ever used an s3 dep in bb? If so we could maybe fix that too in another round, but personally I haven't used this much

bb native tools deps support could be interesting for Polylith. I think the deps resolution is one of the blockers for porting it (partially) to babashka

> I think the deps resolution is one of the blockers how so?

Maybe that desire is only mine though. Just think the boot time could be alot less for many of Polylith tasks. I ported parts of it to babashka before, but that was via monkey patching

you can use the tools-deps-native pod right now as well to get tools.deps working in bb (experimental) but after merging this branch you won't need to

@borkdude https://github.com/polyfy/polylith/blob/master/components/deps/deps.edn

I've wished to share Babashka scripts with friends who don't buy into Clojure, and they have asked why they need a JVM and Clojure when it was supposedly just a script. So my feeling is gratitide! ❤️ babashka ☺️

I would LOVE resolving deps without a jvm!! I share a lot of bb scripts with coworkers, and I've gotten "ew what is this java stuff" messages many times. Most of em don't even have java installed! Uberscripts, which are incredibly cool, have mostly solved the problem. But libraries with assets can't be uberscripted, and I would be very stoked to be able to share portal-powered scripts

Great reasons: co-workers. For @jhane’s with assets you can also make an uberjar or even a self-contained binary https://github.com/babashka/babashka/wiki/Self-contained-executable but I agree that not having to do that dance is better.

Yup, no build step is a great part of working with Babashka.

For ClojureCLR, I am using babashka to resolve and unzip deps according to @pez https://blog.agical.se/en/posts/how-to-create-a-really-simple-clojureclr-dependency-tool/. So this disentangles my CLR setup from the JVM, which is great.

So since this is co-worker driven as well, perhaps (babashka.deps/add-deps ..) and/or bb.edn should have a flag to say: use the JVM-less resolver too (until that's going to be the default after more testing). Having them set an env var isn't good UX for them either

It would be nice if this could be part of the work needed to make procurers in tools.deps pluggable, so the Maven and S3 code could be separated out in a way that allowed for 3rd party implementations for non-JVM implementations. /cc @alexmiller That's been talked about for years, and with the spurt of non-JVM Clojure variants, having ways to support Maven etc without needing the JVM would be great. And finishing off the pluggable procurer work would probably make it easier for, say, ClojureCLR to support nuget deps more easily?

Yes, making the :mvn procurer pluggable is exactly the only spot I had to change :)

the s3 extensions are actually separated out into org.clojure/tools.deps.maven-s3-transporter (which I did during the MIMA adoption) - it's dynamically loaded if available and ignored if not (it has actually always been dynamic to allow tools.deps to be used as a git lib without compilation of the Java parts of the s3 impl). The procurers are already pluggable - there is no additional work to be done in tools.deps. The piece that's not available is pluggability in the CLI which doesn't provide hooks for it.

@borkdude what did you need to change?

Ah, so it's programmatically pluggable, but just not accessible via the CLI? I thought there was more work needed in that lower-level, so that's great.

To be more precise, yes, it's pluggable and I'm using that pluggability but with some minor changes more stuff can be re-used. I'll do a more through investigation/overview later , will let you know.

So let me be more precise. clojure.tools.deps (the .clj) loads the default mvn and pom extensions eagerly, at the bottom of the file. Their ns forms import Aether classes, so without maven-resolver on the classpath, requiring clojure.tools.deps itself fails at load, before any custom procurer has the chance to register. You can shadow those namespaces with your own on the classpath, which is what bb does, but that relies on classpath order and is brittle. So a pluggable mvn procurer as a plain JVM lib, a drop-in replacement for the built-in one, cannot work without also bringing maven-resolver as a dep. Is that right, or am I missing a hook? (there's more but let's start with this) @alexmiller

the mvn and pom extensions could perhaps be conditionally loaded based on the presence of Aether.

that is what I expected your answer to be :) perhaps the auto-loading of procurers could be configurable. it's very convenient when used as a Clojure JVM lib, but I can see how it would be problematic if you want something different

I think the conditionalizing on Aether would fix my problem, but not sure if it would be a more general solution The other issue is that I had to slightly modify the :local/root extension since it loads the maven util namespace. All of these are no big issues, very workaround-able.

Logged https://clojure.atlassian.net/browse/TDEPS-284 - Make auto-loading of procurers configurable

On the local extension - that stuff is minimally need to support local jars with embedded poms. is that a use case you're trying to still cover? that stuff could be more dynamically loaded, or split up differently

logged as https://clojure.atlassian.net/browse/TDEPS-285

Yes, I support it, with a mild re-implementation :)

when I next get a chance, I'll look at these

well, it depends on util maven too, that can't be made portable without an override perhaps coord-deps :jar could be moved to the pom extension the rest in local is just jar file stuff without maven

Removing the JVM dependency for fetching deps would be nice for bbin since it means the CLI and the installed scripts would also no longer require a JVM

ok, lots of benefits I guess ;)

bb edn for a bb-compatible deps-deploy library

{:paths ["src" "test" "."]
 :deps {io.github.clojure/tools.build {:mvn/version "0.10.14"}}
 :tasks
 {test {:doc "Runs the tests with bb"
        :requires ([clojure.test :as t])
        :task (do (require 'babashka.deps-deploy-test)
                  (let [{:keys [fail error]} (t/run-tests 'babashka.deps-deploy-test)]
                    (System/exit (+ fail error))))}
  clean {:exec-fn build/clean}
  jar {:exec-fn build/jar}
  install {:exec-fn build/install}
  deploy {:exec-fn build/deploy}}}

the build.clj is the same as you would use with T:build

Coords for that deps-deploy seem missing?

this IS that deps-deploy lib :)

I've yet to deploy it ;)

Ah, okay

$ bb deploy
Deploying io.github.babashka/deps-deploy-0.0.1 to clojars as borkdude

I consulted with @slipset to see if he was interested in making slipset/deps-deploy bb compatible but his reply was: what if current users just migrated to yours? 😆

Looks great! I would certainly switch if it means bb can run all my build tasks!

I guess it depends what other libs you're using in build.clj but the default tools.build + "my" deps-deploy now works

with the latest dev build that is :)

I'm going to flip the switch to resolver = bb so @markus.agwin doesn't have to guess if this env var works and I don't have to add any printlns. It's going to be opt-out, else I think we'll never get any test coverage

Thank you, it's just that I fooled myself way too often into early victory-declaration which also is bad for everyone else.

The new dev build is up. It should now resolve dependencies without setting any env var. So the new resolution is opt-out

AND it should run tools.build. for deploying deps you can use https://github.com/babashka/deps-deploy. I'm going to test this in my own libraries :)

@seancorfield you thought depstar was dead ... right? But it was still used in the built-in bb uberjar task.... which is there to make uberjars for bb programs itself. But I'll move over to tools.build now, so it will be now completely extinct. ;)

New dev build works! This is huuuuge!

:-D

Just trying out HoneySQL's build.clj with bb (dev) and building a JAR works as expected. Thank you!

what spec? when I implemented nrepl there was barely a spec :) interrupt doesn't work in bb due to Thread/stop not being supported in JVMs anymore

or did fable manage to get this working?

yeah sure I'm interested in getting more of nrepl to work, but please one by one focused issues/PRs and not one giant blob of slop :)

I will work on some targetted PRs, didn't want to just throw anything at you without checking first. Thanks.

Thank you!

I asked it to explain how it implemented the interrupt:

The spike's shape is exactly the Tier‑1 change from the assessment: a per-session eval thread fed by a queue, with the socket reader never blocking on eval. interrupt then becomes ~15 lines: .interrupt → wait on the eval's promise → interrupted | session-idle | (after timeout) abandon+replace, reporting #{"done" "interrupted"} vs. a distinct status so the client can tell the difference.

One possible bb-only advantage worth a follow-up spike: because sci interprets, it could poll Thread/interrupted at loop/recur or fn-call boundaries and throw — giving bb better interrupt semantics than the JVM gets. That'd be a sci change (opt-in flag, small overhead), not a babashka.nrepl change. Worth raising with borkdude if the tight-loop case matters to him.

fable is a big fan btw, even just mentioning your nick elevates the clojure code it outputs.

haha. .interrupt really doesn't get your proper interrupts though, excdept when you do Thread/sleep

Yes it calls that out, it only makes babashka.nrepl the same semantics as the nrepl reference implementation, it's solution is to abandon the thread and spawn a fresh eval thread for the session.

hm, that still leaves your CPU hanging at 100% though

sci has :interrupt-fn now. babashka.nrepl can use that with a pure clojure solution. It won't work for pure host calls, which is where the abandon+replace is the fallback. So nrepl-server could create the sci ctx with :interrupt-fn as long as it's present at sci/init time.

it could but you would sacrifice performance

It would make the nrepl loop about 5% slower without changes to sci though

3. The 5% can be driven toward zero in sci if borkdude cares. Currently the hot path is (when-not (nil? interrupt-fn) (interrupt-fn)) — an indirect call per iteration. Two cheap refinements:

A static volatile boolean interruptRequested checked before the call; the interrupter sets it. Common path becomes a single load‑acquire with no call — sub‑nanosecond on ARM.
Or check only every N iterations via a thread‑local counter. Loses ~N×20 ns of interrupt latency, i.e. nothing.
Both are pure Clojure in sci, no bb/nrepl changes. Given the hook is already marked experimental, that's a reasonable ask alongside the nREPL integration — but even as‑is, 5% on a synthetic tight loop with zero cost elsewhere is well within "ship it" territory for a REPL server where the alternative is an uninterruptible session.

it also needs all included libraries like clojure.core etc to be intstrumented for it to fully work though

It should probably be an option to the nrepl server, I wouldn't turn it on by default

then it sometimes works... not always, depends if you're in interpreted code or not

This is much more complicated than I imagined. I can actually do what I need without interrupt, because I use statecharts on both sides of the connection and am just using nrepl to pass EDN across the wire, I have an event loop on both sides of the connection and can interrupt there, though it has the same semantics as the JVM where you can't really interrupt the compute, I can catch it and restart the process from the statechart to clear the zombie. I still need EDN transport, stdin, and isolated sessions though.

Thanks for the attention on this, I will work on some small incremental PRs for this so it is about the code and not the concepts.

I'm thinking of redesigning the nREPL server such that it can load nREPL middleware from user source. With the current design (based on transducers) that doesn't work. Possibly just a new implementation that isn't baked into the image (except for bencode.core etc which is already) but bundled as source code. Something that has been on my mind for long

I really like the full nrepl in the jvm, it's like tmux for clojure, and let me ship arbitrary compute into a container with no network needed (unix socket). I started exploring doing it with babashka for the fast startup, which would let me make the containers more ephemeral, but hit the implementation of babashka.nrepl only being enough nrepl for editors. With UNIX sockets, edn transport, isolated sessions, and stdin, it would be nearly perfect, a babashka container image or single binary, along with a tiny uberjar would give me a universal solution that could run under most sandbox systems out there.

yeah. there's a unix domain socket PR up for nREPL I think

isolated sessions is probably a small change

edn transport probably as well

etc :)

Yes my tests with AI solved everything cleanly except interrupt, it just made that work the same as jvm nrepl.

we can include interrupt to only work for Thread/interuped for now I guess, at least we have the op working then and can refine later

Doing the PR for interrupt shows that sessions really should be isolated first. Right now every connection gets the same session, so interrupt would require a connection keyed atom outside the sessions atom. If bindings become data, like the jvm nrepl handles things, then interrupt for Thread/interrupted is a pretty small change. Full session support means some core changes to architecture. Like moving bindings into the sessions map.

Bindings as data. Session holds {var → value}; eval takes the map, sci/with-bindings, captures the values back; completions/lookup run on the reader thread with the same map. This is what unlocks tooling ops not stalling behind eval, and what nREPL does. But there's a real constraint: sci has no public get-thread-bindings, so capture means a known var set (sci/ns, *1..*3, *e, :thread-bind) — which is actually what the code already assumes, so it's not a regression, but it's a design decision borkdude will want to weigh in on.

To me writing up an issue with a good problem description / specs / links to resources is WAY more important than a code PR btw, I have claude too ;)

A code PR is still useful to compare various approaches

but the issue > PR

I'm going to fix "all the problems" with bb nrepl by incorporating nrepl.core itself into bb. so I'm not going to merge your stuff, but it'll fix your problems. also edn transport

Thank you, I am using nrepl in clojure for this already, and the "all the problems" quote was not meant to be insulting, the problems exist for my use case which is not standard. babashka.nrepl works great for editors.

oh I didn't even check if you said "all the problems", that quote was just a handwavy summary of mine :)

I certainly meant no disrespect in asking for the changes, so I hope that you did not take it as such. I have nothing but respect for your work ethic and the tools you build. Thank you for the attention on this.

no disrespect perceived at all!

My quote was just a take on "all the things"

Maybe you can give the dev build a spin in 10 minutes from now.

bash <(curl ) --dev-build --dir /tmp

(10 minutes or whenever master CI finishes)

I tested it, everything works for my use case. Nice bonus:

boot → hello → EOF → exit: 0.27s under bb vs 3.9s under the JVM (14×)

nice

The sandbox is now bb + sandbox.jar which is 1.2MB and has sub second start times. It is working for 4 different container systems and seatbelt on macos.

Awesome upgrade to bb! Thank you.

1 second seems a bit slow tbh :) is this loading a lot of extra code?

No it's 0.27s currently.

ah