Clojurians Log v2

I needed to make changes to rebel's source code a bit, and added some classes to bb too, but now I have it working for the first time.

that's awesome! I ended up not looking at it yesterday with the bb support for sqlatom, but I did try the new bb repl and indeed found it very similar

I still need to go through a lot of things and clean things up though

• SCI: deftype now macroexpands to deftype*, matching JVM Clojure, enabling code walkers like riddley • SCI: case now macroexpands to JVM-compatible case* format, enabling tools like riddley and cloverage • SCI: macroexpand-1 now accepts an optional env map as first argument, enabling riddley compatibility • SCI: macroexpand-1 of (.method ClassName) now wraps class targets in identity, matching Clojure behavior • SCI: macroexpand-1 now expands (ClassName. args) to (new ClassName args), matching JVM Clojure • SCI: support functional interface adaptation for instance targets (e.g. (let [^Predicate p even?] (.test p 42))) • SCI: infer type tags from let binding values to binding names • SCI: fix .method on class objects routing to static instead of instance method path • SCI: fix read with nil or false as eof-value throwing instead of returning the eof-value • SCI: fix letfn with duplicate function names crashing with ClassCastException • SCI: fix ns-map not reflecting vars that shadow referred vars • SCI: preserve :tag metadata in copy-var • SCI: fix NPE in resolve when :outer-idens is nil • Support multiple catch i.c.m. ^:sci/error • Fix satisfies? on protocol on proxy • https://github.com/babashka/babashka/issues/1923: support reify with java.time.temporal.TemporalQuery • Fix reify with methods returning int/`short`/`byte`/`float` when Clojure fn returns long/`double` • Fix https://github.com/babashka/babashka.nrepl/issues/71: nREPL server now uses non-daemon threads so the process stays alive without @(promise) • Add clojure.test.junit as built-in source namespace • Add JLine reify support: Parser, Completer, Highlighter, Widget, ParsedLine • Add JLine classes for rebel-readline compatibility: LineReader$Option, Attributes$InputFlag, Attributes$LocalFlag • Add cp437 (IBM437) charset support in native binary via selective GraalVM charset Feature (see https://github.com/babashka/babashka/blob/master/doc/adr/0006-selective-charset-support/decision.md) • Add java.lang.ref.SoftReference • Add java.lang.reflect.Field methods: setAccessible, get, set • https://github.com/babashka/babashka/issues/1919: add java.nio.file.attribute.UserPrincipal and GroupPrincipal • https://github.com/babashka/babashka/issues/1920: add java.nio.file.FileSystemNotFoundException • Bump deps.clj • Bump fs • Bump transit-clj to 1.1.347 • Bump Selmer to 1.13.1

exciting release @borkdude!

congrats on figuring out deftype! that's really big

deftype is still restricted to a few selected patterns, like creating custom map types. it's not fully supported in general

whoa the jline include might mean https://github.com/bhauman/rebel-readline now works in bb

@filipematossilva almost. there are few things I need to address for this. you can find more about that here: https://github.com/babashka/babashka/blob/9b70c1fe672898171b19c84f5ad59edc549b04f1/doc/adr/0001-jline-providers/decision.md#rebel-readline-compatibility the current console repl of bb itself looks a lot like rebel now though

oh I have to try it out then, I had to do some really gnarly process wrangling to get rebel-readline working on invoker for spawned babashka processes

does the bb console repl also work in clj?

I think I could make it work ;)

but then it would be a library which probably would look a lot like rebel

what's stopping it from working right now though?

what gnarly thing did you have to do to get rebel-readline working for spawned bb processes. shell should imo work out of the box and you could also uses exec to hand of process execution to rebel

I just posted a link with the details?

no sorry, I meant "what's stopping the bb console repl from working in clj"

at least I thought the link was about rebel not working yet

it's just being hard-couped to bb's console REPL stuff

and SCI stuff probably

got it

https://github.com/babashka/babashka/blob/master/src/babashka/impl/repl.clj lots of "impl" there. I could extract this out into a library but the autocompletion is also SCI specific here

the gnarly stuff I did with rebel had to do with how invoker will spawn either a bb or clj process with a nrepl server

but rebel is clj only, and wants to own the parent process

I think due to how jline works

ideally I'd be able to spawn a bb process with a nrepl server, and spawn a clj process with rebel connected to the other one

but instead I needed to replace the parent process with the clj process using exec, then spawn a bb process from within the rebel one

I also had to do some weird stuff to support ctrl+c/ctrl+d

rebel does stuff with them, but by default they are also sent to child processes automatically

I think I can make rebel work with bb

from source

I just need to put in some extra support for proxy-super in SCI

and rebel should not use some .impl classes in the non-dev code where it can

is either of those things something I could help with?

on the rebel side I could make a PR that removes some .impl classes

also just wanted to mention how delightful it is that you went the extra mile to make charm work, I was looking at it a while ago and was sad to see it used jline and thus wouldn't work in bb, but now... 😄

yeah on the rebel side help to avoid those impl classes that the link mentions would be good :)

So just: • org.jline.reader.impl.BufferImpl • org.jline.terminal.impl.DumbTerminal

the document gives hints how to avoid those

roger, will take a look and keep you posted

(= "dumb" (.getType...)) is one way to get rid of the DumbTerminal

BufferImpl is only used in dev mode

I'm also looking with Claude at it. It seems the impl.DefaultParser can also be avoided

see https://github.com/borkdude/rebel-readline/tree/babashka

tests still work with these changes

@filipematossilva boy, adding support for proxy-super was super easy, I didn't realize that

ooops, spoke too soon ;)

you jinxed it 😄

wait you have bb support already? ;)

I can do IAtom2 with reify, but can't do the other two so in bb these don't work: add-watch/remove-watch/set-validator!/get-validator/meta/alter-meta!/reset-meta!

how important are these for the core features in your lib?

The validator can still be set with an option, I don't think the atom metadata is very used, but there's no replacement for watchers.

I guess polling it

Guess is could add an option to add the watcher when creating the atom too

I guess we could add support for the general "custom atom" story in deftype like I did for the "custom map" story

assuming you are using deftype

I'm not, I think at some point I tried to but had problems with 2 interfaces implementing Meta or something

But it's no problem to use a different mechanism for bb and another for clj, doing that already with reify/proxy

can you link me to the clojure code that uses deftype

Sure, gimme a sec not at a computer atm

no hurry

can't repro whatever issue I had with meta anymore, here's what a full custom atom impl with deftype in clojure looks like

can you attach this file to the issue and rename the issue to something like "implementing custom atoms in babashka" or so

roger, will make it deftest as well if you wanna use them

https://github.com/babashka/babashka/issues/1931#issuecomment-3919957762

@john wrt perf, added a perf test that resets! and swap! a 20mb edn datascript backup:

filipesilva@m4 ~/r/p/sqlatom (master) [1]> clj test/performance_test.clj
WARNING: Implicit use of clojure.main with options is deprecated, use -M test/performance_test.clj
Reading test/roam-book-club-2026-02-18-11-31-58.edn ...
  File size: 20.3 MB
  EDN parse: 456 ms
  Top-level keys: (:schema :datoms)

reset!: 314 ms
swap!:  649 ms

----- Error --------------------------------------------------------------------
Type:     java.lang.RuntimeException
Message:  com.fasterxml.jackson.core.exc.StreamConstraintsException: String value length (20051112) exceeds the maximum allowed (20000000, from `StreamReadConstraints.getMaxStringLength()`)
Location: /Users/filipesilva/repos/personal/sqlatom/src/filipesilva/sqlatom.cljc:59:3

----- Context ------------------------------------------------------------------
55:              (set-params! stmt (rest sql-params))
56:              (.executeUpdate stmt))))
57:
58: (defn- sql-query [conn sql-params]
59:   #?(:bb  (sqlite/query conn sql-params)
      ^--- com.fasterxml.jackson.core.exc.StreamConstraintsException: String value length (20051112) exceeds the maximum allowed (20000000, from `StreamReadConstraints.getMaxStringLength()`)
60:      :clj (with-open [stmt (.prepareStatement conn (first sql-params))]
61:              (set-params! stmt (rest sql-params))
62:              (with-open [rs (.executeQuery stmt)]
63:                (resultset->maps rs)))))
64:

It's a limit set by Jackson. You can find this in Cheshire github issues

got it https://github.com/dakrone/cheshire/issues/210

looks like I can change the limit, I guess I'll... 10x them

hm tried setting the factory with custom limits but it didn't seem to work

67: (defn- sql-query [conn sql-params]
68:   #?(:bb  (binding [factory/*json-factory* pod-factory]
69:             (sqlite/query conn sql-params))
                ^--- com.fasterxml.jackson.core.exc.StreamConstraintsException: String value length (20051112) exceeds the maximum allowed (20000000, from `StreamReadConstraints.getMaxStringLength()`)
70:      :clj (with-open [stmt (.prepareStatement conn (first sql-params))]
71:              (set-params! stmt (rest sql-params))
72:              (with-open [rs (.executeQuery stmt)]
73:                (resultset->maps rs)))))
74:

----- Stack trace --------------------------------------------------------------
com.cognitect.transit.impl.ReaderFactory/ReaderImpl - <built-in>
cognitect.transit/read                              - <built-in>
babashka.pods.impl/transit-json-read                - <built-in>
babashka.pods.impl/processor/fn--31617              - <built-in>
babashka.pods.impl/processor/fn--31646              - <built-in>

@filipematossilva The JSON limit you're hitting here is with the transit communication back and forth between bb and the pod

I think it would be very interesting to swap out the JDBC usage with @andersmurphy native https://github.com/andersmurphy/sqlite4clj and see if there is a substantial perf difference

.. though since it uses coffi (ffi/ffm) I suppose it wouldn't work in bb 😞

you can do different things for bb and clojure

yeah, already have different codepaths for clj and bb

I think most of the perf here is actually edn serialization, but haven't measured

this is incredibly cool

we already had https://github.com/jimpil/duratom, but as far as I can tell it was always intended to work on a single clj process, not multiple

the idea for this actually came when I was reading through the https://github.com/mtgred/netrunner/blob/5653c24ce6f9cc5304c3746ccd9c3e6320b5a73f/src/clj/web/app_state.clj web.app-state namespace and thought about how so much was being done with just those atoms

hah i based all of that on re-frame, and didn't feel like putting in the effort to keep it consistent with mongodb. this is a much better solution

it's a really neat way of doing things in clojure imho

This in bb would be potent

we were just discussing adding sqlite support to bb, see #babashka in the thread about the announcement ;)

"Values stored as edn" I wonder what it'd look like to build clojure data structures directly over tables.

group-by could be super fast

Oh sweet! I was looking at the pod approach but it felt a bit hard, so I'm definitely interested in supporting bb

take a look at the discussion, it's not that easy and won't happen soon. so pod it is for now

Yeah just finished reading, the zerofs thing also looks like it'd make it very hard to make work cross process

@borkdude what's the recommendation for bb libraries that need pods: should the consumer load the pod, or the library?

library I'd say

Gives a new meaning to persistent data structures Now freeze REPL state, hmmm

i just started using it and it's real slick. might wanna call out that the data it holds has to be serializable (no sets, for example)

I've had this multi process duratom idea ruminating in my head for days now too

@nbtheduke sets should work:

filipesilva@m4 ~/r/s/alex-connections (master) [SIGINT]> clj
Clojure 1.12.3
user=> (require '[filipesilva.sqlatom :as sqlatom])
nil
user=> (defonce *atom (sqlatom/atom :sets nil))
#'user/*atom
user=> (reset! *atom #{1 2 3})
#{1 3 2}
user=> @*atom
#{1 3 2}
user=>
filipesilva@m4 ~/r/s/alex-connections (master) [SIGINT]> clj
user=> (require '[filipesilva.sqlatom :as sqlatom])
nil
user=> (defonce *atom (sqlatom/atom :sets nil))
#'user/*atom
user=> @*atom
#{1 3 2}

metadata works too

it stores a str printed with

(defn- pr-str-meta [v]
  (binding [*print-meta* true]
    (pr-str v)))

(defn- read-edn [s]
  (edn/read-string {:readers *data-readers*} s))

huh, that didn't work for me, so i'm not sure what i was doing wrong

but thank you, that's cool

if you can repro let me know and I'll go fix it, the goal is for it to really be edn-level support

@borkdude I gave bb support a stab. The sqlite pod was no problem at all, but the proxy was. sqlatom currently https://github.com/filipesilva/sqlatom/blob/2481ab0a53de6177041535afc83979a8f17f644d/src/filipesilva/sqlatom.clj#L155 these:

(proxy [Object clojure.lang.IAtom2 clojure.lang.IRef clojure.lang.IReference] []

I may be able to support it, I'd have to look into it. issue welcome

@borkdude here it is https://github.com/babashka/babashka/issues/1931

v1.1.0 is out with babashka support:

## Babashka

The following operations are not supported in Babashka:
- `add-watch`, `remove-watch`
- `set-validator!`, `get-validator`
- `meta`, `alter-meta!`, `reset-meta!

but besides that, the atom stuff works, and you can even use the sqlatom to pass data between clj and bb processes, and the cross-process test now exercises this

Nice. So you could like quickly save some info to the db at the cli with an eval string. I wonder how fast that would be? Under a second?

filipesilva@m4 ~/r/p/sqlatom (master)> time bb test:cross-process
Testing 30 processes x 50 increments = 1500 expected
PASS: counter = 1500

________________________________________________________
Executed in    4.88 secs    fish           external
   usr time   28.97 secs    0.53 millis   28.97 secs
   sys time    2.66 secs    2.45 millis    2.66 secs

(require '[filipesilva.sqlatom :as sqlatom])

(def id (random-uuid))

(let [[dir n-str] *command-line-args*
      n (parse-long n-str)
      a (sqlatom/atom :counter 0 :dir dir)]
  (dotimes [i n]
    (println id i)
    (swap! a inc)))

note that it was executed in 5s, so I expect it should be fast for what you asked

How long does it take 1 proc to assoc a small map?

(if you don't mind me asking - I can download it and try as well)

This seems useful in some agentic cli skills I want to work on.

I sometimes have an agent working in lots of super small bb scripts

(require '[filipesilva.sqlatom :as sqlatom])
(def a (sqlatom/atom ::a-map {}))
(println @a)
(swap! (sqlatom/atom ::a-map {}) assoc :foo (random-uuid))
(println @a)

filipesilva@m4 ~/r/p/sqlatom (master)> time bb test/cross_process_worker.clj
{:foo #uuid "0f673a78-7a6b-488a-8ab8-97c5a47e6d22"}
{:foo #uuid "ed907f5c-a980-4d2b-942f-345452cd8748"}

________________________________________________________
Executed in  117.09 millis    fish           external
   usr time   18.99 millis    0.48 millis   18.51 millis
   sys time   22.82 millis    2.37 millis   20.45 millis

bang bang!

i wonder if it can handle 38k clojure maps lol

ser/der might be problematic

if it's one giant blob

there's faster edn readers/writers than pr-str/`edn/read-string` , as long as they support data readers and metadata it should be fine to change

but yes, giant string

I'd be interested in optimizations if someone can repro it being slow, I imagine it's not that hard, it's just that premature optimization etc etc

Anything large is going to fall over with contention

Which is fine, if it's usually, mostly single user at a time

If you built clojure data structures over the tables, so structure was shared, that'd work

I don't doubt there's a threshold, but started getting a bit skeptical that these thresholds are as low as we expect... computers have gotten really really fast

Well, it's forcing a single point of serialization, right? Callers never actually have parallel access. Which can be fast still.

I think there's some optimizations to be done for the retries over large objects too, like not serializing them more than once etc

readers have parallel access, but single writer

right

Yeah, optimistic write and then try to move the root pointer. If it fails, try to see if your change was associative and graft it on the later snapshot

diff it

i'm at the point where i should just write some sql myself but who wants to do that lol

that sounds possible, and I remember someone that tried something similar over firebase, but I don't quite have the brain capacity for it, and suspect that at that size you want the query leverage of a real db

I think that ship has sailed

lol

@nbtheduke if you want to write some datalog instead, I also made a lib that makes it really easy to do datomic over sqlite

https://github.com/filipesilva/datomic-pro-manager

you can even call that lib from within your app process, and it will download datomic, start it, etc, as shown in https://github.com/filipesilva/invoker#datomic

(ns app
  (:require
   [datomic.api :as d]
   [filipesilva.datomic-pro-manager :as dpm]))

(def db-uri "datomic:")
(defonce *conn (atom nil))

;; 
(def schema
  [,,,])

(defn start []
  (future (dpm/up))
  (dpm/wait-for-up)
  (d/create-database db-uri)
  (reset! *conn (d/connect db-uri))
  @(d/transact @*conn schema))

wow, you are a font of good libraries

I just think local fast dev is nifty!

was looking into it today - turns out it's local to the sqlite connection

see on this test, where they check that inserting into the second connection does not register on the first connections listener https://github.com/xerial/sqlite-jdbc/blob/6e61f29696c14d7d77be8eb971842a6c842629e0/src/test/java/org/sqlite/ListenerTest.java#L100-L136

so this wouldn't do anything for cross process watches, and in-process watches are covered by the atom itself

I expect the json serialization to lose some fidelity, like how right now metadata works with edn

that hook looks really interesting though!

I'll try implementing that

I had a similar idea today so I got a big smile on my face when a slack search for sqlite atom returned this thread! some thoughts: I found out recently that SQLite supports JSONPath syntax and has a bunch of other cool JSON functions: https://sqlite.org/json1.html. I wonder how hard it would be to be able to store atom contents as JSON in SQLite to unlock these. of course having to think about how to serialize sets etc opens a can of worms re add-watch not reacting to database updates, it seems like SQLite exposes a hook you can use to listen to db changes: https://sqlite.org/c3ref/update_hook.html it seems to be supported by the Xerial driver. test example: https://github.com/xerial/sqlite-jdbc/blob/6e61f29696c14d7d77be8eb971842a6c842629e0/src/test/java/org/sqlite/ListenerTest.java#L47-L91

ah mehhhh

on the other idea, I found there's a library that tries to preserve data edn<->json round-trips: https://github.com/wilkerlucio/edn-json the source mentions two shortcomings:

- metadata is lost
- number keys on maps will be turned into strings on the conversion back

Yup that's a very good point! I checked and currently if you have conflicting routes, the behavior is pretty bad. It will match the first route, but the param will be of the last one. So it will match /api/users/:id but then the data you get is {:x "something"} . Definitely not good. I think I'll change the behavior in such a case that if there's multiple, it will always match first AND use the first matches param, but that it would log a warning during compile-time.

You could just throw ambiguous route exception let the user know

I thought about that, but I like software that keeps on working, and tries to self-heal if it can. In this particular case there is an option for a continuing-to-work path.

Well imagine you have a PR for your server that introduces an ambiguous path that is same as existing path. Would you want tests to fail with an exception or would you want tests to pass with a logged warning?

Good point, didn't think from that perspective

While the user-facing API is exactly the same as before, I've still tagged it as a breaking change (2.0) because the behavior changes drastically, but just so you know that upgrading to 2.0, in the best case, should not need any changes from you the user.

this looks cool. what made you want to build this vs use something like reitit?

I wanted something small and that I could use in any clojure-native environment, which basically meant that I needed something that does not rely on any interop to the host language. At the time (4 years ago) I could not find any router that could do that, so I made my own. I'm not sure if there are other routers now that do run on different Clojure runtimes or not, maybe there are, but that's why I made this one. I do have a pending task to get this to run on Jank as well. Last time I tried I couldn't get tests to work, but with Jank being in alpha it might just be time to try again.

I just tagged 2.1.0 which now also runs on Jank. Thus, Ruuter is now a 4-runtime Router!

Perhaps add checks that user didn’t specify multiple conflicting routes, e.g. /api/users/:id and /api/users/:x . only one of those will be in the resulting tree and with large number of routes this can lead to silent failures (i.e. a handler no longer being resolvable).

same thing with optional segments and wildcards

optional segments is an interesting concept, I’ve haven’t seen it in other routing libraries

The idea is that we’re not rendering to a browser at all. We’re rendering to a data structure (a tree-based on that can be converted to either strings or hiccup or used as is) in tests or at the REPL (no browser involved). The goal is fast-running tests that do not involve a browser (if you need e2e tests) and fast interaction with an LLM (since it’s just data that the LLM can navigate directly or with helper functions). You are not typing hiccup or this data structure (you’re still writing Fulcro as-is), but the renderer is replaced by one that just records the render frames as data, and the entire SPA is run in the JVM. Fulcro has always been capable of doing this (I wrote it in CLJC with just these sorts of intentions, before LLMs existed) and never really polished the library itself because most people were happy clicking through the browser. Now that LLMs need better direct access to “what is going on”, I think Fulcro is very well situated to make that better…Also, the verification story for LLMs NEEDs a lot more e2e tests (because humans are used to “trying it in the browser continuously”), so the verification step that reduces hallucination of a “working system” in the current “state of the art” is image based and very token heavy. You could technically have it play directly with the DOM via browser plug-ins or playwright (it can read the console as well), but again that’s pretty heavy on memory, and just loading a browser MCP consumes a lot of tokens.

Here’s a complete test from a demo app using the new async statecharts:

(use-fixtures :once
  (with-test-system {:port 9844}))

(specification "Application startup"
  (let [app (test-client 9844)]

    (h/render-frame! app)

    (assertions
      "Loads the application configuration"
      (get (app/current-state app) :application/config) => {:url ""}

      "the statechart is in the landing page state"
      (contains? (scf/current-configuration app uir/session-id) :dataico.ui.root/LandingPage) => true

      "The welcome page is rendered"
      (hic/find-nth-by-text (h/hiccup-frame app) "Welcome" 0) => [:div {} "Welcome:" "1"])))

(defsc LandingPage [this {:keys [x]}]
  {:query         [:x]
   :ident         (fn [] [:component/id ::LandingPage])
   :initial-state {:x 1}}
  (dom/div "Welcome:" x))

(def system-statechart
  (statechart {}
    (state {:id :state/top}
      (transition {:event :error}
        (script-fn [env data & _]
          (log/error "Unexpected error: " (:_event data))))

      (on-entry {}
        (script-fn [{:fulcro/keys [app]} & _]
          (setup-RAD app)
          [(fops/apply-action assoc :ui/ready? true)
           ;; NOTE: afops BLOCK statechart as-if the async operation were synchronous. This keeps us from trying to route
           ;; until the application config is loaded.
           (afops/load :application/config nil {::sc/ok-event :event/configuration-loaded})]))

      (uir/routing-regions
        (uir/routes {:id           :state/root
                     :routing/root `dataico.ui.root/Routes}

          (uir/rstate {:route/target `dataico.ui.root/LandingPage})  ;; First (deepest) state (default route)

          (state {:id :state/logged-in}
            (sfr/report-state {:route/target AccountList})
            (sfr/form-state {:route/target AccountForm})))))))

I'm having a hard time wrapping my brain around exactly what rendering to hiccup means here and how that's different from "classic Fulcro" (for lack of a better term) which renders a ui tree in html/react to show the corresponding data tree returned from a query.