Clojurians Log v2

Okay, still finishing the jvm side of things, so it's temporarily a little slower than the node side, but here I've split out eve from cljs-thread so it can be looked at in isolation: https://github.com/SeniorCareMarket/eve The scaling numbers are looking stellar.

K, here we go - 4 jvm-clojure-eve threads and 4 node-cljs-eve workers banging on 10 MB and 100 MB file atoms in parallel, with almost the same read/write times. Can theoretically scale to "8 exbibytes." https://gist.github.com/johnmn3/dca4f571a0b310ed2cb58f31983c70d7

I was just kidding

Way back, a long time ago, you expressed surprise that tau.alpha worked. I'm still surprised any of this works at all.

like.... Who would have thought that parallel transducers would Just Work in cljs? As designed? IFAICT, nobody intended for that to happen...

This branch introduces "Extensible Value Encoding" (EVE) The atom abstraction has cooperative gc built in, with epochal snapshots per atom, and there's a deftype abstraction for defining extensible datatypes over that whole system. Eve is currently embedded in this branch of cljs-thread but at some point I'll be extracting eve out into its own lib.

I'm not yet sure how you'd use it outside of cljs-thread, but I think some mmapped file based duratom thing, where multiple flavors of clojure guests hammering on that file in parallel might be pretty swell

like clojure guests as differen os thread procs, banging on a file

@borkdude I'll probably be porting this to squint/cherry/bb/graal/etc soon too

Performance stuff... there's a broken bench suite that I'm going to revive in the next few weeks. In general, we're 10 to 100 times slower that Clojure/JVM for parallel, contended scenarios. I think that will quickly come up to within 10 for most categories. Some edge cases we may come out ahead, because we're on the stack.

@thheller you said this couldn't be done, so I just couldn't let it go 😉

@dnolen many years ago if you'll recall, I was picking your brain on how I might performance test tau.alpha and you recommended a raytracer. I impl'd that but honestly it best shows the parallelism but not the sab backed persistent data structures. The raytracer just wants to be flat tiles of raw ints, so persistent data structures will never be able to compete with raw arrays for that particular job. I have the raytracer demo running here: https://johnmn3.github.io/cljs-thread/raytracer/ Currently broken... gotta fix the coop/coep thing for gh pages

I'll be adding agent soon too

oh, and as you'll see in the docs, we ship with a full dom proxy in this branch, so your "core thread" is now your main thread and your ui thread is your "screen thread" which was the old main... It's a shift of mindset...

@bhauman I have to test this out with figwheel more but I think the kernel strategy is going to work across vanilla, shadow and figwheel

Okay, the raytracer demo is fixed. @borkdude the reagami-counter demo shown in the readme is running here: https://johnmn3.github.io/cljs-thread/reagami-counter/

Also, something to talk to y'all about - my blocking strategy. When you make a "network transaction" - any call like in, future, spawn, etc. - you can do nested calls:

@(in w1 (+ 1 @(in w2 (+ 2 3))))

Also, pool workers always get spun up from network transactions, so their blocks will be parking derefs on those workers, so they don't lock up the task queue on that worker.

The long term goal here: have multiple different clojure flavors running in different wasm runtimes, all banging on shared atoms in parallel, leveraging the runtime benefits of each and coordinating it from cljs. Letting you basically spawn threads of different runtimes for different workloads, and have zero copy access between them.

glojure, clojurewasm, jank, graal versions of clj, etc.

Also, a selfhosted cljs that used a t/atom for its env, and if locals were stored there, then cross-worker locals passing would be transparent, zero copy as well

The key idea here is the atom abstraction is what allows you to constrain alloc/lock-free-gc/read-write-contention to a small domain that can be managed cooperatively between uncoordinated threads.

Also, isomorphic cljs-thread will be possible here. From your cljs nodejs server, you'll be able to do (swap! current-client-state :logged-in? false) and that get's sync'd to the client.

Or

(in current-client
  (-> (d3/select "#bar")
      (.transition)
      (.duration 300)
      (.attr "width" (str 10 "%"))))

You wouldn't get the zero-copy shared semantics over the wire... But everything still works when going full async under the hood. So you can make it work transparently from a dev-exp ergonomics perspective. (clarification - server->client atom syncing and server->client js pushing isn't impl'd yet but those are solved problems that are simple additions)

Oh, and one more point, because we don't allow eve reference types to escape atomic transactions, and they get eve->cljs'd on deref, we don't have any heapspace pointers to those data locations in the atom to keep alive. That's the atom trick. As soon as nobody is reading a given invalidated value from the atom, it immediately gets freed. This way we don't have to maintain complicated weak-maps to track thousands or millions of hamt nodes, that may or may not still be sharing structure from pointers floating out there in the open runtime in heap space.

With clojure atoms, your heap objects are the same objects on the inside of the atom and the outside of the atom. Not so here. These are stack atoms. Maybe you'd call them value typed atoms. Valhatoms lol. Anything you pass into an atom via a swap!/reset!/construction transaction will be serialized into byte data that lives in the atom, decoupled from the version on your heap. Derefing returns a new version of that object. Within an atomic transaction, all protocol methods on clojurey objects will attempt to use protocol methods that sit closest to the stack data, manipulating the zero copy stack data in place where possible.

I'm fairly sure I didn't say this couldn't be done. Pretty sure I said something about trade-offs making this not worth. Great that you got something working. I don't have time to look at any of this, but keep at it if you think its worth.

Here's a deep dive research report from claude on how eve relates to prior art: https://gist.github.com/johnmn3/d2b60b3c6ed08d9972b83c7138e60a6e The tl;dr:

The closest thing to EVE in the wild is LMDB on the mmap+multi-reader-epoch axis, combined with immer on the persistent-HAMT axis — but no single system fuses these. Specifically, cross-process epoch-based GC of persistent immutable HAMT nodes does not appear to exist as prior art outside of EVE. The academic EBR literature (DEBRA, PEBR, crossbeam-epoch) is all single-process/single-address-space. The mmap databases (LMDB, Metall) use different reclamation strategies (freelist B-trees, no GC respectively). The persistent structure libraries (immer, im-rs) are fully in-process.

BTW, Claude had to explain to me that I'm using the term stack wrong lol Nobody told me that off-heap and stack are not the same thing 😆 A more correct name might just be "mmap atoms" or something... "direct memory atoms"