Clojurians Log v2
datahike 2026-07-17
I have landed a bunch of fixes and features over the last week, notably one can now store blobs both in datahike (garbage collected and replicated) and outside https://github.com/replikativ/datahike/blob/main/doc/store-refs.md. Feedback welcome! @maxweber mentioned blob support repeatedly and people have asked for it over the years. I was not sure what the value was initially, but I think it is just much better to keep everything in one store and konserve already supports that. You can also store unindexed documents this way with konserve's document store Clojure API, e.g. k/update-in.
This is really really nice! I have implemented a pattern for this in datomic so many times.
One difference I often do though is use a content addressed pattern for storing the blobs. The id of the blob is its sha256 hash.
This is happening through https://github.com/replikativ/hasch here, too. Right now blob-id is an aliash for hasch.core/uuid https://github.com/replikativ/datahike/blob/main/doc/store-refs.md#the-id-is-content-the-name-is-a-datom.
sha512
Something I wish for but I suspect isn't really possible is to be able to do a konserve kv put in the same transaction as a datahike transaction in the underlying storage. For example in datahike-sqlite it would be very powerful to put the blob and transact datoms in the same underlying sql transaction.
One could build robust and correct durable execution workflows on top a system like that
I have also been working on https://github.com/replikativ/datahike-saas-starter, which demonstrates how to run multi-tenancy at scale of S3 with close to minimal latency. This makes use of the write-amplification improvements like diff-buf support in the persistent sorted set (which secretely turns it into a better hitchhiker-tree 😉 ) and root fusion in Datahike, requiring a single PUT per db update in the optimal case and less than two on average. I would love to hear feedback, I want the starter template to be accessible and I think it is still too technical. My plan is to stabilize the two features and ideally turn them on by default for 1.0.
Nice, did a similar per tenant setup for a project of mine and it worked really well. Curious to see how the experts set it all up
I am sure there is room for improvement in general. The biggest limitation for setups like this so far was the write amplification of B-trees. I finally figured out how to do hitchhiker-trees right and the diff-buf work looks correct and very promising. But I still prefer people testing it and understanding the behavior before turning it on by default.
Super curious about the computer science theory behind it all, but I'm sure it'll go above my head 🥲
It actually got simpler. In-memory we keep a diff when we write to know the changes applied to each tree node. Then when we write the tree we write these diffs instead of writing also the child node if they are small enough. When we later load a child we reapply the diff during loading once. In memory everything stays the persistent-sorted-set (B-tree) from before. The hitchhiker-tree calculated an overlay during operation all the time, which slowed down our query operations.
So, like distributed, in-node WALs? Or is that an oversimplification?
That is why we deprecated it and went back to the persistent-sorted-set. The problem with B-trees is that they are read optimal, but heavy under writes (write amplification). That is why Datahike can take quite a bit of storage atm. when you write a lot and don't use GC aggressively. This work will alleviate that.
Yes, that is a fairly precise way to think of it.
Ah, OK. Datomic had a merge sort that did something similar. But, I believe it was still single threaded and not in process.
Nikita opted for a global log on the root node in DataScript that is reapplied on reads. But since we want Datahike to be easily readable by third parties and joinable across cold cached stores, reapplying a few hundred transactions just to deref a database will have a big latency cost. This is why we do this more interleaved log approach.
Yeah, the distributed reads across CLJS came in clutch a few times for me. But, I never had high pressure/memory situations.
Could try some chaos testing with the SaaS template. There were these skills from the Antithesis folks that apparently can generalize fuzz and simulation testing, but I never tried them in earnest. https://sqlsync.dev/posts/antithesis-driven-testing/
That would be very cool. I have also not really tested the template at scale yet. I think @maxweber tested it on Tigris (not sure), and we think Tigris is probably the best deal rn, but I also need to run it at scale and ideally part of it is a stress test/simulation that helps evaluate workloads also before scaling.
I also am aware of your PR btw. I will merge it soon.
Haha, no pressure. They’re much smaller than what you're working on
@hoertlehner The diff-buf setting used here + root fusion + deactivating of auditable commit history reduce the amount of storage written. diff-buf is by far the biggest lever. diff-buf + root fusion will probably be turned on by default once they are tested a bit more and I am confident they work well for everybody.
You don't have to turn history off, I need to fix this in the doc, this is kind of confusing. History indices are internal and independent of these optimizations. But they do increase the db size in general ofc.