Clojurians Log v2

Color me desired! 🤣 But for your prioritization purposes, know that I’m very early in my journey. The larger use case is for an LLM to query the knowledge graph.

Hey @markaddleman - we've got the underlying relational operator implemented but haven't yet wired it into XTQL (or SQL's WITH RECURSIVE, for that matter), mainly due to nobody having mentioned a desire for it - until now, of course 🙂

Another desirer over here 🙂

Another desirer over here. How much work is left to get https://github.com/xtdb/xtdb/issues/2087? Every search I do for XTDB graph traversal leads me back to https://discuss.xtdb.com/t/xtdbs-sweet-spot/52/9 laughcry

on the SQL side, it'll be the compiler analysis to get with-recursive planning out into the correct relational operators. with-recursive is a bit of a gnarly one in the SQL spec, plenty to consider to be spec-compliant (which we do target). in XTQL we've additionally got the syntax design to do. tbh, though, the blocker is prioritisation rather than time, necessarily - we don't currently have active clients nor design partners with these kinds of use cases, so (much as I'd like to work on it, both for a feeling of completeness as well as it being an interesting problem) I'd find it hard to justify the time at the moment 😐 I will point the prios committee at this thread, though 🙂 and if anyone wants to write up a bit more about their context and use cases, what it'd unlock for you, and send them over to <mailto:hello@xtdb.com|hello@xtdb.com>, that'd definitely help!

Hey @shikhar just catching up with Slack again after some paternity leave 🙂 noticed we'd missed answering your question here - the answer is that XT copes without concurrency control (due to the deterministic replication + single partition usage), although being able to push some of this work to the level of the log probably has good performance benefits for the future

Unrelated, but while I have your attention, I read about Northguard earlier (h/t https://clojurians.slack.com/archives/C05N2M7R6DB/p1751880422581049) - have you seen any insider info on how that API looks? I guess until it's released as OSS it's not a threat for you, but it would be interesting to comparre

is there somewhere I can read about deterministic replication?

no insider info on NG 🙂 but it looks non-trivial to operate, it's meant for LinkedIn. we are all about being a serverless API. so not really comparable. maybe if someone offers it in that fashion...

It's a pretty simple event sourced design (but the events are purely internal) - https://docs.xtdb.com/intro/data-model.html#_a_log_of_transactions Very unlike most SQL systems, which prioritize OLTP concurrency over correctness

> serially and asynchronously, i.e. if you wait a few milliseconds for the processing of a newly submitted transaction to finish, you can generally then read the data from that transaction in your next query (a.k.a. "read your writes"). Ah it is eventually consistent

One way to get strong consistency with S2 in this model – which seems like multiple writers allowed and rely on the log for ordering – is when servicing a read, https://s2.dev/docs/rest/records/check-tail (Kafka: endOffsets) and wait till caught up

Writes are serializable, but reads are "a la carte" consistency thanks to the immutability + as-of querying. Rich Hickey's Deconstructing The Database is a good watch if you're curious about these things and the background of how he designed Datomic

Thanks, will check it out

Oh also, the consistency model for Datomic was also reviewed very favourably by Jepsen fwiw (XT really just carries these same principles to a wider audience) https://jepsen.io/analyses/datomic-pro-1.0.7075