Clojurians Log v2

• Performance: Increase index-parallelism limit from 8 to 128. i bet there is a cool story to why this became necessary!

> What do you think the chances are of a Clojure runtime bump with the Cloud Release? Just give it to me straight! What are my chances?

Clojure 1.12.5 I presume?

Sure 😀

Are re-transacting the same set of :db/add and :db/retract sets then deterministic? So that you could replicate or restore a Datomic db from an existing transaction log. If I remember my experiments correctly this was the only blocker: > Fix: Transactions no longer consume entity ids for tempids that resolve to existing entities through upsert.

Do you mean without using tempids?

@favila yes. One idea would be to do something similar like https://litestream.io/ A transaction listener could upload each transaction to an S3 bucket. The transaction would be the expanded one with :db/add and :db/retract only. Then you could build read replicas by only replaying the transactions from S3. I know Datomic already can do similar things internally and also has read only connections now. However, having a deterministic way to replay Datomic transactions would be great.

"deterministic" is odd phrasing to me: the id-skipping was also deterministic

I think you were frustrated by an entity id high-water-mark? (i.e. you are trying to write to an entity id that the database knows hasn't been minted yet) I'm not sure that what you propose wouldn't still run into this.

there's a fundamental mismatch between what wal-like systems like litestream do and transaction data--transaction input data is a set of commands, subject to db invariant checks and entity-id resolution and minting, and you deliberately have little control over entity ids (you are expected to have got the id from a read, or use a tempid, upsert, or lookup ref to mint a new id or resolve to an existing one by value). The resulting transaction log is datoms--a set of changes to the db state that resulted from the commands. What you want is to fill a transaction log directly with datoms from elsewhere. That can't be done in a semantically coherent way without the entire log being exactly the same, so in essence you just have a clone of the db.

the way you get a clone of a db is backup

so just use backup

But the backup always has a larger gap in comparison to something like litestream. Sure, only relevant if you let Datomic run on something like Sqlite (like we do), where you only have a copy of this database locally. Currently, we use litestream for this. But as I remember according to the Datomic docs it is not sufficient (at least official) to only backup the Datomic storage.

Just backing up the storage itself (not a datomic backup) is hazardous because many storages don't give strong guarantees about the state they backup, and datomic needs a consistent precise snapshot. Ddb can't guarantee this, but many sql dbs can