Clojurians Log v2

Datalevin is deliberately not a history database. The datoms in its persistent indexes do not store the transaction dimension. Its mental model is that of database as mutable state. The author’s experience is that retaining historical datoms makes that model harder to explain. Datalevin doesn’t pay the cost of increased storage, wider indexes, and reads and writes having to account for old states as well as current state. So there are clear pros to this. However, temporal history can be very useful for operations/debugging. Datomic users frequently report things like: “I've goofed up our database a couple of times after a botched data migration... Datomic history was indispensable for figuring out what went wrong and how to fix the problem” “I can't count the number of times that history/metadata has allowed us to understand where bugs are hiding, what went wrong with, e.g. migrations and even to identify developers mistakes running transactions at the REPL (a reality that seems all to common in our business right now).” In Datalevin the guidance offered is “If an application needs audit history, provenance, or temporal semantics, model those facts explicitly.” I’ve been thinking about how we may get some approximation of a temporal history that would help us answer the question “How did that database get into this [possibly incorrect] state?” What follows is sketch of such a solution and I’m curious about others’ experience. The transaction report includes the datoms that were actually added or retracted, and any transaction metadata supplied by the caller. The datom objects include the :tx field - Datalevin's internal numeric transaction id (not stored in the persistent EAV/AVE indexes). The optional tx-meta argument to transact! is also not stored as queryable transaction metadata, but provided by in the transactio report. For each transaction we write what we need from the transaction report to the KV store, in the same transaction. The key is the :tx field and the value is a map of :tx-data (datoms), :tx-meta, and say :tx-timestamp that we add. Then we use the listener to write to an audit-db, which is a separate Datalevin DB. Each datom becomes an entity, which flattens the 5-tuple into EAV that Datalog can query:

{:change/tx  {:db/valueType :db.type/ref}   ; -> tx entity, same DB
 :change/e   {:db/valueType :db.type/long}   ; id from the domain DB, NOT a ref
 :change/a   {:db/valueType :db.type/keyword}
 :change/v   {}                
 :change/added {:db/valueType :db.type/boolean}

 :tx/id  {:db/valueType :db.type/long :db/unique :db.unique/identity}
 :tx/at  {:db/valueType :db.type/instant}
 :tx/meta {:db/valueType :db.type/idoc}}

The query would then have two sources: the domain db and the audit db. Cross into the audit DB on the raw entity id:

(d/q '[:find ?a ?v ?added ?at
       :in $ $audit ?domainid
       :where [$      ?e :domain/identifier ?domainid]
              [$audit ?c :change/e ?e]
              [$audit ?c :change/a ?a] [$audit ?c :change/v ?v]
              [$audit ?c :change/added ?added]
              [$audit ?c :change/tx ?t] [$audit ?t :tx/at ?at]]
     (d/db conn) (d/db audit-conn) "some-domain-id")

DB as-of becomes a fold over changes ordered by :tx/id. Or we can make the change a component on the tx entity:

{:tx/change {:db/valueType :db.type/ref
             :db/cardinality :db.cardinality/many
             :db/isComponent true}}

I am just now starting to think about this and there are probably holes in this solution. I am wondering if people who have been using Datalevin for a while have contended with these questions, what you’ve tried, what worked, what didn’t. As well as any thoughts anyone might have on this. Perhaps if we come up with a good generic solution it can folded into Datlevin as a configurable option? @huahaiy