Clojurians Log v2

Here is what I do. I use d/listen to observe which datoms were added/removed in each tx and save them into postgres. For time travel I select all the datoms in a tx range. To travel back to a point in time I would select all datoms below a target tx.

select * from datom where tx <= ?

I always imagined listening to conn and putting past db references into a list

@sean888 that would work too, but you’ll have to recreate an entire db from datoma each time you want to time travel

Yup, that is exactly what we do. Probably not feasible for all use cases.

> datoma @tonsky Is this a typo, or is the plural of datom datoma?

are the txes already stored by default?

seepel are you saying that you like recreate a database by pushing the tx selection to the bottom (?) of the queries

I've seem "datoms" a lot but I like "datoma"

datoms :) I’m typing on a phone

so the trade-off to serializing a tx log seperately to the DB is like, all writes are duplicated, and any tx-sensitive queries are not benefitting from the 3-index system and queries on them need to rebuild/not use those indexes (?)

and the tradeoff to the niki version is that you'd need to roll your own syntax to do cross-tx queries? but actually that's true for both isn't it, but the Niki version is (# txes) more storage, since no deduplication, full clone each transact

but the Niki version's faster for queries since no rebuilding (?)

My table looks like this

CREATE TABLE IF NOT EXISTS datom (
  id SERIAL     PRIMARY KEY,
  entity        INTEGER,
  attribute     TEXT,
  edn_value     TEXT,
  ref_value     INTEGER,
  tx            INTEGER,
  valid_until   INTEGER,
  owner_guid    UUID
);

(defn save-tx [owner-guid {datoms :tx-data}]
  (when-not (empty? datoms)
    (let [true-datoms (filter #(:added %) datoms)
          false-datoms (filter #(not (:added %)) datoms)
          inserted-datoms (when (not-empty true-datoms)
                            (insert-datoms! owner-guid true-datoms))]
      (doseq [datom false-datoms]
        (execute! (invalidate-datom owner-guid datom))))
      inserted-datoms)))

(defn select-datoms [owner-guid & {:keys [:until]}]
  (->> {:select [:entity :attribute :edn_value :ref_value :tx :valid_until
                 :owner_guid]
        :from [:datom]
        :where [:and [:= :owner_guid owner-guid]
                (if until 
                  [:<= :valid_until until]
                  [:= valid_until nil])]

        :order-by [[:tx :asc]]}
       sql-format/format
       execute!
      (map (fn [{entity :datom/entity
                          attribute :datom/attribute
                          edn-value :datom/edn_value
                          tx :datom/tx}]
                      (let [attr (edn/read-string attribute)
                            val (edn/read-string edn-value)]
                        (d/datom entity attr val tx))))))

(d/init-db (select-datoms owner-guid :until target-tx)

To Niki’s point, this starts to degrade once dbs get big. I’m actually in the process of moving this whole setup to use the storage protocol https://github.com/tonsky/datascript/blob/master/docs/storage.md

We essentially load up a user’s datoms into a db in memory and keep it around until their session “expires”. (Or at least best effort)

I don't want to touch that because I'm in clojurescript and would have to work out how to port the whole protocol/calls to it to be async

or let datascript call an async thing thinking it's sync which sounds like a recipe for trouble

for valid_until to work, you can't ever reuse eids right? dunno whether datascript disallows that, like add eid=1, remove eia=1, add eid=1

Is this on the client or on the server with node? If the db is on the client then I don’t think the storage protocol would help much anyway. You could do something really similar to what I’m doing with indexed-db though.

definitely want to serialize on the server, not sure what I do on the client

thanks a lot for sharing what you do, it's very useful I will try to do something similar, maybe with a kv store instead of SQL

There are certainly some rough edges, but it’s been working pretty well so far.