Clojurians Log v2

are these two different DB files or the same DB file?

Not possible with different DB files. Possible with the same DB file, e.g. inside a with-transaction, you can obtain the LMDB handle, and write to it as the part of a Datalog transaction.

In the end of the day, things in the with-transaction will be written as a single LMDB transaction.

(with-transaction [cn conn]
   (let [lmdb (.-lmdb ^Store  (.-store (d/db cn)))] ...))

I was thinking same DB, yes. So I would use lmdb for low-level LMDB k/v writes, not using the Datalevin k/v API (i.e., transact-kv), right? Nevermind; after skimming the sources I realize lmdb is of the type returned by open-kv on a local path. IIUC opening the same DB simultaneously as a datalog store and k/v store (as in, get-conn and open-kv) is not supposed to work / be safe, just like opening two concurrent datalog connections to the same DB? Or would it be safe as long as I only do k/v writes in with-transaction? Nevermind; I guess I can open it once as a datalog conn, and when I want to query as k/v I just obtain the underlying lmdb as you say.

So the use case I was considering for this is to maintain an immutable log of transactions along with the datalog DB. The log would be maintained as k/v pairs, where k is the timestamp and v is an EDN blob containing the datoms of the transaction and any other metadata I may want to pass to my wrapper over d/transact!. The point is that since I won't be querying these logs in any other way than by timestamp, it seems wasteful to have Datalevin give them entity IDs and index them in any other way. Is this a valid concern, or would it be mostly equivalent to log transactions as single-attribute entities where the entity ID represents the order and everything else (including the timestamp) is in the attribute value as an EDN blob? I understand that this log will dominate the size of my DB. When that becomes a problem I'll add periodical jobs to create snapshots and archive away old logs. Other than that, do you think there's a risk that these log entries may step on Datalevin's toes while it works on datalog in any way? Are timestamp keys likely to clash with the ones Datalevin creates to represent Datalog DBs, caches, etc (I can make them serialized instants if that helps avoid that)? Are they likely to mess with datalog queries in any other way?

Datalog data are stored in their own DBIs. So you KV DBI will not interfere with them. However since these are all in the same DB, the performance is definitely will be impacted. We have a single writer after all. However, I think the impact should not be too big. Adding some data to a transaction doesn't make much of a difference.

Finally, transaction log is on the roadmap

So we will have built-in transaction log and APIs for them in the future.

Exciting, thanks!