Clojurians Log v2
datomic
Just exploring and learning the AVET index. I currently have uuid7s for every entity id (:note/id, :project/id...) across all tenants. Entities are partitioned per tenant. But since partitioning doesn’t directly affect the AVET index, are there any common patterns to optimise for AVET locality? Do you use tuples with the first element as the tenant id/ref? Or for my uuid case, encode the tenant partition info in the uuid?
Both these work
fundamentally you want the left side of the value of V to cluster around your desired locality
do remember that you can't actually leverage locality fully unless you are also making sure peers query with similar locality.
for the example of a tenant, your peers need tenant affinity
Yea, that's essential. I’m thinking of routing my request based on the same or derived hashing function as the one I use for forcing partitions in datomic.
Same hashing function was wrong obviously. I guess it can be as simple as a tenant id mod number of peers. Frequent auto scaling would destroy a lot of the benefits though. Region based clustering of tenant id routing is also something I might have to think about. But those are at scale and independent of the schema which I'd like to remain as stable as possible.
I'm trying to understand as-of. I see that I can give it basically any numeric value (including negative values) and I will get back a db for which d/as-of-t faithfully returns whatever I passed. However, only for some values does d/t->tx actually return a transaction (it always returns a tx ID, but only some of those are "real" in that I can pull a :db/txInstant for them).
If I pass a value that does not map to a real transaction, what are the semantics of the resulting database?
In a non-filtered db, (->> db d/basis-t d/t->tx (d/pull db [:db/txInstant])) will reliably give me a Date representing the latest transaction in the database, but with an as-of db, it won't if the passed in as-of value does not map to a real transaction. For such a database, is there some other way to understand when its most recent transaction occurred? Is it even valid or meaningful to use such a database?
I see that there is similar behavior if you pass as-of a date that does not map exactly to a specific transaction's :db/txInstant.
I sort of expected that maybe as-of-t would return whatever value was passed to as-of but basis-t would return a value corresponding to the actual latest tx in the database. however, it seems like basis-t ignores the filtering completely and always returns the basis-t of the unfiltered db.
as-of is a filter: you are filtering out datoms newer than the as-of-t
if you want actual transaction ids (or corresponding Ts), range over :db/txInstant. This is what d/as-of with an instant is doing under the hood.
If you want to know the details of what T actually corresponds to: https://favila.github.io/2024-05-16/datomic-entity-id-structure/
For such a database, is there some other way to understand when its most recent transaction occurred?The basis-t is always the most-recent transaction the database holds.
That doesn't seem to be true in my testing, or else I'm not understanding what it means.
user> (d/basis-t db)
66
user> (def adb (d/as-of db 1))
#'user/adb
user> (d/basis-t adb)
66
user> (d/as-of-t adb)
1Shouldn't anything after t = 1 be filtered out?
filtered out != database basis/database "holding"
I see
e.g., you could change the as-of and make anything up to basis-t visible
if it were not a filter, but actually restricting what was in the database value itself, then you couldn't do that.
changing as-of would be the same as changing basis
So something like (:e (first (d/rseek-datoms adb :aevt :db/txInstant))) is the way to get the most-recent TX?
yeah, with a bit more caution to make sure you stay in :a
if you specifically want the highest tx < as-of-t, you can add (d/t->tx (d/as-of-t db)) as the starting e. It might be a bit faster, not sure
Yeah, but that doesn't necessarily correlate to a real TX, right? that's my core issue. The background here is I want to give the user (this is basically a Datomic admin UI/browser) some kind of relevant wall time, regardless of what they typed into the as-of field
Whatever you get from that rseek-datoms will be a real transaction
Right. Oh, I see, you mean as the starting point of the rseek-datoms, not an alternative. Sorry, I misunderstood.
specifically (first (d/rseek-datoms adb :aevt :db/txInstant (d/as-of-t adb))) . If returned datom has an :a of :db/txInstant, then that is the highest-visible real TX for given as-of-t. (If it doesn't then no TXs are visible.)
(first (d/rseek-datoms adb :aevt :db/txInstant)) should always produce the same result, I'm just not sure if it does it as quickly