Clojurians Log v2

What are you trying to do?

Hi @favila, I am investigating various way to implement SpiceDB https://authzed.com/docs/spicedb/concepts/caveats, https://authzed.com/docs/spicedb/concepts/expiring-relationships & potentially Scheduled Relationships in https://github.com/theronic/eacl. In practice ~0-5% of Relationships are expiring or scheduled, so one way to avoid changing and expanding EACL's Relationship tuples, is to store validity information on tx-meta, i.e. :rel/valid-until. Using d/filter on tx-meta is appealing for this, but the downside is that I can't batch transact Rels, because all datoms under t share same tx-meta (esp. if mutating prior tx-meta), but can be worked around with some difficult. The hard part is maintaining cache coherence, and EACL relies on "datomic.tx" for this, so if changing tx-meta does not bump t on a datom, it makes coherence hard. EACL bumps :eacl.relation/version for all affected Relations when a Relationship is mutated. It's crude, but it gets the job done and keeps storage costs low. The other option is stuffing more data in the forward & reverse Relationship tuples. Prior versions of EACL used dedicated Relationship entities, but this meant ~5x more datoms per Relationship. The performance and storage costs make tuples too attractive to go back to Rel entities. Arguably valid-from / valid-until is veering into bitemporal XTDB territory, so I am unlikely to implement scheduled relationships, since Spice does not support them either.

EACL's current forward & reverse Relationship tuples (using v7 storage model): https://github.com/theronic/eacl/blob/main/modules/eacl-datomic/src/eacl/datomic/schema.clj#L251-L270

I'll just start by saying that “bump Datom tx” is extremely odd framing. Datoms are immutable; having redundancy just means the index+log gain an additional datom with the same value, and this is indeed what very old versions of datomic do. Redundancy elimination was added because: 1 redundancy can waste a lot of space 2 having multiple “active” assertions of the same e+a+v makes value-asserted ranges in history less clear (ie a common use case of history is harder)

So what datom would you want to bump the tx for?

EACL's kernel uses t as a high watermark to enforce cache coherence under the supported consistency modes, which is why it's convenient when the latest t is readily available on Relationship tuples, which are stored directly on subject & resource entities. So I would like to "bump" the t on Relationship tuple datoms when tx-meta is updated (even though the relationship shape has not changed), but this looks increasingly infeasible. The forward tuple stored on a subject entity is, :eacl.v7.relationship/subject-type+relation+resource-type+resource which contains:

[subject-type relation-eid resource-type resource-eid]

[subject-type relation-eid resource-type resource-eid ?valid-until]

With Caveats, the Relationship tuples would have to grow to 7 components:

[subject-type relation-eid resource-type resource-eid caveat-eid caveat-ctx ?valid-until]

For things that are rare, why not model as a normal entity rather than stuffing into the tuple? Another alternative is another supplemental tuple

Tuple stuffing is the simplest solution and is how SpiceDB does it. One tuple with optional components also allow Relations to migrate between with expiration & without (permanent) or enable/disable Caveats, without moving Relationship data between tuples, or potentially have both active at the same time: permanent + expiry. Becomes an issue for banned relations used in Exclusions. I considered both alternatives options: 1. If expiring relationships are stored on a normal entity, that Rel "Schedule" entity has to repeat all the Relationship components, because Relationships don't have an eid in the tuple model. All reads during traversal & writes need to search for a matching expiring entity to pull out its valid-until value. I also considered a schedule-ref on tuple but might as well store valid-until directly because that's all we need. 2. A larger supplemental tuple (or two exclusive versions v7 & v9) introduces two read & write paths for pretty much every operation. There are some tricky parts around correctness when both a permanent & expiring Relationship exist (don't really want this, but storage model allows it). (What EACL is really reaching for here is more flexible materialized indices. Years ago I yearned for programmable materialized indices in Datomic (similar to Rama), but tuples have mostly fulfilled that need, though tuples are always tied to an entity.)

> I also considered a schedule-ref [..no] because [valid-until] is all we need “All we need” is how you got in the situation of expanding tuples in the first place. What I'm suggesting is not a redundant entity, but an additional final, usually nil ref on the tuple that if not nil references an entity with additional (not overlapping with the tuple) information. I don't fully understand your data model so forgive me if this doesn't get domain entity extents right. > really reaching for materialized indexes I also have long believed this is what you really want. Have you considered trying this? Ie literally have a thing that consumes the tx log and materializes changes while storing its basis; this thing could even be another datomic db.

Thanks, @favila. 1. I initially rejected a trailing qualifier-eid tuple component because of read indirection. However, if we assume conditional (qualified) Relationships are sparse, it is viable and allows for future feature expansion. So I'm going to move ahead with this data structure change for v8. 2. On externalising EACL: the moment you move EACL outside your DB, you might as well use SpiceDB, because you re-introduce all the problems EACL was built to solve: a. Your domain model is immediately polluted by ZedTokens (or :zed/token in EACL) if you want write-read consistency: i. After writes, you need to query Spice using the ZedToken returned from WriteRelationships to get a consistent answer. ii. You need to store that token somewhere in a shared cache or DB (another d/transact polluting tx-log). b. You revert to Eventual Consistency. You can use Full Consistency (`(d/sync conn)`), but it's slow. c. You need a bunch of code to handle retry/rollback/failure with external or semi-external system. Granted, using the same Datomic Transactor with different DBs reduces downtime risk by a lot. d. You need to do multiple txes to write stuff when one d/transact could do. e. You lose real-time UI notifications with easy hydration (a major use-case for EACL), but you could monitor two Peer DBs. The benefits of situated AuthZ is that you have: 1. less complexity 2. one less piece of infra (external AuthZ) 3. at-least local consistency 4. you don't need retries if you can combine d/transact with tx-relationship 5. faster entity hydration due to incidental datom locality 6. suitable for real-time UI updates with good consistency, but this could be traded for an external Datomic DB with some delay.

so you have a normalized form that you transact directly, and a custom index you maintain downstream of changes to it

How to synchronise it across Peers without a shared cache, or writing to Datomic? Hmm, I guess if they all ingest the same tx-log...

T is how, just like d/sync

given any cache with basis T and a db/log, you can determine what catchup (if any) is necessary

"cache" can be a supplemental datomic db, but doesn't have to be

I have been intentionally avoiding draining the tx-log because it could contain many changes unrelated to AuthZ. However, EACL has an in-memory cache (that is local to the Peer), which could maintain these data structures. I am hesitant to add a d/listener esp. for other backends (e.g. that support S3) that would require comms overhead when I would prefer to read from storage (perception scales). I shall think on it more.