Clojurians Log v2

hold my beer 😉

wow, ask and ye shall receive 😅

hehe. lmk what kind of fulltext search you need. feedback on this is very helpful.

I am pretty far away from knowing what I need tbh. Not least of all because I need to brush up on search tech in general to be able to talk about it intelligently 😛

that should already work with datahike

Oh I didn't know that. I'll have a look

you can even stack them; this has limited use ofc., but the abstraction is compositional

the tricky bit is the write through/synch on load logic

i also recommend checking out konserve-sync, which is used by the kabel writer

I was thinking more of using redis as a cache. With tiered store do you need to be able to fit the whole thing?

Or does it just cache a few datoms?

you can decide the strategy, but if you don't replicate everything then you have potential latency cliffs on reads because you need to hit the backend store

Here is an example I created from Wikipedia with embeddings to try (no GPU needed): https://github.com/replikativ/einbetten

> Zero-cost branching - fork indices for experiments without copying data That single sentence totally explained your storage model and the elusive insistence on indexes in your proximum product page, and your earlier comment. I think this sentence/fact deserves to be highlighted earlier in the narrative. Most people do not necessarily understand how indexes can become so much more in databases using structural sharing to keep the history. IIRC datomic indexes store the actual datoms, they're not pointing to another storage area, meaning data is actually duplicated in each index, thus blurring the distinction between data and indexes. Now I understand that in datahike indexes are separated from datoms. Even though they're in the same hitchhiker tree, they're in different nodes, with datoms being in leaf nodes, is that right? For the record, and to ensure we're not pushing the git analogy too far, git commits are stored in objects under .git/objects, while .git/index is confusingly not an index but a staging area that tracks the current working tree. The actual git history is stored with each commit with a parent attribute, meaning index and data are mingled.

Datomic has the same storage model, it just exposes it less. Our indices also store facts/datoms, eavt, aevt, avet in https://github.com/replikativ/persistent-sorted-set (pss). Same for the vector index (but for vectors and an id/metadata binding). I have deprecated the hitfchhiker-tree btw., because its baseline mechanics were slower and we pay for this on every read. The ideas in there are interesting and worth revisiting, but it was better to optimize for reads with the pss.

OIC, I was reading https://medium.com/@csm/datahiking-into-the-cloud-ae9cb8619748#edf1, which is 6 years old indeed, and tbh I did not continuously follow datahike. So are datoms duplicated in each index, or is there some structural sharing in datahike and proximum?

Yes, I am kind of rebooting the Datahike project with a deeper scope atm. I always kept working on the github repo, also during my PhD, but the original company (lambdaforge) that supported it, is not active anymore unfortunately. Datahike is used at scale in the Swedish government though and we did a lot of improvements to make Datahike work well there, most importantly query engine and indexing optimizations.

They are duplicated in indices and that is what you want, because fast retrieval requires the things you need to be accessible through as few memory hops as necessary. Each index is good for a different access pattern, and you directly have the datom when you do the lookup in it. You can break out large values (and you should), e.g. if you store large strings/byte arrays, and then they can be structurally shared between the indices. But many datoms are just a few bytes large and in the indices you can do fast range scans to get a whole bunch of them in one go. Does this make sense?

I am happy to add documentation to explain this better, I just need to make sure people find what they are looking for. There are already quite a few docs now and lately I tried to condense this more.

I don't know if documentation is a priority right now, unless you've reached a stable plateau. My reaction was more about making sense of these sentences regarding proximum: > Zero-cost branching - fork indices for experiments without copying data > • Snapshots — Immutable index snapshots in O(1) (structural sharing). So I suppose the structural sharing mentioned is in memory then, not in storage, right?

Also from a point of view of a person that does not know much about datomic or datahike storage model, it can be surprising to see indexes being the center of attention. So in that regard it may help to have a paragraph provide more context.

And it would help to understand why indexes are loaded by the client, while other DB keep them internal as you said.

My value proposition is that the main benefit of immutable data structures is that they can be shared and cached much better. For instance Datahike could have a konserve store backend that directly hooks into Cloudflare (which is maybe the best cache solution out there) and no matter where you are most of the database would be just a few millisecond fetch away. The same locally on your machine (where it is), whenever any of the persistent index data structures is updated, only a logarithmically sized index delta (the path https://hypirion.com/musings/understanding-persistent-vector-pt-1 ) is written and needs to be sent/fetched, the rest you still have at hand. Does this make sense? This is also the reason why git deltas are very efficient and fast to sync btw.

It definitely makes sense and see where you want to go, I'm all in. What I think would help the reader of https://datahike.io/proximum/ is to have early on a paragraph like: Proximum/Datahike persistence model stores data within indexes. The latter are shared with clients because immutability makes them easy to cache. On the client structural sharing allows for branching into new local forks without duplicating the unchanged data.

Am I right in saying that what Proximum brings to the table is to make these forks also persistent and shareable like git forks?

Yes, exactly.

I guess it's probably better to say branches because a fork implies a new remote, in git-speak

The general sport is to take a useful data structure and carefully introduce copy-on-write during changes to make the git-model possible. I was not sure how generally feasible this is, and there might be cases where it is difficult, but proximum gave me confidence that it can be pulled off very widely. ZFS has this on file system level btw. This is very general and powerful, but you still need to implement the actual in-memory data structure this way to benefit properly (i.e. you can snapshot postgres all the time, but it will still not work well with its connection management, you will have to start processes/containers etc. and it will create much more overhead).

For AI agents people now try to use Docker/OS processes etc., but this is very clunky. It is a reasonable fallback, but not nearly as good.

I think this is where Clojure could actually break through, this is why I am working on this (in part).

Makes sense. So now the introductory paragraph that I imagine would help set the scene for the Proximum page would look like this: Proximum uses Datahike, whose persistence model stores data within indexes. Thanks to their immutability, indexes are easy to share with clients. On the client, they can be used as a cache, enabling local querying, while structural sharing allows for local branches without duplicating the unchanged data. Proximum makes these branches durable and shareable like git branches.

https://github.com/replikativ/datahike/blob/main/doc/versioning.md#query-pattern-extracting-branch-differences

TLDR; you can use Datalog itself to extract the data you want to merge.

I still need to merge the git API in datahike into the main API. This is on my agenda next.

Yeah I get the diffing part and that is straightforward enough. And I guess that side-steps the merge conflict question in that you can always just diff against the target branch. 👍 #2 is the more interesting question though. Does that approach seem reasonable? And if I wanted an audit log of past revisions, it means I would have to either keep all branches around indefinitely or denormalize them as serialized diffs. Right?

Good point. This is like branch metadata. You can track this in the same database, we might also want to standardize this. I am thinking about this atm. The one issue you have with transacting in the same database is that the transaction will be separate from the actual commit, but that is not necessarily a problem.

Separate transaction/commit is not a problem for my use-case I don't think. What would standardizing look like in your mind? I could see it going in lots of directions depending on how faithful to the git model you want to be. I don't have a strong opinion, as long as storing branch-name like this is not against the grain in some way.

I guess I'd be wary of introducing a whole "committer" concept that could diverge from the application-level concept of "user." For example Git has the user.email setting; Bread has :user/emails. So if by standardizing you mean introducing special attributes akin to :db/doc for branches/commits and their metadata, I would definitely keep any opinionated schema as minimal as possible for that reason. It is already "standardized" in the sense that generic Datalog already lets you express metadata in whatever schema you want!

I guess having a clearer idea about branch/commit semantics would clarify my thinking. In Datahike, commits are root pointers, OK...so are branches pointers to pointers? sorry read that wrong...but what exactly are commits then? Is there the same "plumbing vs. porcelain" distinction as in Git? And are commits reified in a similar way to transactions, i.e. :db/txInstant?

yes, similar to transactions; the txInstant is an internal value in the index, while datahike itself also keeps track of parent pointers https://github.com/replikativ/datahike/blob/main/src/datahike/writing.cljc#L161 that can be walked to get the commit graph; this is a newer addition, after we covered the datomic equivalents for history search

datahike's history index for instance provides the ability to query across all transactions (by retaining an index with all changes); this is not automatically given by just writing snapshots with commit ids; the latter is more the basic memory model we have now; as-of is a bit inbetween, we construct it from the history index for historical reasons, and probably that is the right thing to do, but we could also just retrieve the snapshot of that point, it would be more efficient to use it than projecting the history index.

That all makes sense, I think. > datahike's history index for instance provides the ability to query across all transactions (by retaining an index with all changes) right, the t in eavt if I understand correctly > writing snapshots with commit ids...is more the basic memory model we have now; I think you are just describing "place-oriented programming"? rich2 But what I'm asking is are the commits reified in schema the way :db/txInstant is? It looks like they are just checkpoints stored in the metadata though. They don't have relations and can't be queried in the same way.

What I'm getting at is, in my example above, the :revision/branch attr is of type keyword. The db doesn't know it happens to represent a branch or that it has any special meaning at all. But if there were, say, a :db.type/commitRef type, maybe there would be some advantages to that. I don't know if that would work, it could be utter nonsense, that is just where my brain went when you mentioned standardizing

We could add the commit-ids into the database, yes. The one thing to keep in mind though, that you can gc them if you don't want to keep all these snapshots around, in which case they would be gone. That can be fine, then the commitRef points to nowhere which will indicate it was gc'ed. I am just saying this because we don't gc things inside the indices, only snapshots.

ah ok that makes sense

The snapshots are a more fundamental memory model that is different to Datomic or DataScript (although the storage support I helped add there makes it similar), because we keep the snapshots around for distributed/remote readers and sync them as needed in parallel. Datomic needs a transactor overlay which is the most complicated piece in its memory model and a central point of failure. Datahike always writes the full index to storage and any process can read it, even if the writer (transactor) is down.

E.g. you can give somebody only access to your S3 bucket, and they can join your db from S3 with Datahike against their own databases.

Most people don't want to be too permissive with their DB, but this should not be conflated with a limited memory model which forces all participants to meet in a MVCC context (e.g. SQL dbs, LMDB etc.). Clojure actually tried to get away from this, but Datomic still bought back into it a little, which in the end made it very complex, I think.

yeah totally agree

DataScript also keeps a transaction log, which I thought about, too. The upside is less write amplification and overhead. The downside is that on every db deref you need to retransact it and load potentially many index fragments from storage for that. I think the write overhead is not that bad, you can just do gc regularly and reclaim the storage and the upside is that you always have optimal indices available for each snapshot you want to use.

We don't have problems with write throughput in general, so this is an ok price to pay in my opinion.

It might be annoying if you are on DynamoDB and have a lot of small transactions.

Because of the pricing model.

It is always possible to add a transaction log though.

Anyway, this is a bit of a digression, but maybe it helps to understand what I am prioritizing.

It seems like a good tradeoff in general. Appreciate the context! For my use-case I will probably just keep branches around indefinitely by default, and store the branch names as keywords. That seems like the simplest way, and I don't think it incurs much storage overhead above and beyond what you'd incur for keeping history around generally? Also, denormalizing would be bad for GDPR, and would probably mean a bunch of other weird edge cases down the line! At some point Bread could optionally GC open db branches after some time, and/or denormalize on merge if data excision doesn't matter to the user.