Clojurians Log v2

Thanks for the detailed explanation! So. I've got a push based architecture. Using CQRS. Effectively, if there is a change then connected users get the latest view pushed. But, the server controls the rate of rendering. So Command -> update db -> change goes into a throttled queue -> clients get updated. Clients share common components. E.g; a leaderboard view. I can cache that in user land or do reactive signals. But that does add a bunch of complexity in user land. Then I got thinking, if I had database as a value for that update tick, I wouldn't need to think about user land caching, as the largest piece of work is hitting the db. But if I'm doing 10000 leaderboard queries against the same db value I get amazing cache usage and I don't have to think about explicit caching. I basically need database as a value for 100ms ticks. So I was curious if there was a way to get that in Datalevin. But by the sounds of it the page based caching is probably good enough regardless of updates coming in and changing things during those 100ms tick. I'm still exploring a simple way to do user land caching if I need it. Seeing the same state would be nice. But the downsides of page growth doesn't sound great. As this read tx would be open every 100ms for the amount of time it takes to do the work for all clients. So probably a bad time.

I think I'll just implement user land caching if I need it! Thanks again for the detailed explanation as always!

A leaderboard use case is a great fit for the KV store of Datalevin. You can open a :temp? true KV store as your cache. It can easily do a million read/writes per second on an m3 MacBook Pro if you do some batching. So a 100ms time window would allow 100K read/writes. Probably sufficient for most use cases.

Completely missed the :temp? true feature. That's awesome.

exactly what I need!

with-transaction works with a single read/write transaction, so it does block write.

I don't know what would be the benefit for what you are trying to do though.

LMDB caching is on the basis of pages, it has nothing to do with transaction.

If a page is in the cache, it's in the cache, regardless which read transaction brought it in.

the snapshot is not a real thing, it's conceptual.

it's a lot like immutable data structure, the "copy" is conceptual, not a real thing.

The design motivation of LMDB is precisely NOT to manage buffer cache of its own. It's all OS, which knows nothing about LMDB's transactions. So really, caching has nothing to do with transaction.

In conclusion, running a bunch of queries against the same read transaction accomplishes nothing in terms of caching. But it does ensure they see the same data. But if there's a write transaction happening at the same time, they will not be seeing the change. That's may or may not be what you want. In any case, I don't want to expose the transaction object to the users. It just brings too much trouble for questionable gains.

None of our API functions has explicit transaction objects in it, it's a deliberate decision. I want Datalevin to be simple to use. Ergonomics is the main driven force.

When there's a functional necessity, I did introduce with-transaction, but that's for atomic operations. For reading, I don't see a need for it.

So, what exactly motivated you to want to do this? I am curious. You feel the read is not fast enough or what? Maybe we should look into that instead.

Grouping a set of queries will have two benefits: 1. these queries see the same data. 2. saving the cost of reset/renew read transactions. So it does improve performance for these reads slightly. I can be persuaded to add a with-read macro if there's compelling use case for point 1 above. . The disadvantage is that a long lived read transaction will keep pages from being recycled so the DB can grow large very fast. So I really need to see some compelling use cases.