Clojurians Log v2

A few more updates on write latency: File backend locally d/transact 28ms+-5 with fsync, 13ms+-3ms without fsync (relying that the OS will still be able to flush to disk in most crash scenarios and you will only lose data if the OS or data center has an error), memory backend d/transact is ~2ms+-2. In comparison Datomic on dynamo supposedly has an optimal latency for around 15ms (according to ChatGPT, this will need checking) for tiny transactions. Atm. even the underlying PutItem does not show such low latency for me. We can half the dynamo transaction time by changing the transact logic. This is generally possible for stores that can transactionally update multiple values. Hacking that in will not be hard, properly abstracting over it will require a bit of thought.

So I think you can track the state of real-time games with 60 FPS with the memory backend, have fairly snappy local durable databases and then cover the spectrum all the way to some commodity bulk storage such as S3 or whatever other cloud backends you have. The important aspect of this is that from the query perspective they all form one space. You can just join over all of those as fixed snapshots in your application logic and if the query caches are kept warm (something that is not automatically done through streaming of updates from the writer yet, but is also not super hard to do) then the readers will not pay for the latency unless they access data they have not cached yet. So you can read with in-memory speeds if you are somewhat smart about your cache management (which is not that hard for immutable values). For instance it is not hard to add local SSD caches for S3 as well (just a few lines of code), which will significantly reduce read latency if you cannot keep everything in memory.