Clojurians Log v2
datahike
Headsup: In preparation of distributed synching support and also reflecting the store-id config issues people had in the past the newest konserve version now requires you to declare a globally unique UUID per store in the config under :id : https://github.com/replikativ/konserve. This will make it possible to replicate the same store between different media cleanly (e.g. server S3 to browser IndexedDB, etc.). I will update all the backends in the next days to follow the same pattern. More importantly konserve is now fully abstracted over, i.e. you can inject a dependency for a new backend (maybe one you wrote privately) and the whole database stack will work transparently on top. This is not done in Datahike yet, but I hope to do this either with the cljs PR or right after. This will also render all the datahike-jdbc, datahike-redis, ... etc. repos obsolete, which is very nice.
Whoa big news!! So basically I should install konserve-jdbc instead of datahike-jdbc and it'll "just work"? I don't call any code in datahike-jdbc, just require it and use it "ambiently" I suppose
yes
it is not super magical, it is just that konserve now has lifecycle methods for store management that were in the datahike bindings before; this was a bit annoying, because while konserve users could use the protocols in a store agnostic way, they had to explicitly import constructors/destructors etc. for a store from a backend, tieing their code to a store implementation; this is now generally not necessary anymore; you just need to require the konserve backend once somewhere in your project and everything can use the transparent constructor/destructor multimethods without knowing about backends; this is good for any database/storage solution building on top of this stack, not just datahike
@alekcz360 @hoertlehner this is in particular relevant for you guys. @alekcz360 I will open a PR for konserve-jdbc, it would be cool if you can review that.
All store backends are now updated (JDBC PR pending). konserve also now has support for CAS (compare and swap) for backends that support it, this means you can apply k/update-in safely from different machines. S3 + GCS support this now. This works with retries, so it only makes sense if you don't have high write contention between machines, but it is very nice to have if you build distributed database solutions. https://github.com/replikativ/konserve-s3/?tab=readme-ov-file#optimistic-locking-for-distributed-updates
So... if I use datahike-jdbc now it will be sane to wait konserve-jdbc PR approve before datahike version bump?
yes
I also did extensive tests on the memory model of konserve and added jimfs tests for the filestore, which uncovered an issue with the exception handling in the keys listing on it, which is also fixed now. I can simulate all kinds of storage failures now and check that the DefaultStore implementation is correct (which is used by most backends to provide the high-level API). The notable exception is the new LMDB backend which ties the high-level API directly into the memory mapped layout of LMDB (providing full edn and custom handler support), yielding very high performance if you do a lot of small operations on a local store https://github.com/replikativ/konserve-lmdb/. This is still WIP, I have been benchmarking on it extensively and removed a bunch of issues, I think it is a sound design, but there might still be some issue somewhere (I cannot prove the memory correctness the same as for the DefaultStore without modeling LMDB behaviour).