Clojurians Log v2
datahike 2024-11-10
@sasha_bogdanov_dev I have added a DynamoDB backend https://github.com/replikativ/datahike-dynamodb, please try it out if you have the time and provide feedback (in particular about latency). One still has to get rid of the reflection warnings for native compilation https://github.com/replikativ/konserve-dynamodb/blob/main/src/konserve_dynamodb/core.clj#L26 and I am not sure about the capacity settings for Dynamo, when I was trying to transact too much data I think there was an error that was not propagated properly and the transact call hang. It should properly throw this error ofc.
Oh wow) so fast! With transactions could be the limit of 100 records or 4 MB of data. If reached, need to batch request.
Implementing backends isn't that difficult, we put quite a bit of effort into making it simple despite providing convenient storage abstractions. But maintaining a lot of backends and making sure they work really well can be a burden. That is why I am usually asking first why it should be implemented.
If you upload snapshot to clojars it would be very nice. I have a little problems with git dependencies right now
@timok I probably should not bother you with this all the time and just set up the releases myself. What needs to be done for that?
Figured it out: https://clojars.org/io.replikativ/datahike-dynamodb
PSA: My db now has 600M datoms and everything is working fine:muscle: I hadn't checked in ages.
Wow that sounds really promising
*Correction: 16M datoms. I thought the output from backup was datoms turns out it's file size.
Nice!
I will probably purge some though. My backup is 600MB
I am exploring a transaction log to reduce the amount of data written with every commit. Alternatively/Complementarily we could auto-gc automatically in the background as well. How do you feel about handling that? Is it a concern for you?
This isn't a datahike issue. It's some of the design choices we've made. Some have been down right terrible, but that's the startup life haha:joy:
Hehe 🙂