thanks, I am not currently using datahike but lurking on the channel. There are some changes happening at work and it's an oportunity to work on storage. I thought I would explore datahike for this usecase, but I kind of need replication.

That sounds cool! Can you explain a bit more what your ideal replication setup would look like? Our approach atm. is to leverage immutability in the same way as Datomic. You can scale out reading (queries) horizontally without any coordination (but compared to Datomic you don't even need to have a transactor running for reads), and if you want to write you run one transactor process (currently this would be datahike-server, in the future there also could be other implementations, e.g. on top of Kafka or so). This already works seamlessly with PR 332, it is just not documented outside the PR yet, but I can walk you through the process (while documenting it). This is the repo that provides the dispatch for datahike-server https://github.com/replikativ/datahike-server-transactor (you just need to require the namespace for the multimethods/protocol implementation to be registered).

well, we have some collections of articles. They are updated infrequent. Most once per quarter. Some daily / weekly / monthly.

honestly, I think a replication like syncthing provides would be great

https://docs.syncthing.net/specs/index.html

I am using syncthing with some files and I am very happy with it. I do have it on my todo to try implementing the protocol in clojure as an exercise

the syncthing replication spec is simple to follow - if you have time to check it out - please let me know if it's suitable to implement for datahike

it might provide the motivation to start working on this

syncthing is cool, i have played with it in the past and this is what we had in mind in our blog post. Any mechanism that synchronizes files is sufficient (even rsync would do), the only thing you need to ensure is that there are no multiple writers in the system at any point in time, otherwise syncing would probably overwrite the changes of one of the writers with the ones of the other. For this case you need a single writer in form of datahike-server atm. (while you can still replicate with syncthing).

Another subtlety is that your syncing mechanism should synchronize files in chronological order in which they have been written, otherwise parts of the indices might become visible before all of their pointers are reachable. This would not cause any data loss, but it might corrupt the reader state.

The latter problem can also be avoided by refreshing the in-memory connection only after a synchronization completed.

yeah, having in app sync should do work as to keep system consistent

Also, datahike can seamlessly join over multiple databases, so instead of letting mutliple processes write to a single db, you can shard and then syncing is not a problem.