Hi @U2M723H42 you will need to use the open-tx-log API for this, and scan the log for put operations (or whatever operations you are interested in)

Ah interesting, then I would filter that down to just the document types I'm looking for right?

exactly, yep 🙂

Great, I'll give it a shot, thanks for your help 🙂

Hey @US03ZP2F5 in theory such a setup could work great - RocksDB has an OpenAsSecondary capability for exactly this use-case https://github.com/facebook/rocksdb/wiki/Read-only-and-Secondary-instances But the team here hasn't yet attempted to configure XTDB like this before or measure the impact of that sort of clustering setup. Let alone on EFS specifically. Instead I tend to steer people towards the naive setup of each node having its own RocksDB instance on as fast a local SSD as possible for maximum performance and redundancy. In contrast a shared instance running on EFS could be a interesting option for very dynamic workloads where peak performance is less critical and you want to overhead the checkpoint-downloading overheads. When I spoke to the team at https://www.speedb.io/ last year they suggested that EFS + OpenAsSecondary actually performed very well, so maybe it's worth the exploration

regarding EFS, if running as an ECS task with an EFS Mount that is not a Read Replica, it means that every instance in the service needs its own new mount point, and for a task restart that means "downtime", since the old task has to release the EFS Mount before the new task can start up. So there cannot be scalability (automatic or manual count updates) without first creating the additional EFS targets. For the cases when we know it is readonly usage of XTDB, it would be interesting to be able to have snapshots (for restore and reading updates of the index) being able to be shared so any new instance, in addition to restarted instances, could instantly be up to date without that lag time of releasing the EFS mount or creating new EFS mounts if need to scale out

I didn't see anything in the documentation, but wanted to check if there was support already

Thanks!!!

interesting, well I would certainly be keen to hear how you get on with EFS if you do try it - and happy to help advise on the config/changes to make OpenAsSecondary work 🙂