Clojurians Log v2

There has been progress here.... We contracted with @kenny last Fall to update and improve https://github.com/kennyjwilli/datomic-backup and we can confirm that it works nicely for dumping a DC database into a fresh copy. Not so much a backup as a replication. We use it to copy a DC database in one AWS account to a fresh copy in another account and we have done that successfully about a dozen times over the past six months. Our source database is almost the exact same size as yours. The copy proceeds through a (REPL) intermediary. There is a mode to incrementally append to the copy, but it has not been battle tested (by us at least). It was added to address the frustration that can come from spending 10 hours making a copy only to have a couple hundred datoms be added to the source that would be really really useful in the copy. We would love to have others use it; kenny and Sun Tribe (RECmint) were totally in sync about making it open source and free.

I wonder if approaching backup from the S3 level would work, you could always just copy the data out of the bucket periodically. You'd have to stop datomic to ensure it’s consistent though

Unfortunately, not all the data is in S3, some is exclusively in DynamoDB.

Some more info about the approach we are using with @kenny’s tool... to copy (so decant and pour into a fresh db) takes 10 hours for us (100M datoms). We use the current-state-restore option so it does not include history, but you could try the incremental-restore option which does include history (from a given point forward -perhaps Kenny can elaborate). That was the focus of Kenny's most recent work and it was tested once or twice late last year on our db IIRC.

If you're halting writes to clone the s3 bucket, you could just as easily dump the dynamodb tables at the same time

then for some cases of audit trailing is good to have the old db as a reference, right? that's ok :))

but we don't have previous retractions with incremental-restore , right?

IIRC, you will have "previous" retractions as long as they were transacted after the current-state-restore and before the incremental-restore. If that is really important to you, perhaps @kenny can expound on a process to force the current-state-restore to be from t0 (=> nothing) and then apply only incremental restores thereafter. I imagine it would be slow, but it might get you all the history of all your retractions.

^ I would love to know if this reliably works.