Why you don’t want s3 bucket or something with compatible s3 interface?

we’re on GCP, not aws

If your storage snapshot is transactionally consistent, you can snapshot. Many (eg dynamodb’s) are not

The advantage of the datomic-level backup is storage independence and automatic garbage collection (it only restores non-garbage)

Fwiw on both gcp and aws at two different companies we eventually converged on using a persistent backup because it’s faster and cheaper, although more work to set up

*persistent backup volume

One big problem with backups is there’s no way to remove old ones, so you need some sort of rotation anyway. Might as well be a persistent volume vs a directory of your bucket

@U3ZUC5M0R GCS has a https://cloud.google.com/storage/docs/interoperability#xml_api

thank you @U09R86PA4 and @U74MQ2347 - super helpful

s3 backups are nice because they are incremental you can run a script, once every hour, and have a nice and hot backup 🙂 (once in a month, copy the "currnet-backup" into "aug-backup")

You can do this just fine with a persistent volume also. We run a backup to a volume continually (i.e. runs backup in a loop, snapshots the volume), and every week we spin up another backup instance and volume with a few hours overlap to allow a full backup to complete

We use persistent volumes mostly for cost and speed: s3 operation cost adds up and is much slower to complete a full backup (> 10 billion datoms here). An ebs volume is much cheaper and faster.

On GCS we did this because we had no other choice at the time

(might still not have a choice--I don’t know if you can do the hostname override needed to make “s3 compatible” backends usable by the backup process.)

What are you doing that would require a permission change to the Lambda?

Consuming an SQS queue. Needed permission to read and delete messages.

The Lambda function should just be a pass through to code running on the node. How are you modifying code on the Lambda?

We’re not modifying code on the lambda. But the https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html#services-sqs-batchfailurereporting apparently needs permissions beyond the default. Not surprising in some ways since Cognitect can’t know my SQS queues in advance, right? And it definitely needs to read (pull!) and delete potentially sensitive messages.

Ah. This seems like a valid use-case, and it does not seem like there’s a supported way to do this now. The created lambdas currently have an execution role, right? If so, and this a little gross, you could read the resource in using your IaaC tooling, grab the role arn, and attach your custom policy to it.

I ended up modifying the node policy. Not nice.

I think the only length limit that is actually enforced (i.e., a transaction will be rejected) is a 256 limit on strings inside tuples.

I strongly caution you not to casually write any string longer than say 1-2k

and use an attribute predicate to enforce length limits

it is very difficult to detect, correct and recover from perf problems related to too-long strings and large segments down the road.

We now enforce the 4K limit in cloud.