Hi 🙂, How are people doing Datomic Backups on AWS? Anyone managed to get it to work with Lambda?

@mbutler I had a successful proof-of-concept of a scheduled AWS Lambda launching a short-lived EC2 instance to run the Datomic Backup.

@stuartsierra This was what i assumed would be necessary (rather than being able to run the backup directly in a Lambda), how were you letting Lambda know that the backup was complete (http req maybe)? Any pitfalls I should be aware of? Or is it as simple as Trigger Lambda to bring up box with S3 access, make sure datomic files present on instance, run backup, trigger lambda to kill box.

@mbutler we have a small ec2 instance running backups continuously

I have Jenkins task that makes the backup

Remember backups are strictly incremental so can be run very frequently with little harm

Yes, had also considered running from an application server, just seemed that Lambda might be a nice solution. however the extra step of launching and managing an instance makes it a little less appetising.

https://github.com/robert-stuttaford/terraform-example

-whistle-

don’t run backups from an app server. it’s a java process, it’ll eat all your ram 🙂

Allowances had been made for the extra RAM usage but I agree its a messier solution. 🙂

i’m fairly certain that the cost of an ec2 instance doing continuous backups will be less than the DDB read spikes you’d see by doing bigger infrequent backups. i have no actual facts to back this up

beause of ec2's hourly pricing, spinning up ec2 instances doesn't make a lot of sense to my mind

you might as well keep one going continuously

I confess from a practical standpoint I'm inclined to agree @robert-stuttaford, Lamda just seemed like the right solution for a "backup" job.

(now if only aws introduced by-second billing for long-running tasks, that'd be a game changer)

@pesterhazy Doesn't hourly pricing incentive's the launching/killing of EC2 instances?

only if you back up less than once an hour 🙂

I suppose with the incremental backup, there's an argument for smaller windows, point well made 🙂

we do it continuously because AWS. don’t want to be at the mercy of DDB being down in that region and our business coming to a halt

we’re not in US-EAST-1 but still

Great points, thanks 🙂

@mbutler I think I had the Lambda launch the instance, then the instance destroy itself when it was done.

Gotcha 🙂

I have a weird problem. I'm trying to transact with a single, pretty simple datom, and am getting :db.error/not-a-data-function.

I first tried with (d/transact (conn) [[<eid> :my/attr "value"]]), then with (d/transact (conn) [[<lookup-ref> :my/attr "value"]])

Both times it complains that the value which is not a data function is the eid of the thing I'm updating.

(so it resolved the lookup ref).

Oh, god. :db/add.

hahaha. Ignore me.

That one has bitten me a few times as well