Fork me on GitHub
#datomic
<
2020-09-24
>
tatut06:09:56

any recommendations for backup/restore in datomic cloud? the feature request hasn’t seen activity in a long time. we’ve been rolling our own application level export/import but that unfortunately makes every entity id change

tatut06:09:30

mainly for disaster recovery or just migrating the database to a new environment

steveb8n09:09:10

basically there isn’t a solution except to build your own. the (un)official answer seems to be that backups are not required because s3 is so reliable and, because there’s no excision, no data is ever lost

😮 3
steveb8n09:09:34

I can see the sense in that response but I think it doesn’t account for our customers who don’t understand our powerful new toolset. it forces us to take our customers out of their comfort zone which is not great for conservative (i.e. many enterprise) customers

6
steveb8n09:09:07

the new local dev client supports importing cloud data to local if that’s one of your use-cases

steveb8n09:09:21

but for migration, you have to roll your own

steveb8n09:09:03

I’ll be happy to be corrected on any of these interpretations. FWIW it doesn’t change the fact that I really like the cloud managed service.

steveb8n10:09:01

I have a uuid attr on every entity. If you have this, you don't care about entity IDs changing. Is that something you have tried?

kennytilton11:09:51

Not sure if you have the luxury of time, but from what little I know it would be a good idea to do what it takes to abide entity-ids changing. My understanding is that backup/restore will not guarantee entity-ids being unchanged.

joshkh21:09:21

i am in the exact same boat right now, and am finding it challenging to justify to our enterprise customers that we can't "simply" backup/restore a db from storage to meet their (i.e. not our) DR requirements. and unfortunately for us, dev-local is not yet an option because we have string values that exceed dev-local's max character limit. that being said, i did a small test with dev-local to replay the demo mbrainz db transaction log into a new db and it worked well, but the t values are of course different which is a real shame.

steveb8n02:09:34

@U0GC1C09L exactly! despite the fact that we might not need backup/recovery, in an enterprise sales cycle this can be a real problem. Even worse if it’s an RFP situation because the prospect writing the RFP might consider common DR techniques to be a must have. It doesn’t matter that we can explain it away, the internal politicians in the customer can simply use this as a battering ram to avoid choosing our product. It’s not always a technical question: I hope one day Cognitect will provide an answer for export so that Datomic Cloud can be used without this risk in the sales cycle. @U05120CBV any comments on this?

steveb8n02:09:27

what’s interesting is that Datomic provides DR features that other dbs cannot e.g. you can recover a single tenants data to any point in time, even in a multi-tenant system. Update in-place dbs cannot do this. So we are technically superior for DR. But that doesn’t always work in enterprise sales.

Nassin02:09:44

Curious, why does your application relies on entiti ids ?

tatut05:09:31

DR is also to protect against human error, customers want to have a backup file stored in a completely different AWS account S3 bucket… or even have it downloaded to their own infastructure

tatut05:09:27

and when you have backups you need to be able to restore from them… now we’ve rolled our own and fixed some mistakes we’ve made in relying too much on :db/id values

steveb8n06:09:22

that’s true. you could accidentally delete your Datomic s3 bucket and then you’d be finished! goodbye biz 😞

steveb8n06:09:40

I wonder if some kind of s3 level backup would be supported by Datomic to guard against this?

stuarthalloway19:09:29

@U0GC1C09L @U0510KXTU We hear you and working on things in this area.

👍 6
🙏 3
joshkh19:09:03

cheers. i know that we're in good hands. 🙂

daniel.spaniel18:09:29

This is mega important, not just for enterprise, but for any datomic cloud user who wants to preserve their sanity. The sooner the better Stuart. This is huge deal.

joshkh21:09:41

for the sake of testing query performance, is there a way to flush/bust the query cache in Datomic Cloud other than by renaming bound variables?