Fork me on GitHub
#datomic
<
2021-09-23
>
daniel.spaniel11:09:30

Will Datomic / Can Datomic -> ever allow us to retract / change <- attribute definitions that are not used yet? I mess up quite often in defining an attribute as integer instead of double, and then once it's transacted , Datomic does not allow change of attribute type or even to retract it completely. Seems like if the attribute is not even used yet, we could make that change?

jaret14:09:58

You can never alter _:db/valueType_, _:db/tupleAttrs_, _:db/tupleTypes_, or _:db/tupleType_. But you can change the schema of an attribute. Is there a specific example you have? I ask because you can change the :db/ident of an attribute and in that way you can technically re-purpose or re-use the ident you are attached to. We discuss this in https://docs.datomic.com/cloud/schema/schema-change.html:

jaret14:09:08

> We don't recommend re-purposing an old `:db/ident`, but if you find you need to re-use the name for a different purpose, you can define the name again as described in attribute-definition. This re-purposed `:db/ident` will cease to point to the entity it was previously pointing to and ident will return the newly installed entity instead.

jaret14:09:43

If you have never used the attribute this kind of re-purposing through re-using the :db/ident seems like what you would want to do.

jaret14:09:51

@dansudol are you using Cloud or on-prem?

daniel.spaniel14:09:33

@jaret we are using Cloud. I don't want to repurpose the :db/ident. I want to alter the :db/valueType flat out. But only when it's unused anywhere. That seems reasonable right? No datoms using db/ident = ok to change that valueType?

daniel.spaniel14:09:16

I know this is an iilegal proceedure ( believe me I know ) and it hurts me alot because if I make one mistake on schema .. like this .. there is no do over

jaret15:09:45

@dansudol I feel for you on this. Schema design is hard in any DB and I strongly recommend to anyone that you work out new schema in a staging environment or a test DB before moving schema to production. Even better you can experiment at https://docs.datomic.com/cloud/dev-local.html. Use import-cloud to get your DB locally. If you haven't used the attribute in question, here is exactly what you can do to save the name and add a new schema attribute with the valuetype you want:

;start with your attribute foo/bar
{:db/ident foo/bar
:db/valueType :db.type/long}

;archive foo/bar because you realize later that you do not want this attribute to be a long

[{:db/id :foo/bar
  :db/ident :foo/bar-no-longer-used}]

;transact your new schema using your favorite name foo/bar with a new value type double

{:db/ident foo/bar
:db/valueType :db.type/double}

daniel.spaniel13:09:24

As you probably already know @jaret this worked .. I am alittle shocked . but I will take this win .. thanks again!

jaret13:09:02

@dansudol It's important to understand that you aren't getting rid of your initial schema mistake with this approach. The original schema is still there. So you can always see the history of this move.

jaret13:09:37

So I definitely recommend again the whole testing schema in non-production etc

daniel.spaniel13:09:41

for sure for sure .. we test on staging always ..

daniel.spaniel15:09:25

Gee .. that looks pretty magical.. I will try it. And we use dev-local. And we use staging environment too. I just sometimes push things to staging that have bad schema. Sure, I could blow away staging, but sometimes its nice to not blow it away. This trick I have not tried quite like this but thanks for the tip!

Daniel Jomphe20:09:29

Datomic Cloud backup &amp; restore with an external lib With a small DB seed of a few thousand datoms, we were able to use @tony.kay's https://github.com/fulcrologic/datomic-cloud-backup lib to backup and restore our DB in different scenarios. 🎉

datomic 3
Daniel Jomphe20:09:58

Solution We spent more time operationalizing these ops than playing with Tony's lib. We're now able to remotely connect to our DB in each environment, take or incrementally update a backup on our local computer's hard drive, and restore to a new DB in any environment by replaying the backed up transactions there. We're also able to dynamically change an environment's connection from its original DB to its restored DB, without going through a full Ion deployment cycle. Thus we can restore to a new DB in some system, and when the DB's ready, have the instance(s) switch their connection to the new DB and seamlessly continue.

Daniel Jomphe20:09:19

Advisability With that said, I'm not sure it's the best way to proceed. For example, if we switch DBs dynamically, we need to remember to git-commit our config changes so that the next Ion deployments won't switch us back to the previous DB. Also, the CloudFormation template has a parameter to pre-heat a chosen DB, and we should remember to update this parameter after having switched.

Daniel Jomphe20:09:47

Performance Measures of a restore of a few thousand datoms: • ~3 seconds using dev-local. • ~3 minutes using Client, remotely connected. • ~? minutes using Client, Ion-locally connected: I'm yet to try this 1️⃣. It surprised me that it took ~3 minutes to replay this small number of transactions and datoms. We're soon to be in production with our first client and we wonder how long it would take to restore bigger, more real amounts of txes and datoms. Scalability Of course, thanks to Tony's good work, we could try: • 1️⃣ storing our backups in S3 and setting up a streaming restore server to continuously-incrementally prepare a restored & ready, unused DB. But that would imply many more efforts on our part.

Daniel Jomphe20:09:30

cc @U072WS7PE re: our (now archived) discussion about backups & restores for Cloud.