How do people handle schema migrations with datomic cloud? I was looking at conformity but it doesn't support the client api.

Hi everybody, We are having an issue with the way datomic tx works. Our application allows user to register driver registrations with a start and end date for a certain license plate via xml files. If a new driver registration overlaps with the start and end date of the old driver registration, we cut off the old dates in our timelines. Our timelines are sorted on the highest tx. Just to be clear about the overlaps. Here's an example: driver registration A inserted at 2019-09-09 {:driver-registration/id "1XXX001-2018-01-01" :driver-registration/license-plate "1XXX001" :driver-registration/start #inst"2018-01-01" :driver-registration/end #inst"2019-01-01" :driver-registration/driver {:person/first-name "John" :person/last-name "Doe"}} driver registration B inserted at 2019-09-10 {:driver-registration/id "1XXX001-2018-06-01" :driver-registration/license-plate "1XXX001" :driver-registration/start #inst"2018-06-01" :driver-registration/end #inst"2020-01-01" :driver-registration/driver {:person/first-name "Mary" :person/last-name "Jane"}} timeline: 2018-01 2018-06-01 2020-01-01 |--driver John Doe---|-----driver Mary Jane------| As you can see the end date of driver registration A (John Doe) was cut off by the start date of driver registration b (Mary Jane). We retrieve this data by the following query: (d/q '[:find ?tx (pull ?dr [: {:driver-registration/person [:]}]) :in $ ?license-plate :where [?dr :driver-registration/license-plate ?license-plate] [?dr :driver-registration/id _ ?tx]] db "1XXX001") We sort the list by the value of ?tx and cut off the dates were necessary. This works fine for most cases but now image the user has made a mistake and wants the end date of driver registrations as the cut off date. Like this: 2018-01 2019-01-01 2020-01-01 |----driver John Doe-----|-----driver Mary Jane------| When the user uploads a new xml with the exact same data of driver registration A, we expect that driver registration A would now have the highest tx. But due to datomics redundancy elimination, datomic will filter out the data of the transaction and never update the tx of driver registration A. When the user asks for the driver registration timeline, he will still receive the old one. Is there a way to solve this issue on the query side? One of the solutions would be to add a field :driver-registration/last-upload with a date value to driver registration but that feels as if I'm rebuilding the db/txInstant system.

is bigdec safe from floating point errors?

Datomic cloud documentation mentions: > [:db/unique] attribute must have a :db/cardinality of :db.cardinality/one. https://docs.datomic.com/cloud/schema/schema-reference.html#db-unique However, in my project, I’ve been using the following schema definition just fine:

{:db/ident       :user/email
    :db/unique      :db.unique/identity
    :db/valueType   :db.type/string
    :db/cardinality :db.cardinality/many}

(d/transact conn {:tx-data [{:user/email "calvin"}]})
=> {:tx-data [#datom[13194139533323 50 #inst "2019-10-17T13:44:42.951-00:00" 13194139533323 true] #datom[10740029580116059 78 "calvin" 13194139533323 true]]

(d/transact conn {:tx-data [{:user/email ["jenny"]}]})
=> [#datom[13194139533325 50 #inst "2019-10-17T13:47:13.676-00:00" 13194139533325 true] #datom[13453624277467228 78 "jenny" 13194139533325 true]]

(d/pull db [:user/email] [:user/email "calvin"])
#:user{:email ["calvin"]}

@cjsauer can you do the following:

(d/pull (d/db conn) '[*] :user/email)

#:db{:id 78,
     :ident :user/email,
     :valueType #:db{:id 23, :ident :db.type/string},
     :cardinality #:db{:id 36, :ident :db.cardinality/many},
     :unique #:db{:id 38, :ident :db.unique/identity}}

fascinating

what version of datomic? (cloud or onprem)?

This is cloud. Checking on version (what would be the fastest way to see that? CF?)

yep

This is in the output section of my compute stack

DatomicCFTVersion	512
DatomicCloudVersion	8806

thanks

i’m looking into it

I also did a split stack deploy, if that matters

@cjsauer you’re correct that it can be done. Dont. 🙂 The semantics of unique identity are such that having a card-many attr there is pretty dicey I’ll look into filing this as somethign that should potentially throw or warn

@marshall ha okay, thanks for checking. I’m a bit puzzled tho. Email addresses seem to challenge those semantics. Is there a technical reason that unique-many attributes can’t exist? Regardless, throwing an exception there would be great.

i suppose not a technical reason so much as a semantic one unique identity says “this specific attr/value pair is unique in the database”

being able to assert multiple values for that is…. complicated?

>“this specific attr/value pair is unique in the database” This could still hold for a card-many attribute theoretically. I opened an issue/PR for datascript before posting here on this subject, and @tonsky mentioned: > Upsert would not work because it’s not clear which value to look at. Or we must look at all provided values and make sure they all resolve to the same entity. This might be the complication. When upserting multiple values one must ensure that they do indeed resolve to the same entity. https://github.com/tonsky/datascript/issues/320

well, you definitely open up to conflicts

i.e. what if you have two separate “entites” in your transaction, each using one of your unique emails, but they both contain a separately conflicting datom for some other attr

since they both resolve to the same entity based on that unique id, you then have a conflict of the other datom

It would seem appropriate for that transaction to fail in that example. The two entities are resolved to one, and then can be constrained from there.

Maybe the detection of that conflict is the hard part tho. I’m ignorant of the details.

Given this limitation, is it possible to model users being uniquely identified by multiple email addresses? This felt like a very natural way to model my domain, where a user can be part of multiple teams/orgs, each with their own email domains. It would be great if those emails did indeed resolve to the same entity.

You have the same conflict if you use different identity attributes on the same entity.

What is the conflict? I’m not disagreeing, just failing to conceptualize. Is there a minimal example of a transaction that shows this?

Mostly tho, semantically, card-many-unique attributes seem very natural. Using Game of Thrones as an example, royal figures can have many identifying titles: Robert Baratheon, first son of XYZ, slayer of ABC, builder of QRS, etc etc etc.

Can you build 2 datomic cloud instances in the same region with different names? can you share the key?

Is this wise?

which key @hadilsabbagh18?

The key named datomic.

can you be more specific? some cloudformation parameter?

I am going to create a separate ssh keypair. I think this is wise...

oh that's the AWS EC2 Key Pair -- yes you can share those

Ok.

Thanks!

we put one Datomic system in one region, we attach a couple query groups to it, and it hosts hundreds of databases

but you can put two isolated systems in the same region, too

the AWS EC2 Key Pair is only used if you need to ssh to the actual datomic box

the Bastion/Analytics Gateway uses a different key pair

I started deploying to us-west-2 according to instructions and I repeatedly get this error:

Embedded stack arn:aws:cloudformation:us-west-2:962825722207:stack/stackz-StorageF7F305E7-13QLTET3W9OAQ/ad7f7950-f0ff-11e9-b33c-02a77ed54d64 was not successfully created: The following resource(s) failed to create: [DatomicCmk, CatalogTable, FileSystem, LogGroup, LogTable].

https://docs.datomic.com/cloud/troubleshooting.html

you'll need to look at your Event Log in the CF Stack

Thanks @ghadi!