Fork me on GitHub
#datomic
<
2018-02-17
>
uwo17:02:02

We’re doing change data capture from our legacy database, and we found ourselves making an assumption that was silly in retrospect. We would construct a tree of txdata to insert, where the root was uniquely identified and thus would upsert, but where the branches were component references. By asserting this as is, we were orphaning the component references. Clearly, we were in a document mindset. Whoops! In order to remedy this, we’re going to try the following: 1. Determine novelty by diffing the txdata prepared for assertion with the corresponding entity in the db and, recursively, its component references. 2. assoc in db/ids to the relevant component refs to ensure upserts I know cardinality many component references will require additional work, but I wanted to run this past y’all to make sure I’m not making a poor design decision or missing something obvious.

steveb8n22:02:08

@uwo I have a “public” id (a uuid) in all entities, including component entities for this a many other benefits. Also in data layer tests I have a fn which checks the ids persisted against the before and after db and it can report on which ids were inserts. this technique allows me to know exactly when I am (accidentally) adding new entities and it reports on component entities just as well as top level entities

uwo16:02:01

Thanks @U0510KXTU! In this case the data from our legacy system doesn’t map directly to the new data model. Most of the data that becomes component refs doesn’t have a primary key in the old sql database, so the only thing that identifies it is a unique attribute on the parent entity. When the data from the change data capture is prepared for assertion, we effectively want to merge it with the same structure that we can d/pull from datomic.

uwo16:02:52

While adding a uuid to all our component refs may have other benefits, it doesn’t offer any additional leverage here unless we were to also add it as a primary key to the legacy database

steveb8n23:02:38

Hmm, is there no way to compose the parent PK and some child attribute to make a unique string? That upsert behaviour is too good to miss out on

roklenarcic23:02:29

I'm not sure if datomic has an equivalent of select for update. To give an SQL example, let's say you have Accounts table. In one transaction an account wires $100 to 3 other accounts ($30 $20 $50). In SQL what you can do is select for update all 4 rows, calculate the "after-transaction" balances and update all of them. What would the equivalent operation in Datomic look like?

steveb8n23:02:47

@roklenarcic if you want to do the select and write with no other txns in the middle then a transactor fn is the way to go. https://docs.datomic.com/on-prem/database-functions.html I’m not sure if they are available in Cloud (yet). you can also look into :db.fn/cas