consider creating a composite unique-value attribute

I do that, plus usually an associated tx-fn

seems to work for me?

at least using mem db

(-> "datomic:" d/connect
    (d/transact [[:db/add (d/tempid :db.part/user) :db/ident :foo] [:db/add "bar" :db/ident :foo]]) deref)
=>
{:db-before datomic.db.Db,
 @5e4de82f :db-after,
 datomic.db.Db @b6765e2a,
 :tx-data [#datom[13194139534312
                  50
                  #inst"2019-06-13T14:19:50.056-00:00"
                  13194139534312
                  true]
           #datom[17592186045417 10 :foo 13194139534312 true]],
 :tempids {-9223350046623220289 17592186045417, "bar" 17592186045417}}
(-> "datomic:" d/connect
    (d/transact [[:db/add (d/tempid :db.part/user) :db/ident :foo] [:db/add "bar" :db/ident :foo]]) deref)
=>
{:db-before datomic.db.Db,
 @b6765e2a :db-after,
 datomic.db.Db @276d2771,
 :tx-data [#datom[13194139534314
                  50
                  #inst"2019-06-13T14:20:00.447-00:00"
                  13194139534314
                  true]],
 :tempids {-9223350046623220291 17592186045417, "bar" 17592186045417}}

maybe the tempid is in a different partition?

actually that works too, at least where [:db/ident :foo] exists already

seems to work for initial write also, but honestly the partition difference seems like it should be an error. I don't know how it would choose a partition

(-> "datomic:" d/connect
    (d/transact [[:db/add (d/tempid :db.part/db) :db/ident :bar] [:db/add "bar" :db/ident :bar]]) deref)
=>
{:db-before datomic.db.Db,
 @28ac123 :db-after,
 datomic.db.Db @a8ad04f9,
 :tx-data [#datom[13194139534319
                  50
                  #inst"2019-06-13T14:22:26.281-00:00"
                  13194139534319
                  true]
           #datom[63 10 :bar 13194139534319 true]],
 :tempids {-9223367638809264717 63, "bar" 63}}

this one, I edited the above post

how do you know it doesn’t exist yet? In fact it works even if it preexists:

user=> (d/transact conn [{:u/k :preexisting}])
#object[datomic.promise$settable_future$reify__4751 0x62cd562d {:status :ready, :val {:db-before datomic.db.Db@1d500de1, :db-after datomic.db.Db@c2499ebc, :tx-data [#datom[13194139534328 50 #inst "2019-06-13T15:10:42.368-00:00" 13194139534328 true] #datom[17592186045433 64 :preexisting 13194139534328 true]], :tempids {-9223301668109598113 17592186045433}}}]
user=> (d/transact conn [[:db/add "tmp1" :u/k :preexisting] [:db/add "tmp2" :u/k :preexisting]])
#object[datomic.promise$settable_future$reify__4751 0x14b5752f {:status :ready, :val {:db-before datomic.db.Db@c2499ebc, :db-after datomic.db.Db@98f067bc, :tx-data [#datom[13194139534330 50 #inst "2019-06-13T15:11:01.592-00:00" 13194139534330 true]], :tempids {"tmp1" 17592186045433, "tmp2" 17592186045433}}}]

preiexisting makes sense because both unify to the eid

asserting ":u/k :bar" on a tempid would trigger replacement of tempid with realid

then all future uses of that tempid would also be replaced with realid

similarly, if [:u/k :bar] didn't exist and was asserted, kinda makes sense to say that every other tempid trying to assert that would unify to the same newly-minted eid

but when :u/k :bar could unify to a real eid, to ask a different tempid asserting a new [:u/k :foo] to unify to the what-is-now :bar but what will be :foo seems like too much mind-reading

this works ok

"works ok" in what sense? no tx error?

I'm highlighting the potential ambiguity, I'm actually concerned that it works

actually nm this isn't what I was thinking of

This is the case I was thinking of (:refa and :refb are unique-identity)

@(d/transact conn [{:db/id "t1" :refa :a1 :refb :b1} {:db/id "t2" :refa :a2 :refb :b2}])
@(d/transact conn [[:db/add "t3" :refa :a1] [:db/add "t3" :refb :b3]])
=>
{:db-before datomic.db.Db,
 @20fead49 :db-after,
 datomic.db.Db @c13cf2d2,
 :tx-data [#datom[13194139534316
                  50
                  #inst"2019-06-13T15:43:29.444-00:00"
                  13194139534316
                  true]
           #datom[17592186045418 64 :b3 13194139534316 true]
           #datom[17592186045418 64 :b1 13194139534316 false]],
 :tempids {"t3" 17592186045418}}

IMO this should be an error

it works because resolution to a real eid happened first

this btw is also the tx ops from expansion of the tx map `{:db/id "t3" :refa :a1 :refb :b3}`

I think this syntax makes the ambiguity more clear

this was probably either a mistake or too-clever code

If I were going back in time, I think I would make upserting attributes work like lookup refs that may not resolve against the db-before

e.g. {:db/id [:refb :b3] :refa :a1} would (if :b3 didn't exist) reliably make a new eid, assert :refb :b3 on it, and assert :refa :a1 on it

tempids would always make new eids

actually you may be able to replicate some of that behavior by consistently hashing the same ref lookup to the same string-for-tempid

it’s an integrity violation you are merging two existing entities

if :refa2 existed prior, would it still be an integrity violation?

you have to decide whether forms like [:db/add "t1" :refa :refa2] are primarily a lazy way of resolving tempids or a way to assert a new ident

you mean if it didn’t exist? because it exists {:db/id 67890 :refa :refa2}

when both are possible, allowing it to guess doesn't seem like a good idea

in general when I want my tx to be an update rather than a upsert, I will use the ident or lookup ref as the eid

I will not rely on unification through the assertion

yes but in case b, it's possible I made a mistake in my tx

wait, what is case 2

is that [:ref :A] does not resolve to eid?

it’s [:ref :A] resolves to another-eid

so you would expect a conflict? then why the bug report?

I expect a conflict too

I thought you expected "tmp" to resolve to eid (not another-eid)

i.e. the value :ref :A will resolve to after [:db/add eid :ref :A] is applied

No

A transaction is data describing the change you want done

To prepare that data, you may have read values out of the db

so your changes are prepared assuming a certain state of the db

the problem is by the time that tx data gets to the transactor, your assumptions may be wrong

thus invalidating your transaction

:db/cas is a way to assert that something still has the value you read at the moment the write occurs

the transactions themselves are applied serially, but the transaction data was not prepared serially (i.e. it was prepared by uncoordinated peers reading whatever they read)

I see, thank you @favila. So then, in a Cloud system, the “compute group” actually can prepare datoms (i.e. run tx fns) in an uncoordinated fashion, but the actual mutation of storage is always serial?

This actually would explain why :db/cas is a built-in, because there must be some storage-level magic happening to ensure that CAS’s promise is kept.

I don't think tx functions are run outside a tx

unless this is a cloud vs prem difference

it's a surprising difference if so

they are not

the same description for the use/need/purpose of CAS for on-prem that Francis provided above is true for Cloud

Okay so to check my own understanding, if I prepare the tx outside of a tx function, then :db/cas might be required. However, if I query for the dependent data inside a tx function, then I shouldn’t need CAS, correct?

more or less yes

take a look at the link i dropped in the other thread

:db/cas is a transaction function you can use it within a custom transaction function you write, but you don’t have to you can also reimplement it (or something like it) as a transaction function yourself However, the general use of CAS is more frequently for optimistic concurrency applications - i.e. https://docs.datomic.com/cloud/best.html#optimistic-concurrency

This is because transactions run serially. the transaction data expansion and application to make the new DB value begins with the previous db value. All tx functions receive that previous db value. No other transactions are expanded/applied during this process (in essence, the transaction has a lock on the entire database). The datoms that result from db are applied to the previous db value to make the next db value. then the next tx is processed

there is no opportunity for a tx function to get a stale read

right ^

the tradeoff is that whatever work you’re doing in your transaction function is happening in the single-writer-thread of Datomic

so if you try to do something expensive (like call a remote service — eek!) from within the transaction function, all writes are going to wait on that work

if you instead do that work locally in your client (or peer), you can avoid that cost on the transaction stream but you need to ensure that no one has changed your relevant data out from under you in the meantime, so you can often use CAS for that

cas (and it's general technique of "assert what I read hasn't changed") allows the opposite tradeoff: possible parallel tx preparation, but a stale read is expensive to recover from. (You need to catch the tx error, detect it was a CAS error, and reprepare your tx using a newer db, and reissue hoping you don't race with some other write)

Ahhh okay, that link coupled with these explanations have totally cleared up my confusion. I was totally missing the fact that concurrency was in the hands of the developer. So putting simple query logic into a tx function is totally valid, but when that logic becomes expensive (e.g. remote call as @marshall said), it might make more sense to perform that work outside the tx fn to keep the tx stream clear, and rely on CAS to uphold consistency.

yep

Thank you both very much

If you are familiar with clojure atoms: this is roughly the difference between (swap! db apply-ops (inc-something db)) and (swap! db (comp apply-ops inc-something))

Makes sense. Former performs the inc transformation outside the swap, and the latter performs the inc/apply inside the swap (iiuc).