Yes, and your “related literature” links show the same problem. Tempid resolution (including via upserting) happens before composite tuple re-computation; composite tuple recomputation happens at the very end after all datoms are expanded to make sure nothing else could change the values of the attributes the tuple is indexing. To resolve tempids using the value of composites which itself depends on resolved tempids would require a more complex multi-pass strategy

My advice is to use a transaction function and not rely on upserting

we are using transaction functions, but do you mean, we should hand-roll the decision, whether a certain nested entity map (or a set of positional arguments) represent an insert or an update and emit different tx-data accordingly?

yes. Something like [:myfn/make-x-y-ref tempid x y] It can see if x and y exist already, and use an existing tuple value to resolve tempid to an existing value accordingly

> To resolve tempids using the value of composites which itself depends on resolved tempids would require a more complex multi-pass strategy i think, i also had issues using both lookup-refs and keywords representing :db/idents as tuple components. could those be the consequences of this single-pass algorithm too?

no, that’s just that d/entid doesn’t resolve them

this is the problem that you need to resolve the tempid (possibly create a new entity) before you can resolve another tempid

you would need to make the tx itself multi-stage, instead of collect tempids, resolve them, commit

txs right now are atomic, they’re not gathering db changes “as it works”

> ... [:myfn/make-x-y-ref tempid x y] ... what is tempid in this case? u mean something like this:

[{:db/id "x" :db/ident "x"}
 {:db/id "y" :db/ident "y"}
 [:myfn/make-x-y-ref "?" ? ?]]

I was thinking tempid was the tempid of the joining entity, so you could use it as a reference elsewhere in the tx

ok, so i have to split the tx into too, like i showed in my forum entry?

no, just make the tx fn accept more args

[:myfn/make-x-y-ref joining-entity-tempid {:x-tempid "x" :x-unique-value "x"} {:y-tempid "y" :y-unique-value "y"}]

and the return value of this tx-fn would be what? different shapes, based on whether [:x-unique-value "x"] & [:y-unique-value "y"] exists already or not?

yes

well, that's what i meant by "we should hand-roll the decision, whether a certain nested entity map (or a set of positional arguments) represent an insert or an update"

> yes. Something like [:myfn/make-x-y-ref tempid x y] It can see if x and y exist already, and use an existing tuple value to resolve tempid to an existing value accordingly yeah, I’m agreeing 🙂

okay, thanks a lot for confirming it! that's how we were doing it in the past, it's just a bit tedious to do it for every composite key 😞

honestly I stay away from upsert, especially the upsert + unique combination

I think it’s too surprising

If datomic were designed with lookup refs from the beginning, I’m not sure it would exist

or at least I wouldnt have put it in

(especially when one entity has more than one upsert attr on it)

we are mirroring data from existing systems and it seems straightforward in that case. the other use-case is storing oauth access and refresh tokens for a combination of our system's user, oauth system's user and the oauth resource they authorized. both of these would work with how datomic does the upsert anyway, if only it would resolve the tempids in tuples 🙂

using tempids in this scenario allows us to use different unique keys for representing the oauth user and the oauth resource, since it varies across systems, yet have the same uniqueness constraint.

have you considered either not having a composite tuple at all (use an entity predicate), or using a heterogenous value tuple and manually maintaining it?

possibly with values instead of refs

yes, if i would use values, i would need to customize the same overall token refresh & revocation logic, depending on oauth provider, so i would need different heterogen tuples based on whether an oauth resource is a uuid or just a string

you can always flatten to something common, or leave some slots nil

but if this isn’t for lookup, you may not need one at all

an entity predicate can ensure the invariant without needing a unique composite-tuple attr as an index

today was the 2nd time we considered using entity predicates for a smaller use-case. previously we were able to solve the problem without any entity predicates, just with tx-fn. but now that u are mentioning it, i will consider them more seriously.

we did consider leaving the oauth resource dimension in the uniqueness constraint nil for one of the oauth providers, so we would be utilizing that capability.

yeah, but the uniqueness constraint on an explicit attribute makes upsert possible... :)

well a txfn can do the same I guess

saves us writing boring, parochial code

it comes down to whether you think races will be uncommon enough that just protecting against them causing invariant volations is enough (entity predicate), or common enough that you want to make it commute (tx fn or upsert)

i'll sleep on it. thanks a lot again for the exhaustive reply!

happy to help!

(let [conn (mk-conn schema)
        txr0 (tx! conn [{:db/id "x" :x/id "x"}
                        {:db/id "y" :y/id "y"}])
        {:strs [x y]} (-> txr0 :tempids)
        ent {:db/id "ent"
             :ref/x [:x/id "x"]
             :ref/y [:y/id "y"]
             :key   [[:x/id "x"] [:y/id "y"]]}]
    (tx! conn [(-> ent (merge {:attr 1}))])
    (tx! conn [(-> ent (merge {:attr 2}))]))

Execution error (ExceptionInfo) at datomic.core.error/raise (error.clj:55).
:db.error/invalid-tuple-value Invalid tuple value

no, it’s just d/entid doesn’t do it

It’s just annoying; I don’t see any technical limitation, they probably just overlooked it

im using datomic cloud, btw, so no d/entid available there

ah; well there’s still an internal function doing the same thing

does it worth creating an entry on the datomic forum about this case, then?

I mean, I would love it if it supported this, but I think they know about it already

we noticed it right when tuples were released

i mean, would it help to get this feature implemented eventually in datomic?

🤷

I guess it wouldn’t hurt

alright, i will look into it then! because it would be extremely convenient during REPL explorations...

you might have to do some reflection in your txfn if you don’t have d/entity sadly

long -> entid; vector -> query, normalize to entid, etc

if I were redesigning this from scratch (hah!) I would replace existing upsert behavior with something that only triggers when it gets a lookup ref in a spot where it expects an entity id

and your application should just always use lookup refs in its transactions if it wants to not care about entity id and only about “unique-identity”

and the system would emit the assertion if the lookup ref failed to resolve and the attr was an upsert type

i have the gut feeling, that we could bolt something on top of the current behaviour, but such logic should know about the schema too... which is available in tx-fns...

yep, everything it needs is there, it’s just not made for you

i really don't care about how non-performant would it be, because the convenience would trump that drawback 🙂

I don’t think it would be slow

at least with on-prem, all that schema and ident stuff is aggressively cached in memory all the time (not object cache, faster)

but the apis to use those fast paths aren’t in cloud (d/entid, d/ident, d/attribute)

but I suspect they still exist in the cloud implementation; they just couldn’t be fast over a network

thus are not in the client api