Fork me on GitHub
#datomic
<
2020-03-17
>
pmooser12:03:07

It's a shame that when they added support for strings as temp-ids, they didn't make strings work as temp-ids for the entity function, so it's easy to have tx-fns that use entity explode in your face. I'm not sure I think they would consider this a bug, but it's the kind of sharp edge that feels all-too-common in datomic.

matthavener13:03:50

you mean like (d/entity some-db "a-tempid-string") ?

matthavener13:03:31

we usually just transact, and then do (d/entity after-db (get tempids some-tempid))

pmooser13:03:36

@matthavener Yes, that's what I mean. But I have some transaction functions that internally use entity , which can't handle those string tempids, so the transaction will abort. I don't see how transacting is an option, since the problem is that the tx-fns won't run.

pmooser13:03:15

That's why as far as I can tell the solution is just not to us those string ids if you have a situation like this.

favila13:03:28

@pmooser that’s really a type error: those tx fns cannot accept a tempid as an argument

favila13:03:50

whatever work they are doing requires knowledge they cannot have--silently doing nothing seems wrong

favila13:03:02

explosion seems better

pmooser13:03:09

@favila I mean it depends on what you think it means for datomic to support strings as tempids - I would say the error is incompletely supporting it. It's not like the docs clearly state you can use them only in certain places as strings.

favila13:03:39

you can use tempids anywhere they are not resolved

pmooser13:03:05

(entity db (tempid :db.part/user)) works, even for an unresolved tid

pmooser13:03:14

(entity db "anything") doesn't work

pmooser13:03:31

That means strings aren't tempids and don't behave like them.

favila13:03:35

yeah, I consider the former a bug, I remember complaining about that back in the day

pmooser13:03:09

I mean it's hard to say what is intended in this case, since (as often happens with datomic) it's hard to know which of these edge cases are intentional and which aren't, and the documentation does little if anything to clarify.

favila13:03:36

I think this is a clojure culture problem in general

pmooser13:03:11

In any case, in terms of concrete things, in my situation it just means we either have to walk some data structures and replace strings with temp-ids, or just avoid the strings where possible. In this case it's a little unfortunate, since strings as tempids are a convenient affordance for things like client-generated data.

favila13:03:26

edge cases are often not called out or checked and you only know they are edges by experience and intuition

favila13:03:40

don’t you have to walk anyway though?

pmooser13:03:12

I think it's also just that despite clojure being a great language, they don't have the impression that this is a significant sort of quality problem. There are some truly shocking bugs in things like core.async - and they're known, and they'll probably never be fixed.

favila13:03:32

[{:db/id tempid :upserting-attr bar} [:txfn tempid]] is wrong whether tempid is a string or a record.

pmooser13:03:51

@favila Not in general, no. I mean, I don't walk these structures I'm transacting unless I'm doing some modification of the data anyway.

pmooser13:03:31

I'm not sure what you are implying - that I can't use a tempid of any kind as an argument to a tx-fn?

pmooser13:03:46

That's demonstrably false, I'm fairly sure.

pmooser13:03:25

(I probably just don't understand the particularity you're trying to illustrate with upsert)

favila13:03:26

I mean if :txfn needs to read tempid, these two transactions are “equivalent” if [:upserting-attr :bar] exists:

favila13:03:39

[{:db/id tempid :upserting-attr :bar} [:txfn tempid]]

favila13:03:56

[[:txfn [:upserting-attr :bar]]

favila14:03:09

but one of them, :txfn can deref, the other cant

favila14:03:26

even though the state of the db is the same and arguably the tx should do the same thing

favila14:03:58

that’s why I argue, if a txfn is expected to read an entity provided as input, it should hard-fail if that input is not resolveable (i.e. a tempid)

ghadi14:03:21

I agree with the last point ^

pmooser14:03:51

I'm still trying to chew on this and understand it.

favila14:03:06

transaction functions are transaction “macros”

favila14:03:29

they can read the environment (&env, the db), but they cannot read “in progress” transactions

favila14:03:00

what tempid resolves to is not known until the very end, and then all fully-expanded adds and retracts are applied simultaneously, set-wise

✔️ 4
pmooser14:03:02

I think it's a little more subtle than that.

pmooser14:03:19

They can't "read" the current tx, because it essentially has not happened yet.

favila14:03:31

yes, that’s fine

favila14:03:38

my point is that there’s a syntatic transformation occurring, and if that transformation needs to read the environment to perform the transformation, it can only do so with what is available to it syntatically

pmooser14:03:59

@favila It's going to take me a couple minutes to respond, because I want to go experiment with what you are saying a little bit.

favila14:03:00

and tempids are not resolveable syntatically. some other opaque tx fn could emit an assertion which changes its resolution

favila14:03:31

only when the full set of primitive asserts/retracts is known and the syntax is fully expanded can tempids be resolved

pmooser14:03:47

Ok, so I sort of understand what you mean, but I don't think it's even quite correct.

pmooser14:03:10

You're right that the tempid passed to your tx-fn won't be magically converted to the db/id of the upsert,

pmooser14:03:26

but if the tx-fn uses the tempid to assert some things, datomic will correctly eventually resolve that tid to the upserted db/id.

pmooser14:03:54

So I don't completely understand the point you were trying to make, or what it has to do with my original point, which is that half-implemented features (ie, temp-ids as strings) are unfortunate sources of complexity and sharp edges.

ghadi14:03:31

you're thinking about it wrong and presuming it's half implemented

pmooser14:03:52

I suppose that's a convenient point of view, from the standpoint of the implementer.

ghadi14:03:01

I'm not the implementor, I'm a user

pmooser14:03:13

I don't mean you, I'm just saying, it suits the creator, not the users.

ghadi14:03:15

a tempid isn't resolved to a real entity id before the transaction commits

pmooser14:03:45

Ok, I'm probably just being really dense, but so? What does it have to do with the changelog that says you can use strings as tempids, when they are not substitutable?

pmooser14:03:15

Like it's hard for me to understand the idea of that the implementation of entity isn't wrong in the sense that either it should work for all representations of temp-ids, or for none of them. I'm not quite sure how you think some other solution serves the users.

ghadi14:03:28

they can appear in transaction data, but they cannot appear as an argument to d/entity

pmooser14:03:29

The minimum acceptable version would be for it to be documented.

pmooser14:03:52

Well, what you just said isn't true, as it happens, depending on what you mean by temp-id, which is my point.

favila14:03:43

I concede absolutely, that a string tempid and a record tempid should fail in the same way; however, whichever way they fail, you still need to do the same checks

pmooser14:03:58

My code isn't wrong and isn't failing to check

favila14:03:59

because at the end of the day, you still need to throw to abort the tx

ghadi14:03:01

zoom out and re-examine what you're trying to achieve. calling d/entity on a tempid is a detail, what is the overarching goal?

ghadi14:03:17

I haven't read the full scrollback

ghadi14:03:28

feel free to link something if you've already said it

pmooser14:03:41

It's just a discussion of wishing the API behaved consistently

pmooser14:03:00

My code works, I worked around the fact that entity freaks out if you call it using a string on a tempid

pmooser14:03:42

Maybe I mistakenly gave you guys the impression that I'm trying to fix a bug - the bug is fixed. It's just yet another case of having to understand datomic from its behavior than from any kind of real specification.

ghadi14:03:48

well I can't help with wishes 🙂 what semantics would it even have to call d/entity on a tempid? like what would it return?

pmooser14:03:49

Exactly what it does if you call it on a tempid. In fact, entity confuses me in that it will return something for any integer value you pass it, even if the entity doesn't exist. I can't explain that, just as I can't explain why it only works for certain representations of tempids but not others.

favila14:03:17

what does it mean for an entity to exist?

pmooser14:03:25

What I would have it do, if I could control it, is: whatever entity does, do it for all representations of tempids, and not behave differently depending on representation of tempid.

pmooser14:03:55

@favila Is that a trick question?

favila14:03:16

datomic stores datoms, not entities. d/entity provides a projection of datoms as a map. So, what does it mean for an entity to exist?

pmooser14:03:37

If we accept your definition, presumably it must mean an entity with id E exists if at least one datom has ever been asserted with E in the first position of the EAVT tuple.

favila14:03:57

ok, entity-maps are lazy. your definition requires at least some eagerness

pmooser14:03:33

What is your (presumably more correct) definition?

favila14:03:16

I don’t think it’s a meaningful question. an entity is a key upon which to join datoms

favila14:03:44

it’s not like an id in a row in a relational store

favila14:03:55

the row exists, so the thing exists

pmooser14:03:01

That is essentially what I said.

pmooser14:03:13

A row existing is isomorphic to some assertion having been made.

pmooser14:03:18

But you rejected it, for some reason.

favila14:03:43

it matters for what d/entity returns. (d/entity 9999) => {:db/id 9999} (assuming 9999 has no assertions)

favila14:03:53

or (d/entity 9999) => nil

favila14:03:55

which is right?

pmooser14:03:57

I must just be communicating very poorly

pmooser14:03:03

Let me ask you a direct question which will help clarify.

pmooser14:03:19

Why should entity behave differently for different representations of the same concept?

pmooser14:03:54

(I have no idea the relevance of anything else you're talking about to THIS, which is my fundamental point/question. As far as I can tell, all of these existential questions about entities have absolutely no bearing on this, which is fundamentally a behavioral question.)

favila14:03:34

to me, the bug is (d/entity db (d/tempid :user/foo)) should be nil

favila14:03:52

it’s an implementation detail that it’s not, namely that tempid records encode a negative number, which represents a tempid

favila14:03:24

the argument to d/entity got passed to d/entid (or moral equivalent) at some point, and that’s why you have what you have

pmooser14:03:34

I would accept that as well, because to me, the confusion is from the inconsistency.

favila14:03:49

but they probably did a long? check on the result, instead of positive-long (or 0)

pmooser14:03:22

The utility of having entity work for a tempid is that you can still use entity to check if there are existing values for an attribute that you intend to set (and there's actually no worry about them being asserted elsewhere in the tx, since the existence of that would create multiple assertions, which would abort the transaction anyway).

favila14:03:23

so I would sure like it if d/entity returned nil to indicate any unresolveable state

pmooser14:03:34

Now, if I haven't communicated that clearly, I don't think that I can really explain it any better.

pmooser14:03:06

If entity returned nil for tempids, it would be fine, I'd just make sure that I would query for the existing values, and you'd have EXACTLY the same problem, as the query couldn't see anything else you asserted in the same TX but hasn't been committed yet.

pmooser14:03:17

Behaviorally, there IS utility in this defintion.

pmooser14:03:27

But you have to be cautious with using it this way.

favila14:03:30

which is why I was arguing, in both cases, the only appropriate thing is to check and abort

pmooser14:03:40

Right, and that is where we disagree.

pmooser14:03:55

I don't need to abort - the transaction in my particular case is well-formed and meaningful - even with the result of entity being passed a temp-id.

favila14:03:58

the check changes and is much more convenient if d/entity is consistent (really d/entid)

favila14:03:20

so you are using this tx fn in such a way that you have some guarantee from the application that it will never try to change something the tx fn will read in a way that would affect its functioning, nor ever compose this tx fn in the same tx with anything else that might do the same?

pmooser14:03:25

I agree completely on all counts that entity should behave consistently - and ideally, the behavior would be documented or specified somewhere.

pmooser14:03:52

Let me answer that question in 2 parts:

pmooser14:03:18

1. In this particular case, the tx-fn makes assertions of a particular attribute. Something else making an assertion of the same attribute for the same entity somewhere else in the tx would be an error anyway, since you're not allowed to make ambiguous transactions. So no, that's not a problem. 2. Do you think that all functions we write, and especially transaction functions, are arbitrarily and infinitely composable? I assure you this is not a general property of transaction functions.

pmooser14:03:26

The fundamental problem here, as far as the argument you're making, isn't even about entity behavior, because as I said above, even if this didn't work with entity at all, you'd just have to query for existing attribute values, and that would have the same potential issue you're worried about.

pmooser14:03:50

I think the problem that concerns you, then, is whether we can really have tx-fns that work on both existing and new entities, correctly in both cases.

favila14:03:00

for 1) I don’t know exactly what you are doing so perhaps you are avoiding this case, but the upserting case from earlier is what I was thinking of. 2) absolutely not, which is why the caution about throwing if someone accidentally supplies a tempid to a tx-fn that is expected to read the value of the tempid. Correct to your last two paragraphs.

pmooser14:03:56

What I will say is that the tx-fn in question that I wrote is pretty specialized, and I acknowledge the wisdom in what you're saying in general - at the very least we have to be careful, and in many circumstances, having entity return non-nil when something isn't really there could create problems.

favila14:03:27

not knowing your problem specifically, I would prefer using a sentinel value to a tx fn (say, nil) to indicate “I am minting a new entity id, it has no assertions before this point” rather than a tempid to communicate the same.

pmooser14:03:16

That's an interesting idea.

favila14:03:41

since a tempid doesn’t make that guarantee

pmooser14:03:04

I like the clarity of your idea - in my particular case, it would just require slightly different code.

favila14:03:20

although of course, you may still need the tempid in order to hang new assertions off of it

favila14:03:43

I guess my point is really that the intent should be communicated out of band somehow

favila14:03:09

I’m also starting to like using :db/ensure to check invariants afterwards

favila14:03:29

it can catch many cases of “accidental composition” of transactions

pmooser14:03:53

Yeah, if I had upserting attributes in the entity, what I'm doing wouldn't work at all.

pmooser14:03:31

Since I do not have that case (currently), that means the presence of a tempid does actually indicate a new entity.

pmooser14:03:47

If that changed in the future, it could break things.

mruzekw21:03:18

How do I get all attributes on a matched entity? Say I have a database with movies. And I want all movies before a certain year. How would I do that?

(d/q '[:find ?movies 
        :where [?movies :movie/release-year ?myear] 
               [(< ?myear 2013)]] db)
Right now this only returns the entity id. How would I change this to include all attributes on that entity?

favila21:03:00

You probably want pull

favila21:03:47

:find (pull ?movies [*])

favila21:03:09

for your sanity, though, production code should generally be explicit about attributes

mruzekw21:03:17

So typically you’d only bind certain attributes?

favila22:03:43

yes, whatever you needed

favila22:03:57

or you would separate query from pull explicitly (using pull-many)

mruzekw22:03:22

Hmm I honestly haven’t looked at pull yet, very new to Datomic

mruzekw22:03:50

Could I bind multiple attributes from a query into a map rather than a list?

mruzekw22:03:08

I know I can do (d/q '[:find ?title ?year

favila22:03:23

(pull db [:foo] entity ) => {:foo "bar"}

favila22:03:47

it’s already a map

mruzekw22:03:49

Okay, I will look into pull then. It seems to be what I’m looking for

mruzekw22:03:01

Thank you 🙏

favila22:03:19

there’s also :keys in query, if you just want a natural name for each member of the result tuples

mruzekw22:03:42

Cool, I’ll look that up as well

favila22:03:06

:find ?a ?b ?c :keys a b c => [{:a 1 :b 2 :c 3} ,,,]

mruzekw22:03:12

user=> (d/q '[:find (pull ?movies [:movie/title :movie/directors]) :where [?movies :movie/release-year ?myear] [(> ?myear 2013)]] db)
[[#:movie{:title "Paddington", :directors [#:db{:id 17842874695549022}]}]]
How would I go further and retrieve the directors’ as well?

mruzekw22:03:01

Never mind

mruzekw22:03:04

user=> (d/q '[:find (pull ?movies [:movie/title {:movie/directors [:director/name]}]) :where [?movies :movie/release-year ?myear] [(> ?myear 2013)]] db)
[[#:movie{:title "Paddington", :directors [#:director{:name "Paul King"}]}]]

mruzekw22:03:16

That is super sweet!

🙂 4
mruzekw22:03:31

It’s like GraphQL built in

mruzekw21:03:46

Other attributes include :movie/title :movie/directors :movie/actors