Clojurians Log v2

We solved it, it was an issue with our Docker DynamoDB system.

I am triggered whenever someone says "when the entity exists"

entities only have identity, not existence, and join facts about themselves

testing for entity "existence" should always be testing for an assertion of a specific attribute you use to signal existence

I'm not saying that the pull return behavior is correct, but I am saying that pull (indeed no datomic function) can never provide an "entity does not exist" signal because the information model doesn't contain that notion.

With one exception: entity lookup refs and idents inherently test for a matching datom, and may not find one; so these could be transmitted as a "could not find a matching entity" (different from "entity not existing").

I'm happy to restate the problem to avoid that imprecise wording: The ion client and the cloud client behave differently when pulling. Specifically, pulling when the provided lookup ref refers to an entity (=> the provided AV yields an E), but the pull expression does not intersect with the entity's attributes returns nil with the ion client and and empty map with the cloud client. This is really dangerous behavior if one is, per idiomatic Clojure, using nil-versus-anything-but-false as a condition. Is this a known bug?

So while I agree that proving an entity doesn't exist is difficult, that's not the situation here. Given that an entity does exist (which is easy to prove), the behavior of pull differs across the clients. That is the problem.

> This is really dangerous behavior if one is, per idiomatic Clojure, using nil-versus-anything-but-false as a condition. My contention is this idiom should never be used for the result of pull. What does "nil" mean?

That the lookup ref found no entity.

and only that? Not "the pull expression does not intersect with the entity's attributes"?

That is a separate concern.

But, even if you can't see daylight between those two concerns, there should at least be consistency across the clients.

Not disagreeing; but I am noting that d/pull doesn't guarantee when it returns nil; if anything, the bug is that it returns nil

https://docs.datomic.com/client-api/datomic.client.api.html#var-pull

"Returns a map."

I can accept that, and if there had been consistency across the clients in returning an empty map, I would not have had a production bug.

IOW, IMO that is a reasonable model.

(I'm curious what peer does, but I don't have access to a peer installation)

> This is really dangerous behavior if one is, per idiomatic Clojure, using nil-versus-anything-but-false as a condition. This is the part I'm harping on: one should never use "nil" in this situation as a signal of anything

the reason is multilayered; usually it's a misguided notion of "entity existence"

peer version (also d/entity) sometimes returns nil; docstring doesn't say when it returns nil

Perhaps at the database level the concept of entity existence is misguided, but at the domain level it is perfect valid and idiomatic. For a US State to exist in my domain, it must have a USPS two-letter abbreviation. Thus it's reasonable to ask the database, using a lookup ref, "does there exist a state with this USPS abbreviation". BUT THIS IS ALL A SECONDARY CONCERN!! Can we please have consistency across ion, db-local and cloud clients?

(or, if not, loud warnings)

devil's advocate here: why is the inconsistency important if nil puns to empty map?

if it was for the sake of a falsy-testing idiom, we've already said why that shouldn't be done even if the return values were consistent.

FWIW, If I were writing the spec for d/pull and d/entity, I'd have this behavior:

(d/pull db [:some-attr] eid-without-assertions) => {}
(d/pull db [:db/id :some-attr] eid-without-assertions) => {:db/id eid}
(d/pull db [*] eid) => {:db/id eid-without-assertions}

(d/pull db [:some-attr] not-found-lookup-ref-or-ident) => nil
(d/pull db [:db/id :some-attr] not-found-lookup-ref-or-ident) => nil
(d/pull db [*] not-found-lookup-ref-or-ident) => nil

and it still has the "gotcha" of "entity existence". I fully expect someone to go (d/pull db [:something] eid-they-just-retractEntitied) and expect nil

FWIW, that ^ is exactly the behavior I would like to see. And, with the inconsistent (across clients) and undocumented behavior it is very easy to let nil represent an unresolvable lookup ref (domain level: the entity does not exist).

is the case where you saw "nil" the result of an unresolved lookup? Your OP didn't mention that, just said "when the entity exists but the pull expression does not intersect with the entity's attributes"

I saw nil where the lookup ref resolved (==> domain entity exists) but the pull attrs did not intersect. But only in the ion client.

My test was for a map and it failed unexpectedly.

getting specific here, is this what you see?

;; :server-type :ion or :cloud
(d/pull db [:identifier] [:identifier "foo"]) => {:identifier "foo"}

;; :ion
(d/pull db [:unasserted] [:identifier "foo"]) => nil

;; :cloud
(d/pull db [:unasserted] [:identifier "foo"]) => {}

Yes.

are there any other behaviors you don't expect?

IIRC, yes. Defaults come to mind.

I mean about the "empty" case

or is this still about the empty case?

empty case -> empty map as return when the lookup ref does not resolve?

I don't think so. Just a two level concern about empty: inconsistent across clients and not matching your (and my) preferred semantics (without which I have to make assumptions about pull attr intersection to ascertain existence of a domain entity).

> (without which I have to make assumptions about pull attr intersection to ascertain existence of a domain entity). Don't do that. Doc says "returns a map"; treat result like a map, and nil like an empty map (as one does in clojure idioms). Don't treat it like a value you can test for truthiness.

How do I test if the state of XX exists?

(use case: validating user input)

(let [x (d/pull db [:identifier] [:identifier "foo"])]
 (if (:identifier x) ;; exists
 )

or go through query

Sadly, that convolution is the conclusion I come to.

(d/q '[:find (pull ?e ,,,) :in $ ?id :where [?e :identifier ?id]])

And I only come to that conclusion through experience. (missing) Documentation and working with ion apps could lead you to believe there is a better world.

question, what should these do?

(d/pull db [(:unasserted :default "unasserted")] [:identifier "foo-doesnt-exist"])
(d/pull db [(:unasserted :default "unasserted")] eid-with-no-assertions)

First should return nil. Second can return {:db/id eid-with-no-assertions}.

> Documentation and working with ion apps could lead you to believe there is a better world. This bit I don't understand; the docs for d.client.api/pull clearly say "Returns a map". Peer is less forgiveable because the docs don't say

So, you get a different result depending on what input you supplied? You realize how that may surprise people who expect it not to matter what entity reference type you use?

One could easily argue behavior should always be as if you "found" an entity with no assertions on it

(which is what "returns a map" seems to imply)

(FWIW I don't know what actually happens for those :default cases above)

At the domain level, that makes no sense when you have an identity attribute. A state without a two letter USPS abbreviation is not a state.

And if I assert it does exist (with a lookup ref) and that is proven false, it would sure be useful if the API signalled that my assertion was false.

IOW, I think lookup refs (and idents) deserve special treatment.

And the ion client lulls you into a false sense that the special treatment is real.

First priority: consistently across the clients, or clear documentation to the contrary (lack of documentation is not sufficience). Second priority: an API that makes it "easy" to ascertain existence of a domain entity by a pull expression that is also pulling attributes of that entity (if found).

Suppose this function exists:

(defn do-something [db id]
  (let [r (d/pull db [:get-stuff] id)]
    ;; do stuff with r
    ))

Suppose you have one caller that calls it like this:

(defn http-handler [db {:keys [identifier]}]
  (let [id [:identifier identifier]]

    (do-something db id)))

Then change to this:

(defn http-handler2 [db {:keys [identifier]}]
  (let [{eid :db/id} (do-other-stuff db identifier)]
    (do-something db eid)))

what should do-something assume about what d/pull returns? It didn't control whether id was a lookup ref or an eid

If lookup ref and eid should do return something different, you've put additional burden on caller of d/pull to check two different things

if they do the same thing, then it can consistently do the "convolution"

In many situations, the value of ascertaining-existence-and-pulling-useful-attrs-for-further-work in one step is more valuable than abstracting of the eid form. In such situations, you would not write code that allows an arbitrary eid like you have shown.

A function that takes an arbitrary pull selector and a lookup ref is fine in that world.

that means lookup refs and eids are not substitutable, which seems bad

We rarely pull with db/ids... and we very frequently want to pull-and-ascertain-existence

well, maybe in your codebase; not so in other's I've used (which eagerly resolve to an eid and flow it)

I would vastly prefer a pull that supported lookup refs and idents only and returned nil on failure to resolve.

But even without that, your proposal above is a very close second.

But I hope you understand now that it is a preference, and not without downsides?

I totally appreciate the db/ids behave differently.

(and definitely not behavior guaranteed by the api)

And using them interchangeably has risks (and not just in pull).

But I maintain that behavior of pull should be consistent across clients and there is high value in being able to pull and ascertain existence in one step (particularly in cloud).

"that behavior of pull should be consistent across clients" yes, I'll get that filed; but this bug is not "dangerous behavior" if you don't assign nil a special meaning (which you shouldn't). "high value in being able to pull and ascertain existence in one step" agree, but this is almost certainly a feature request.

The ability of pull to synthesize an entity (via defaults) that does not exist in the db (=> not found via the eid assertion) is, IMO, nearly worthless.

Somehow, I feel that the ability ascertain existence and pull attrs in one API was sacrificed to support default. If so, that was a bad tradeoff.

(but maybe I'm just wrong)

Thanks for filing the bug. Long ago I internalized the disappointment over the inability to ascertain existence while pulling attrs. But the client mismatch caught me off guard yesterday in production.

> Somehow, I feel that the ability ascertain existence and pull attrs in one API was sacrificed to support default. If so, that was a bad tradeoff. I doubt this was conscious, and I think whether this tradeoff was bad is very debatable, even though I have my preference. (I also don't know if there's actually a tradeoff--I still don't know what :defaults does in the failed-lookup case.)

It gets very very confusing as to what will be returned with the eid type and defaults (not to mention client!). In my namespace providing existence-while-pulling semantics, there are a lot of caveats. Defaults and db/ids require special treatment.

Here's peer (I don't have a cloud handy):

(d/pull db '[:db/id] [:identifier "doesnt-exist"])
;; => {:db/id nil}
(d/pull db '[:db/id :db/doc] [:identifier "doesnt-exist"])
;; => {:db/id nil}
(d/pull db '[:db/doc] [:identifier "doesnt-exist"])
;; => nil

(d/pull db '[(:db/id :default "db/id")] [:identifier "doesnt-exist"])
;; => {:db/id nil}
(d/pull db '[(:db/doc :default "db/doc")] [:identifier "doesnt-exist"])
;; => {:db/doc "db/doc"}

I have to step out, but I'll run that same thing on cloud and db-local. I might even get around to testing on ion.

Schema for :identifier

{:db/ident :identifier
                    :db/valueType :db.type/string
                    :db/cardinality :db.cardinality/one
                    :db/index true
                    :db/unique :db.unique/value}

I assume db.unique/value and db.unique/identity are treated the same... I don't have any db.unique/value in my cloud db.

FWIW, db-local behaves like the ion client AFAICT.

(I am aware that there are other sharp edges around pull with an entity id versus a lookup ref, but I don't want to muddy the waters of this discussion with that bug).

@jaret, any thoughts?

correct. All metrics with a map value are statistical samples summaries

Thank you