I have noticed many cases that if the attribute is not statically known as a keyword, non-entid parameters will not be resolved to entids before pattern matching. So here, your :auth x/User may not be getting resolved. Try resolving it first in the first case then passing that entid in as ?visited

Yep, that solves the problem:

(let [db (d/db conn)]
         (d/q {:query '[:find (pull ?e [*])
                        :in $ %
                        :where
                        , [?x :db/ident :authz/User]
                        , (walk :st.authz/children ?e ?x)]
               :args [db '[[(walk ?attr ?start ?visited)
                            [?start ?attr ?visited]]]]}))

This issue has progressed from an annoyance at the REPL to a full blown “bug” in my code. @U1QJACBUM, is there any sensitivity to this issue at the mother ship? Is there a work-around that doesn’t remove the generality of querying with “e” values bound transparaently to either entity ids or lookup-refs or db-idents?

The workaround I always use is normalize ref inputs to entity ids with datomic.api/entid

As a cloud user, I am jealous. I could use d/pull, but due to a known limitation of selectors, you can’t request just the entity db/id … so for general use you are stuck with pulling all attributes, which just seems crazy.

Hm yuck

You could implement it as a rule

Guard the three impls with long? / keyword? / vector? for the eid/ident/lookup ref cases

I could see that working for long? and keyword?, but lookup refs might be a bit harder… might need some untuple application to pull out attr and value. I might give this a try…

It’s not hard [(entid [?ref] ?eid) [(vector? ?ref)][(untuple ?ref) [?attr ?val]][?eid ?attr ?val]]

You could add a bit of safety by asserting that attr is unique in the schema

You beat me to it! Thanks, @U09R86PA4. It’s a shame this is required at all, but wrapped up in a rule it could be a handy tool in the toolbox.

Id double check it uses Avet lookup via IO-stat

If it doesn’t, you can drop down to d/datoms to make sure it uses the right index

I can’t find syntax that allows a predicate to act as a guard. IOW, the predicate does not seem to short-circuit the evaluation of rules in the same ruleset and, as a consequence, run-time errors are produced when arguments don’t match. For example, despite doing precisely what I want when given a db/ident keyword (binding ?eid to an entity id) this ruleset will throw when given a long (eid) or a vector (lookup ref):

[(entid [?ref] ?eid) [(keyword? ?ref)] [?eid :db/ident ?ref]]

Seems to work for me, and I use predicate guards in polymorphic rules all the time:

(d/q '[:find ?ref ?eid
       :in % [?ref ...] $
       :where
       (entid ?ref ?eid)]
     '[[(entid [?ref] ?eid)
        [(keyword? ?ref)]
        [?eid :db/ident ?ref]]

       [(entid [?ref] ?eid)
        [(int? ?ref)]
        [(identity ?ref) ?eid]]

       [(entid [?ref] ?eid)
        [(vector? ?ref)]
        [(untuple ?ref) [?attr ?v]]
        [?eid ?attr ?v]]]
     [1
      :ident
      [:unique/attr "foo"]]
     [[1 :attr "val"]
      [2 :db/ident :ident]
      [3 :unique/attr "foo"]]
     

     )
=> #{[:ident 2] [1 1] [[:unique/attr "foo"] 3]}

This issue has progressed from an annoyance at the REPL to a full blown “bug” in my code. @U1QJACBUM, is there any sensitivity to this issue at the mother ship? Is there a work-around that doesn’t remove the generality of querying with “e” values bound transparaently to either entity ids or lookup-refs or db-idents? Thank you so very much, @U09R86PA4 -I’ve confirmed I get the same results as you with cloud & dev-local when using your example. For anybody following along, here’s a dev-local equivalent to @U09R86PA4’s example that operates on transacted entities:

clojure
(let [client (d/client {:server-type :dev-local :storage-dir :mem :system (str (gensym "db"))})]
  (d/create-database client {:db-name "test"})
  (try (let [conn (d/connect client {:db-name "test"})
             db (d/db conn)
             schema [{:db/ident :unique/attr
                      :db/valueType :db.type/string
                      :db/cardinality :db.cardinality/one
                      :db/unique :db.unique/value}
                     {:db/ident :common/attr
                      :db/valueType :db.type/string
                      :db/cardinality :db.cardinality/one}]
             data [{:db/id "A" :common/attr "val"}
                   {:db/ident :ident}
                   {:unique/attr "foo"}]]
         (d/transact conn {:tx-data schema})
         (let [{{a "A"} :tempids db :db-after} (d/transact conn {:tx-data data})]
           (d/q '[:find ?ref ?eid
                  :in $ % [?ref ...]
                  :where (entid ?ref ?eid)]
                db
                '[[(entid [?ref] ?eid)
                   [(keyword? ?ref)]
                   [?eid :db/ident ?ref]]

                  [(entid [?ref] ?eid)
                   [(int? ?ref)]
                   [(identity ?ref) ?eid]]

                  [(entid [?ref] ?eid)
                   [(vector? ?ref)]
                   [(untuple ?ref) [?attr ?v]]
                   [?eid ?attr ?v]]]
                [a :ident [:unique/attr "foo"]])))
       (finally (d/delete-database client {:db-name "test"}))))

So why do I get exceptions when trying the exact same rules and invocation as above but in the scope of an existing database and (slightly) more complex query? The exceptions clearly indicate that the guard predicate is not effective so the only possible explanation I can imagine is that the Cloud/dev-local query engine is re-ordering my query expression so the dangerous expression is evaluated before the guard. The conditions that trigger this presumed re-ordering are not clear. This query fails:

(let [{conn :datomic/connection :as system} *system*
        db (d/db conn)
       lr [:st.generator/sysid "gen00000"]
       rid (-> (d/pull db [:db/id] lr) :db/id)
       e-rules '[[(entid [?ref] ?eid)
                  [(keyword? ?ref)]
                  [?eid :db/ident ?ref]]
                 
                 [(entid [?ref] ?eid)
                  [(int? ?ref)]
                  [(identity ?ref) ?eid]]
                 
                 [(entid [?ref] ?eid)
                  [(vector? ?ref)]
                  [(untuple ?ref) [?attr ?v]]
                  [?eid ?attr ?v]]]]
   (d/q {:query '[:find ?rid
                  :in $ % [?resource ...]
                  :where
                  , (entid ?resource ?rid)]
         :args [db e-rules [rid]]}))

=> [[96757023248743]]

(let [{conn :datomic/connection :as system} *system*
        db (d/db conn)
       lr [:st.generator/sysid "gen00000"]
       rid (-> (d/pull db [:db/id] lr) :db/id)
       e-rules '[[(entid [?ref] ?eid)
                  [(keyword? ?ref)]
                  [?eid :db/ident ?ref]]
                 
                 [(entid [?ref] ?eid)
                  [(int? ?ref)]
                  [(identity ?ref) ?eid]]
                 
                 [(entid [?ref] ?eid)
                  [(vector? ?ref)]
                  [(untuple ?ref) [?attr ?v]]
                  [?eid ?attr ?v]]]]
   (d/q {:query '[:find ?rid
                  :in $ % ?resource
                  :where
                  , (entid ?resource ?rid)]
         :args [db e-rules rid]})))
Execution error (ExceptionInfo) at datomic.core.error/raise (error.clj:55).
:db.error/not-a-keyword Cannot interpret as a keyword: 101155069759847 of class: class java.lang.Long

Does :query-stats true show you the reordering when you supply a keyword rid?

Ah I bet it fuses it into a predicate

Hm that is tricky. I guess you could alias

WIth the rules in place, I always get an exception (=> no query stats) because 2/3 of the entid clauses are “dangerous” regardless of input.

Or maybe [(ground true) ?true][(keyword? ?ref) ?true]

Every possible input type gives you an exception?

I’m weak with query stats… need to dig in a bit on this.

So far, yes. The guards are presumably reordered out of effectiveness and the “dangerous” expressions are evaluated first. I might be able to contrive one that works… hang on.

You can always take out the irrelevant impls for your input to see query-stats

I can’t see query stats yet. I’m guessing they are not supported in dev-local. I’m going to try to generate a cloud test. BTW, https://docs.datomic.com/cloud/api/query-stats.html#background is an interesting comment about predicates being “pushed” -but I’m not sure how to interpret it. I almost seems like “pushing” would put predicates (= > guards) first and avoid the problem I am seeing..

A thing that can happen is that predicate forms are pushed into the datasource, so that they filter values before the datalog engine sees them. This is usually good because it can reduce result set size significantly and cheaply, or the datasource can interpret the pred to get a more efficient lookup (eg turning (< v ?x) into direct access into a sorted list at the position before v)

In query stats you can see this as a clause with :preds

If that is happening, and assuming there is no way to turn it off, then it seems like the rule approach of “correcting” Datomic’s inability to resolve entity ids/lookup-refs/db-idents generally is not feasible for Datomic Cloud. @U1QJACBUM, any thoughts on this?

A way to “turn it off” is what I suggest above—make it not a predicate anymore by binding the output. Agree that we are several steps way too far into a workaround forsomething that should be much more straightforward

Ah! I see what you mean now. And totally agree that this is too deep into the rabbit hole.

@U0698L2BU a lot to unpack here. Please forgive me if I am missing some things. I would generally not expect to get two different results when you have logic variables bound v unbound, if there is a difference it would be performance related. Query stats would certainly supply more information and we did not support query stats in local as it has its own limitations, but perhaps we should as it would be useful here in what you and Francis cooked up for testing, just to see the order. Were you ever able to run a test on a cloud system and get query-stats from that? Generally, could I get some context on what you are trying to do? Specifically, if you split up ref/entity/ident flows into separate queries do you get the expected results? You mentioned at the start of this thread that you expected the rule to go to the VAET index? While the authz/User is an ident, it is not a ref to another entity, correct? You would only go to the VAET index when it is a ref to another entity.

I would be happy to dig into the specific problem you are encountering and provide an alternative for what you are trying to do. + it would give us a chance to document features other problems we should address from this case. I can say generally, we have a few stories about ref resolution (i.e. around tuples) and adding the entity API to client cloud... but we do not have a story about preserving ref/entity/ident ambiguity through many APIs. I would like to explore that more for a documented feature request. I think its a hard feature to do correctly though given we can currently rely on users to separate out those requirements per their needs.

This problem started out (in the top post, about a month ago) as differing results depending on whether an expression was inlined or expressed as rule and the entity was specified with an ident (`:authz/User`). When expressed as an ident the results were unexpected. Francis correctly identified this problem as one I have faced before: sometimes Datomic does not recognize idents as entity references.

Then we moved onto trying to compensate for this failure by adding a rule that resolves the three ref types (eids, idents and lookup refs) into eids. That attempt failed due to, presumably, Datomic reordering guard predicates such that lookup ref vectors were destructured out of a long (BOOM!). This “problem” is maybe more understandable (Clojure will reorder queries sometimes, and when guard predicates are reordered, things will break.

But the net total is a depressing place: the ability to transparently substitute eids, idents and lookup refs is broken and there is not a nice way out*. Perhaps even scarier is that I have found scenarios where this breakage is not overt (no exception thrown). Instead, the results are just wrong. • @U09R86PA4 points out that an even more complex rule binding “predicate-like” output to unification vars might solve the problem.

What can I do to help resolve this problem? The lack of query stats in dev-local is not a big deal.

What would help me to talk about this with the team is: mostly what you already have in the thread, but perhaps we can clean it up with a real example from your system including the query stats from cloud? Could you share that with me via a support case or here? I can probably make this other workaround part example tonight on an internal test system. What you posted with Francis looks complete enough to get that, but I don't have the ability to replicate your query results where you actually encounter "wrong results". I.e. documenting when you encountered wrong results and then I will need to connect what you and Francis did a.ka. "how building complex verification/validation via rules" is too hard to navigate as a solution. The thread points pretty heavily to entid api being a pattern that would solve all this pain for you, but perhaps I am missing something and when you reference scenarios where this breakage is not overt and results are just wrong, I want to hold that up to the light because i agree that is scary and I want to learn everything about those scenarios. As an interim solution for you, Is this in an ion? Are you able to use pull to retrieve the entity-id of the ident? but something like: (:db/id (d/pull db '[:db/id] :authz/User)) or are you outside of an ion and thus not wanting to incur every additional call in cloud over the wire? I just note in the thread you ruled out pull "due to a known limitation of selectors, you can’t request just the entity db/id." Does the above not work?

With or without the query-stats from cloud I am going to try to talk to Stu about this tomorrow. I'll keep you posted, but let me know if that pull is do-able as an interim solution to your problem or if you can get the query-stats.

> (:db/id (d/pull db '[:db/id] :authz/User)) or are you outside of an ion and thus not wanting to incur every additional call in cloud over the wire? I just note in the thread you ruled out pull “due to a known limitation of selectors, you can’t request just the entity db/id.” Does the above not work? Apparently on cloud, [:db/id] as a pull pattern behaves like [*] , i.e. returns everything

https://clojurians.slack.com/archives/C03RZMDSH/p1689347405750469

I wrote this up into an orderly “code” narrative (clj script file), but this exercises on-prem

Case 3 and Case 4 are the “WAT” cases that exist on on-prem as well. Note that I don’t see any “:preds fusion” for any of the rule implementations, and my rule-based entid seems to work fine.

FWIW I think case 3 and 4 are bugs: I don’t have any intuition as to why they should be different from case 1. I expect in both cases the datasource is getting an attr value and so can introspect it to determine if the v slot could be some alternate entity identifier.

Sorry, I’m just now seeing this. @U1QJACBUM, Yes, I am using Ions and d/pull is a workaround. It would be a “good” workaround were it not for the bug @U09R86PA4 correctly references above (`:db/id` is equivalent to *).

Francis’ documentation of the “wrong” results depending on what type of entity reference seems like the same bug that inspired this thread initially and pushed me to trying the rules approach. I don’t think Cloud and On-Prem differ in these unusual results.

w.r.t. to the rule work-around not working for me, I think that is due to query optimization. In the simple cases, the rule approach works fine (using dev-local, but presumably in cloud too). However, with a “rich” schema and existing database I get exceptions because the guard predicates are not effective.

So I see two problems/bugs: A. Entity references are not interchangeable and wrong results are silently yielded. This issue appears in dev-local and on-prem and probably cloud too. I think Francis’ documentation should be sufficient, but please let me know if you would like me to package up a dev-local equivalent (roughly matching the original post of this thread, but with details). B. Query expression re-ordering renders guard predicates useless. This issue doesn’t seem to be affecting Francis (On-Prem) but it is easily reproducible for me in dev-local (see above example). Getting a reproduction in Cloud is possible for me if needed but I imagine you could do that faster than me with my example code.

@U09R86PA4 & @U1QJACBUM, let me know if you think that subdivision is right, and if there is anything I can do to help get these addressed (open ticket, provide test case, provide access to my system, whatever).

I started working on this last night. I will keep you updated. I talked briefly with Stu about this and he agrees there may be multiple problems reported here and I want to break them out in the 2 problems or potentially n problems. Because in addition to your subdivision, I see the pull problem as extremely relevant. That's the way I would have expected you to be able to resolve to the entity.

Thanks -and yep, I forgot about the pull problem.

BTW, @U1QJACBUM, this is related: https://ask.datomic.com/index.php/703/pulling-db-id-will-pull-the-entire-entity

@U0698L2BU I have this issue reproduced and I understand the pull problem aspect. Pulling the db-id in client pulls the entire entity including all the other attributes you were not intending to pull. We have an open ticket to address this in client. What would help me with getting priority for dev is to begin to understand the implications of this behavior. One problem i have included in my story is that unexpectedly pulling more = memory issues. Kenny had to restart his REPL. In your case, I believe this contributes to inability to get ref resolution, but I want to understand why. My thinking was even though pull is giving you more than the db/id can you just take the db/id and move forward? i.e.:

(:db/id (d/pull (d/db conn)
        [:db/id]
        db-id))

In the context of the first query, where the issue is really “I have not-normalized entity identifiers and the query gives me different results if I fail to normalize”, there are two problems: one, the number of entities you pull may be very high, so not just one d/pull but d/q or d/qseq. Granted this is just the same as “takes more memory”, but it also takes more network and IO and peer resources, etc, that it may be uncomfortable in some situations. two: in the context of that query, the entid normalization is needed inside the query, so you would potentially have to split one query into 3 (two round trips) or subqueries.

IOW pull [:db/id] returning too much is already a second-order bug in an attempt to workaround the real problem, which is silently inconsistent entity identifier normalization and no blessed high-perf way to do that in client

I agree with all three statements above: it’s a memory issue. It’s a memory+network issue in the case of multiple entities. Most importantly, it is a second-order problem stemming from entid normalization issue. I’ll add that some of the queries I am writing are generalized for any entity (it’s an authorization subsystem dealing with “principals” and “resources”) and the entity+components size is open-ended and I have no way to know the cost impact. That makes it “scary” to just pull :db/id and pluck it from the whole locally. Still, the performance implications (for me at least) and manageable for now because of my small scale. The entid normalization OTOH just frustrates me -twice in the past year I’ve ended up chasing my tail hours before realizing (or having @U09R86PA4 remind me) that the (silent) normalization failure is probably the root cause of my bug.

Once again bitten by the severe irregularities in pull with :db/id … a missing entity returns nil unless :db/id is part of the pattern. @U1QJACBUM, has there been any movement on this group of bugs around :db/id?

@U1QJACBUM, any idea of when a fix for some (all?) of the pull issues will arrive? https://clojurians.slack.com/archives/C03RZMDSH/p1692628698852119?thread_ts=1688916479.328169&amp;cid=C03RZMDSH