Question about not clause and cardinality many refs. If an entity has 2 favourites, one where the type is :like and another where it is not (lets say :thumb-up) will this query return the entity or not? In my testing is appears to return the entity. In english I think what I’m asking for is a never satisfies clause.

[:find e?
 :where [?e :favourites ?f]
        (not [?f :type :like])]

@teng I think I have the answer. The retrieved attributes are those that can change. The :db/id never change, it’s the actual timeline for the entity!

@teng It's also because .keySet doesn't return :db/id, which is because :db/id is technically not an attribute of the entity.

@rauh ok!

@mbutler not sure what you're trying to achieve, could you reformulate?

I think the question is, given that :favourites could have many ?f results, and the desire is that if any of them match a clause, then it's not a matching ?e. Does that make sense?

@dominicm yeah totally, just wanted to be sure

Seems to work, not entirely sure why though 😄

@val_waeselynck Thanks so much 🙂, I thought not-join was about specifying which variables unified. Why in this case does it work for what I need?

@mbutler basically, your previous not clause did not work because it filtered out [?e ?f] pairs, not [?e] singletons

I'm not sure not-join is entirely necessary, maybe the following would work:

[:find e? :where [?e :favourites ?f] (not [?e :favourites ?f1] [?f1 :type :like])]

I just prefer not-join because I find it easier to understand what it does at a glance

@val_waeselynck I think I understand, could you help explain why it works on all of the favourites rather than match on any favourite that isn’t the one specified in the not block?

Because there is a favourite that belongs to ?e that doesn’t have a :type :like

So why does datomic think that satisfies the query, I’m happy it doesn’t but am a little unclear on the logic.

I'm sorry I don't get what you wrote 🙂 probably too used to Datalog now. Maybe you could show the results with example data and show me what surprises you ?

?e has 2 favourites {:type :like} {:type :dislike} The reading of that datalog query seems to say match where ?e has a favourite where its :type isnt :like. which is correct for one of the favourites, bother of which belong to ?e so why doesn’t that return ?e.

one of e?s favourites is :type :dislike which isn’t :like so why does the not work across all the favourites

I agree that is does, and its good that it does just surprising.

No, the query says to match ?e where ?e has a favourite, and ?e does not have a favourite which type is :like

the key difference is that the two mentioned favourites don't have to be the same

Okay, so not works in the way I had hoped I just needed to structure my query differently 🙂 When applied across cardinality many refs it works like saying none? Is this correct?

(not 
          [?e :favourites ?f1]
          [?f1 :type :like])

@mbutler yes and no, it depends if ?f1 is externally bound

(not-join [?e]
  [?e :favourites ?f1]
  [?f1 :type :like])

^ this query definitely says so

I think I’m now on the same level, final but that leads me to thinking why doesn’t this work

(not 
          [?e :favourites ?f]
          [?f :type :like])

The [?e :favourites ?f] is because we want to say that the [?e :favourites ?f] is a relation we want?

I don't know exactly if this is a theoritical limitation or a practical one. The Datomic team will be of more help than me for that

I’m going to say that [?e :favourites ?f] gets you an entity with favourites, then the

(not-join 
          [?e :favourites ?f]
          [?f :type :like])

I am wondering, are connections cached by datomic itself, or does it make sense to cache the connections in an atom in the application?

> Connections are > cached such that calling datomic.api/connect multiple times with > the same database value will return the same connection object.

from the doc of datomic.api/connect

Ah thanks 🙂

@mbutler It helps me to think of not clauses along the lines of: consider everything in the not in ‘isolation’ - it will match a set of datoms; remove that set of datoms from the set matched by everything else in the query. in other words, the stuff matched by the ‘not’ is “subtracted” from the result set

Datalog question: are namespaced symbols officially supported for rule names ?

(already asked above but not sure people saw it)

@marshall Absolutely, thats how I began to think of it, like 2 queries where the return of one was removed from the other 🙂

(d/q '[:find ?e
           :in $ ?a
           :where
           [?e :favourite ?i]
           (not-join
             [?e]
             [?e :favourite ?i]
             [?i :size ?x]
             [(> ?x ?a)])]
         db 10)

(d/q '[:find ?e
           :in $ ?a
           :where
           [?e :favourite ?i]
           (not-join
             [?e]
             [?e :favourite ?i]
             [?i :size ?a]
          db 10)

(d/q '[:find ?e
           :in $ ?a
           :where
           [?e :favourite ?i]
           (not-join
             [?e ?a]
             [?e :favourite ?i]
             [?i :size ?x]
             [(> ?x ?a)])]
         db 10)

The comparator (>) requires both arguments to be bound before it can execute The second example will match all possible values to ?a

if you don’t have ?a in the list of bound variables it is effectively a different variable inside the not clause

so to bind it to 10, you must include it in the not-join join list

@marshall Aha yes, thanks, turns out i was just matching on any val of ?a in the second example it was just giving the expected behaviour anyway. That makes much more sense and is consistent 🙂 Thanks again 🙂

no problem :thumbsup:

@marshall Is a query such as the 3rd example capable of being expressed as a rule before I try to do so?

sure. pretty much any set of datalog clauses can be in a rule from the docs: "rules can contain any type of clause: data, expression, or even other rule invocations."

Yep I read that and had concluded it was possible but thanks for the reassurance 🙂

@hunter Yes, if you’re restarting the transactor the peer will report a lost connection (what you’re seeing)

you should see retry and reconnection once your transactor is replaced or becomes available again

@marshall All but one peer did not retry/reconnect, however it's transaction queue was still operational

If it helps, the peer that is not reconnecting is using an asOf database filter, however the t in the log is increasing with each message from the transaction queue

if it’s getting new novelty from the replacement transactor, it has reconnected. that transition may have occurred seamlessly without reporting an error

but if the original transactor is gone and a peer is getting new transactions, it must be connected to the newly active transactor

Here's a question: Is there a way to make something like this work?

(d/q '[:find ?out
       :in $
       :where
       (or-join [?e ?out]
                (and [?e :a1]
                     [(unify ?e ?out)])
                (and [?e :a2]
                     [?out :subcomponent ?e]))]
     [[123 :a1 "foo3"]
      [456 :a1 "foo4"]
      [456 :subcomponent 789]
      [789 :a2 "bar"]])

unify is obviously not a thing. What I'm kind of wanting to do there is say (== ?e ?d)

(ground)

Point being: I've found some entity that may be a top-level type, or it may be a subcomponent. But I want to return the top-level type.

erm. ground requires a const tho

yeah.... 😕

[(identity ?e) ?out]

i think that might do it, although i’m not crazy about it

I guess the obvious thing to do is just run separate queries. Just wondering if there's some functionality I'm missing.

Hmmm, if you're not crazy about it, I'm gonna avoid that then.

i.e. use a built in clojure function that returns the value right back to you and assign that value to a variable

well, it’s not the use, it’s more the putting conditional logic in a query

gotcha

i’d probably tend toward separate queries

with conditional logic in your application code

@potetm I’m not sure I follow what you’re doing in that query.

is:

(d/q '[:find ?out
       :in $
       :where
       (or-join [?out]
                [?out :a1]
                (and [?e :a2]
                     [?out :subcomponent ?e]))]
     [[123 :a1 "foo3"]
      [456 :a1 "foo4"]
      [456 :subcomponent 789]
      [789 :a2 "bar"]])

@bkamphaus Yeah that is. I apparently oversimplified the example.

Something like this:

(d/q '[:find ?out
       :in $ [?e ?a]
       :where
       [?e ?a]
       (or-join [?e ?out]
                (and [?e :a1]
                     [(identity ?e) ?out])
                (and [?e :a2]
                     [?out :subcomponent ?e]))]
     [[123 :a1 "foo3"]
      [456 :a1 "foo4"]
      [456 :subcomponent 789]
      [789 :a2 "bar"]]
     [789 :a2])

(Actual use case is we found some tx-data in the log by attr, and want to find the top-level entity.)

hmm, something somewhere feels off to me, but not entirely sure. I guess it’s just the fact that as Marshall indicated it feels more like branching logic than a set union, which is why identity seems like a hack there. But is the contrived example a match, where you only care about :a1 and :a2 on ?e — and ?e ?a are passed from tx data? Why provide ?a — aren’t you only passing in ?e values you know you care about without additionally limiting by some other attribute which occurs with the datom?

So, in the real case, I'm passing in ?a to scan the logs for changes to certain attributes. In the end, I want all ?e that have ?a, or that has children that have ?a, where ?a is bound to a collection of attrs.

So yeah, having typed that out, that sounds a lot more like branching logic than set unions.

that description seems a bit of a mismatch for the queries you showed, maybe I’m just missing something. I.e. the last description sounds like:

(d/q '[:find ?out
       :in $ [?a ...]
       :where
       (or-join [?out ?a]
                [?out ?a]
                (and [?e ?a]
                     [?out :subcomponent ?e]))]
     [[123 :a1 "foo3"]
      [456 :a1 "foo4"]
      [456 :subcomponent 789]
      [512 :subcomponent 332]
      [332 :idontcare "snafu"]
      [500 :subcomponent 123]
      [789 :a2 "bar"]]
     [:a2 :a1])

if you just want to drop the subcomponents with that kind of exhaustive logic, something like:

(d/q '[:find ?out
       :in $ [?a ...]
       :where
       (or-join [?out ?a]
                [?out ?a]
                (and [?e ?a]
                     [?out :subcomponent ?e]))
       (not [_ :subcomponent ?out])]
     [[123 :a1 "foo3"]
      [456 :a1 "foo4"]
      [456 :subcomponent 789]
      [512 :subcomponent 332]
      [332 :idontcare "snafu"]
      [500 :subcomponent 123]
      [789 :a2 "bar"]]
     [:a2 :a1])

Yeah dropping attrs is probably too much. Not worth it just to avoid doing separate queries. I'm only interested in a small number (4) attrs in the tree.

Hmm... I'm doing very poorly at this. How about I start with the top-level use case?

So I'm looking through the log for 4 attrs that have changed. Two of those attrs are on sub-entities, two are on the top-level entity.

But I'm only interested getting the top-level entity.

This example might be more realistic? I have 2 data sources, one that represents the db, one that represents the log.

(d/q '[:find [?out ...]
       :in $d1 $log
       :where
       ($log or
             [?e :parent/attr]
             [?e :child/attr])
       ($d1 or-join [?e ?out]
            [?out :parent/attr]
            (and [?e :child/attr]
                 [?out :parent/child ?e]))]
     [[123 :parent/attr "foo3"]
      [456 :parent/attr "foo4"]
      [999 :parent/attr "shouldn't show"]
      [456 :parent/child 789]
      [789 :child/attr "bar"]]
     [[123 :parent/attr "foo3"]
      [123 :parent/attr "foo2"]
      [789 :child/attr "bar0"]
      [789 :child/attr "bar"]
      [000 :dont-care "blerg"]])

=> [123 456 999]

I'm probably missing something, but it seems like I need to say (== ?out ?e) in that first or-join.

it’s because you don’t bind the :parent/attr and :child/attr distinctly.

Ah, so ?p ?c instead of both as ?e

still thinking through query structure.

sorry, don’t think that fixes the issue but it seems incorrect to treat them the same initially then try to split them later. In that case it seems to come down to not being able to use different data sources within one or.

I agree that it seems incorrect. Yeah, multiple sources in an or might fix it, but I agree that it's kind of a ridiculous ask.