Fork me on GitHub
Thomas Moerman10:07:47

Hi, we bumped into this problem upgrading from Crux to XTDB: We're not sure whether this is a regression or a previously undetected mistake in the query?

"* XTDB circular dependency problem"
(db/q (db/db node) '{:find  [?proj]
                     :in    [?search-string]
                      :where [[?proj :nexus.project/id]

                             ;; WORKS
                             #_(project-name-like? ?proj ?search-string)

                             ;; WORKS
                             #_(or-join [?proj ?search-string]
                               [(text-search :nexus.project/name ?search-string) [[?proj]]])

                             ;; THROWS: Circular dependency between ?proj and ?search-string
                             ;; {:reason,
                             ;;  :node ?proj, :dependency ?search-string}
                             ;; at (dependency.cljc:87)
                             ;; using XTDB version 1.21.0
                             ;; HOWEVER: this works with Crux 21.02-1.15.0-beta
                             (or-join [?proj ?search-string]
                                (project-name-like? ?proj ?search-string)
                                 ;; ...might add other rules here...
                     :rules [[(project-name-like? [?proj ?search-string])
                              [(text-search :nexus.project/name ?search-string) [[?proj]]]]]}
  "ATR") => #{[(db-ref :nexus.project/id 1)]
              [(db-ref :nexus.project/id 3)]}
ā€¢ Using XTDB 1.12.0, there appears to be a problem using or-join in combination with the rule project-name-like? . ā€¢ It does work using Crux 21.02-1.15.0-beta Any thoughts? šŸ™


Hi @U052A8RUT your rule head looks like it needs tweaking as I think the ?proj needs to be an output, not an input:

(project-name-like? [?proj ?search-string])
(project-name-like? [?search-string] ?proj)
(remember to switch the arg ordering in the invocation clause also)


That said, I'm not sure how that could have been working previously :thinking_face:

Thomas Moerman08:07:52

I tried you suggestion, however the result remains the same.

"* XTDB circular dependency problem"
        (db/q (db/db node) '{:find  [?proj]
                             :in    [?search-string]
                             :where [[?proj :nexus.project/id]

                                     ;; WORKS
                                     #_(project-name-like? ?search-string ?proj)

                                     ;; WORKS
                                     #_(or-join [?proj ?search-string]
                                         [(text-search :nexus.project/name ?search-string) [[?proj]]])

                                     ;; WORKS
                                         [(text-search :nexus.project/name ?search-string) [[?proj]]])

                                     ;; THROWS: Circular dependency between ?proj and ?search-string
                                     ;; using XTDB version 1.21.0
                                     ;; HOWEVER: this works with Crux 21.02-1.15.0-beta
                                     (or-join [?search-string ?proj]
                                       (project-name-like? ?search-string ?proj))

                                     ;; THROWS: Circular dependency between ?proj and ?search-string
                                     ;; using XTDB version 1.21.0
                                     ;; HOWEVER: this works with Crux 21.02-1.15.0-beta
                                       (project-name-like? ?search-string ?proj))

                             :rules [[(project-name-like? [?search-string] ?out)
                                      [(text-search :nexus.project/name ?search-string) [[?out]]]]]}
          "ATR") => #{[(db-ref :nexus.project/id 1)] [(db-ref :nexus.project/id 3)]}
This is the full stack trace:
Actual: clojure.lang.ExceptionInfo: Circular dependency between ?proj and ?search-string
{:reason, :node ?proj, :dependency ?search-string}
 at (dependency.cljc:87)
    xtdb.query$calculate_join_order$fn__94131$fn__94135.invoke (query.clj:1302)
    clojure.core.protocols$fn__8249.invokeStatic (protocols.clj:168)
    clojure.core.protocols/fn (protocols.clj:124)
    clojure.core.protocols$fn__8204$G__8199__8213.invoke (protocols.clj:19)
    clojure.core.protocols$seq_reduce.invokeStatic (protocols.clj:31)
    clojure.core.protocols$fn__8236.invokeStatic (protocols.clj:75)
    clojure.core.protocols/fn (protocols.clj:75)
    clojure.core.protocols$fn__8178$G__8173__8191.invoke (protocols.clj:13)
    xtdb.query$calculate_join_order$fn__94131.invoke (query.clj:1300)
    clojure.lang.PersistentVector.reduce (
    xtdb.query$calculate_join_order.invokeStatic (query.clj:1293)
    xtdb.query$calculate_join_order.invoke (query.clj:1283)
    xtdb.query$compile_sub_query.invokeStatic (query.clj:1493)
    xtdb.query$compile_sub_query.invoke (query.clj:1468)
    xtdb.query$compile_sub_query$fn__94457.invoke (query.clj:1537)
    xtdb.query$compile_sub_query.invokeStatic (query.clj:1536)
    xtdb.query$compile_sub_query.invoke (query.clj:1468)
    xtdb.query$build_sub_query$fn__94579.invoke (query.clj:1609)
    xtdb.cache.second_chance.SecondChanceCache.computeIfAbsent (second_chance.clj:68)
    xtdb.cache$compute_if_absent.invokeStatic (cache.clj:10)
    xtdb.cache$compute_if_absent.invoke (cache.clj:9)
    xtdb.query$build_sub_query.invokeStatic (query.clj:1603)
    xtdb.query$build_sub_query.invoke (query.clj:1593)
    xtdb.query$build_or_constraints$iter__93492__93496$fn__93497$fn__93498$or_constraint__93505$iter__93519__93523$fn__93524$fn__93525$fn__93530.invoke (query.clj:1107)
    xtdb.query$build_or_constraints$iter__93492__93496$fn__93497$fn__93498$or_constraint__93505$iter__93519__93523$fn__93524$fn__93525.invoke (query.clj:1090)
    xtdb.query$build_or_constraints$iter__93492__93496$fn__93497$fn__93498$or_constraint__93505$iter__93519__93523$fn__93524.invoke (query.clj:1084)
    clojure.lang.LazySeq.sval (
    clojure.lang.LazySeq.seq (
    clojure.lang.RT.seq (
    clojure.lang.LazySeq.sval (
    clojure.lang.LazySeq.seq (
    clojure.lang.RT.seq (
    xtdb.query$build_or_constraints$iter__93492__93496$fn__93497$fn__93498$or_constraint__93505.invoke (query.clj:1115)
    xtdb.query$build_sub_query$constrain_result_fn__94630$fn__94631.invoke (query.clj:1625)
    xtdb.query$build_sub_query$constrain_result_fn__94630.invokePrim (query.clj:1624)
    xtdb.query$build_sub_query$constrain_result_fn__94630.invoke (query.clj:-1)
    xtdb.index.NAryJoinLayeredVirtualIndex.seek_values (index.clj:265)
    xtdb.index$layered_idx__GT_seq$step__91864.invokePrim (index.clj:317)
    xtdb.index$layered_idx__GT_seq$step__91864.invoke (index.clj:-1)
    xtdb.index$layered_idx__GT_seq.invokeStatic (index.clj:325)
    xtdb.index$layered_idx__GT_seq.invoke (index.clj:302)
    xtdb.query$build_sub_query$fn__94634.invoke (query.clj:1635)

Thomas Moerman08:07:58

AFAIU the problem occurs when combining an or or or-join with a Lucene query embedded in a rule.

Thomas Moerman08:07:53

The problem does not occur with Xtdb 1.20.0


Hey @U052A8RUT apologies for the late reply. Having reproduced this locally and reflected a little, I think this is a more general regression than just Lucene (it affects all such predicate functions), and warrants raising as a bug which I'll do next week. However as a workaround for now, you should be able to add an extra lvar + explicit unifying clause, e.g.:

(or-join [?search-string ?proj2]
         (project-name-like? ?search-string ?proj2))
[(== ?proj ?proj2)]

šŸ™ 1

I explored the problem a little further and just opened this: (but have to park it for now)


If it's a non-trivial problem to have to adjust your existing queries with the workaround then it might be a good idea for us to sync this week to figure out a timeline - I'm always happy to chat in any case šŸ™‚ Otherwise I'm going to have to park this for the time being (sorry!)

Thomas Moerman16:07:11

It's not an urgent matter for us at the moment. For the time being I can work with the previous xtdb version or try out your workaround. Thanks for offering help! :thumbsup:

šŸ™ 1
Thomas Moerman09:07:21

FYI: the workaround solves it for us using the latest version :thumbsup:


Fabulous! I'll keep you posted on a fix šŸ™‚


hello folks lets say I have a few records ingested in the database like [{:category "foo" :xt/id 1 ...} {:category "bar" :xt/id 2 ...} ā€¦] given that I have an array of categories like ["category1" "category2" ...], whats the best way to fetch all the records that the category appears in this list? something like an sql IN

Jacob O'Bryant15:07:26

This should do it:

(q db
   '{:find [doc]
     :in [[category ...]]
     :where [[doc :category category]]}
   ["category1" "category2"])


thank you šŸ™‚


(q db
   '{:find [(pull ?doc [*])]
     :in [[category ...]]
     :where [[?doc :category category]]}
   ["c1" "2"])
works great, and the result looks like this
#{[{:category "c1",
    :xt/id #uuid"2b0ca3ed-d392-44b7-9f9e-a5bddcd35695"}]
  [{:category "c2",
    :xt/id #uuid"d413c4e4-4532-44f4-8e50-738bfcc2b497"}]}
is there a way of tweaking the query to return a vector of maps instead of a set with each element being a vector containing a single map? its trivial to do in clojure but maybe im missing something on datalog side of things

Jacob O'Bryant17:07:54

At the moment you just have to do it in Clojure; I think this is maybe something the XT team will add in the future though? Hopefully at least šŸ™‚. For myself I made a helper fn that I always use instead of xtdb.api/q: With that fn, if you omit the vector around the :find value, e.g. (q db '{:find (pull ?doc [*]) ..., then it'll wrap the results in a (map first ...) for you.

thanks3 1
šŸ™ 1