Fork me on GitHub

Hello, I’ve got a data model that has a container like record and a series of children. I was wondering if it is possible to filter using pull syntax on joins. Something like:

(xt/q (xt/db node)
   '{:find [(pull e [* {(:item/_wishlist {:as items}) [*]}])]
     :where [[e :wishlist/id]
             [i :item/wishlist e]
             [i :item/price p]
             [(<= 150 p)]]})
Where this query returns all wishlists with their items embedded, but only items that are over $150 in price. I’ve got a similar query in my app and it returns all of the ‘items’ even if they don’t match the price filter (e.g. some item that only is $100).


Hey @U03DWT00WUB in short 'no' - pull (projection) is executed explicitly after the main query is finished. I think you can work around it by either: 1. doing more work inside the query (e.g. including building up the hash-map nested result) 2. running the pull in a subquery, and then filter'ing out the results of the pull in an outer query 3. performing the filtering in your app


Happy to help with examples if one or more of those seems of interest 🙂


this is actually a semi common thing, pull is so convenient, but lacks this feature

📝 1

luckily you can do multiple queries from the with the same basis and use clojure code to combine results… but still I have certainly wished for a filtered pull many times


> you can do multiple queries from the with the same basis and use clojure code to combine results ah yep, of course, how did I forget that 4th option 🙂


I've never actually tried it, but I've wondered before about whether 's EQL pull mechanism could play nicely "out of the box" embedded in a Datalog query (essentially a sub-option for my option (1) above). It wouldn't be able to perform joins though like the actual pull, but only work on the individual document structures


Hello @U899JBRPF I'm curious about those first two solutions you mentioned. @U11SJ6Q0K can you show me what you mean by within the same basis and combine them? I just finished the available tutorials yesterday so still trying to get my footing. Thanks for your responses.


as you call (xt/db node) you can run multiple queries on that returned db and know that they will point to the same tx (consistent results)


Oh okay, so something like pull all the container records in call to (xt/q) A, pull all the items in call to (xt/q) B, and then do some kind of merge?


as you are pulling form local indexes, the “roundtrips” don’t really matter


Just had another question, if the pull happens after the query is executed, would it be possible to change it so that it’s operating against “a world” that matches all the where clauses. So in my case if I put in a where clause that says items must be priced higher than $150, then the pulls join only can only look at items that satisfy the where clause?


not in any practical way afaict


Bummer. I came up with something like this after some tinkering.

(xt/q (xt/db node) 
        '{:find [(pull e [*]) items]
           :where [[e :wishlist/id]
                   [(q {:find [(pull i [*])]
                        :where [[i :item/lot lot]]}) items]]})
This maybe works? Though all the items come back as a tuple with a single map in it. Not hard to get around with some regular clojure code, but a little awkward and seems like it would waste a ton of CPU cycles if people are requesting large data sets. I could maybe do the :keys [] and specify fields explicitly, but that might get a little hairy as in my real data set I want to joins around 4 levels deep. Might just have to do some rethinking about how to get this data?


I’d say benchmark first… a little formatting doesn’t sound like a problem even for big results, but memory might be, You could also stream results with open-q depending on how the data is consumed

Mikko Harju05:09:55

Hi all! What is the best procedure for CORS using the (Docker image standalone-rocksdb) HTTP server version of XTDB? Just don't do it and put it behind a proxy? 😄


Hey @U32ST9GR5! Great question, I had to double-check this one 😅 we definitely don't support CORS in the latest versions of XT and we only did so initially to try and get a cljs console off the ground (but we later changed tack) I think a proxy is the way to go (short of forking the http-server module), and this might serve for inspiration:

Mikko Harju09:09:15

Yeah, I solved it by just putting this to my backend and integrated the routes from the xtdb.http-server to my route table, along with the cors-middleware 🙂 Seems to be working nicely!

Mikko Harju09:09:16

I wanted our web client to be able to make requests directly to the xtdb as well since this is an internal tool – saves a lot of effort 😄

Mikko Harju09:09:25

Thanks for answering!


Awesome, always happy to help ☺️

Mikko Harju10:09:53

Another question – Where can I find documentation (if anywhere?) about the encoding of the in-args-json -array – I'd need to send UUIDs in them, should they work as just strings or how does it work?

Mikko Harju10:09:31

Eg how to do this through HTTP :

    (xt/db node)
    '{:find [(pull ?user [*])]
      :where [[?user :core/knows ?skill]
              [(get ?skill :core/skill) ?skill-id]]
      :in [[?skill-id ...]]}
    [#uuid "e206855a-2e83-4742-a793-8b8816a61dc7"
     #uuid "62e8cb0d-8cf2-480c-8919-3d90b9fff482"
     #uuid "f8051e4e-8281-4d3f-96f3-54b80a1f8fd5"])


in-args-json implicitly can't understand actual UUID types, so you'd have to pass and store them as strings (which is lossy and inefficient). Could you use in-args-edn instead? That should work fine with #uuid literals


> Where can I find documentation about the encoding of the in-args-json suggesting to browse the source/tests is certainly not a viable suggestion for non-Clojurists, but since you're posting here I'm presuming that's not a complete turn-off 🙂 see: and This is the main docs page for the HTTP API though

Mikko Harju11:09:21

Definitely can work my way from there. Thanks! 🙂

Mikko Harju11:09:49

Tried EDN – This fails with HTTP 500?


The error handling isn't terribly helpful here, and by default obscures the underlying error, but I think you just need to add an extra [ ] around your in-args-edn value In any case, this works:

curl -G \
--data-urlencode "queryEdn={:find [e] :in [a b] :where [[e :xt/id]] :limit 10}" \
--data-urlencode "inArgsEdn=[\"9dde518b-01df-4e64-b370-13ee3ba6c248\" \"5c44fc5f-cb5c-4f19-af1b-2e99672a0053\"]" \                     


For reference, I debugged my own example there by adding some try/catch printing within this existing function in http_server/query.clj:

(defn data-browser-query [options]
  (fn [req]
      (let [{query-params :query body-params :body} (get-in req [:parameters])
            {:keys [valid-time tx-time tx-id query-edn in-args-edn in-args-json]} query-params
            query (or query-edn (get body-params :query))
            in-args (or in-args-edn in-args-json (get body-params :in-args))]
        (-> (if (nil? query)
              (assoc options :no-query? true)
              (run-query (transform-req query req)
                         (assoc options
                                :valid-time valid-time
                                :tx-time tx-time
                                :tx-id tx-id)))
            (transform-query-resp req)))
      (catch java.lang.AssertionError e
        (prn e)))))


We could probably add some tactical log/debug logging in there (to avoid any such ad-hoc REPL patching), short of a review of more comprehensive error handling / logging changes

Mikko Harju14:09:00


🙏 1

Just to confirm - did that [ ] do the trick?

Mikko Harju05:09:12

Yes it did! 🙂