This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2022-09-02
Channels
- # adventofcode (6)
- # announcements (6)
- # babashka (21)
- # babashka-sci-dev (18)
- # biff (6)
- # clara (4)
- # clj-commons (2)
- # clj-kondo (7)
- # cljdoc (4)
- # clojure (9)
- # clojure-berlin (8)
- # clojure-europe (23)
- # clojure-gamedev (3)
- # clojure-indonesia (1)
- # clojure-nl (1)
- # clojure-norway (10)
- # clojure-poland (1)
- # clojurescript (27)
- # community-development (1)
- # conjure (32)
- # etaoin (6)
- # events (20)
- # fulcro (5)
- # graalvm (1)
- # helix (19)
- # hyperfiddle (14)
- # introduce-yourself (2)
- # music (1)
- # nbb (24)
- # off-topic (37)
- # pathom (2)
- # polylith (14)
- # reagent (11)
- # releases (1)
- # remote-jobs (1)
- # reveal (22)
- # shadow-cljs (16)
- # sql (3)
- # squint (11)
- # test-check (2)
- # xtdb (36)
Hello, I’ve got a data model that has a container like record and a series of children. I was wondering if it is possible to filter using pull syntax on joins. Something like:
(xt/q (xt/db node)
'{:find [(pull e [* {(:item/_wishlist {:as items}) [*]}])]
:where [[e :wishlist/id]
[i :item/wishlist e]
[i :item/price p]
[(<= 150 p)]]})
Where this query returns all wishlists with their items embedded, but only items that are over $150 in price. I’ve got a similar query in my app and it returns all of the ‘items’ even if they don’t match the price filter (e.g. some item that only is $100).Hey @U03DWT00WUB in short 'no' - pull
(projection) is executed explicitly after the main query is finished. I think you can work around it by either:
1. doing more work inside the query (e.g. including building up the hash-map
nested result)
2. running the pull
in a subquery, and then filter
'ing out the results of the pull in an outer query
3. performing the filtering in your app
luckily you can do multiple queries from the with the same basis and use clojure code to combine results… but still I have certainly wished for a filtered pull many times
> you can do multiple queries from the with the same basis and use clojure code to combine results ah yep, of course, how did I forget that 4th option 🙂
I've never actually tried it, but I've wondered before about whether https://github.com/lilactown/pyramid 's EQL pull mechanism could play nicely "out of the box" embedded in a Datalog query (essentially a sub-option for my option (1) above). It wouldn't be able to perform joins though like the actual pull
, but only work on the individual document structures
Hello @U899JBRPF I'm curious about those first two solutions you mentioned. @U11SJ6Q0K can you show me what you mean by within the same basis and combine them? I just finished the available tutorials yesterday so still trying to get my footing. Thanks for your responses.
as you call (xt/db node)
you can run multiple queries on that returned db and know that they will point to the same tx (consistent results)
Oh okay, so something like pull all the container records in call to (xt/q)
A, pull all the items in call to (xt/q)
B, and then do some kind of merge?
Just had another question, if the pull happens after the query is executed, would it be possible to change it so that it’s operating against “a world” that matches all the where clauses. So in my case if I put in a where clause that says items must be priced higher than $150, then the pulls join
only can only look at items that satisfy the where clause?
Bummer. I came up with something like this after some tinkering.
(xt/q (xt/db node)
'{:find [(pull e [*]) items]
:where [[e :wishlist/id]
[(q {:find [(pull i [*])]
:where [[i :item/lot lot]]}) items]]})
This maybe works? Though all the items come back as a tuple with a single map in it. Not hard to get around with some regular clojure code, but a little awkward and seems like it would waste a ton of CPU cycles if people are requesting large data sets. I could maybe do the :keys []
and specify fields explicitly, but that might get a little hairy as in my real data set I want to joins around 4 levels deep. Might just have to do some rethinking about how to get this data?I’d say benchmark first… a little formatting doesn’t sound like a problem even for big results, but memory might be, You could also stream results with open-q
depending on how the data is consumed
Hi all! What is the best procedure for CORS using the (Docker image standalone-rocksdb) HTTP server version of XTDB? Just don't do it and put it behind a proxy? 😄
https://github.com/xtdb/xtdb/issues/471 found this issue
Hey @U32ST9GR5! Great question, I had to double-check this one 😅 we definitely don't support CORS in the latest versions of XT and we only did so initially to try and get a cljs console off the ground (but we later changed tack) I think a proxy is the way to go (short of forking the http-server module), and this might serve for inspiration: https://github.com/wkok/re-frame-crux
Yeah, I solved it by just putting this to my backend and integrated the routes from the xtdb.http-server to my route table, along with the cors-middleware 🙂 Seems to be working nicely!
I wanted our web client to be able to make requests directly to the xtdb as well since this is an internal tool – saves a lot of effort 😄
Thanks for answering!
Another question – Where can I find documentation (if anywhere?) about the encoding of the in-args-json -array – I'd need to send UUIDs in them, should they work as just strings or how does it work?
Eg how to do this through HTTP :
(xt/q
(xt/db node)
'{:find [(pull ?user [*])]
:where [[?user :core/knows ?skill]
[(get ?skill :core/skill) ?skill-id]]
:in [[?skill-id ...]]}
[#uuid "e206855a-2e83-4742-a793-8b8816a61dc7"
#uuid "62e8cb0d-8cf2-480c-8919-3d90b9fff482"
#uuid "f8051e4e-8281-4d3f-96f3-54b80a1f8fd5"])
in-args-json
implicitly can't understand actual UUID types, so you'd have to pass and store them as strings (which is lossy and inefficient). Could you use in-args-edn
instead? That should work fine with #uuid
literals
> Where can I find documentation about the encoding of the in-args-json suggesting to browse the source/tests is certainly not a viable suggestion for non-Clojurists, but since you're posting here I'm presuming that's not a complete turn-off 🙂 see: https://github.com/xtdb/xtdb/blob/20a2632a18a36dae2e915ab032307ce0c5b3e150/modules/http-server/src/xtdb/http_server/query.clj#L53 and https://github.com/xtdb/xtdb/blob/44303aa35ff5843b2e1130d1bb4a943bc3f8b3be/modules/http-server/test/xtdb/http_server/json_test.clj#L122 This is the main docs page for the HTTP API though https://docs.xtdb.com/clients/http/openapi/1.21.0/#/paths/~1_xtdb~1query/get
Definitely can work my way from there. Thanks! 🙂
Tried EDN – This fails with HTTP 500?The error handling isn't terribly helpful here, and by default obscures the underlying error, but I think you just need to add an extra [
]
around your in-args-edn value
In any case, this works:
curl -G \
--data-urlencode "queryEdn={:find [e] :in [a b] :where [[e :xt/id]] :limit 10}" \
--data-urlencode "inArgsEdn=[\"9dde518b-01df-4e64-b370-13ee3ba6c248\" \"5c44fc5f-cb5c-4f19-af1b-2e99672a0053\"]" \
localhost:3000/_xtdb/query
For reference, I debugged my own example there by adding some try/catch printing within this existing function in http_server/query.clj
:
(defn data-browser-query [options]
(fn [req]
(try
(let [{query-params :query body-params :body} (get-in req [:parameters])
{:keys [valid-time tx-time tx-id query-edn in-args-edn in-args-json]} query-params
query (or query-edn (get body-params :query))
in-args (or in-args-edn in-args-json (get body-params :in-args))]
(-> (if (nil? query)
(assoc options :no-query? true)
(run-query (transform-req query req)
in-args
(assoc options
:valid-time valid-time
:tx-time tx-time
:tx-id tx-id)))
(transform-query-resp req)))
(catch java.lang.AssertionError e
(prn e)))))
We could probably add some tactical log/debug
logging in there (to avoid any such ad-hoc REPL patching), short of a review of more comprehensive error handling / logging changes
Done https://github.com/xtdb/xtdb/commit/a5723508b741034a2f4fa59c445e320d47fe0923 🙂
Yes it did! 🙂