This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2024-03-19
Channels
- # announcements (5)
- # babashka (49)
- # beginners (11)
- # biff (5)
- # calva (123)
- # clerk (9)
- # cljdoc (5)
- # cljs-dev (9)
- # clojure (62)
- # clojure-europe (32)
- # clojure-nl (1)
- # clojure-norway (54)
- # clojure-uk (3)
- # clojurescript (30)
- # community-development (5)
- # cursive (9)
- # devops (5)
- # events (1)
- # fulcro (35)
- # graalvm (10)
- # gratitude (3)
- # hyperfiddle (9)
- # jobs (3)
- # keechma (1)
- # lsp (10)
- # malli (14)
- # off-topic (42)
- # overtone (1)
- # releases (3)
- # shadow-cljs (66)
- # squint (153)
- # xtdb (19)
I wonder if xtdb guarantees commit so the recorded data can be pulled/queried right after the recording transaction is committed. For sure I am doing something wrong, my tiny web app makes a query but cannot pull the data that was recorded prior to this step. It seems it can after a few milliseconds. My xtdb is on AWS postgres. Can someone please let me know what I am missing.
If you pass the result of xt/submit-tx
to xt/await-tx
and then get a new DB value, you should be guaranteed to have the transaction results in the DB. e.g.:
(let [tx-result (xt/submit-tx node ...)
_ (xt/await-tx node tx-result)
db (xt/db node)]
...)
This query times out despite there not being more than a few thousand records with :type :timecard
. How can I optimise it? Are there any docs about how to speed up slow queries?
(xt/q
(xt/db node) '{:find [tc p1 user d value dd]
:where [[tc :type :timecard]
[d1 :project p1]
[p1 :value value]
[tc :email user]
[tc :duration d]
[tc :date dd]
[(clj-time.core/within? interval dd)]
]
:order-by [[dd :desc]]
:in [interval]}
interval)
Is one of d1
, p1
or value
supposed to join onto something from tc
?
It's possible it's timing out because it will be returning every combination of tc
and p1
, for example:
(xt/submit-tx xtdb-node
[[::xt/put {:xt/id :a :type :one}]
[::xt/put {:xt/id :b :type :one}]
[::xt/put {:xt/id :c :type :two}]
[::xt/put {:xt/id :d :type :two}]])
(xt/q (xt/db xtdb-node)
'{:find [id1 id2]
:where [[id1 :type :one]
[id2 :type :two]]})
; => #{[:a :d] [:b :d] [:a :c] [:b :c]}
I’ve now rewritten this query like so, which is a lot quicker, but still not as fast as I’d like:
(xt/q
(xt/db node) '{:find [tc p1 user d value dd]
:where [[tc :deliverable d1]
[d1 :project p1]
[p1 :value value]
[tc :email user]
[tc :duration d]
[tc :date dd]
(matches tc interval)
]
:rules [[(matches [tc] i)
[tc :type :timecard]
[tc :date dd]
[(clj-time.core/within? i dd)]
]]
:order-by [[dd :desc]]
:in [interval]}
interval)
I tried replicating your issue with some generated data:
(let [projects
(for [id (range 10)]
{:xt/id (str "p-" id)
:value (str "value-" id)})
deliverables
(for [id (range 100)]
{:xt/id (str "d-" id)
:project (str "p-" (rand-int 10))})
timecards
(for [id (range 10000)]
{:xt/id (str "tc-" id)
:type :timecard
:deliverable (str "d-" (rand-int 100))
:date (t/date-time (+ 2019 (rand-int 3)))
:email (str id "@email.com")
:duration :something})]
(xt/submit-tx xtdb-node
(->> (concat projects deliverables timecards)
(map (fn [doc] [::xt/put doc])))))
(time
(xt/q (xt/db xtdb-node)
'{:find [tc p1 user d value dd]
:where [[tc :deliverable d1]
[d1 :project p1]
[p1 :value value]
[tc :email user]
[tc :duration d]
[tc :date dd]
(matches tc interval)]
:rules [[(matches [tc] i)
[tc :type :timecard]
[tc :date dd]
[(clj-time.core/within? i dd)]]]
:order-by [[dd :desc]]
:in [interval]}
(t/interval (t/date-time 2019) (t/date-time 2021))))
; (out) "Elapsed time: 479.920167 msecs"
But I can't seem to replicate it timing out
With a bit of fiddling though just removing the [tc :type :timecard]
give a nice speedup:
(time
(xt/q (xt/db xtdb-node)
'{:find [tc p1 user d value dd]
:where [[tc :deliverable d1]
[d1 :project p1]
[p1 :value value]
[tc :email user]
[tc :duration d]
[tc :date dd]
(matches tc interval)]
:rules [[(matches [tc] i)
;[tc :type :timecard]
[tc :date dd]
[(clj-time.core/within? i dd)]]]
:order-by [[dd :desc]]
:in [interval]}
(t/interval (t/date-time 2019) (t/date-time 2021))))
; (out) "Elapsed time: 301.287542 msecs"
Btw, this query is just as fast and more readable imo:
(xt/q (xt/db xtdb-node)
'{:find [(pull tc [:email :duration :date
{:deliverable [{:project [:value]}]}])
dd]
:where [[tc :date dd]
[(clj-time.core/within? interval dd)]]
:order-by [[dd :desc]]
:in [interval]}
(t/interval (t/date-time 2019) (t/date-time 2021)))
Hmm, how is your node setup?