Fork me on GitHub

Is it possible to use a binding for an attribute in a where clause?

> (xt/q (xt/db xnode) '{:find [e a v]
                        :in [a]
                        :where [[e a v]]}
Execution error (IllegalArgumentException) at xtdb.error/illegal-arg (error.clj:12).
Query didn't match expected structure


The query engine relies on all attributes being declared prior to compilation, but you can enumerate them and generate queries dynamically like this


Very cool! Thanks for the response. I'm thinking about developer tools that integrate with xtdb which might find that type of query useful.

☺️ 1

Related, is there a way to scan through the entities in the db?


Oh. Looks like I can do:

(xt/q (xt/db xnode) '{:find [e id]
                      :where [[e :xt/id id]]})
Since every entity has an :xt/id


and do incremental scans using:

(xt/q (xt/db xnode) '{:find [e id]
                      :in [last-id]
                      :where [[e :xt/id id]
                              [(pos? ?comparison)]
                              [(compare id last-id) ?comparison]]
                      :limit 10}


Well, if you can be sure that nothing inserts ids to the middle while you are scanning


right, the scan should probably run on a snapshot rather than calling xt/db on every iteration


I'm pretty new to xtdb, but it seems like you should be able to run a scan on a snapshot and then scan the transactions added since the snapshot you used.


Sure yeah, if you have a job where you can keep a handle on it, and it's not the usual HTTP backend where the request for the next page might not even come


Or perhaps have the tx id or tx time as query param, so that the pagination can be continued at any time


(I have no idea how we will do it, for now we will just not paginate 😬 )


> it seems like you should be able to run a scan on a snapshot take a look at open-q πŸ™‚

πŸ™‚ 1

in 1.21.0 I'm getting a weird exception when match fails

user> (xt/submit-tx node [[::xt/put {:xt/id "foo" :bar 42}]])
#:xtdb.api{:tx-id 37655,
           :tx-time #inst "2022-05-19T07:05:55.250-00:00"}
user> (xt/submit-tx node [[::xt/match "foo" {:xt/id "foo" :bar 42}] [::xt/put {:xt/id "foo" :bar 666}]]) ; match succeeds
#:xtdb.api{:tx-id 37656,
           :tx-time #inst "2022-05-19T07:06:11.561-00:00"}
user> (xt/entity (xt/db node) "foo")
{:bar 666, :xt/id "foo"}
user> (xt/submit-tx node [[::xt/match "foo" {:xt/id "foo" :bar 42}] [::xt/put {:xt/id "foo" :bar "not updated"}]]) ; match fails
#:xtdb.api{:tx-id 37657,
           :tx-time #inst "2022-05-19T07:06:37.979-00:00"}10:06:37.982 [bus-listener-4] ERROR  - Uncaught exception:
java.lang.NullPointerException: Cannot invoke "java.lang.Number.doubleValue()" because "x" is null
	at clojure.lang.RT.longCast(
	at xtdb.metrics.index_store$assign_doc_meters$fn__53722.invoke(index_store.clj:35)
	at xtdb.bus.EventBus$fn__51683$fn__51684.invoke(bus.clj:105)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(
	at java.base/java.util.concurrent.ThreadPoolExecutor$
	at java.base/


so because no docs were ingested, there is no count to update?


hey @U11SJ6Q0K - I'd be interested in a repro for this if you have one?


do you mean the node config? the steps to repro are in the snippet


aha, sorry, haven't started my morning coffee yet 😳

🍡 1

alright, so I have the following, which gives the same error - cheers πŸ™‚

(with-open [node (xt/start-node {:xtdb.metrics.jmx/reporter {}})]
  (xt/submit-tx node [[::xt/match "foo" {:xt/id "foo" :bar 42}]])
  (xt/sync node))


have a fix, popping it via CI and will get it pushed to master


pretty much what you speculated, (when (seq doc-ids) ...) required in the metrics plugin


Ref., I find it a bit unexpected that xt/with-tx is faster than just running transactions to an in mem db. What does it do differently to process the transactions into the temporary db instance compared to "permanent" in mem node?


idk if it does the same as datomic, it doesn't update indexes for real... so queries will be slower using with-tx returned db value


or so I would expect, but we are using in-memory dbs in our tests and it has been fast enough


I think in this case the author is assuming that you have a larger base set of data that you ingest once, and then run several different tests on top - i.e. ingest, with-tx, test, with-tx, test, ...

πŸ‘ 1

I suspect with-tx is faster simply because there's no tx-log or doc-store in the picture. It is a real in-memory KV index though


how does eviction work with tx fns now? I see there was some discussion in but doesn't look conclusive


@U797MAJ8M is the main change here - transaction functions replace their arg docs with the result of the tx-fn, and any resulting documents are then put in the doc-store, to be evicted in the usual way


ah right, that way it can still work with later-evicted data. I think match doesn't work that way? Is there still a problem with evict and migrations as in


indeed it doesn't, and it should - good spot πŸ™‚ it's been on our radar for a while, but unfortunately is a more involved, potentially tx-log breaking change, so not one we'll be making lightly...