xtdb 2022-05-19 | Slack Archive

Is it possible to use a binding for an attribute in a where clause?

> (xt/q (xt/db xnode) '{:find [e a v]
                        :in [a]
                        :where [[e a v]]}
      :user/name)
Execution error (IllegalArgumentException) at xtdb.error/illegal-arg (error.clj:12).
Query didn't match expected structure

refset00:05:37

The query engine relies on all attributes being declared prior to compilation, but you can enumerate them and generate queries dynamically like this https://gist.github.com/refset/93135adcbf41fccab9b641638ab10997

phronmophobic01:05:52

Very cool! Thanks for the response. I'm thinking about developer tools that integrate with xtdb which might find that type of query useful.

☺️ 1

phronmophobic01:05:18

Related, is there a way to scan through the entities in the db?

phronmophobic01:05:16

Oh. Looks like I can do:

(xt/q (xt/db xnode) '{:find [e id]
                      :where [[e :xt/id id]]})

Since every entity has an :xt/id

phronmophobic01:05:30

and do incremental scans using:

(xt/q (xt/db xnode) '{:find [e id]
                      :in [last-id]
                      :where [[e :xt/id id]
                              [(pos? ?comparison)]
                              [(compare id last-id) ?comparison]]
                      :limit 10}
      :a/b)

Hukka05:05:49

Well, if you can be sure that nothing inserts ids to the middle while you are scanning

phronmophobic05:05:49

right, the scan should probably run on a snapshot rather than calling xt/db on every iteration

phronmophobic05:05:38

I'm pretty new to xtdb, but it seems like you should be able to run a scan on a snapshot and then scan the transactions added since the snapshot you used.

Hukka05:05:02

Sure yeah, if you have a job where you can keep a handle on it, and it's not the usual HTTP backend where the request for the next page might not even come

Hukka05:05:59

Or perhaps have the tx id or tx time as query param, so that the pagination can be continued at any time

Hukka05:05:29

(I have no idea how we will do it, for now we will just not paginate 😬 )

refset08:05:12

> it seems like you should be able to run a scan on a snapshot take a look at open-q 🙂

🙂 1

tatut07:05:01

in 1.21.0 I'm getting a weird exception when match fails

user> (xt/submit-tx node [[::xt/put {:xt/id "foo" :bar 42}]])
#:xtdb.api{:tx-id 37655,
           :tx-time #inst "2022-05-19T07:05:55.250-00:00"}
user> (xt/submit-tx node [[::xt/match "foo" {:xt/id "foo" :bar 42}] [::xt/put {:xt/id "foo" :bar 666}]]) ; match succeeds
#:xtdb.api{:tx-id 37656,
           :tx-time #inst "2022-05-19T07:06:11.561-00:00"}
user> (xt/entity (xt/db node) "foo")
{:bar 666, :xt/id "foo"}
user> (xt/submit-tx node [[::xt/match "foo" {:xt/id "foo" :bar 42}] [::xt/put {:xt/id "foo" :bar "not updated"}]]) ; match fails
#:xtdb.api{:tx-id 37657,
           :tx-time #inst "2022-05-19T07:06:37.979-00:00"}10:06:37.982 [bus-listener-4] ERROR  - Uncaught exception:
java.lang.NullPointerException: Cannot invoke "java.lang.Number.doubleValue()" because "x" is null
	at clojure.lang.RT.longCast(RT.java:1284)
	at xtdb.metrics.index_store$assign_doc_meters$fn__53722.invoke(index_store.clj:35)
	at xtdb.bus.EventBus$fn__51683$fn__51684.invoke(bus.clj:105)
	at clojure.lang.AFn.run(AFn.java:22)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)

tatut07:05:52

so because no docs were ingested, there is no count to update?

jarohen08:05:54

hey @U11SJ6Q0K - I'd be interested in a repro for this if you have one?

tatut08:05:31

do you mean the node config? the steps to repro are in the snippet

jarohen08:05:57

aha, sorry, haven't started my morning coffee yet 😳

🍵 1

jarohen09:05:23

alright, so I have the following, which gives the same error - cheers 🙂

(with-open [node (xt/start-node {:xtdb.metrics.jmx/reporter {}})]
  (xt/submit-tx node [[::xt/match "foo" {:xt/id "foo" :bar 42}]])
  (xt/sync node))

jarohen09:05:08

have a fix, popping it via CI and will get it pushed to master

jarohen09:05:09

pretty much what you speculated, (when (seq doc-ids) ...) required in the metrics plugin

Hukka08:05:16

Ref. https://www.juxt.pro/blog/testingwithxtdb, I find it a bit unexpected that xt/with-tx is faster than just running transactions to an in mem db. What does it do differently to process the transactions into the temporary db instance compared to "permanent" in mem node?

tatut08:05:59

idk if it does the same as datomic, it doesn't update indexes for real... so queries will be slower using with-tx returned db value

tatut08:05:17

or so I would expect, but we are using in-memory dbs in our tests and it has been fast enough

jarohen08:05:52

I think in this case the author is assuming that you have a larger base set of data that you ingest once, and then run several different tests on top - i.e. ingest, with-tx, test, with-tx, test, ...

👍 1

refset08:05:23

I suspect with-tx is faster simply because there's no tx-log or doc-store in the picture. It is a real in-memory KV index though

tatut09:05:07

ah, ok

nivekuil23:05:45

how does eviction work with tx fns now? I see there was some discussion in https://github.com/xtdb/xtdb/issues/121 but doesn't look conclusive

jarohen06:05:16

@U797MAJ8M https://github.com/xtdb/xtdb/pull/881 is the main change here - transaction functions replace their arg docs with the result of the tx-fn, and any resulting documents are then put in the doc-store, to be evicted in the usual way

nivekuil09:05:41

ah right, that way it can still work with later-evicted data. I think match doesn't work that way? Is there still a problem with evict and migrations as in https://github.com/xtdb/xtdb/issues/1386

jarohen09:05:20

indeed it doesn't, and it should - good spot 🙂 it's been on our radar for a while, but unfortunately is a more involved, potentially tx-log breaking change, so not one we'll be making lightly...

2022-05-19

Channels