Fork me on GitHub
Slack Archive

This page is not created by, affiliated with, or supported by Slack Technologies, Inc.

2024-01-07

Channels

#xtdb
<
2024-01-07
>
Soumil Shekdar09:01:15

Logical Variable Naming Impact on Query Execution Was wondering if there are certain logic variables that should be avoided since they would interfere with query execution? My understanding was that it should not matter, but we have observed that the following query takes 28s to execute 

:find [(pull [?e]), ?e, ?search-value, ?search-score]
:where [[(text-search :field "term") [?e, ?search-value, ?search-score]]]
But just changing it to the following reduced execution time to 3s 
:find [(pull [?e]), ?e, ?v, ?s]
:where [[(text-search :field "term") [?e, ?v, ?s]]]

😵 1
💡 1
refset10:01:48

Thanks for sharing @U05EY098T09 - this is over HTTP presumably? How many results?

James Hamlin19:01:00

@U899JBRPF would the logic variable names actually appear in the result set? i'd expect to receive only the values in the results, making the variable naming equivalent wrt the result set

refset12:01:43

Oh, apologies, I didn't read the query carefully enough. You're right though @U04CAK74QG4 that the result should be unaffected in this case (unless you were to use the :keys option on the query) I'm struggling to see anything else that could explain the performance difference between the two queries :thinking_face:

refset12:01:03

Would be useful to know if it's happening over HTTP or not though - I'll try to repro later today

thanks3 1
Soumil Shekdar18:01:54

This is over HTTP 🙂

kokonut16:01:48

This might be a newbie question. I have an app that has xtdb in the local postgres. When I start up the app, it shows dozens (if not hundreds) of similar debug logs like the following. Does this mean that I set up anything wrong? Otherwise, I'd wonder why transactions need be indexed and committed on start up. 

2024-01-07 11:38:36,086 [xtdb-tx-processor-1] DEBUG xtdb.tx - Indexing tx-id: 12 
2024-01-07 11:38:36,091 [xtdb-tx-processor-1] DEBUG xtdb.tx - Transaction committed: {:xtdb.api/tx-id 12, :xtdb.api/tx-time #inst "2023-12-31T03:03:09.817-00:00", :xtdb.tx.event/tx-events [[:crux.tx/put #xtdb/id "9ef81a98db039656b88da9e1789608fb2693c5db" #xtdb/id "0839d00ff55b5bd01e2f91ec4461635cfa6b17a3"]], :xtdb.api/submit-tx-opts {}} 
2024-01-07 11:38:36,092 [xtdb-tx-processor-1] DEBUG xtdb.tx - Indexing tx-id: 14 
2024-01-07 11:38:36,092 [xtdb-tx-processor-1] DEBUG xtdb.tx - Transaction committed: {:xtdb.api/tx-id 14, :xtdb.api/tx-time #inst "2023-12-31T03:04:20.407-00:00", :xtdb.tx.event/tx-events [[:crux.tx/put #xtdb/id "e90723d7668cad355690592b3e14ab1e73d3801a" #xtdb/id "ca29ad1a89fda662a1f9943664c9d7961b8b86ed"]], :xtdb.api/submit-tx-opts {}}

refset18:01:02

Hey @U022N1DU7GQ do you have an index-store backend configured (e.g. using RocksDB)? if not then the node will be replaying from the log each time

kokonut18:01:27

Nope, I only use postgres and nothing else.

👌 1
kokonut18:01:10

I just wanted to see if it is normal. Thanks.

👍 1
kokonut14:03:20

On my setup, this replaying logs makes my startup time quite longer. Would it be recommended to use RocksDB for index store?

kokonut15:03:12

I am basically referring to this documentation, https://v1-docs.xtdb.com/storage/rocksdb/ but also wondering what documentation can be helpful on deploying on a cloud.

refset17:03:03

Hey @U022N1DU7GQ yes RocksDB is the recommended choice for index store regardless of where you are deploying 🙂

refset17:03:18

which cloud are you thinking of deploying to?

kokonut19:03:25

I am currently using AWS postgres (through heroku) only. I am open to cloud choices for DB.

kokonut19:03:58

I am currently trying to learn how to deploy/use rocksdb on clouds. Doesn't seem as easy as using postgres if on cloud. 🙂

refset22:03:18

ah, you will probably want to keep postgres around also (for the tx-log and doc-store), Rocks is really only intended for the index-store in serious deployments (i.e. where HA is needed)

kokonut15:03:26

yes, I am getting it. I added this (although I am not sure if this is okay when deployed to heroku) in my db setup 

:xtdb/index-store
          {:kv-store {:xtdb/module 'xtdb.rocksdb/->kv-store
                      :db-dir (io/file "/tmp/rocksdb")}}

kokonut15:03:52

So the setting looks like 

(xt/start-node
         {:xtdb.jdbc/connection-pool
          {:dialect {:xtdb/module 'xtdb.jdbc.psql/->dialect}
           :db-spec db-spec}
          :xtdb/tx-log
          {:xtdb/module 'xtdb.jdbc/->tx-log
           :connection-pool :xtdb.jdbc/connection-pool}
          :xtdb/document-store
          {:xtdb/module 'xtdb.jdbc/->document-store
           :connection-pool :xtdb.jdbc/connection-pool}
          :xtdb/index-store
          {:kv-store {:xtdb/module 'xtdb.rocksdb/->kv-store
                      :db-dir (io/file "/tmp/rocksdb")}}})

kokonut15:03:08

locally, it works perfectly as wanted.

kokonut15:03:21

on cloud, it looks okay - I am monitoring.

kokonut15:03:51

Please correct me if any of this looks wrong. Otherwise, thanks much for help @U899JBRPF

refset20:03:29

nice - looks good to me! 🙂

👍 1
jarohen22:01:58

For those of you on the other side of the pond (it'll be 2am for us in the UK!), the #clojure-losangeles group are running a collaborative learning session on XT2 this https://meetu.ps/e/MNYjn/vx7CV/i - enjoy 😊

🚀 3
dorab23:01:22

We'll record the session. Of course, you're welcome to join us in person as well.

Stef Coetzee01:01:38

Is the event recording available somewhere?

dorab16:01:54

Not yet. But soon.

gratitude 1