Fork me on GitHub
#xtdb
<
2020-08-21
>
Nejc11:08:50

Are there any guidelines on tuning the performance? I imported 1mio records into Crux (with Kafka topology ---&gt; single Kafka node, single partition). 200k of "nodes" and 800k of "relationships". Each relationship is represented by its own document (cause it has its own properties too) with :start and :end properties which contain crux.db/id of the document to which it relates. Is this correct approach for representing relationships? Or should I have all ids to related nodes on the "node" document itself (problem with relationship props)? The query itself looks like this and it takes around 17 seconds to complete:

'{:find [foo, bar]
          :where [
                  [foo :id "someIdPropertyValue"]
                  [foo :eType "typeFoo"]
                  [foo :crux.db/id fooId]
                  [rel :end fooId]
                  [rel :eType "RELATED_TO"]
                  [rel :start barId]
                  [bar :crux.db/id barId]
                  ]})
I'm new to Datalog, so I'm not sure if I could write the query more efficiently. The index store used is default (RocksDb) defined by
:crux.kv/db-dir "/tmp/crux-store/indexes"
My guess would be, that the problem is with index DB.

refset11:08:04

There are no general guidelines yet but hopefully we can point you in the right direction. How many types of relationships are there? And how many results does that query return? There's nothing explicitly wrong with modelling relationships as independent documents, like a more typical property graph model, though you do lose some benefits of sorted index seeks. That may well be the bottleneck here but let's see 🙂

Nejc11:08:57

There is ~30 relationship types. But this foo in the sample above has only 1 relationship pointing to it. IOW

'{:find [foo, rel]
          :where [
                  [foo :id "someIdPropertyValue"]
                  [foo :eType "typeFoo"]
                  [foo :crux.db/id fooId]
                  [rel :end fooId]
]}
will return only one foo and only one rel Strangely enough, if I remove eType match of rel the query executes immediately:
'{:find [foo, bar]
          :where [
                  [foo :id "someIdPropertyValue"]
                  [foo :eType "typeFoo"]
                  [foo :crux.db/id fooId]
                  [rel :end fooId]
                  [rel :start barId]
                  [bar :crux.db/id barId]
                  ]})
I'm thinking that [rel :eType "RELATED_TO"] doesn't work on the subset of the previous statement?

Nejc12:08:57

I'm re-importing data ATM and will report back if the issue persists.

jarohen12:08:18

you don't need the explicit [foo :crux.db/id fooId] and [bar :crux.db/id barId] - without these you could request [rel :end foo] [rel :start bar] and this would be an equivalent query

👍 3
jarohen12:08:40

I'd expect removing those to make the query quicker but not significantly so - the most likely cause is then the Crux query planner choosing the wrong query plan for this query and your dataset would you mind setting the log level of the crux.query namespace to debug and sending us the result?

jarohen12:08:10

this will print out (amongst other things) a list of all the attributes in your Crux node - if these are sensitive, feel free to DM @U899JBRPF or me instead 🙂

Nejc12:08:31

Ack.. will get you the logs asap

Nejc12:08:49

Updated query as you suggested:

'{:find [foo, bar]
          :where [
                  [foo :id "someIdPropertyValue"]
                  [foo :eType "typeFoo"]
                  [rel :eType "RELATED_TO"]
                  [rel :end foo]
                  [rel :start bar]
                  ]})
executes in miliseconds. I think I mislead the Crux query planner by extracting and manually matching those ids

👍 3
Nejc12:08:23

Thank you very much on those hints. I believe you don't need the logs now right?

Nejc13:08:04

What are the options on KV store? I can only find samples for embedded RocksDB which doesn't suit my need since I'll be running multiple clients in the distributed environment. And I'm unable to use it in a way that samples suggest (having it in a folder on a local file system) - also the DB is locked by the first client. I can see some projects with Cassandra and RocksDB. Am I getting "too far" as of current Crux roadmap?

refset14:08:22

It would still be interesting to see the logs showing the join order but it's no problem if you have more valuable things to be doing! 🙂 As for a distributed KV store, the way the Crux query engine works is quite "chatty" so having the KV store local & in-process is how everything is currently envisioned for optimal query performance. It's certainly plausible that a shared KV store can still work efficiently, especially given that disk IO latency is often higher that network hops to nearby machines. There's also "rocksdb-cloud" (from Rockset, founded by a team of ex-Facebook RocksDB engineers) which we have been looking at for a while and could also solve the problem quite elegantly. However, ultimately I think there is always going to be some trade-off for query performance. We are certainly open to exploring some of the possibilities as part of the near term roadmap or collaborate somehow. Happy to chat if it's easier.

Nejc12:08:52

Hi @U899JBRPF! Thank you for the answer. I'll get you the logs, once my schedule clears a bit. So I'm trying to understand how the indexing works and the documentation says this:

The main query API is eager but you may also consume the results lazily as the entire Crux query engine and index structure is lazy by design.
So if I get this right... I can connect multiple clients to my Kafka cluster (using Kafka topology) using default :crux.kv/db-dir setting and each client will build it's indexes lazily? On related note: how does the re-indexing looks like if we somehow break it?

👍 3
refset13:08:37

Ah, so the indexes are not constructed lazily. In retrospect that is ambiguous wording in the docs, apologies. The indexes get built eagerly by consuming from the Kafka tx-log as fast as possible. Each node maintains independent indexes. Re-indexing is simply a matter of starting from the beginning, although we do have a checkpoint API coming in the next release so you can replay from a copy of the indexes. Every so often, when we make a breaking index change, a full re-index will be required.

Nejc13:08:54

So if I have node1 inserting the data to Crux, it will also have it's index ready once the data is inserted right (if I use node.awaitTx() ? If I than start another node (`node2`) on different HW, this node needs to consume the transactions topic to build it's own local KW index? Does this happen automatically when I start the node (if so, how do I know that the index is ready?) or I need to consume tx topic manually?

refset13:08:18

All that is correct. As long as the tx-log configuration is the same (i.e. same topic name) then the new nodes will automatically synchronise when you run start. You can know the index is "ready" by running sync and waiting for it to return, but you probably want to set a reasonable timeout as the default could easily be too short for a full re-index if your database is large

Nejc13:08:06

Thank you very much on this answers. That all sounds great. I'll be testing on our big test dataset in the following days.

refset13:08:46

No problem and good luck! How big is big?

Nejc13:08:15

I'll be generating them.. ATM I have ~1mio documents (170k of "nodes" and 900k of "relationships" which sums up to around 150MBs of "raw" data. I'll be "dummy" multiplying this data to different amounts. However, what scares me, is that the index DB for current dataset comes out at around 1GB. If the index DB size will go up in proportion to the number of documents in the Kafka I'm getting a bit worried. Since there is network in between and each client must re-build its own index, which could easily take hours.

👍 3
refset14:08:35

Hmm, well perhaps the checkpoint API will be able to help somewhat. It will help to get a new node up to speed a lot faster than consuming from Kafka directly

Nejc14:08:18

Of course, we will figure something out.. Like maybe using some K8s persistent volume to keep the RocksDB there. I just had different picture in my mind by reading

Nodes can come and go, ... 
...part of the documentation 🙂. If you are running K8s deployments for clients "come and go" is not that cheap anymore. We should instead (probably) run dedicated statefulset node just for entry point to the data.

refset14:08:38

So we did some early k8s work with statefulsets and an original "backup" checkpointing setup which fulfilled the promise somewhat, but we need to revisit it more comprehensively. In can you are looking for inspiration: https://github.com/danmason/crux/blob/ac52f7128771366a6e7d901c21aa9e361136804d/docs/example/standalone_webservice/kubernetes_templates/app_deployment.yml#L2 (this example has since been removed from master)

Nejc14:08:22

Thx! Will take a look.