Fork me on GitHub
#xtdb
<
2020-02-12
>
ordnungswidrig09:02:57

I tried crux with a hosted confluent kafka but I’m a little lost regarding the settings. What’s the recommended KV-store? I had the experience that sometimes queries would not return any results although the transactions where published successfully. I’m pretty sure I’m missing something fundamental here. Is the kv-store required to be persistent? I used memdb for simplicity.

jarohen09:02:34

hey @U054UD60U 🙂 re 'recommended KV store' - it's possible to persist the memdb, although we'd recommend other stores (e.g. RocksDB, LMDB) in production usage - are you able to paste your node config? re missing transactions - once you've submitted the transaction it's then indexed asynchronously. to ensure your node's up-to-date with the transaction you've submitted, you can call sync using the transaction-time returned - e.g. (sync node (:crux.tx/tx-time submitted-tx) timeout)

ordnungswidrig09:02:07

I see! Maybe that was the issue.

jarohen09:02:26

(the latest documentation, showing await-tx , unfortunately isn't yet released - 1.7.0 will hopefully be coming in the next week or so 🙂 )

👍 4
ordnungswidrig09:02:33

(crux/start-node {:crux.node/topology 'crux.kafka/topology
	                    :crux.node/kv-store 'crux.kv.memdb/kv
                      :crux.kafka/replication-factor 3
                      :crux.kafka/kafka-properties-file "kafka.properties"})

ordnungswidrig09:02:46

For a freshly started node, how can I away for the latests tx to be processed? sync with current time?

jarohen10:02:23

sync without a time, (sync node <timeout?>) will wait for the new node to catch up

jarohen10:02:43

or, yes, if you know which transaction you want to sync to, you can provide that tx-time

jarohen10:02:03

(if you provide the current time, it'll wait for a transaction at that time, which likely doesn't exist)

jarohen10:02:55

about the persistence - memdb will persist if you supply a :crux.kv/db-dir property in that map, although I wouldn't recommend its use in production. RocksDB is also perfectly suitable for local dev, we use it a lot here

jarohen10:02:24

RocksDB is :crux.node/kv-store 'crux.kv.rocksdb/kv in the config

ordnungswidrig10:02:40

I see. Trying this now.

dotemacs14:02:23

Trying to put ~3,000,000 docs into Crux, where Kafka is running on Docker hosted on a local machine. So it’s not the speediest thing in the world. But any examples of how you’re doing something like this? Thanks

refset15:02:03

Batching up multiple puts into a single transaction is the main learning there

refset15:02:17

Incidentally we've also been doing a lot of work to improve ingestion recently, but please let us know how you get on!

👍 8