Fork me on GitHub
#xtdb
<
2020-11-04
>
voltecrus11:11:42

Greetings! Any way to speed up the building the indeces? I'm running crux on JDBC, both transaction log and document store. I have approximately 30k transactions stored, and it takes several minutes to build them on RocksDB.

voltecrus11:11:47

I should mention that I'm familiar with checkpoints - though I'm running crux on google cloud unfortunately.

refset13:11:29

Hi 🙂 you can trying disabling the Rocks write-ahead-log by adding :disable-wal? true as an option https://github.com/juxt/crux/blob/14046b78efbf4ff09866042ea2a611265671b6e3/crux-rocksdb/src/crux/rocksdb.clj#L154A Have you got a ballpark disk write speed (MB/s) that you've observed? There are many options for tuning Rocks for bulk loading which we've not fully explored. You can ever go as far as to build your own SST files: https://rockset.com/blog/optimizing-bulk-load-in-rocksdb/

refset12:11:53

@U0FLJ4UEQ (just making sure you've see the reply ^)

voltecrus11:11:37

Sorry to have taken long! It's been somewhat stressful few days 😅 > you can trying disabling the Rocks write-ahead-log by adding :disable-wal? true as an option It seem to have had no effect 😞. > Have you got a ballpark disk write speed (MB/s) that you've observed? 220 MB/s(average SSD) Had tested all the options for index storage with some generated data, all seem to have about the same time to index a transation - 30 ms. 30 seconds to bring up an instance given 1000 tx log difference is a significant amount of time to add to startup time the way our infrastructure is setup unfortunately. It would seem it has more to do with the JDBC than the index storage choice.