Fork me on GitHub
#xtdb
<
2023-02-17
>
Namit Shah08:02:21

For a read heavy use case, which one is more preferable, RocksDB or LMDB. Can anyone help me regarding this dilemma.

tatut10:02:31

If I remember correctly, LMDB should be faster… but I don’t know if such a blanket statement is applicable and in what scenarios

tatut10:02:48

if you can, try both and measure

Namit Shah10:02:16

Sure will try that. Thank you.

dgb2311:02:43

What I personally gathered so far: LMDB is more optimized for reads rather than writes. RocksDB is more optimized for mutations/writes. As with anything performance related there is of course always a big "it depends". You'd have to measure and compare. Secondly. if I remember correctly, it is not safe to backup/copy a RocksDB db while it's running, but it might be with LMDB. Haven't tested that though. And I couldn't find the reference that I read in a hurry.

tatut12:02:23

what do you mean “backup/copy”? checkpointing works fine with RocksDB

refset12:02:32

> LMDB is more optimized for reads rather than writes. > RocksDB is more optimized for mutations/writes. This definitely the case. Our own benchmarking will often show up to 3x faster queries with LMDB > it is not safe to backup/copy a RocksDB db while it's running, but it might be with LMDB as @tatus mentioned, XT's https://docs.xtdb.com/administration/checkpointing/ system is able to take copies of both stores while running, and it accomplishes this using their respective APIs for doing so, see https://github.com/xtdb/xtdb/blob/a231be4c4bfabb5ec75552be074bb056827551e8/modules/lmdb/src/xtdb/lmdb.clj#L133 and https://github.com/xtdb/xtdb/blob/a231be4c4bfabb5ec75552be074bb056827551e8/modules/rocksdb/src/xtdb/rocksdb.clj#L251

👍 2
refset12:02:09

There may however be workloads where LMDB is slower, e.g. potentially in low-memory environments, or where RocksDB's compression across your specific data is very effective at minimising I/O (LMDB doesn't really have compression)