Fork me on GitHub
#xtdb
<
2020-09-05
>
David Pham13:09:55

What are the capacity limits of crux in terms of number of entities identities/datoms for each respective backend? I would be interested in PostgreSQL in particular.

refset14:09:38

It's an interesting question 🙂 We haven't hit the limits for any of the backends yet so it's hard to be sure. I gather Postgres has a fixed single table size limit of 32TB (which needs to fit all docs + transactions) but I haven't seen so far that Rocks or Kafka impose any such theoretical limits...so I think the practical limits will mostly depend on how nicely those backends interact with different filesystems. We know a single RocksDB instance is able to operate happily in the 1-5TB range, which I reckon can probably fit somewhere in the range of 10-50B triples. There are a few improvements we will eventually make to the jdbc backend so that you can store the docs elsewhere (e.g. S3), in which case Postgres' 32TB limit could be dedicated purely for transactions.

David Pham08:09:50

In terms of performance though, is there a limit before seeing a clear degradation?

refset10:09:46

Not that we're aware of. The query engine heavily relies on efficient KV prefix seeks & iteration, so that is the main potential bottleneck that could degrade overall query performance. Fortunately RocksDB seems to have things figured out pretty well. This blog has some great posts describing how Rocks behaves in reality (LMDB is rather different): http://smalldatum.blogspot.com/2015/11/read-write-space-amplification-pick-2_23.html