Hi @U7BEY9U10 - lots of memory is always a good idea for a database, although Crux still works quite well with small amounts of memory due to the way it lazily performs the joins (i.e. unlike other systems that really on intermediate result sets having to fit in memory). In terms of storage I doubt the Kafka/JDBC log storage will ever be the bottleneck, instead it will be the cost of SSDs for the nodes. We have been working on a crux-compactor module that can help decrease the amount of storage required for a given node based on the configurable range of history you're actually interested in querying (e.g. don't store multiple document versions for entities older than 1 year). I recommend benchmarking your node storage utilisation with 1-2GB of approximately representative data and extrapolating from there. I've seen evidence that RocksDB scales pretty seamlessly into the 10s of TB range but we've not tested at that scale yet.

Thanks for the info, @U899JBRPF. That helped but didn’t quite answer my question. Again, what data is stored on the node? All data is stored in Kafka, right? Are full indexes stored on each node? And does that effectively mean that all data must be stored on each node as well (because of multiple EAV, AEV, etc. covering indexes)?

Yes, I could fire up a full topology and try to measure it all as I add data to the database, but I’m trying to get some idea of how things scale up by understanding the architecture first. Absolute numbers can come later.

Ah, sorry - fair point 😅 Assuming you're looking at using Kafka for the tx-log AND document-log, here are some not-completely-unrealistic ratios to have in mind: 1GB tx-log in Kafka (single partition) + 6GB document-log in Kafka (multiple partitions, S3 is also plausible but we're still cooking it) = 20GB indexes per node (yes, all data must be stored on each node as the nodes are essentially independent, although you could potentially have a mix of nodes with different crux-compactor policies if your entity histories have a lot of churn)

Thanks, @U899JBRPF, that helps. And how much of those indexes are stored in memory versus on disk? Obviously, with DBs, more memory is always better, but is it a requirement to store a large fraction (all?) of it in memory, or is it all on disk and memory just helps performance because it allows higher caching?

It's the latter, everything is on-disk (in Rocks/LMDB) but the KV stores do their own caching and Crux has a few LRU-type caches also.

As mentioned, the amount of memory required per query is significantly lower than with Datalog engines that execute clause-by-clause, so you should also get better parallelism for the same amount of memory

OK, thanks. That helps me understand. Much appreciated.