Fork me on GitHub
#xtdb
<
2023-12-13
>
Christian Pekeler17:12:24

When importing data, is there any downside to putting it all into one very large transaction? Is there a maximum transaction size?

refset17:12:43

Hey @UDQE4G9B2 there's no hardcoded maximum, but you will eventually OOM if you push it too far. Our usual advice for batching is 1000 docs/tx, anything higher doesn't really save on any per-tx overheads

refset17:12:16

It will also be much easier to monitor import progress given reasonably chunked batches

jarohen17:12:22

@UDQE4G9B2 as Jeremy says regarding 1k docs/tx, but also if you're using Kafka for either your tx-log or doc-store, it has a (configurable) message size limit

jarohen17:12:05

doc-store should be ok as it's one doc per message; the tx metadata will eventually get too large for one message

Christian Pekeler18:12:56

This is using Postgresql and RocksDB; no Kafka. Cool, then I’ll use 1000docs/tx. Thanks!