This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2023-12-13
Channels
- # adventofcode (37)
- # announcements (11)
- # babashka (46)
- # beginners (35)
- # biff (1)
- # clojure (44)
- # clojure-austin (1)
- # clojure-europe (23)
- # clojure-nl (2)
- # clojure-norway (8)
- # clojure-uk (5)
- # conjure (3)
- # cursive (22)
- # data-science (13)
- # docker (11)
- # events (8)
- # hyperfiddle (7)
- # joyride (1)
- # juxt (9)
- # malli (7)
- # matrix (4)
- # pedestal (3)
- # podcasts-discuss (1)
- # portal (1)
- # re-frame (62)
- # reitit (2)
- # releases (1)
- # schema (3)
- # sql (14)
- # squint (3)
- # xtdb (6)
- # yamlscript (4)
When importing data, is there any downside to putting it all into one very large transaction? Is there a maximum transaction size?
Hey @UDQE4G9B2 there's no hardcoded maximum, but you will eventually OOM if you push it too far. Our usual advice for batching is 1000 docs/tx, anything higher doesn't really save on any per-tx overheads
It will also be much easier to monitor import progress given reasonably chunked batches
@UDQE4G9B2 as Jeremy says regarding 1k docs/tx, but also if you're using Kafka for either your tx-log or doc-store, it has a (configurable) message size limit
doc-store should be ok as it's one doc per message; the tx metadata will eventually get too large for one message
This is using Postgresql and RocksDB; no Kafka. Cool, then I’ll use 1000docs/tx. Thanks!