Hi all. Recently experienced a Datomic transactor OOM during a large index merge: the JVM had a 10 GB heap, while Datomic’s object cache was 4 GB, memory index 1 GB, and embedded ActiveMQ Artemis appeared to default global-max-size to roughly 50% of -Xmx. GC pressure escalated until the transactor self-terminated. We have since moved to 32 GB RAM and are considering an explicitly bounded Artemis setting such as -Dbrokerconfig.globalMaxSize=2147483648, alongside a larger heap and reduced Datomic cache limits.
Does anyone have experience tuning embedded Artemis in Datomic, and can confirm whether this property is supported and applied correctly in Datomic 1.0.7622, plus any risks or recommended sizing guidance? Relevant Artemis docs: https://artemis.apache.org/components/artemis/documentation/latest/paging.html and https://artemis.apache.org/components/artemis/documentation/latest/configuration-index.html.
Hey Mike! I'd be happy to look at what specifically happened by gathering logs, potentially a heap dump etc. Datomic support is free with no SLA response time and we're interested in helping folks so we can uncover any issues with Datomic. If you would like to log a ticket we can start requesting info to give you the best advice and understanding of the OOM. you can e-mail <mailto:support@datomic.com|support@datomic.com> or log a ticket from https://support.cognitect.com/hc/en-us/requests/new I would want Transactor Logs from the active and standby transactor at the time of the OOM and thehttps://docs.datomic.com/clojure/index.html#datomic.api/db-stats from your DB along with any information you have about why there was a large index job. Did you turn on indexing on a new attribute? if so, which one and what is that attributes count on the overall db? I am highly skeptical that Artemis played any role in the OOM so can't advise you to change those settings. I also believe capping the size here would not have prevented the OOM.
Will do. Thanks!