Hello 👋 I am designing an ETL written with Clojure that is intended to run in a Datomic cluster, it’ll pull entries from a MySQL db located in the same AWS account and push them in the Datomic db. My first goal is to process 9M entries as fast as possible, hopefully in less than 5hrs. The strategy I’d like to implement is the following: • pull a first batch of entries (with offset and limit) from the MySQL db • process those entries leveraging parallelism via the pmap function to push them into Datomic db • pull the next batch of entries • process that next batch • etc. until there is no more entries returned by the MySQL query In my first POC I discovered a few things: • sometimes Datomic returns errors such as :cognitect.anomalies/busy, Busy indexing • other times it returns errors such as :cognitect.anomalies/fault, :datomic.client-spi/exception java.lang.NullPointerException If I remove parallelism, changing pmap for map those issues won’t happen though the ETA is about 30hrs. Is it a bad idea to try to parallelize the ingestion? Please does anyone has an idea why this could happen? Any suggestion?

What's the usual way to get the datoms using a transaction id? I'm thinking of using range-tx