To bulk load data, the fastest way that also avoid OOM is to create datoms yourself and load then with conn-from-datoms This will avoid the transaction logic, which is very expensive and not well optimized yet.

If you insist on transacting them, there are two things to note. If you want to transact all that data in a single transaction, your memory needs to be much much larger than the data as many copies of the data are used during transaction. You can always tune your JVM to use more memory, which you are not using obviously. The easiest is to set a bigger -Xmx

Another easy fix, is to transact the data in smaller batches, e.g. using partition function.

I've been playing with various parition sizes on the load, currently running with about 128. I just now partitioned the retractions, which let them succeede.

My issues using 'with-datoms' is that this is a puzzle involving about 100 or so different batches of data that need to be merged into a single "pile" at the end of it all

the transaction logic is not optimized at this moment, we will get to that after query engine rewrite is finished.

I have no issue with the speed, only the size.

Is there any sense to sort of "restarting" the connnection periodically to overcome the swells?

I had some better luck "partitioning" the retracts, it seems to enjoy that better

don’t think “restarting” helps. Another thing is that when you transacting bulk data, one should turn off caching, we can add that option

let me file an issue

yes, all of this stuff is historical from the 80's. once it gets to disk, it can be mostly forgotten

If you want to remove all data, you can use clear function, that remove all data and the schema.

It will be much faster.