I am doing a very large datomic import (over 14 million rows). I believe I’m doing all the ‘best practice’ things: 1) batch transactions 2) pipeline 3) initial schema w/o indexes 4) set transactor for import. I’m noticing however, that the transactions come in relatively quickly at the start (~ 4 sec for 100 transactions) and then gradually degrade as the import proceeds (now over 1 min for 100 transactions after 23,000 transactions. What are some other things to consider that might cause this? Transactor is running locally as dev and I’m using client api to a peer server.

@donmullen may i suggest using the peer library instead? what storage backend are you using? what are the threshold values for your transactor? higher = bigger indexing jobs

@robert-stuttaford I’m running locally against a dev disk storage. Would local-ddb be more efficient? Transactor settings memory-index-threshold=32m / memory-index-max=512m / object-cache-max=64m // running -Xms1g -Xmx1g -XX:+UseG1GC -XX:MaxGCPauseMillis=50

I will try switching to peer library.

I seem to be running into memory / GC issues. Likely some rookie clojure dev mistake in the code somewhere (only recently ramping back up in clojure).

datomic:mem will store on ram. 1Gb is enough for data + index's + datomic code?

@donmullen - I'm pretty sure you're holding on to the head of your data sequence. As more of the sequence gets realized, the GC can't free up any of the first elements because you still have a reference to the head (your data binding in your code). This can take some work to get right, and it's also a bit frustrating to debug because the feedback cycles are long. But the good news is that it's a fairly common problem in Clojure when dealing with large datasets, so there should be some good resources out there to learn more about it. Let me know if this doesn't make sense or if you're not sure where to go from here.

@jeff.terrell Thanks - was reaching the same conclusion.

@donmullen degradation on import is normal as the amount of data reaches indexing thresholds; indexing (done by the transactor) slows down the import process because it consumes cpu and io.

@favila Is there indexing happening even if no attributes have :db/index true?

yes, there are still :eavt :aevt and :vaet (for refs) indexes

:db/index true only controls the presence of :avet for a given attr

FYI: http://docs.datomic.com/capacity.html#data-imports

you can check your logs for backpressure alarms. that would at least tell you that the slowdown is because the transactor is applying backpressure

(transactor logs)

@favila - thanks - not seeing alarms - so I think the issue likely GC/mem in my code.

I don't see any obvious head-holding in your code

normal process monitoring should tell you which process is consuming cpu

something like jstat or jvisualvm can tell you what is happening in each process