got it thanks

@bkamphaus: is there any way to know how much memory an index will take up?

I'm seeing

Transaction error clojure.lang.ExceptionInfo: :db.error/transactor-unavailable Transactor not available {:db/error :db.error/transactor-unavailable}

it always recovers but this is a bit worrying. (This is using AWS, official AMIs with dynamo)

could this be related to GC pauses in the peer?

yes, that could trigger it

same way as a broken network

the GC pauses we see are only 5 seconds, though -- would that be sufficient?

not that 5 second GC pauses aren't indicative of a problem in our code

is there a timeout parameter of some sorts?

@pesterhazy also large transactions on the peer or indexing not keeping up on transactor. gc pause of just 5 seconds could possibly impact it if times poorly or in quick succession.

this peer is processing hardly any transactions

Timeout tolerance can be set by upping transactor heartbeat (Datomic level), or on peer changing datomic.peerConnectionTTLMsec to be higher (HornetQ level)

@pesterhazy: if the peer isn’t processing many transactions, and the transactor (verified from metrics or logs) is heartbeating fine and not reporting alarms, peer GC is most likely culprit. If you’re not using non-default JVM GC settings on peer app, you could adopt some similar to those on transactor if goal is to avoid pauses. Or tolerate the GC by upping one or both of the settings mentioned above.

like stu halloway says in his debugging talk, the culprit is always the GC

looking at the transactor metrics, the heartbeating looks fine

I guess there's no way around finding where those GC pauses are coming from

thanks for your help!