Hey @U09R86PA4, can you share (however you'd prefer) the flags used during your backup process?

there are no flags used

I just added backupPaceMsec and no change, still crashes

I’m beginning to suspect GC--does newer version use more memory?

• Yeah, you may have more memory pressure because because we increased some branch-level concurrency in the backup process. I imagine because you've got a large enough database you're going to be taking advantage of that concurrency. • The ddb timeouts are likely because of the new ddbRequestTimeout setting, which now defaults (IIRC) to 1s and previously was 1m. • Try increasing mem • If increasing mem doesn't solve it, set -Ddatomic.ddbRequestTimeout=10000

You might be getting DDB ReadThrottled if you're using provisioned capacity.

right, which was my initial suspect, but why was this not an issue on 6610?

by all appearances the network activity from 7075 seems to have decreased

vs 6610

I’m comparing jstats now from the two versions

https://docs.datomic.com/pro/changes.html#1.0.7010 • "New System Property: ddbRequestTimeout ..."

yeah, so you’re saying the old value was much higher?

(old, hidden value)

Yeah, old value was the SDK Default of 1 minute (If I could only share the things I've seen...)

I think it’s all memory pressure

running on 6610, jstat shows GC isn’t as active. :backup/segment :msec values remain very low

on 7075, GC is very active, and msec values degrade over time. I suspect from gc

Are they being skipped or copied. Since you're doing an incremental backup they may just be getting skipped.

(Not disagreeing, just an alternative hypothesis for what you're seeing)

overall network throughput from 6610 is higher according to CW metrics from the instance

they are getting skipped, but 7075 never got past that. 6610 just finished a 30 minute incremental in < 5 minutes

7075 just falls over in the middle

6610 invoked full gc 5 times, 7075 invoked it > 150 times

so yeah, I think it’s memory pressure

7075 uses more memory

For those following along at home, I was using a 2g heap with 6610; using a 4g heap with 7075 appears to be enough for it to complete.