Fork me on GitHub
#datomic
<
2017-03-14
>
chrisblom14:03:26

What can I expect when not running datomic.api/gc-storage regularly?

chrisblom14:03:46

Will it degrade performance, fill up storage etc?

marshall15:03:42

@chrisblom it will result in accumulation of garbage segments in storage

marshall15:03:07

unlikely to affect performance substantially, although that may depend on the specific storage

marshall15:03:19

will take up storage space.

chrisblom15:03:26

@marshall thanks, we're using DynamoDB, does it affect performance there?

marshall15:03:30

i wouldn’t expect it to have much if any perf effect

marshall15:03:52

but why are you hesitant to run gcStorage?

chrisblom15:03:30

oh, i want to run it, but we didn't run it for a long time as our gc task was misconfigured

marshall15:03:45

ah. you should be fine - just go ahead and start running it again

chrisblom15:03:11

when I ran it manually, i got a AlarmGCStorageFailed, which kills the transactor

chrisblom15:03:02

should it just try with an older timestamp?

chrisblom15:03:54

btw, we're using a backup transactor, so a transactor going down is not a problem

marshall15:03:08

how far back did you try running it?

marshall15:03:38

did you have anything in the log? an exception possibly?

kirill.salykin15:03:02

Why not Datomic runs gc-storage on regular basis?

kirill.salykin15:03:10

why it should be scheduled outside?

marshall15:03:42

mostly likely you are getting throttled by dynamo

chrisblom15:03:39

@marshall i found the exception, it was indeed throttling by dynamodb

chrisblom15:03:43

14:19:39 2017-03-13 14:19:38.972 INFO default datomic.garbage - {:event :garbage/collect, :dbid "production-2017-01-09-c-cf03505f-7aac-4653-acab-913f4b3a086b", :older-than #inst "2017-01-13T10:31:35.781-00:00", :msec 1.3699999999999998E7, :phase :end, :threw java.util.concurrent.ExecutionException, :pid 9, :tid 1121}
14:19:39 2017-03-13 14:19:38.974 WARN default datomic.garbage - {:message "Cluster gc failed", :pid 9, :tid 1121}
14:19:39 java.util.concurrent.ExecutionException: com.amazonaws.services.dynamodbv2.model.ProvisionedThroughputExceededException: The level of configured provisioned throughput for the table was exceeded. Consider increasing your provisioning level with the UpdateTable API. (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: ProvisionedThroughputExceededException; Request ID: STKU1ATRLK677TKLOARC6BVP
14:19:39 Caused by: com.amazonaws.services.dynamodbv2.model.ProvisionedThroughputExceededException: The level of configured provisioned throughput for the table was exceeded. Consider increasing your provisioning level with the UpdateTable API. (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: ProvisionedThroughputExceededException; Request ID: STKU1ATRLK677TKLOARC6BVPBBVV4KQNSO5AEMVJF66Q9ASUAAJG)
14:19:39 2017-03-13 14:19:38.975 WARN default datomic.garbage - ... caused by ...
14:19:39 com.amazonaws.services.dynamodbv2.model.ProvisionedThroughputExceededException: The level of configured provisioned throughput for the table was exceeded. Consider increasing your provisioning level with the UpdateTable API. (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: ProvisionedThroughputExceededException; Request ID: STKU1ATRLK677TKLOARC6BVPBBVV4KQNSO5AEMVJF66Q9ASUAAJG) at com.ama

marshall15:03:44

you may want to turn down your gcStorage throughput settings

marshall15:03:50

and/or up ddb provisioning

marshall15:03:44

datomic.gcStoragePaceMsec

marshall15:03:53

as a command line argument when you launch the transactor

chrisblom15:03:51

thats for deleting db's right? i was using datomic.api/gc-storage for live databases

marshall16:03:02

yes, that region of the docs is about deleted databases, but the gcStoragePaceMsec applies to both