Fork me on GitHub
#datomic
<
2015-11-09
>
Ben Kamphaus14:11:34

@gurdas if you can share the schema it would help.

Ben Kamphaus14:11:43

mainly, I want to verify cardinality of ref attrs (am I ok to infer it from plural vs. singular naming convention? i.e. node/tree line/dataset = card one, line/nodes card many?

arohner18:11:48

For those of you running on AWS + dynamo, how frequently do you get “transactor not available”, as an intermittent error?

arohner18:11:32

oh, fun. It looks like if you get the dynamo ‘throughput exceeded’ error, the transactor kills itself and starts over

gurdas18:11:29

Appreciate the help! Let me know if a dump of the datomic db i'm working with would help as well and I can get that out to you

pesterhazy19:11:15

@arohner: the transactor restarting is actually not a bad reaction to seeing errors

pesterhazy19:11:53

though it probably won't help much if the issue is due to dynamo's throughput limit simple_smile

arohner19:11:56

it also took some poking around to find out that’s what happened

pesterhazy19:11:55

how did you find out? I find it hard to understand what the AMI does

pesterhazy19:11:24

there are logs on S3 but they only seem to be updated once a day

arohner19:11:45

my S3 logs were up to date

arohner19:11:55

not sure why mine are and yours aren't

arohner19:11:42

looks like it pushes logs when a transactor restarts

arohner19:11:53

I have the full logs from each transactor that died

pesterhazy19:11:39

it's possible I didn't look close enough

pesterhazy19:11:15

I've also run into the issue that the AMI restarts continuously (every minute or so)

pesterhazy19:11:41

probably some misconfiguration of the auto-scaling group

robert-stuttaford19:11:26

arohner: we went through quite some fun with this. you have to get your write prov, and memory-index-threshold, memory-index-max values tuned

robert-stuttaford19:11:59

if your m-i-* values are too high, you can slam storage with a BIG amount of writes in an otherwise sleepy system

robert-stuttaford19:11:02

you want small, frequent indexing jobs, so small m-i-threshold. ours is 32mb. @bkamphaus is a wizard at reading CloudWatch, so if you’ve Pro, you should totally spend an hour with him, and have him read your account’s entrails

robert-stuttaford19:11:58

we had transactor-not-available issues all over until we got this right, and we did have two instances where things had to restart

marshall19:11:17

@arohner: What size EC2 instance are you running the txor on?

arohner19:11:03

@robert-stuttaford: thanks. Not sure what my m-i-threshold is, I’ll check it out

arohner19:11:14

@marshall c3.large, but that’s probably overkill

marshall19:11:54

@arohner: what do you have Dynamo write throughput set to?

marshall19:11:16

i suspect that is the issue

marshall19:11:44

the transactor will need sufficient write throughput to handle both incoming transactions as well as background indexing and heartbeats

marshall19:11:21

That page indicates starting values for common sstems

marshall19:11:35

the lowest recommended DDB write is 75

marshall19:11:48

and that would be for a fairly small system (one supported by an m1.small)

robert-stuttaford20:11:34

@arohner: m-i-threshold is my laziness; i mean max-index-threshold as set out in your transactor.properties file when booting your transactor instances

arohner20:11:55

I understood simple_smile

robert-stuttaford20:11:17

our write throughput is 400 😐

thosmos20:11:56

kind of absurd I know, but I got an example app + transactor using postgresql running on a free heroku dyno: https://github.com/clojurous/shouter-datomic-heroku You can see it in action here: https://calm-castle-4835.herokuapp.com