Fork me on GitHub
#datomic
<
2017-03-10
>
dominicm17:03:13

What are the trade-offs to adding an index to an attribute? Is it just storage?

dominicm17:03:24

Thinking a string e.g. emails

stuartsierra17:03:17

Storage & indexing costs.

stuartsierra17:03:57

Generally we recommend starting with db/index=true, since it's easier to remove an index than add one later.

dominicm17:03:56

Oh really? Interesting!

phillipc18:03:52

In a project I'm currently involved in, I'm using datalog to pull a subset of data from a datomic database into an edn file. An issue arises when pulling anything typed bigint. It seems to render as type long in the edn file (no trailing N). The project errs when attempting to load the edn file as it expects a bigint but is getting a long. Is there anything in datalog that would allow me to force the bigint typing on export to the edn file?

alexandergunnarson19:03:59

Hey! So I just got this fun error: ("heartbeat failed")

2017-03-10 18:27:30.006 INFO  default    datomic.lifecycle - {:event :transactor/heartbeat-failed, :cause :timeout, :pid 7943, :tid 15}
2017-03-10 18:27:30.009 ERROR default    datomic.process - {:message "Critical failure, cannot continue: Heartbeat failed", :pid 7943, :tid 1722}
2017-03-10 18:27:30.010 INFO  default    datomic.process-monitor - {:tid 1722, :AlarmHeartbeatFailed {:lo 1, :hi 1, :sum 1, :count 1}, :MemoryIndexMB {:lo 1, :hi 1, :sum 1, :count 1}, :AvailableMB 428.0, :RemotePeers {:lo 1, :hi 1, :sum 1, :count 1}, :HeartbeatMsec {:lo 5000, :hi 9692, :sum 44694, :count 8}, :Alarm {:lo 1, :hi 1, :sum 1, :count 1}, :pid 7943, :event :metrics, :SelfDestruct {:lo 1, :hi 1, :sum 1, :count 1}, :MetricsReport {:lo 1, :hi 1, :sum 1, :count 1}}

alexandergunnarson19:03:23

It’s not immediately apparent to me what caused the heartbeat failure. How should I read the log?

marshall19:03:35

most likely storage unavalabl

alexandergunnarson19:03:37

Thanks for responding so quickly @marshall! I was thinking it might be a memory insufficiency, because the transactor is simply transacting to DynamoDB

alexandergunnarson19:03:29

But I suppose DynamoDB might be read/write throttling, or maybe some latency was introduced

marshall19:03:34

transactor on aws or local?

alexandergunnarson19:03:06

So I’m following bad practice and co-locating the transactor with the server JVM on the same AWS (EC2) instance.

marshall19:03:17

also, look around/just before that in the transactor log

marshall19:03:29

if ddb is throttling you’ll see that

alexandergunnarson19:03:29

Memory creeps down steadily but not steeply

alexandergunnarson19:03:02

What should I look for if it’s throttling? It’s just a bunch of heartbeats

marshall19:03:25

you’d see throttling notifications

marshall19:03:29

or storage backoff

alexandergunnarson19:03:36

Yeah, not seeing any notifications of that kind

marshall19:03:45

could also just be a transient network failure

marshall19:03:12

are you running HA?

alexandergunnarson19:03:29

No, for cost reasons

alexandergunnarson19:03:32

This is a small project

marshall19:03:38

yeah, so that is the case for HA

alexandergunnarson19:03:40

But times like this make me wish we had HA...

marshall19:03:47

if your primary goes down

marshall19:03:52

for somethign like a network hiccup

marshall19:03:54

HA takes over

marshall19:03:07

but you really only get the advantage if you’re running the pieces on separate instances

alexandergunnarson19:03:26

Right — which is more expensive 😕 Though I suppose not by much

alexandergunnarson19:03:42

It’s just myself and another person on the team and we have no revenue yet haha so we’re trying to minimize costs

alexandergunnarson19:03:13

Otherwise I’d say, pay the extra few dollars a month and save ourselves the headache because developer time is much more expensive

alexandergunnarson19:03:32

I’ll look around for some other feasible AWS options

alexandergunnarson19:03:08

Might have to buckle down and get two super-small AWS instances large enough for the two transactors (one active, one failover/backup) as well as the server instance

marshall19:03:21

what size instances are you running now?

alexandergunnarson19:03:34

Let me check — it’s the 4GB memory with 2 cores

alexandergunnarson19:03:14

We don’t have autoscaling or replication either so if that server goes down, I’m on duty 😅 Oh, to have money so permanently crossing fingers wouldn't be necessary

marshall19:03:43

yeah, you can get away with a smaller instance for the transactor, but you have to be careful about load

marshall19:03:08

well, worst case shoestring you could put the transactor up on a single instance ASG

marshall19:03:18

then if it goes down you have temporary outage while the ASG brings it back up

marshall19:03:28

it’s not HA failover, but at least it’s automatic

alexandergunnarson19:03:45

Here’s my transactor memory settings — pretty minimal

memory-index-threshold=32m
memory-index-max=128m
object-cache-max=64m

zane19:03:50

What kind of performance characteristics does datomic.api/since have?

alexandergunnarson19:03:23

ASG? Auto-scaling … G?

alexandergunnarson19:03:38

Yeah that’s true

alexandergunnarson19:03:48

Not HA, but a smart idea

marshall19:03:15

just make sure if you’re not using our provided CF/launch scripts that you terminate the instance when the transactor goes down

marshall19:03:21

that way the ASG replaces it

alexandergunnarson19:03:50

I used the CF/launch scripts at a previous startup when we had HA, which was nice, but we don’t have that luxury (really, necessity) now sadly

alexandergunnarson19:03:05

Thanks for the tip about termination!

marshall19:03:22

yep. and yeah, the CF/AMI we have will work that way too - just set the ASG size to 1

alexandergunnarson19:03:43

So I can use the CF/launch scripts to create an ASG for the transactor without having to use HA?

alexandergunnarson19:03:57

Oh I get what you’re saying now — you just answered that

alexandergunnarson19:03:17

Sounds good — thanks so much!

alexandergunnarson19:03:11

Do you think a t2.micro instance is big enough? We might swing HA if we had two t2.nanos...

marshall19:03:41

i think people have done it, but you definitely can’t push much data through ti

alexandergunnarson19:03:59

With the t2.micro it’s hard too?

alexandergunnarson19:03:18

I guess the bandwidth is really small

marshall19:03:39

1gb isn’t much memory

marshall19:03:50

you figure you need at least a few hundred mb for OS

alexandergunnarson19:03:52

That’s true, especially since that’s including the OS

marshall19:03:57

so you can’t even have a 1gb heap

alexandergunnarson19:03:11

Yeah so t2.nano is infeasible I suppose

alexandergunnarson19:03:41

I remember trying to get a transactor running while setting the max heap size to something like 384MB and it crashed right away

alexandergunnarson19:03:58

Hmm, well this gives me something to think about — thanks!

timgilbert22:03:40

I was never able to get a transactor up with less than 4GB (t2.medium), personally. It still relatively cheap though, $34 / mo according to http://www.ec2instances.info/