Fork me on GitHub
#datomic
<
2016-08-09
>
podviaznikov00:08:33

hit problem with aws deployment. I have micro ec2 instance with clojure app that uses datomic. I’m getting following error:

got an error java.lang.IllegalArgumentException: :db.error/not-enough-memory (datomic.objectCacheMax + datomic.memoryIndexMax) exceeds 75% of JVM RAM
Is this problem with machine where my app runs or machine where transactor runs?

marshall00:08:38

Are you running both peer and transactor on the same instance?

bvulpes00:08:42

podviaznikov: naively provisioned t2.micro will struggle to run both a transactor and a clojure app for precisely reasons of RAM allocation. using swap to increase RAM can help.

marshall00:08:55

If so, you'll definitely want to run them on separate instances. Regardless, that error indicates that the system you're running on has too little memory to run with your configured settings. You can try reducing the object cache size in your object cache max http://docs.datomic.com/capacity.html#peer-memory

podviaznikov00:08:10

sorry, I have two instances. transactor on m3.medium and my app on t2.micro. I’m getting error above in my clojure app. What I’m not sure about whether problem with datomic instance or my app instance

podviaznikov00:08:32

thanks for the System.setProperty("datomic.objectCacheMax", "256m"); tip. Trying now

marshall00:08:36

And @bvulpes is correct. A micro is pretty small for a datomic app, although not impossible

marshall00:08:53

Specifically regarding the error you saw, the sum of object cache max and memory index max (set in transactor properties file) cannot exceed 75% of JVM heap

bvulpes00:08:47

bezos' artificial ram constraints are easily gotten around with an SSD drive and swap space.

podviaznikov00:08:48

yeah, I’m assuming that is from my app. I was confused because I google that error and some people were complaining about transactor. So I wanted to double check

podviaznikov00:08:44

I’d like to change instance type, but now lost how to do that. I’ve launched instance using amazon container service and there is no easy way to change instance type. Trying to figure that out

marshall00:08:52

It's affected by transactor memory settings because the peer will always use the same memory index values as the transactor

marshall00:08:40

@bvulpes clever. I like it:)

bvulpes00:08:38

heh i can't take much credit for it but ty

marshall00:08:20

Not sure if the JVM will let you provision a larger than ram heap or not. I'll have to try it tomorrow

marshall00:08:05

I would worry about GC doing that though

podviaznikov00:08:32

It would be good if “java.lang.IllegalArgumentException: :db.error/not-enough-memory (datomic.objectCacheMax + datomic.memoryIndexMax) exceeds 75% of JVM RAM” error included current JVM RAM limit. Hard to tell now what is missing. I changed instance type to t2.small with 2GB of RAM and my single docker container has limit of 1.8GB. That should be enough right to connect to empty datomic database?

bvulpes00:08:34

podviaznikov: usually one provides heap size explicitly when booting a jar

bvulpes00:08:38

how are you booting this app?

bvulpes00:08:28

yrva << rot13'd guess

podviaznikov00:08:34

yeah, I think I’m missing that parameter. My last limes of the Dockerfile:

RUN mv "$(lein uberjar | sed -n 's/^Created \(.*standalone\.jar\)/\1/p')" api.jar
EXPOSE 3000
CMD ["java", "-Dfile.encoding=UTF-8", "-jar", "api.jar”]

bvulpes00:08:57

yeah xmx and xms are the flags i think you want

bvulpes00:08:27

"-Xmx 1g" for instance

bvulpes00:08:36

(re: rot13, i was wrong :P)

podviaznikov00:08:01

thanks! trying

bvulpes00:08:21

podviaznikov: the flag is "-Xmx1g"

podviaznikov00:08:39

yeah, I did added it without space

marshall01:08:31

If you have 1.8gb, I'd probably try to get 1.5g heap

marshall01:08:05

I.e. -Xmx1500M

podviaznikov01:08:01

I don’t have that error anymore. but now need problem:

Aug 09, 2016 1:01:46 AM org.hornetq.core.client.impl.ClientConsumerImpl$Runner run
ERROR: HQ214000: Failed to call onMessage
java.io.EOFException
	at org.fressian.impl.RawInput.readRawByte(RawInput.java:40)
	at org.fressian.FressianReader.readNextCode(FressianReader.java:927)
	at org.fressian.FressianReader.readObject(FressianReader.java:274)
	at datomic.hornet$create_deserializer$fn__6615.invoke(hornet.clj:353)
	at datomic.hornet$create_rpc_client$fn__6648.invoke(hornet.clj:412)
	at datomic.hornet$set_handler$reify__6602.onMessage(hornet.clj:288)
	at org.hornetq.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:1116)
	at org.hornetq.core.client.impl.ClientConsumerImpl.access$500(ClientConsumerImpl.java:56)
	at org.hornetq.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:1251)
	at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:104)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Happens when I call (d/connect data/uri)

marshall01:08:23

Does your peer instance have correct role etc to reach transactor?

podviaznikov01:08:55

I can’t reach transactor from laptop in the shell. I enabled ingress for all addresses. I assume I don’t need to do anything specific to my peer instance in that case, right?

marshall01:08:09

If you're on aws you'll need to setup aim roles

marshall01:08:47

Perhaps not of you've enabled access from anywhere. Best way to

marshall01:08:08

Check is to try accessing transactor instance from peer instance

marshall01:08:02

You'll want to make sure you can reach the transactor from the peer using either host or alt-host, as specified in transactor properties file

podviaznikov01:08:47

last link suggest checking out transactor logs. I didn’t configure saving logs initially. Is there is tutorial how to update configuration?

marshall01:08:08

Transactor logs default to log directory in datomic distribution. http://docs.datomic.com/configuring-logging.html Peer logs need to be enabled

podviaznikov01:08:15

right, but now it’s unclear how to ssh into ec2 instance with transactor. KeyPair wasn’t added when instance was first created. Is there a way to specify keypair when created cloudformation template for datomic transactor?

marshall01:08:58

Not if you're using the provided AMI. You'd need to set up log rotation. Were you able to test host and alt-host from the peer machine?

marshall01:08:55

If neither of those resolve to the transactor machine that is the issue.

podviaznikov01:08:39

how do I test test host and alt-host from the peer machine? It wasn’t clear to me how to do that based on http://docs.datomic.com/deployment.html#peer-fails-to-connect

marshall01:08:42

If you can ssh to the peer machine, try pinging the host and alt-host from it

marshall01:08:04

Make sure at least one of those resolves to the transactor machine

marshall01:08:31

If peer is in a container you should test from within the container

podviaznikov01:08:48

and by host you mean public DNS for ec instance? Like “http://ec2-54-152-44-27.compute-1.amazonaws.com”?

podviaznikov01:08:52

what is alt-host?

marshall01:08:12

No, host and alt-host are specified in the transactor properties file

marshall01:08:42

They are the values the transactor writes to storage that peer uses to locate and connect to live transaxtor

podviaznikov01:08:45

those on top of the file

protocol=ddb
host=localhost
port=4334
?

marshall01:08:57

Are you running in a vpc? If so, are both transactor and peer in the same vpn?

marshall01:08:20

Right, what is the alt-host value?

podviaznikov01:08:47

there is no alt-host value in the ensured-transactor.properties file

marshall02:08:46

You might want to review http://docs.datomic.com/storage.html#provisioning-dynamo The host should be specified by the scripts, but there are manual configuration directions below if necessary

podviaznikov02:08:05

I followed today this video: https://www.youtube.com/watch?v=wG5grJP3jKY Is this still up to date resource how to setup datomic in aws?

podviaznikov02:08:49

also you are correct, my app and datomic are in different vpss

podviaznikov02:08:57

I assume that is the main issue now

marshall02:08:27

The video may not be up to date. I'd highly recommend following the steps in the docs here http://docs.datomic.com/storage.html#provisioning-dynamo They're definitely the most up to date

podviaznikov02:08:10

I see now that docs mention setting up host

host=<FINAL-HOST-NAME>

Chris O’Donnell02:08:10

FWIW, I had no trouble setting up dynamo with the linked instructions a few weeks ago

Chris O’Donnell02:08:45

so I don't think anything is out of date

marshall02:08:21

@podviaznikov: the ensure scripts should handle all of that. Did you run ensure-transactor as described?

marshall02:08:09

I just reviewed the video you linked. I believe it is up to date. You'll probably want to resolve the vpc thing

marshall02:08:19

Also, if you did use the provided cloud formation template your transactor should be rotating logs to s3 of you want to check them for the values of host and alt-host. They should be printed to the logs on syartup

podviaznikov02:08:30

yeah, but video doesn’t mention changing “host” property, right? Or I didn’t pay attention. Also I don’t think I’ll have s3 logs. I have # aws-s3-log-bucket-id= commented it out. It was like that by default

marshall02:08:45

Ah. Yeah, you'll need to uncomment that to get logs. Correct, you don't need to manually specify host if you use the provided scripts

marshall03:08:13

The test he shows in the video of connecting from his laptop to the local transactor then to the remote transactor is a good one. It helps you narrow down the issue to the peer vs the transactor config

marshall03:08:52

If you ran that test successfully, then it's most likely the vpc issue and/or IAM config issue for the peer instance. The ensure scripts should have created a datomic peer role. You need to grant it to the peer instance

podviaznikov03:08:01

yeah, I run that test both from shell and running my clojure app locally. I was able to connect to datomic on aws. I already fixed issue with datomic peer role and now fixing issue with VPC. Hopefully that will work

marshall03:08:29

Sounds like you're on the right track then. If you can connect from your local machine with your app it's definitely an issue with the peer instance. Sorry for the meandering to get there :)

bvulpes05:08:54

fwiw i vastly prefer 'manually' provisioning datomic into my clusters.

bvulpes05:08:59

podviaznikov: ^^

bvulpes05:08:15

but i also don't use spacemacs

mrmcc308:08:57

:address/city-id -1000001 should that be :address/city-id #db/id[:db.part/user -1000001]

tengstrand09:08:46

Now it works. Needed to change ":person/address-id -1000002” also.

mrmcc309:08:13

:thumbsup:

yonatanel11:08:18

Is there a single page infographic for "managers", emphasizing datomic advantages and presenting it as less scary, or comparing to sql for instance?

yonatanel13:08:51

Thanks, stuart. I saw it but I guess I need something a little technical and more schematic.

Lambda/Sierra14:08:42

(Both are links from http://datomic.com homepage)

yonatanel14:08:46

Do I need to call shutdown when developing in the REPL? Currently I'm using Component and just sets :connection to nil on Lifecycle/stop.

bostonaholic14:08:09

I’ve always just used (read-string (slurp (io/resource f)))

marshall14:08:03

http://docs.datomic.com/javadoc/datomic/Util.html#readAll-java.io.Reader- readAll will read multiple items (i.e. multiple transactions) from an edn file and return a list

marshall14:08:37

the Day of Datomic repo uses it to get multiple transactions out of a file

bostonaholic15:08:58

ah ok, I guess usually what I’m reading in (a small schema) is one tx

sdegutis16:08:52

Do medium-complexity queries basically have the same performance profile as starting with an entity and navigating it via entity relationship psuedo-keys (such as :foo/bar and :bar/_quux, etc.)?

podviaznikov17:08:31

Still having problems with deploying talking from clojure app to new datomic on aws. This time both instances are in the same VPC and I’n getting error

clojure.lang.ExceptionInfo: Transactor request timed out {:db/error :peer/request-timed-out, :request :start-database, :result #object[java.lang.Object 0x697bf0e8 "java.lang.Object@697bf0e8”]}
. Also side question. What is the recommended way to deal with situation where I want to change e.x. ensured-transactor.properties. Should I regenerate cf.json and then update cf stack with it?

bvulpes19:08:45

podviaznikov: can you telnet to the transactor ip/port from your app instance?

podviaznikov19:08:49

telnet  4334
Trying 172.31.23.102...
Connected to .
Escape character is '^]'.
Connection closed by foreign host.

podviaznikov19:08:09

is this correct? port 4334, right?

jgdavey20:08:04

@podviaznikov: You may be missing an entry in your transactor’s security group setting.

podviaznikov20:08:55

I have automatically created sg for datomic instance:

22	tcp	0.0.0.0/0	✔
4334	tcp	0.0.0.0/0, sg-d34eb2a9	✔

podviaznikov20:08:09

something is missing there?

jgdavey20:08:31

Nope that seems legit

colindresj20:08:33

Just getting started with Datomic and trying to wrap my head around limiting access to particular data when querying. As a hypothetical, given a person entity and a company entity, how would I get the full collection of company members (person entity ref) only if the given person entity is contained within that collection of company members, otherwise return an empty collection?

podviaznikov21:08:15

I see this line in the docs: 'The transactor writes the value of the host transactor property in storage’. How can I see what are the host and alt-host values? are they somewhere in the dynamo db now? I see dynamo db key pod-coord, is it the place?

bvulpes21:08:28

podviaznikov: i'm going to gently suggest doing your own devops instead of wrestling the cloudformation. this'd be trivial for you to run down if you had access to the transactor.properties on disk.

atroche22:08:16

@colindresj: how about https://gist.github.com/atroche/a8e731202cdc01d7e0bbe0c4102704b9 ? i’m making some assumptions about your data model, but hopefully that helps.

atroche22:08:44

sdegutis: i’d say it depends on factors like a) how many entities are you doing the navigation on? b) how much of that data is cached on the peer already?

atroche22:08:28

do you have a specific example you’re wondering about?

atroche22:08:55

sdegutis: i’ve been running datomic in production for ~9 months on https://www.bookwell.com.au/ and my approach has been to “make it nice, then make it fast”. and it’s super rare that i have to do the second step 🙂

podviaznikov23:08:33

@bvulpes: setting up infrastructure manually sounds like a good idea. But I’m not sure I’d be able to do everything correctly since I don’t know what is the problem now. Datomic error messages and documentation for troubleshooting is not excellent for sure

bvulpes23:08:04

hey now buddy nobody said "manually"

podviaznikov23:08:45

how to do that without cloudformation?

bvulpes23:08:04

podviaznikov: ^^