hit problem with aws deployment. I have micro ec2 instance with clojure app that uses datomic. I’m getting following error:

got an error java.lang.IllegalArgumentException: :db.error/not-enough-memory (datomic.objectCacheMax + datomic.memoryIndexMax) exceeds 75% of JVM RAM

Are you running both peer and transactor on the same instance?

podviaznikov: naively provisioned t2.micro will struggle to run both a transactor and a clojure app for precisely reasons of RAM allocation. using swap to increase RAM can help.

If so, you'll definitely want to run them on separate instances. Regardless, that error indicates that the system you're running on has too little memory to run with your configured settings. You can try reducing the object cache size in your object cache max http://docs.datomic.com/capacity.html#peer-memory

sorry, I have two instances. transactor on m3.medium and my app on t2.micro. I’m getting error above in my clojure app. What I’m not sure about whether problem with datomic instance or my app instance

thanks for the System.setProperty("datomic.objectCacheMax", "256m"); tip. Trying now

And @bvulpes is correct. A micro is pretty small for a datomic app, although not impossible

Specifically regarding the error you saw, the sum of object cache max and memory index max (set in transactor properties file) cannot exceed 75% of JVM heap

bezos' artificial ram constraints are easily gotten around with an SSD drive and swap space.

yeah, I’m assuming that is from my app. I was confused because I google that error and some people were complaining about transactor. So I wanted to double check

I’d like to change instance type, but now lost how to do that. I’ve launched instance using amazon container service and there is no easy way to change instance type. Trying to figure that out

It's affected by transactor memory settings because the peer will always use the same memory index values as the transactor

@bvulpes clever. I like it:)

heh i can't take much credit for it but ty

Not sure if the JVM will let you provision a larger than ram heap or not. I'll have to try it tomorrow

I would worry about GC doing that though

It would be good if “java.lang.IllegalArgumentException: :db.error/not-enough-memory (datomic.objectCacheMax + datomic.memoryIndexMax) exceeds 75% of JVM RAM” error included current JVM RAM limit. Hard to tell now what is missing. I changed instance type to t2.small with 2GB of RAM and my single docker container has limit of 1.8GB. That should be enough right to connect to empty datomic database?

podviaznikov: usually one provides heap size explicitly when booting a jar

how are you booting this app?

yrva << rot13'd guess

yeah, I think I’m missing that parameter. My last limes of the Dockerfile:

RUN mv "$(lein uberjar | sed -n 's/^Created \(.*standalone\.jar\)/\1/p')" api.jar
EXPOSE 3000
CMD ["java", "-Dfile.encoding=UTF-8", "-jar", "api.jar”]

yeah xmx and xms are the flags i think you want

"-Xmx 1g" for instance

(re: rot13, i was wrong :P)

thanks! trying

er

no space

podviaznikov: the flag is "-Xmx1g"

yeah, I did added it without space

If you have 1.8gb, I'd probably try to get 1.5g heap

I.e. -Xmx1500M

I don’t have that error anymore. but now need problem:

Aug 09, 2016 1:01:46 AM org.hornetq.core.client.impl.ClientConsumerImpl$Runner run
ERROR: HQ214000: Failed to call onMessage
java.io.EOFException
	at org.fressian.impl.RawInput.readRawByte(RawInput.java:40)
	at org.fressian.FressianReader.readNextCode(FressianReader.java:927)
	at org.fressian.FressianReader.readObject(FressianReader.java:274)
	at datomic.hornet$create_deserializer$fn__6615.invoke(hornet.clj:353)
	at datomic.hornet$create_rpc_client$fn__6648.invoke(hornet.clj:412)
	at datomic.hornet$set_handler$reify__6602.onMessage(hornet.clj:288)
	at org.hornetq.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:1116)
	at org.hornetq.core.client.impl.ClientConsumerImpl.access$500(ClientConsumerImpl.java:56)
	at org.hornetq.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:1251)
	at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:104)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

Does your peer instance have correct role etc to reach transactor?

I can’t reach transactor from laptop in the shell. I enabled ingress for all addresses. I assume I don’t need to do anything specific to my peer instance in that case, right?

If you're on aws you'll need to setup aim roles

http://docs.datomic.com/aws.html

Perhaps not of you've enabled access from anywhere. Best way to

Check is to try accessing transactor instance from peer instance

http://docs.datomic.com/deployment.html#peer-fails-to-connect

You'll want to make sure you can reach the transactor from the peer using either host or alt-host, as specified in transactor properties file

last link suggest checking out transactor logs. I didn’t configure saving logs initially. Is there is tutorial how to update configuration?

Transactor logs default to log directory in datomic distribution. http://docs.datomic.com/configuring-logging.html Peer logs need to be enabled

right, but now it’s unclear how to ssh into ec2 instance with transactor. KeyPair wasn’t added when instance was first created. Is there a way to specify keypair when created cloudformation template for datomic transactor?

Not if you're using the provided AMI. You'd need to set up log rotation. Were you able to test host and alt-host from the peer machine?

If neither of those resolve to the transactor machine that is the issue.

how do I test test host and alt-host from the peer machine? It wasn’t clear to me how to do that based on http://docs.datomic.com/deployment.html#peer-fails-to-connect

If you can ssh to the peer machine, try pinging the host and alt-host from it

Make sure at least one of those resolves to the transactor machine

If peer is in a container you should test from within the container

and by host you mean public DNS for ec instance? Like “http://ec2-54-152-44-27.compute-1.amazonaws.com”?

what is alt-host?

No, host and alt-host are specified in the transactor properties file

They are the values the transactor writes to storage that peer uses to locate and connect to live transaxtor

those on top of the file

protocol=ddb
host=localhost
port=4334

Are you running in a vpc? If so, are both transactor and peer in the same vpn?

Right, what is the alt-host value?

there is no alt-host value in the ensured-transactor.properties file

You might want to review http://docs.datomic.com/storage.html#provisioning-dynamo The host should be specified by the scripts, but there are manual configuration directions below if necessary

I followed today this video: https://www.youtube.com/watch?v=wG5grJP3jKY Is this still up to date resource how to setup datomic in aws?

also you are correct, my app and datomic are in different vpss

I assume that is the main issue now

The video may not be up to date. I'd highly recommend following the steps in the docs here http://docs.datomic.com/storage.html#provisioning-dynamo They're definitely the most up to date

oh, I see

I see now that docs mention setting up host

host=<FINAL-HOST-NAME>

FWIW, I had no trouble setting up dynamo with the linked instructions a few weeks ago

so I don't think anything is out of date

@podviaznikov: the ensure scripts should handle all of that. Did you run ensure-transactor as described?

I just reviewed the video you linked. I believe it is up to date. You'll probably want to resolve the vpc thing

Also, if you did use the provided cloud formation template your transactor should be rotating logs to s3 of you want to check them for the values of host and alt-host. They should be printed to the logs on syartup

Startup

yeah, but video doesn’t mention changing “host” property, right? Or I didn’t pay attention. Also I don’t think I’ll have s3 logs. I have # aws-s3-log-bucket-id= commented it out. It was like that by default

Ah. Yeah, you'll need to uncomment that to get logs. Correct, you don't need to manually specify host if you use the provided scripts

The test he shows in the video of connecting from his laptop to the local transactor then to the remote transactor is a good one. It helps you narrow down the issue to the peer vs the transactor config

If you ran that test successfully, then it's most likely the vpc issue and/or IAM config issue for the peer instance. The ensure scripts should have created a datomic peer role. You need to grant it to the peer instance

yeah, I run that test both from shell and running my clojure app locally. I was able to connect to datomic on aws. I already fixed issue with datomic peer role and now fixing issue with VPC. Hopefully that will work

Sounds like you're on the right track then. If you can connect from your local machine with your app it's definitely an issue with the peer instance. Sorry for the meandering to get there :)

fwiw i vastly prefer 'manually' provisioning datomic into my clusters.

podviaznikov: ^^

but i also don't use spacemacs

:address/city-id -1000001 should that be :address/city-id #db/id[:db.part/user -1000001]

Now it works. Needed to change ":person/address-id -1000002” also.

Thanks!

:thumbsup:

Is there a single page infographic for "managers", emphasizing datomic advantages and presenting it as less scary, or comparing to sql for instance?

@yonatanel: maybe http://www.datomic.com/benefits.html

Thanks, stuart. I saw it but I guess I need something a little technical and more schematic.

Then http://www.datomic.com/rationale.html perhaps

(Both are links from http://datomic.com homepage)

Do I need to call shutdown when developing in the REPL? Currently I'm using Component and just sets :connection to nil on Lifecycle/stop.

no

what is the benefit of using datomic.Util/readAll https://github.com/Datomic/day-of-datomic/blob/053b3bd983d165b8fa7c0c039712fb1cb75eddf3/src/datomic/samples/io.clj#L18

I’ve always just used (read-string (slurp (io/resource f)))

http://docs.datomic.com/javadoc/datomic/Util.html#readAll-java.io.Reader- readAll will read multiple items (i.e. multiple transactions) from an edn file and return a list

the Day of Datomic repo uses it to get multiple transactions out of a file

ah ok, I guess usually what I’m reading in (a small schema) is one tx

Do medium-complexity queries basically have the same performance profile as starting with an entity and navigating it via entity relationship psuedo-keys (such as :foo/bar and :bar/_quux, etc.)?

Still having problems with deploying talking from clojure app to new datomic on aws. This time both instances are in the same VPC and I’n getting error

clojure.lang.ExceptionInfo: Transactor request timed out {:db/error :peer/request-timed-out, :request :start-database, :result #object[java.lang.Object 0x697bf0e8 "java.lang.Object@697bf0e8”]}

podviaznikov: can you telnet to the transactor ip/port from your app instance?

telnet  4334
Trying 172.31.23.102...
Connected to .
Escape character is '^]'.
Connection closed by foreign host.

is this correct? port 4334, right?

@podviaznikov: You may be missing an entry in your transactor’s security group setting.

I have automatically created sg for datomic instance:

22	tcp	0.0.0.0/0	✔
4334	tcp	0.0.0.0/0, sg-d34eb2a9	✔

something is missing there?

Nope that seems legit

Just getting started with Datomic and trying to wrap my head around limiting access to particular data when querying. As a hypothetical, given a person entity and a company entity, how would I get the full collection of company members (person entity ref) only if the given person entity is contained within that collection of company members, otherwise return an empty collection?

I see this line in the docs: 'The transactor writes the value of the host transactor property in storage’. How can I see what are the host and alt-host values? are they somewhere in the dynamo db now? I see dynamo db key pod-coord, is it the place?

podviaznikov: i'm going to gently suggest doing your own devops instead of wrestling the cloudformation. this'd be trivial for you to run down if you had access to the transactor.properties on disk.

@colindresj: how about https://gist.github.com/atroche/a8e731202cdc01d7e0bbe0c4102704b9 ? i’m making some assumptions about your data model, but hopefully that helps.

sdegutis: i’d say it depends on factors like a) how many entities are you doing the navigation on? b) how much of that data is cached on the peer already?

do you have a specific example you’re wondering about?

sdegutis: i’ve been running datomic in production for ~9 months on https://www.bookwell.com.au/ and my approach has been to “make it nice, then make it fast”. and it’s super rare that i have to do the second step 🙂

@bvulpes: setting up infrastructure manually sounds like a good idea. But I’m not sure I’d be able to do everything correctly since I don’t know what is the problem now. Datomic error messages and documentation for troubleshooting is not excellent for sure

hey now buddy nobody said "manually"

how to do that without cloudformation?

i use https://aws.amazon.com/sdk-for-java/

podviaznikov: ^^