Fork me on GitHub
Slack Archive

This page is not created by, affiliated with, or supported by Slack Technologies, Inc.

2017-11-15

Channels

#datomic
<
2017-11-15
>
andrethehunter03:11:50

Anyone else having issues testing with datomic? Running locally "datomic:... we get the error: 

datomic.api/create-database                 api.clj:   19
                                                        datomic.Peer.createDatabase               Peer.java:  117
                                                                                ...
                                                       datomic.peer/create-database                peer.clj:  764
                                                       datomic.peer/create-database                peer.clj:  772
                                                    datomic.peer/send-admin-request                peer.clj:  752
                                                 datomic.peer/send-admin-request/fn                peer.clj:  760
                               datomic.connector/create-transactor-hornet-connector           connector.clj:  320
                               datomic.connector/create-transactor-hornet-connector           connector.clj:  322
                                            datomic.connector/create-hornet-factory           connector.clj:  142
                                               datomic.connector/try-hornet-connect           connector.clj:  110
                                      datomic.artemis-client/create-session-factory      artemis_client.clj:  114
org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory  ServerLocatorImpl.java:  799
org.apache.activemq.artemis.api.core.ActiveMQNotConnectedException: AMQ119007: Cannot connect to server(s). Tried with all available servers.
    type: #object[org.apache.activemq.artemis.api.core.ActiveMQExceptionType$3 0xacdff3a "NOT_CONNECTED"]
                                        clojure.lang.ExceptionInfo: Error communicating with HOST 172.31.8.99 or ALT_HOST 52.64.171.90 on PORT 4334

axrs06:11:25

We found the problem. We had used a backup of our database that still had heartbeat information. Unfortunately the transact or had died and could no longer be resolved on the last used ip addresses. (A new transactor replaced the crashed one with new ips)

jaret16:11:21

Andre, I believe you logged a case last night and I replied, but can you confirm that the peer machine can communicate to the HOST and ALT HOST (set in transactor properties file)? http://docs.datomic.com/deployment.html#peers-fail-connect-txor

Desmond03:11:40

im trying to go through the day-of-datomic tutorial but i dont know where to run the repl from. when i run it from the root of the day-of-datomic repo the first require statement fails because theres no datomic on the classpath. and if i run it from the bin/repl where i installed the free datomic in the datomic getting started guide it complains that there is no datomic.repl on the classpath.

souenzzo04:11:03

(require '[datomic.api :as d]) @Desmond

Desmond05:11:53

@U2J4FRT2T that's the line that fails because datomic isn't on the classpath

Desmond05:11:34

ha! ok I guess I wasn't at the root of the repo after all. working now

Desmond05:11:45

thanks anyway for the quick response @U2J4FRT2T

Desmond03:11:03

anyone know where i should be running the repl from?

foobar11:11:50

Setting custom trustStore gives ssl errors trying to connect to datomic

Empperi12:11:32

I’m trying to optimize Datomic peer performance and going around with different JVM options, was wondering if there’s any “known to be good” values where I could start from?

Empperi12:11:41

to me it looks like +UseParallelGC seems to be better for performance with less deviation than +UseG1GC which was kinda surprising to me

Empperi12:11:15

just by switching that I get about 2x performance boost and about half the deviation

Empperi12:11:49

another thing I’ve noticed is that Datomic seems to enforce the amount of query threads to be equal to the amount of cores it sees

Empperi12:11:24

I wonder if that value can be tuned somehow? I do not see any properties in the Datomic documentation here http://docs.datomic.com/system-properties.html

mpenet15:11:07

Might be the setting for core.async or the agent thread pool

conan15:11:19

I'm having trouble connecting to a transactor running on AWS using the CloudFormation template. Does anyone know what value should I have for the host in my transactor.properties?

conan15:11:10

I'm trying to connect from a peer running in Elastic Beanstalk. It's using the datomic-aws-peer instance profile, which includes four dynamo permissions in its policies.

conan15:11:28

There are only INFO logs in s3 for the transactor, so I think it's running fine.

conan15:11:11

The transactor instance is running in a security group called datomic, which allows TCP ingress on port 4334.

jaret16:11:44

the host should be set to the externally reachable IP of the transactor machine. You’ll want to confirm you can reach that host IP from your peer machine.

jaret16:11:15

you can put the machine VM hostname in the host property and optionally use the internal/private hostname IP in the alt-host property to ensure that it uses both.

marshall16:11:09

If you’re using the provided create and start -template tools

marshall16:11:23

then the host and alt host get populated automatically by the startup script

marshall16:11:41

@U053032QC if you’re launching the stack ‘manually’ then you’ll need to do what Jaret mentioned

marshall16:11:54

If your transactor is up and running (can confirm with CW metrics and/or logs), then you only need to specify the storage URI in your peer

marshall16:11:05

it will lookup the transactor endpoint from storage

conan17:11:14

I used the scripts, and it just has localhost in there.

conan17:11:58

I don't really see how I can set it to the IP of the machine it's running on, as the transactor.properties file is used to create the CloudFormation stack that in turn spins up the machine, so there's no way to know the IP in advance.

conan17:11:35

I've read that the peer gets the URI from the storage, in which case what is the host property for?

marshall17:11:37

if you run the stack with the bin/datomic create-cf-stack command it will put in the correct address

marshall17:11:49

it won’t use the one listed in your local copy of the properties file

conan17:11:02

So the one in the file doesn't get used?

marshall17:11:16

not if you’re using the included build/launch tools

conan17:11:32

Oh ok, so maybe I'm barking up the wrong tree. Thanks!

marshall17:11:45

did you launch the transactor with bin/transactor create-cf-stack ?

marshall17:11:58

yeah, then it’s not that

conan17:11:26

the transactor is running fine. maybe it's a security problem, the docs said all the security groups were also set up by those tools but maybe there's something that isn't working

marshall17:11:35

you should launch your peer process with the aws role specified in the configuration

marshall17:11:33

the aws-peer-role

marshall17:11:41

that is listed in your properties file after you run ensure-transactor

conan17:11:42

that role just gives access to the storage though, nothing else

marshall17:11:46

that’s correct

marshall17:11:56

the peer reads storage

conan17:11:57

so i still can't connect to the transactor

marshall17:11:10

what error are you getting

conan17:11:51

ActiveMQNotConnectedException[errorType=NOT_CONNECTED message=AMQ119007: Cannot connect to server(s). Tried with all available servers.]

conan17:11:24

i think it makes sense to me, there is nothing to say that the peer should be able to connect to the transactor

marshall17:11:11

what security group is it using?

conan17:11:28

some ebs-generated one

conan17:11:33

the peer, that is

marshall17:11:46

you’ll need to allow ingress to the transactor security group from that one

marshall17:11:06

on your txor port

marshall17:11:08

(default 4334)

conan17:11:14

i figured that datomic-aws-peer role would provide access but i now realise that's just an assumption i made up

marshall17:11:06

the role is for storage access

marshall17:11:15

the system should also make a peer security group

conan17:11:19

yep, that's probably it

marshall17:11:20

but you’d have to be using it

conan17:11:14

thanks for your help, it would have taken me a while to find that role

conan10:11:00

btw i've been able to connect from EBS now, and if i allow ingress from the internet I can connect locally as well, so it's all working perfectly. thanks!

marshall14:11:21

glad to hear it

marshall16:11:48

You can specify the query pool thread count with the datomic.queryPool property. For example: -Ddatomic.queryPool=6

Empperi16:11:54

Ok, good to know, that's not in the documentation

marshall16:11:58

Our official recommendation is the G1GC. I can imagine varying peer workloads benefiting from other GC options, but since the peer is your application, that will definitely depending on your specific code and usage

marshall16:11:25

It’s a relatively new addition; I’ll see about adding it to the docs

Empperi16:11:40

Thanks 👍

marshall16:11:57

Default object cache is 1/2 of the heap Again, depending on what your workload looks like you may want to tweak that

Empperi16:11:21

Yeah, that I found already about

Empperi16:11:07

Our data sizes is at least currently relatively small but the structure is complex and so are the queries

Empperi16:11:08

We are building a knowledge graph solution on top of Datomic and trying to find out how well it works out

Empperi16:11:39

So very query intensive stuff

Quan Nguyen18:11:36

Had a random conceptual question. Is it possible to do atomic transactions that involve a read and a write? E.g. suppose I have multiple threads or peers trying to increment an int attribute on some entity ? All the examples I've seen with transactions are simply asserting some new value, but that doesn't take into account the current value. Thanks!

potetm18:11:27

I'm sure your example was just that, an example, but, to be clear, storing an incrementing number is probably not the best fit.

Quan Nguyen18:11:15

@potetm thanks for the tip!