Fork me on GitHub
#datomic
<
2019-12-03
>
Brian Abbott14:12:24

Is it possible to run Datomic Cloud on Fargate?

ghadi15:12:53

You can connect fargate clients to datomic cloud though, @briancabbott

Brian Abbott15:12:24

Sorry, that is what I mean. Is there somewhere that I could find some documentation on how to do that?

Brian Abbott15:12:53

Does anyone here on this channel have experience doing it?

ghadi15:12:12

Yep, there’s a few things to take care of: right subnet && right IAM policy on the fargate task role

Oleh K.08:12:37

when I create fargate service I need to create subnets for it. But in order to connect to datomic a service must be in the datomic subnet (as the documentation says). If I create a service and put in it datomic subnets then the service fails to even start. Can you give some insights where to look for the problem?

ghadi09:12:12

Confirm that the datomic subnets and the fargate subnets are mutually routable

ghadi09:12:38

And confirm that the “nodes” security group is augmented to include ingress from your new subnets

ghadi09:12:18

Are you doing peering @U5JRN2PU0 or same vpc?

Oleh K.09:12:06

I wasn't able to connect datomic with fargate in the same VPC, so now I'm trying peering

Oleh K.09:12:52

what do you mean by "Confirm that the datomic subnets and the fargate subnets are mutually routable" ?

ghadi09:12:34

Can they send packets to each other?

ghadi09:12:58

Peering works too, but there are different steps (see the documentation)

Oleh K.11:12:23

I've managed to do connecting within the same VPC from different subnets, thanks you!

ghadi12:12:16

nice -- what did you have to do @U5JRN2PU0?

ghadi12:12:26

just for the other readers that might be watching this thread

Oleh K.12:12:30

I just allowed ingress traffic in <system>*-nodes security group from my services subnets' cidrs

ghadi15:12:53

Other than that the code is the same in the jvm

bartuka15:12:40

hi, I have some questions about datomic analytics.. what is the "active workers" that we see at the presto server gui ? I have 3 instances for my query-group but I still get 1 active worker and 0 worker parallelism

marshall16:12:56

The presto server itself runs on your access gateway instance

marshall16:12:24

it will use parallelism on that instance, but analytics support doesnt currently use multiple presto workers

bartuka16:12:12

ahn, cool. I am still trying to proper configure the query-group for my analytics needs. i) I noticed that on the cloud watch dashboards no action was happening on the query-group[I run the x-ray table from metabase on a 100M datoms database] and the bastion server had 100% cpu and this single worker node was the bastion server indeed.

marshall16:12:18

you can choose a larger instance type for the access gateway instance

bartuka16:12:04

but if this is the case, I dont understand the purpose of the query-group itself. I thought those instances were performing the hard-work.

marshall16:12:04

some of the work, yes (the datomic DB work) but some of the work (the SQL processing) happens on the gateway instance

marshall16:12:25

I believe there are 3 or 4 choices

bartuka16:12:26

however, I found an error on the sync -q <query-group-name> config and fixed it [it was not recognizing the parameter and was taking the system name instead]. But now, the S3 bucket for analytics/ has two folders, one for my system-name and another with query-group-name. Is it right?

markbastian16:12:34

I have a very minor feature request for the datomic team. When you issue a push request the response contains the deploy command (e.g. clojure -A:ion-dev '{:op :deploy, :group elided, :rev \"0876503319a40bafffc8525f0597b1355b94b587\"}'). The rev entry contains escaped quotes so I always have to paste this into a shell and then cursor over to the slashes and remove them. Any chance we can get a version in a future release that doesn't include the slashes? The status-command output does not require doing the above.

❤️ 12
marshall16:12:31

@iagwanderson seems fine - what was the thing you changed in the script?

bartuka16:12:42

the problem was not in the script, I think I was calling it using shinstead of bash.

bartuka16:12:25

but how do I tell the bastion to use the folder of the query-group-name? Or I shouldn't?

bartuka16:12:46

I'm poking around and I deleted the folder system-name and everything stop working 😃 Presto complains that no catalog was found. It seems it was always looking at the system-namefolder

marshall16:12:05

@iagwanderson you need to set the AnalyticsEndpoint in the primary compute group

👍 4
marshall16:12:20

when you go into the cloudformation for the primary compute group

marshall16:12:37

there is a parameter for

marshall16:12:38

“Analytics Endpoint Provide the name of a query group if you’d like analytic queries to go to a different endpoint. Defaults to system name.”

marshall16:12:44

put your query group name in there

marshall16:12:21

that will cause the access gateway to direct its analytics queries to the query group you specify

bartuka16:12:42

I cannot find this option when I click in update in the cloudformation. It should be done when I first create the primary compute group?

marshall16:12:09

it is available either way Are you running a split stack? you can’t do it with a master stack system

bartuka16:12:42

I have a master stack system.. 😕

marshall16:12:52

you’ll want to split the stack

bartuka16:12:45

great! thanks for the help. I think would be nice to have these infos in the documentation Analytics Support -> Configuration. It seems I only needed to add the -q <query-group-name> . I don't know, I am using this stack for over a month now and probably missed some instructions in the docs too.

marshall16:12:59

Yeah, we’ll fix that

bartuka17:12:15

@marshall just to recap the question about the usefulness of the query-group with analytics-support in mind. Makes sense to say that for the analytics support, the access-gateway should be priority when we talk about machine-sizing rather than the query-group instances itself? As I understand, the presto can only make few operations before it start the SQL-processing of the data in memory. Maybe an access-gatewaymore optimized for memory-intensive tasks should be a good call?

tyler21:12:55

Are http-direct requests each executed in their own thread?

em22:12:16

Curious about this as well, and about the threading of ions in general

markbastian22:12:06

Hey all, I've got a datomic ion I'm trying to deploy and I keep getting a runtime error: "Syntax error (ClassNotFoundException) compiling new at (core.clj:79:38). com.fasterxml.jackson.core.exc.InputCoercionException". This class was added in 2.10 (http://fasterxml.github.io/jackson-core/javadoc/2.10/com/fasterxml/jackson/core/exc/InputCoercionException.html). In my deps.edn file I specify com.fasterxml.jackson.core/jackson-core {:mvn/version "2.10.1"} in my :deps map. However, when I push with clj -A:ion-dev '{:op :push}' I get a dependency conflict warning for jackson-core listing com.fasterxml.jackson.core/jackson-core #:mvn{:version "2.9.8"} as the version being used. This leads me to believe my specified version isn't taking. Any ideas as to how I specify/force the runtime version of a library in my datomic instance?

markbastian22:12:38

I was able to change my deps.edn file version of the jackson libs to 2.9.8 and it appears to be unbreaking. I'll just have to watch for version issues when pushing. I'm still interested in knowing if there's a way to control the deployed versions of the ion so that it uses the latest jars in the cloud.

jaihindhreddy09:12:21

Ahh. Good 'ol jackson.