Fork me on GitHub
#datomic
<
2021-03-19
>
zendevil.eth02:03:51

@joe.lane I was able to deploy on beanstalk, but when I use the following line:

(d/client {
                     :server-type :ion
                     :region "us-east-1" ;; e.g. us-east-1
                     :system "humboi-march-2021"
                     :creds-profile "humboi"
                     :endpoint ""
                     :proxy-port 8182
                     })
I get “Unable to connect to localhost:8182” in the beanstalk logs

zendevil.eth02:03:52

Here are the logs. Notice line 363.

zendevil.eth02:03:49

I’m guessing that’s because the following command isn’t run on beanstalk:

./datomic client access humboi-march-2021 -p humboi -r us-east-1
But then how can one run this command on beanstalk?

Joe Lane12:03:27

You need to run beanstalk in the same vpc as your system as a client application https://docs.datomic.com/cloud/operation/client-applications.html https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/vpc.html Remove the creds-profile and proxy-port entries from your config when running in beanstalk. You do not need to run the access gateway cli commands when you are in the same vpc.

onetom03:03:28

datomic solo up <system name> --wait fails with

Upping <system name>-TxAutoScalingGroup-89CAXXFQ591C
Upping <system name>-BastionAutoScalingGroup-128DD1GX31OU7
Waiting for gateway to start.
Execution error (ExceptionInfo) at datomic.tools.ops.process/sh! (process.clj:64).
Shell command failed
it doesn't throw this error without the --wait option though, so at least it's usable. anyone with the same error or anyone who is using it successfully? couldn't find an posts on http://forum.datomic.com about this issue, so i will make one eventually.

onetom03:03:11

ehhhh, it's shelling out to the aws cli command, which i don't have in my specific environment:

[{:type clojure.lang.ExceptionInfo,
    :message "Shell command failed",
    :data
    {:args
     ("aws"
      "ec2"
      "wait"
      "instance-running"
      "--filters"
      "Name=tag:Name,Values=xxx-datomic-system-bastion"
      "Name=tag:datomic:system,Values=xxx-datomic-system"
      "Name=instance-state-name,Values=running"),

onetom04:03:14

i got this error from the stack trace which was saved into /var/folders/dm/bjgtcwgx7nqfh3flbpq7m0qc0000gn/T/clojure-927549602744154670.edn such trace file paths are always printed at the end of clojure cli errors, but i've noticed that ppl often forget to look inside them.

tatut07:03:53

anyone using divert-system? I’m not really sure what it means actually… I’ve only used local dev with locally created database in file storage. Now I’m looking into having test envs that have a copy of a cloud database as basis

tatut07:03:05

what does divert actually do? do queries copy data from the diverted system

Alex Miller (Clojure team)08:03:12

you use import-cloud to import (a subset of your) data from your cloud system to your local storage, then divert-system will direct queries to be answered via the local storage instead of prod

tatut08:03:16

yeah, I read that page but it wasn’t clear to me what it does… ok so import-cloud is the one I need

mikejcusack17:03:47

If you are importing an existing db. If you just want to create a local test db divert-system will direct calls to local.

Joe Lane12:03:27

You need to run beanstalk in the same vpc as your system as a client application https://docs.datomic.com/cloud/operation/client-applications.html https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/vpc.html Remove the creds-profile and proxy-port entries from your config when running in beanstalk. You do not need to run the access gateway cli commands when you are in the same vpc.

kenny15:03:11

The DesiredCapacity knob is a bit strange when deploying a query group. We have our query groups deployed with auto scaling, so the ASG is "managing" the desired count. If I change a parameter (e.g., MaxSize) and have DesiredCapacity set, CloudFormation will actually set the DesiredCapacity to the value passed in. This is pretty nasty, particularly in the situation where you'd be increasing the MaxSize (e.g., DesiredCapacity is set to 2 and MaxCapacity is set to 4. You're currently running at MaxCapacity and need to immediately increase MaxCapacity to meet current demand. You set MaxCapacity to 6 and update the CF script. CF will set MaxCapacity and DesiredCapacity to 2. This makes the high demand situation worse!). The safest workaround seems to be to always have DesiredCapacity set to MaxCapacity. Does the Datomic team have any advice on how to handle a situation like this? FWIW, we deploy our services onto Fargate via Pulumi. Pulumi has the ability to set "ignoreChanges" on certain properties (e.g., ignoreChanges: ["desiredCount"]). This lets me set an initial desiredCount for my Fargate service and ignore any changes that have occurred since initialization. It would see a similar knob for query groups would solve this issue.

ghadi15:03:06

@kenny are you making use of drift detection?

kenny15:03:24

No. I've seen it in the console before but never used it and I am not familiar with it. Does it help in this situation?

ghadi15:03:05

yes it helps identify drift between reality and the CF template

ghadi15:03:16

CF manages the ASG which manages the instances

ghadi15:03:37

the ASG is not the head honcho

ghadi15:03:12

I would recommend consulting the drift detection before making manual changes to resources

kenny15:03:21

In the case of DesiredCapacity, why do I care about "drift"?

kenny15:03:00

A change in DesiredCapacity doesn't seem like drift.

kenny15:03:38

Drift sounds like something unexpected. I fully expect DesiredCapacity to change 🙂

ghadi15:03:34

desired capacity is an ASG parameter

ghadi15:03:51

it's part of the ASG API, right?

ghadi16:03:15

I get that it's the thing most likely to be manually tweaked outside of source control

kenny16:03:29

I read https://aws.amazon.com/blogs/aws/new-cloudformation-drift-detection/ on drift detection. It sounds like a heck of a lot of extra work. When updating an ASG, you don't need to set desired capacity again.

kenny16:03:51

Drift detection also doesn't solve the problem. Say I run the drift detection and see the actual ASG desired capacity is different than the capacity set in the CF parameters. What action am I supposed to take? Change my CF DesiredCapacity param update to match the current state? That's not what I actually want. I want to update MaxCapacity and leave DesiredCapacity unchanged. I don't want all future updates to set DesiredCapacity to the value I am forced to set it to for this particular update. Worse, DesiredCapacity could have changed from the time I ran the drift detect and the time I run the CF update.

ghadi16:03:49

drift detection is just a tool that exposes changes made to resources under management by a cloudformation stack Is there a particular reason you don't want to change Min/Max/Desired via CloudFormation?

ghadi16:03:12

just trying to understand, not suggest anything specific

kenny17:03:00

I am changing the Max via CF. I'm saying, perhaps poorly 🙂, that changing via CF has undesirable side effects.

kenny18:03:02

With the exception of DesiredCapacity, all query group CF parameters are managed (updated/changed) via a CF update. The DesiredCapacity is only ever controlled by the ASG scaling policy.

Joe Lane18:03:24

So, is this a feature request?

kenny18:03:40

Or bug report. Was trying to start from the problem to ensure that the problem was actually a problem.

ghadi18:03:03

I thought Desired was part of the cf stack params

ghadi18:03:21

That would explain our misunderstandings

kenny18:03:00

It’s a required param.

Joe Lane17:03:05

@kenny this is happening in what context? A datomic version upgrade A datomic parameter update Something else?

kenny17:03:24

Changing MaxCount from 4 -> 6.

kenny17:03:39

The example I gave is the exact thing that happened to us 🙂

Joe Lane18:03:29

What does this have to do with datomic cloud? Are you reporting a bug or bemoaning how CF works?

Joe Lane18:03:07

(I'm not saying it doesn't have anything to do with Datomic cloud, I just don't understand in what usage scenario you're running into this.)

kenny18:03:12

I think Datomic's CF implementation leads to undesirable results (though, I am not a CF expert so it could be a problem with CF itself). If DesiredCount were an optional parameter, I think this would not be an issue.

Joe Lane18:03:06

So you're saying that the QueryGroup CF template always sets the DesiredCapacity and the MaxCapacity equal to the same value? • When upgrading a stack to a new version? • When updating the stack for some reason? • Something else is happening?

kenny18:03:16

The problem is that CF is trying to manage the DesiredCount parameter (ensure actual DesiredCount matches the DesiredCount set in the params). This is problematic because the DesiredCount is entirely managed by the ASG.

kenny18:03:12

Not exactly. The problem is that the CF template is always setting the DesiredCount param.

kenny18:03:12

Besides the very first run, I never want CF to touch the desired count parameter.

Joe Lane18:03:24

And it shouldn't because....?

Joe Lane18:03:17

And so it's "Always setting the DesiredCount param" • When you're upgrading a version? • Updating a stack with some new config value (like an env-var)? • Something else?

kenny18:03:23

This is the situation: DesiredCapacity is set to 2 and MaxCapacity is set to 4. You're currently running at MaxCapacity and need to immediately increase MaxCapacity to meet current demand. You set MaxCapacity to 6 and update the CF script. CF will set MaxCapacity and DesiredCapacity to 2. This makes the high demand situation worse

kenny18:03:36

Updating MaxCount.

kenny18:03:56

I'm not sure on the other situations.

kenny18:03:55

Intuitively I would expect the same behavior (i.e., DesiredCount gets set to the CF param). Would need to test it to verify ofc.

Joe Lane18:03:44

> DesiredCapacity is set to 2 and MaxCapacity is set to 4. In the ASG or CF? > You set MaxCapacity to 6 and update the CF script Again, ASG or CF?

Joe Lane18:03:36

What is "CF script"? Are you referring to your scripts or the QG CF template parameters? This is an update right?

kenny18:03:27

Oh, sorry. In all cases I mean query group CF template.

kenny18:03:57

And yes, an update.

Joe Lane18:03:53

So, In the template you have the DesiredCount at 2, MaxCount at 4. You're at your max, because you manually changed the ASG DesiredCount to 4 (to match your load). The problem is that when you update your MaxCount from 4 to 6 but leave the DesiredCount at 2, you think it should keep the 4 you manually set in the ASG console? Am I close? If I'm wrong, can you copy the above prose and edit it, then paste it back here in this thread?

kenny18:03:57

So, In the template you have the DesiredCount at 2, MaxCount at 4. You're at your max, because the ASG scaling policy scaled up the group DesiredCount to 4 (to match your load). The problem is that when you update your MaxCount from 4 to 6 via a CF template update but leave the DesiredCount at 2, I think it should keep the 4 the ASG scaling policy set.

Joe Lane18:03:58

What is the ASG scaling policy based upon? Why has it scaled up to 4 machines? CPU, mem?

kenny18:03:28

Target track 50%.

kenny19:03:48

Worth opening an ask.Datomic topic on this?

em19:03:44

> You set MaxCapacity to 6 and update the CF script. CF will set MaxCapacity and DesiredCapacity to 2. This makes the high demand situation worse Curious, if you set MaxCapacity to 6, why would CF set MaxCapacity to 2? The problem makes sense though, as a dynamic parameter DesiredCapacity is modified by ASG to control behavior. But CF template updates want to set it as a default param. Maybe an optional tick box in the template would solve the issue without breaking changes?

jaret19:03:15

@kenny how are you updating? from CFT? CloudFormation does not change your ASG settings when you update or upgrade. It reads your ASG settings. Maybe I am missing something here, but adding machines is going to potentially bounce the process monitoring CPU or lower the CPU average, right?

kenny19:03:24

Yes, the objective would be lower cpu. Since I’m not fiddling things in the console directly, there could be something in our deployment process affecting this. Let me create a minimal repro and get back to you in a couple hours.

jaret20:03:17

@kenny what do you have set for your warmup time?

jaret20:03:49

So the default, then

jaret20:03:58

and did you disable scale in?

Joe Lane20:03:08

And you're using i3.xlarges?

Joe Lane20:03:46

Have you timed how long they take to come up and start accepting traffic? If you start reporting metrics before that, they may start reporting CPU metrics against your ASG policy, lowering the utilization, and killing instances.

kenny20:03:12

I have not. I do not think that was what happened. I recall checking in the ASG activity log and saw a DesiredCount change due to CF (I think). Need to double check. Not at a computer atm and AWS mobile console is hard to navigate.

jaret20:03:11

@kenny I think I understand your issue, let me know if this clarifies it for you. I think the missing piece is that AWS CF looks at parameters it sets. It only knows about parameters it has set. If you have changed your ASG settings outside of CF (i.e. your policy changed your settings) the CF defaults the parameter to the last CF-set parameter. Does that make sense? So you need to manually set these parameters or update your script to query for the currently set "DesiredCount" in you asg and inject that into your DesiredCount CloudFormation parameter.

kenny20:03:05

Yes, I think that’s exactly what happened. I don’t think those extra steps are necessary though. Why is DesiredCount a required parameter?

kenny20:03:29

(Your solution is inherently racy 🙂)

jaret13:03:02

Kenny, I think it's required because we are setting up an ASG with the CFT (per AWS requirements) when you launch the CFT. I will double check that with the team.

kenny18:03:58

From the doc ^ > If you do not specify a desired capacity when creating the stack, the default is the minimum size of the group.

kenny17:03:58

Since this conversation is likely to fall off of the Slack retention window, I have added a question here: https://ask.datomic.com/index.php/603/desiredcapacity-optional-parameter-query-group-template