This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2020-09-11
Channels
- # announcements (7)
- # aws (52)
- # babashka (16)
- # beginners (49)
- # bristol-clojurians (1)
- # calva (2)
- # chlorine-clover (26)
- # cider (6)
- # clara (1)
- # clj-kondo (79)
- # cljfx (15)
- # clojure (82)
- # clojure-berlin (2)
- # clojure-czech (1)
- # clojure-europe (26)
- # clojure-france (91)
- # clojure-germany (48)
- # clojure-nl (7)
- # clojure-norway (99)
- # clojure-uk (54)
- # clojurescript (18)
- # code-reviews (9)
- # data-science (2)
- # datalog (15)
- # datomic (15)
- # depstar (20)
- # emacs (4)
- # events (1)
- # fulcro (30)
- # funcool (1)
- # graphql (1)
- # helix (5)
- # jobs (6)
- # kaocha (12)
- # leiningen (8)
- # luminus (1)
- # malli (13)
- # off-topic (73)
- # pathom (12)
- # portal (11)
- # portland-or (1)
- # re-frame (10)
- # reagent (1)
- # reitit (44)
- # remote-jobs (1)
- # ring (19)
- # shadow-cljs (64)
- # tools-deps (32)
An interesting discussion on Twitter about Clojure deployment on AWS https://twitter.com/pesterhazy/status/1304131064835772416?s=20
I’m doing a deployment now and I’m seeing these metrics: • Env. update is starting -> registering first batch to load balancer and waiting to be healthy: 40s • First batch passed health checks: 5m • Deploying to second batch and waiting for health checks: 36s • Second batch passed health checks: 5m
@orestis thanks for bringing this up here
what deployment config are you using?
I've written up my findings and experiments, with some alternatives sketched (but I haven't figured this out yet by any means): https://gist.github.com/pesterhazy/d0030f559f600d0ce1b3a090173c9c9c
Any comments appreciated
“Currently we use Rolling policy with BatchSize=1” -> that means you’re doing the update one-by-one, have you tried using a percent-based batch size? 25% would do two instances at a time, so it would half your deployment time.
I feel your pain though. We were hosted on rackspace before, using plain old VMs and updates took seconds.
There’s a few things I mean to try but it’s low priority for us ATM: 1. There’s a new split traffic update mechanism https://aws.amazon.com/about-aws/whats-new/2020/05/aws-elastic-beanstalk-traffic-splitting-deployment-policy/ 2. There’s a (literally yesterday) ability to share a non-EB load balancer between different EB environments: https://aws.amazon.com/blogs/containers/amazon-elastic-beanstalk-introduces-support-shared-load-balancers/
Do you think these have the potential of speeding up deployments?
The split traffic will probably fix the “mixed results” problem. I’m using session stickyness to overcome it, but I don’t like session stickyness in general.
The shared ELB gives you great flexibility, since you can mix-and-match environments — but I don’t think you can get faster deployments without significant engineering investment in automation, so it’s probably not that relevant.
Elastic Beanstalk was nice to get us some peace of mind with minimal ops investment, as we migrated to AWS. But it feels creaky. OTOH, it is Dockerless, and there was some movement lately which makes me hopeful that it’s actively developed and improved.
> have you tried using a percent-based batch size Yeah that's definitely something we'll try, along with RollingWithAdditionalBatch. I figured we'd try Immutable first, on the assumption that it'd be faster on principle, because it spins up all 8 instances concurrently. But that doesn't seem quite true
Another thought I've had is this: • in the normal day-to-day deployment case, 26 min is probably acceptable; so we can keep using Rolling deployments (or Immutable deployments) • when there's a problem, however, and you need to deploy a hotfix or roll back a change, 26 min is unacceptable. In this case we could manually switch to AllAtOnce before uploading the new app version
Ah wait — the configuration policy is different than the new application version policy. So you could keep the configuration policy at AllAtOnce indefinetely.
Just tested this. You can manually switch to AllAtOnce in the console. It takes about a minute. Then deploying a new application version takes less than a minute. The downside is that you have downtime, in our case 12 minutes
This tradeoff may be acceptable when hotfixing a customer facing bug
@ghadi how long does it take for traffic to hit a new version of code, and how fast is rolling back? Also, does Fargate necessitate Docker? Does it have a nice console to get started? Is it a good fit for “monoliths”? How do you develop locally? What about monitoring things like traffic, memory use etc etc? Do you pick instance sizes eg if you want large machines with lots of memory? What about static assets, do you bundle nginx together with a JVM or do you make separate containers?
I’m clueless and suspicious about containers and beanstalk gave me an entry point where many things are reasonably automated, but we’re outgrowing it :)
monolith is broad, so I can't assess if it's a good fit, but if you can containerize your app, it's a start
A few more questions: do you use AWS cli to deploy new versions? Is a new version a fresh docker image that you push to a registry, then notify Fargate to run it? How does auto scale work? Is it also suitable for “background” jobs eg if I have some cron jobs can I keep a Fargate container running forever?
What about the subjective stuff instead, are you happy with it? Would you choose it for a new project?
upstream build job makes a docker image, downstream deployment job updates cloudformation stack
since java is so easy to deploy, sometimes I use a stock container that downloads the jar upon startup
autoscaling in ECS/Fargate https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-auto-scaling.html
Subjectively, I like Fargate, but I've used it for several years now and have some expertise
We used to use Fargate, and I quite liked it. Had to switch due to lack of persistent storage at the time. It's gotten some very nice features since, however. They added capacity providers so using the spot market is easy. You can now attach EFS volumes to a task for persistent storage. We use Datadog for metrics, logs, APM. If you use something similar, all tasks need a sidecar container running which is a small additional added cost per task replica. Deploy everything using Pulumi. Overall Fargate experience has been fantastic. Would also recommend.
Ah, right, so different vendors. Interesting POV, I thought that using AWS for everything was the norm but it seems not.