Clojurians Log v2
datomic 2026-09-07
I have a question for this group. What should I consider when needing to "increase capacity", as a very vague term. My old app had database and webserver on the same EC2 instance, and every November when a client announced their event and every grandmother in the Salt Lake Valley got on to buy tickets, my server crashed unless I changed the instance size to something very powerful. Now I have clojure web server on its own instance, inside an auto scaling group, and I think I can handle the traffic by bumping the number of instances there. But what to do on the database side? Should I also scale the compute group to multiple instances? Or have one very beefy instance? What do you with experience suggest?
Are you using Datomic or cloud?
Cloud
And you're not using ions , just remote client API against the 1-2 node primary compute group? What kind of traffic are we expecting? 90/10 reads/writes? Something different?
Also what kind of instances are the cloud nodes running?
I am using ions. Lots of them. I think this situation I'm expecting in November will be lots of writes, because there will be order creations, purchases, etc. I don't know what the ratio would be but I would expect it to be closer to 50/50 than 90/10. My clojure app (the web server) runs on t3.medium. The datomic cloud compute stack also on t3.medium.
“My clojure app ( the web server)” is therefore not an ion and is using the remote client api, correct?
Right.
I have about 30 tx-fn's and about half of those are related to and might be actively used somewhere along the way during a customer purchase. The others are related to set up of events/pricing/whatever sort of like content management stuff.
I guess I sort of confused ions for tx-fn's when I was reading your questions.
To get a tx-fn in place it's "ion-deploy" so that is my excuse. :-)
Ok so I would point those queries at a query-group that you can scale independently and leave the primary group for just transactions. The query groups forward transactions to the primary group as needed.
Don't auto scale your primary group, only do that for query groups
OK. I wasn't planning on autoscaling per se. I know when this will happen, so I was going to get the instances in place a bit earlier, get through the tsunami, and then scale back down. So you are saying set up a query group and let that be the absorber of the traffic. I could also change the compute group to something larger than t3.medium? That would help?
An i3.large and bigger will include a valcache that is also reachable as a cache over the network by the query groups nodes. Perf will be much better as long as those nodes aren't replaced.
It depends on how big your db is though, how much traffic, etc though.
Do I need to worry about a particular order of operations? For example, do I resize the compute group instance first, then increase the query group instance count? Does it matter? Do I do things in reverse order when I'm ready to go back to normal?
Eg. What is “lots of writes” in txes/sec, datoms/sec, etc.
I can go back and examine the traffic patterns from last year and make some sort of educated guesses, and I will do that. But also, this only has to be in place for a couple of hours, so I'd rather go way bigger than needed and just handle the traffic. The costs won't be too bad if it's only for a short duration.
Ok, if you just want to scale up and scale back down do primary first the query groups and the reverse when you’re done.
Using autoscaling groups (both for Datomic and my own app) makes things much smoother to do these transitions. Before it was shut completely down while I resized my one and only instance.
Remember, when launching the query groups, they get their own query api endpoint that you’ll need to configure in your client-cfg
Later this month I'll go through a dry run of the whole thing. I probably won't have a way to simulate the traffic, but I'll do all the scaling up and down and make sure it all works as expected.
We’re here if you want to chat about upfront capacity planning for your launch. I used to work at a company based out of Lehi that did weddings and other “game day” events so I’m familiar with the stakes (heh, pun intended)
This is the annual Nutcracker ballet put on by Mountain West Ballet and I can't risk angering the grandmothers. Seriously.
Have you looked at io-stats and query stats before? I think you may find analyzing them very valuable for capacity planning for your event
I haven't looked at those.
So it is a flash sale
How many total tickets and how many grandmothers from last year
I’ve got some time right now to hop on a video call if you want to chat in higher bandwidth
sure.
It's like a holiday. Aren't you supposed to be bbq-ing or something?