Fork me on GitHub
#datomic
<
2020-03-22
>
steveb8n06:03:43

Q: I have a stock prod topology and I have started load testing readonly loads. when I max it out, the cpu tops out at approx 50%. Am I right in understanding this is because only one of the two server nodes is handling queries?

steveb8n23:03:00

@U05120CBV could you comment on this?

marshall23:03:49

How are you "maxing it out"

marshall23:03:11

Is the read load all coming from a single source?

steveb8n23:03:46

I have a cljs lambda which calls two Ion fns. Both are read only. I use Gatling to create increasing levels of load. It reaches an asymtote pretty quickly and the CPU chart in the dashboard never goes above 50% when it hits the asymtote

steveb8n23:03:19

(I can’t spell asymptote)

steveb8n23:03:24

so I guess it is a single source but I’m not sure how lambdas scale out. are you suggesting that, if > 1 lambda is making the call, it will distribute across both nodes?

marshall23:03:39

I suspect it has to do with traffic routing on a per database basis for the primary compute group, but i dont know for a fact. I would suggest trying it against a query group

marshall23:03:30

But also, the thing that matters is your read throughput, not the distribution of cpu usage

steveb8n23:03:18

digging in. my cljs lambda invokes the Ion lambda via the npm aws client. not sure if that is relevant

steveb8n23:03:45

also my Ions work with N datomic databases, not just a single one. I’ve segregated database based on usage requirements

steveb8n23:03:18

ok. I’ll try to grok “read throughput” a bit more on the next load test

steveb8n23:03:26

in my startup status (i.e. low budget) I’m trying to minimise costs. adding a query group implies extra cost right?

marshall23:03:47

I dont know of the top of my head how the dashboard reports cpu usage. You can always look at individual ec2 instance cpu metrics if you want to examine things further

steveb8n23:03:11

yeah. I thought about watching ec2 node cpu as well. I’ll do that also

steveb8n23:03:21

thanks for the ideas. I’ll refine more on the next test

steveb8n01:03:47

here’s a thought. would the routing system be different if I started using http-direct for this Ion invocation? It’s something I’ve been meaning to do anyway

marshall13:03:12

shouldn’t change HTTP direct goes to the LB the same as a lambda

steveb8n05:03:06

@U05120CBV ok. I have since verified that under load, only one of the two nodes are being used for reads. with the OOTB topology, should the load be spread across both? if so, what’s the next step to diagnosing this? Is it possible that my cljs/aws api Ion invocation is somehow affecting the load balancer?

steveb8n05:03:36

(interestingly the architecture diagrams don’t show an NLB behind the lambdas or direct. I trust your statement despite that 🙂

steveb8n05:03:29

happy to log a ticket to pursue this further if it makes sense?

David Pham09:03:56

Can we translate number of datomic system with the number of transactors?