I'm monitoring RAM usage, and it never gets close to 100%. When I give it 5 CPU's, it will get pretty close. My team doesn't want to allow me 5CPU's due to budget requirements. Even with 5 CPU's it can't hit 1000k concurrent. One thing I've done quite a bit is re-establish DB connections instead of using the same connection. It will take quite a bit of time to ensure I'm using the same DB connection on every request, I know that's best practice, but I would only like to take on things that will affect load right now. Would that help ? Any other things to monitor ?

Just broad advice: you should provision a datomic peer similarity to how you would provision a read-only db replica (vs a web application), because in a sense that’s what it is.

important metrics to look at are objectcache and memcached hitrates

If the hit rates are low (esp objectcache), you are paying significant IO and cpu cost

how do i get those metrics ?

and what's low ?

If you have extra Jvm headroom, consider increasing the bias towards objectcache. (Default is half xmx)

i'm giving it 4GB of Ram and my dynamo table is only 1.5GB

so wouldn't half more than cover it ?

Object cache is decoded objects, so they are much much larger

The stuff in dynamo and memcached is fressian-encoded and gzipped

ok

If your objectcache hit rate is <99%, then you k ow that it is a bottleneck

Because it has to go to memcached or storage at least some of the time

ok

so if i have 4GB of RAM, I might consider giving it 3GB ?

to objectcache

Depends on what your application and query code needs. You can probably give it at least half the headroom you see

ok

But really you should look at the metrics to make sure OC hitrates are even a problem

If it’s already 100%, obviously more won’t help

Is there a way to get an overview of the OC hitrate or do you need to do it query by query like this

(d/q {:query '[:find (count ?a)
                   :in $ ?aname
                   :where [?a :artist/name ?aname]]
          :args [db "The Beatles"]
          :io-context :artist/by-name})

https://docs.datomic.com/on-prem/operation/monitoring.html

that io-stat stuff is brand new. The older mechanism is described there

and it is the behavior of the peer as a whole

The metrics are also logged in the datomic.process logger iirc

but you will want to use a callback to get it in your monitoring stack in production eventually

ok, so i could just (println request) in the callback handler

and tail the log and watch

very cool thanks

?

no I mean the java logging of datomic

I’ll look up an example

Here’s one

2023-02-03 13:01:58.094 76941657 [Datomic Metrics Reporter] INFO  datomic.process-monitor - {:tid 29, :CacheRepair {:lo 1, :hi 1, :sum 51, :count 51}, :MemcachedGetTimeoutMsec {:lo 16.55, :hi 16.55, :sum 16.55, :count 1}, :ObjectCacheCount 12945, :PeerAcceptNewMsec {:lo 0, :hi 0, :sum 0, :count 448}, :MemcachedGetFailedMsec {:lo 0.43, :hi 65.85, :sum 152.07, :count 51}, :MemcachedPutSucceededMsec {:lo 0.52, :hi 10.43, :sum 83.29, :count 51}, :AvailableMB 850.0, :Memcache {:lo 0, :hi 1, :sum 1258, :count 1309}, :MemcachedGetSucceededMsec {:lo 0.38, :hi 30.44, :sum 1327.46, :count 1258}, :StorageGetMsec {:lo 4, :hi 15, :sum 370, :count 51}, :ReaderCachePut {:lo 1, :hi 1, :sum 51, :count 51}, :pid 4920, :event :metrics, :ObjectCache {:lo 0, :hi 1, :sum 9004, :count 10389}, :MemcachedGetMissedMsec {:lo 0.43, :hi 3.21, :sum 36.24, :count 48}, :MetricsReport {:lo 1, :hi 1, :sum 1, :count 1}, :PeerFulltextBatch {:lo 1, :hi 2, :sum 448, :count 445}, :StorageGetBytes {:lo 7448, :hi 79049, :sum 1736271, :count 51}, :DirLoads {:lo 1, :hi 1, :sum 24, :count 24}}

Oh i thought I could do

-Ddatomic.metricsCallback=db/stats-handler 

(defn stats-handler [request] 
  (pritnln request) 
)

that looks great that's what i need

oh, I didn’t understand what you meant by “request handler”

yeah, you can do that

cool

the reporting interval is like once a minute

nice

Are you using mbrainz in your load test?

That’s like the fake dataset ?

Is the server side work of the load test pure query?

The query you posted above looks like mbrainz

Oh that was copy pasted from docs

What are the io-stats, query-stats, and process monitor metrics for your queries during the jmeter load test?

i haven't set up the i/o stats yet. doing that now

{:PeerAcceptNewMsec {:lo 0, :hi 0, :sum 0, :count 7}, :MetricsReport {:lo 1, :hi 1, :sum 1, :count 1}, :PeerFulltextBatch {:lo 1, :hi 1, :sum 7, :count 7}, :AvailableMB 2970.0, :ObjectCacheCount 523}

My report has far less data than yours @U09R86PA4

It doesn’t report 0

so this looks like an idle machine

it's def not idle... i can see web traffic coming in

Do you have any io-stats for your query ?

(What is your query?)

no, this was just the result of

-Ddatomic.metricsCallback=lms.db/stats-handler

(defn stats-handler [request]
    (println request))