Charting this responses correlates with a 'Read usage' metric in DynamoDB, so I'm thinking the problem might be that each new server asks data from the transactor which then pulls from Dynamo.

@U3UFFB420 check out https://docs.datomic.com/on-prem/overview/caching.html Your options are memcached or valcache.

nit: the peer server is not reading from the transactor, it's reading directly from Dynamo. In addition to adding another caching layer like memcached or valcache, if you know the data that the peer is going to need to serve web requests, you can try to force it to be loaded into the object cache by reading that data in your app before it starts handling traffic.

@U060QM7AA so peers may read straight from Dynamo? Yes, that seems like the more sensible idea. Something like preloading the database may help.

That’s my understanding. Writes are centralized through the transactor but each peer can read from storage.

Thank you!

no. i just saw someone clicking around and inspecting records in AWS dynamo console, and i noticed the update button, and it dawned on me…. “this would be really easy to make my day horrible”

that's a good point... i think we are also overly liberal with our AWS access permissions...

I understand that the entity essentially memoizes its access to (d/datoms :eavt e) but at what scale does this start to matter especially with SSD cache 1ms away?

It really only pays off if you repeat reads to the same attributes, because that guarantees you will never go even to the object cache. If you’re always reading new attributes there’s no difference.

d/touch forces all attributes to be read if you want to do that eagerly

do you know what the cost difference is between object cache and SSD?

It’s the difference between pure B+tree pointer chasing (assuming every pointer is loaded) vs io scheduling and decoding fressian into objects on miss

And is it true that for "large" dbs that the object cache might be entirely evicted and rebuilt on a request to request basis due to the object cache being significantly smaller than the actual indexes?

if your entire workload does not fit in OC, something will get evicted

also new indexes create new segments, and are an inherent source of eviction

is that common for the workload to not fit in OC ina prod cofiguration (so permitting eng to partition the query load across multiple query boxes if that is a thing people do to make datomic fast)

You’re asking if people use big or small dbs with datomic. 🤷

I know that for us, we have 17+ billion datom db, and routing request loads for locality became essential for performance at < 4 billion datoms

and is it even possible to make it so OC is mostly not thrashing every request

but we have a lot of dumb select *-like workloads which probably thrash the cache horribly anyway, and we have new indexes every 10-15 minutes

yes it is, certainly, by controlling your read workload and locality

thanks

watch your object-cache hitrate