In on-prem only

Create a bunch of partition entities, eg 256

Assign each patient one randomly and evenly (eg with a hash function of something stable like a patient number)

Now any time you make a new entity, use d/tempid, use the partition belonging to the patient that the entity “belongs to” for partitioning purposes

It’s very much a “roll your own” system entirely in the application code. Datomic doesn’t care about partitioning at the data level; it’s just to get better segment locality at the storage level

Thx, what is maximum realistic number of partitions?

The partition space is 20 bits, but that uses idents too

You really don’t need more than a few hundred

Partitions only help when your working set and object cache are significantly smaller than your entire db

It is - let say i have 1-10M patients - each 100k medical records ~ 1M datoms

~1M datoms

My dream was - each patient is a partition, < 1sec cold start from S3 archived blocks to get pt datoms in memory

S3 archived?

Partitions are not hard boundaries, they just influence sort order to increase locality

1mil is not a lot of datoms

What is your oc hit rate?

For some perspective, we have 256*2 partitions on a >10 billion datom db, and we put a load balancer in front of peers that keys by partition

We have perf problems still, but it’s not from lack of partitions

And if you are on cloud this is all academic because it doesn’t let you control partitions (it doesn’t have d/tempid)