what other stores are you comparing?

Typically I would use something like Postgres for this usecase

we’re doing way than 200M datoms per day, so it’s possible to transact a fairly large throughput. Hard to say more without knowing more about your query needs, but features of Datomic worth considering are: historical audit, flexibility (graphs and relational APIs), cache locality, real transactions. If your project is ETL based, you may not benefit from some of these

I’m surprised and pleased to hear this. In the past I have heard warnings of poor query performance with more than 10 billion datoms in the database. You would have hit that limit in less than two months, so I’m guessing modern hardware and improvements in the product have made that less of an issue.

IME the partitioning strategy is really important. We experienced a lot of IO pain until we got that in order. One thing that’s nice about Datomic is that it’s operational tradeoffs are fairly clear

Still, it’s not necessarily the best fit for non-transactional ETL workloads, just depends

^^ This. At billions of datoms, you really, really need to know what your query locality looks like and make sure you used partitions accordingly

it’s not something you can easily fix later

What do you mean by non transactional workloads?

I think he means OLTP vs OLAP workloads

^ sure

datomic is better for OLTP

passable at OLAP because of column-oriented indexes (AEVT)

but still not great

ie if your use-case doesn’t actually have the need to say, ensure global uniqueness of a user’s email, or lock a bank account during a transfer, you’d be putting data through a single thread that could otherwise be spread across a cluster like elasticsearch

hmm, yes it makes sense, thank you. My use case is not entirely OLAP in the sense that I don’t need to do a lot of (if at all) analytics, but also don’t think I need some of the features you mentioned above either :thinking_face: It’s more about getting graph querying capabilities and time tracking

another item I’d look out for is if the CSVs have any large strings in the cells. Due to indexing design Datomic doesn’t do so well with large string content in the datoms themselves

> CSVs have any large strings in the cells. Due to indexing design Datomic doesn’t do so well with large string content in the datoms themselves interesting, that is actually the case. Do you happen to know if xtdb handles this better?

when you say doesn’t do well, can you give an example perhaps? Wouldn’t lucene integration help with full text search?

it’s not about fulltext, it’s just row storage in the segment

datomic doesn’t have any kind of large-value offloading in its on-disk encoding like e.g. postgres TOAST

Datomic uses “covering” indexing, so the actual datoms themselves, instead of a reference to them, are stored in the B-tree (let’s say B-tree-like). This means that if a datom has say a 50mb string in it, it will be serialized into the index node and will need to be downloaded, de-serialized etc

This is, I believe, regardless of whether the v in the E A V T is indexed

I see! I don’t think it would be that large, likely in the kB range. Perhaps <8kb is ok, given that Postgres doesn’t trigger TOAST below those size anyway IIRC

I’m guessing I’ll need to test to be sure :thinking_face:

datomic cloud’s limit is 4k, in our 10bil datom db we keep everything under 2-4 with attribute predicates (i.e. no cheating)

I thought the TOAST threshold was about 2kb?

10bil, nice. How long did it take to get to that size? 🙂

for the size of the tuple

8+ years, although that’s with a decant in the middle

(it was 17bil as of the beginning of this year, went back down to 9 after a cleanup)

I thought the TOAST threshold was about 2kb?you’re right, thanks for the correction! (`TOAST_TUPLE_THRESHOLD` is mentioned to normally be 2kb in docs)

@U09R86PA4 what kind of sustained throughput do you do? txps and dps , and on what data store

also Q for everyone, what batch size (datoms per tx) and parallelism is optimal for bulk loading