Clojurians Log v2
datomic 2026-09-03
Does Datomic have a limit to the number of entities in a transaction? I'm uploading an EDN file to my server, it is creating/inserting/transacting a large object and when the dust settles, there are exactly only 1000 items inside. I had a different EDN file and tried that and again, exactly 1000. At first I thought my browser or web server was limiting filesize, but if that were the case, I would get an incomplete/mangled EDN file and have all kinds of different errors. I'm not getting any errors, just an object with only 1000 child items. Any ideas?
Maybe this isn't a limit on transactions, but on cardinality/many? Can I only have 1,000 of them?
Pull has a default limit of 1000, that is probably what is happening. https://docs.datomic.com/query/query-pull.html#limit-option Transactions don't have a limit like 1000 entities.
I just did some searching and found that was it. I was coming back to answer my own question, but you got here first. Thanks.
Why does Datomic Pro limit tuples to 8 components? Is the limitation due to hashing, comparators or storage? Could it be raised to 16 or 32 for non-string components? (since Datahike does not have this limit). Specifically, EACL is approaching the Datomic tuple limit to retain parity with SpiceDB, and the alternative is reverting to modelling Relationships as entities, which would 5-15x Relationship storage costs in Datomic.
Got a diagram of how exactly you’re trying to use tuples?
Hi @joe.lane, I don't have a diagram but it's explained in this thread with @favila: https://clojurians.slack.com/archives/C03RZMDSH/p1788526428293519?thread_ts=1788516102.259579&cid=C03RZMDSH It's not dire, I just want more tuple components and I don't see a good reason why Datomic should limit finite (non-string) tuples components. I suspect there is a better design lurking here involving (consistent) arbitrary materialized indices, but I don't think it's gonna be on the Datomic roadmap any time soon.
What are the properties you'd like to see from arbitrary materialized indices? tx-log -> reducer + snapshots as init What kind of index do you need? Does it need range scan? Need to be larger than memory? Does it need to be datoms or something else?
I want programmable indices: not just :eavt, :avet, :vaet
In heaven we want subscribeable differential dataflow,
It's all about reducing the number of d/(r)seek-datoms. Tuples minimize indirection serial page scan indirections.
In Datomic heaven I want a Programmable Transactor and more flexible indices, but I won't venture into areas Nubank won't support. Tuples are the main "materialized view" feature right now.
My main complaints are that Datomic tuples are limited, esp. Composite Tuples:
1. I want Composite tuples that support a mix of entity attrs & scalars, not just attrs on the entity.
2. I want Tuple Retraction Policies so that when a ref in a composite tuple is retracted, the entire tuple can be retracted instead of turning one component nil, which causes havoc if tuple has uniqueness constratint. Same work, more consistency. Turning tuple ref components -> nil is a known footgun.
Everything I'm asking is about reducing the number of d/(r)seek-datoms for EACL and reducing the complexity of maintaining cache coherence.
On the topic of materialized indices, I want to program the Transactor to maintain derived indices, and I want to be able to wait on them being ready (so as not to block the single-writer). This can be achieved with separate DBs, but it hugely increases overhead. I don't have a good code example on-hand. Basically I want more of Rama in Datomic.
Can the need for these views be solved by keeping a memory only index (that gets updated by either a bespoke transact function or, probably better - a listener on the transaction queue)? I'm looking into this right now with EACL-type features, where at system start I "compile" an initial view with things like "all but X" realized. Then when the transaction log gets updates, I need to build something that keeps this view in sync with reality (or recompile it if the operation is not super expensive).