Additionally, are there suggestions on tx size if you’ve got control over it? I arbitrarily chose to import my documents in 10000 document tx/batches and boy does it take a while to index one of those. I reasoned it might be higher total throughput than a bunch of small transactions though.

Hi 🙂 as it happens we've done quite a bit of work on improving ingestion performance since the last release. Depending on the shape of your documents/operations/transactions I think you may find the current dev-SNAPSHOT Release Candidate (available via Clojars) to be as much as twice as fast. A 1.13.0 release that includes these improvements will be available very soon. > What’s the usually the bottleneck in indexing documents? The biggest cost is in decoding the Nippy-ified documents, splitting them into AV pairs and then re-encoding them into various index combinations (ECAV, AVE, AEV, AV, AE etc.). Currently this work happens serially, even in the Release Candidate, but we are looking at adding some parallel processing logic here next. Being able to saturate Rocks is definitely the long-term objective. > Additionally, are there suggestions on tx size if you’ve got control over it? Your reasoning about large transactions being better for throughput is sound although we've not examined the practical upper-bound for this, so I definitely suggest measuring carefully with your setup. We tend to run performance tests with batches of 1000 (see https://github.com/juxt/crux/search?q=partition-all+1000)

For anyone who’s curious, those indexing prs are magic. The open (as of yesterday) pr alone was a 1.6x improvement. I was already on the dev branch, so 2x sounds pretty reasonable for those going from 1.12-1.13

I also tested partition sizes of 10-100-1000-10k-100k with the nyc cabs dataset. The inner values there were all about the same speed. Ymmv based on backend, as it looked like all the documents in a tx were requested from the store at the same time.

> The biggest cost is in decoding the Nippy-ified documents, splitting them into AV pairs and then re-encoding them into various index combinations (ECAV, AVE, AEV, AV, AE etc.). Currently this work happens serially, even in the Release Candidate, but we are looking at adding some parallel processing logic here next. Being able to saturate Rocks is definitely the long-term objective. Sweet, that’s pretty much what I was hoping for 🙂 Internals question about that statement: are documents really nippy-ified when delivered to the index? I thought the document store would return already decoded documents. I ask because I was going to experiment with nippy compression settings.

Cool, thanks for giving the (now merged) PR a spin too! Looking at the code I can't see that we do handle batching of doc-store fetches differently depending on backends: https://github.com/juxt/crux/blob/master/crux-core/src/crux/tx.clj#L293 It does seems likely that there could be some benefit in batching those requests, to avoid waiting for everything before the next stage of the process starts, but I imagine it will (further) complicate the indexing of the operations if the indexer has to frequently ensure that various doc requests have resolved correctly. It might well be a good idea to consider in the next round of ingestion work 🙂 /cc @U050V1N74 > are documents really nippy-ified when delivered to the index doc-store? (note: amended for precision, unless I'm missing something) Yep, we lean on Nippy extensively to handle serialisation of arbitrary documents for the doc-store and serialisation of transaction events for the tx-log in the first instance (during submit-tx). This is the main ns to understand the other ways in which we use Nippy https://github.com/juxt/crux/blob/2027cadfc64fc9fc4ff3fb5f93f1e606707bbf5c/crux-core/src/crux/codec.clj > I thought the document store would return already decoded documents That's right, all implementations of the document store protocol do the Nippy thawing/decoding before returning. I can't see that we have ever tried changing Nippy compression configs ourselves and it's probably something you could manage to override without our direct input (that said, I'm not sure if it's fundamentally a good idea or not...). For reference, there is a wider issue open about pluggable serialisation which we don't currently have any plans to address: https://github.com/juxt/crux/issues/151

> Looking at the code I can’t see that we do handle batching of doc-store fetches differently depending on backends: I figure it would be different amounts of expense to fetch from (for example) jdbc than s3 based on how they batch within the doc store, right? I was actually asking about how documents go from doc store -> index. For example (I think this is the right spot on indexing? I’m really only familiar with doc-store code) https://github.com/juxt/crux/blob/a9b67f38560e70012364cf72845773dac62fcb9c/crux-core/src/crux/tx.clj#L293 makes it so we can implement fetch-docs however we like, including messing with different nippy configurations when writing to the physical store itself, as long as fetch-docs returns thawed objects. I was considering inline compression like that might be useful for backends that can’t compress transparently, but since it’s just document-level compression it’ll probably only be marginally useful unless you’ve got some really large documents. No clue what the performance hit would look like, but it might be a way to save on storage costs.

> I figure it would be different amounts of expense to fetch from (for example) jdbc than s3 based on how they batch within the doc store, right? Yeah that makes sense to me. If you had the ability to configure the batch size then you'd probably want to tune the it based on the latency & throughput of the doc store Implementing your own doc store backend is pretty straight-forward. You could study the s3, azure-blobs and Redis implementations for inspiration: https://github.com/juxt/crux/blob/master/crux-s3/src/crux/s3.clj https://github.com/juxt/crux/blob/master/crux-azure-blobs/src/crux/azure/blobs.clj https://github.com/crux-labs/crux-redis-tx-and-doc/blob/master/src/io/dominic/crux/redis.clj#L134

For large nested documents that are only ever changing slightly, but relatively frequently, creating some kind of structural-sharing doc-store backend would be really neat. I don't think anybody has attempted such a thing yet 😉

> Looking at the code I can't see that we do handle batching of doc-store fetches differently depending on backends We do give each backend the chance to batch if it can - fetch-docs is passed a transaction's worth of documents at a time. S3's a good example of batching here, within that function. Admittedly, what we don't do particularly well yet is pipelining those transactions, that's certainly an area that should bring us a speed and utilisation boost