Clojurians Log v2

thank you for the compliments 🙂

Thanks for answering so quickly. Appreciate life is busy so no expectation on my part (open source is hard enough as is). Though its really cool to hear the indexes are modular. What sort of size transactions and what frequency are we talking about here? For context, I'm running an application that's written in a CQRS style, so it amenable to and is already doing transaction batching based on the refresh/resolution rate of the view. e.g if the view is say only updated every 100ms then every 100ms of data can be batched into a single transaction. Would that help write throughput of asami? Do multiple reads scale reasonably well with on disk-storage? Thanks for the tips on copying/replicating data. That was what I was hoping for.

I'll keep this discussion to the current on-disk storage module. (not anything planned, and not the in-memory module) The main cost comes when there are lost of high-speed, small transactions. These start expanding the files quickly. Asami storage was not designed for that use-case. The larger the transaction, the less overhead. Asami's storage is based on Mulgara (though with a number of improvements), and Mulgara had similar performance characteristics re: small transactions. One user reported that they were loading billions of triples, and so they didn't lose their work they were breaking into chunks and loading each of those in their own transactions. This approach was terrible for large data sets, as it led to wide scale duplication, over and over. Then again, having checkpoints is valuable too. Ideally, Asami likes working with large transactions, as this minimizes overhead and duplication. The fixed-size blocks contain slack-space, and this only contributes to the overhead. (You can probably tell what I want to do with the new index, just based on what I've described here)

That for writing

Multiple readers work quite well. All of the files are memory mapped in 2GB partitions (this is a JVM limit). If a process needs an area of the file that hasn't been mapped, then the memory mapping gets automatically expanded. Access to all data is then through MappedByteBuffers. This offloads disk caching to the operating system. Given that transacted data is all read-only, then this means that multiple readers (even multiple readers across multiple applications) can all access the same memory pages without contention. The operating system loads the pages, and anyone asking for data will get an instance of a defrecord that is backed by the byte buffer, with no copying.

Multiple writers will also work, though they have to be serialized. (Theoretically, writers could operate in parallel, creating git-like branches, but then you'd end up with lots of branches in your database, and the semantics of merging make me balk). To enforce the serializing, I've put a file-lock on the index when writing. I have not measured the performance characteristics of doing that, and I suspect it could have an impact on lots of high speed transactions. Then again, every commit forces data to disk, so it's probably negligible.

Oh, that's a writing gotcha… finishing a transaction forces data to disk. The on-disk index is not updated until it's all there. This maintains on-disk consistency, but it also means that you get NO benefit to having lots of RAM if you are doing lots of small transactions. They get pushed to disk, and that is necessarily slow. This is part of the OLAP approach of Asami: consistency is more important than speed.

Thanks for the detailed breakdown. Ok so better for large ingests of data. Is there a batch size limit? Should that user have been loading a billion triples in one go (is that even possible)? What sort of batch size would you recommend in that sort of context? Honestly, single writer is the way to go for speed in a lot of systems I've encountered. I'm used to putting writes on a queue to batch a serialise even with system that can handle concurrent writes. Nice, I was hoping reads would be some sort of memory mapped mechanism that leans on the OS. I've actually been curious to dive in to a clojure/java implementation in practice so I might do some source diving there. Commits forcing to disk might be ok in my context, the consistency is important and it's more likely to be read heavy than write heavy (with the bulk of the data being bulk ingest). I'll have to see how that plays out though in practice. Thanks again for the super detailed answers. I really appreciate it!

I have to head out for the evening, so I can't write after this response (sorry). There is no batch size limit, though whatever mechanism you're using to give triples to Asami may be limited. e.g. it's common for things to end up in vectors, for instance. Well, to be honest, I tried to handle it as streaming, but I never stress tested this to load something larger than RAM in a single transaction. Maybe I need to do that. So it should be unlimited, but I can't guarantee it. I "protected" against multiple writers, but I think doing that would be a bad idea. But multiple readers are an expected use-case. I tried to abstract the memory mapping away, so some of the layering may get clunky. It's open-source, so please feel free to let me know of anything I could be doing better! I'm tempted to allow commits to not require a force-to-disk, as a configuration option. In that case, the disk will be eventually consistent. Or maybe the user could issue an operation to force it. Either way, if the power were suddenly cut, then it's possible to have an index file that refers to blocks that were never written. On the other hand, if such a thing were to happen, there should be a transaction somewhere in the history where the data is valid. Either way, a user may want the opportunity to sacrifice guaranteed consistency for performance.

I've got plenty go on now. Thanks again for the thorough answers. I'll let you know how it goes. 🙂