Fork me on GitHub
#xtdb
<
2021-06-03
>
stathissideris12:06:15

general question: is it OK to store large things (like images and PDF documents) in crux, or is it a bad idea (like for Datomic)?

refset13:06:42

crux-core can generally handle medium sized blobs quite well (there are no hardcoded limitations, anyway), but the performance costs of going through the query engine & indexes means it is probably not suitable for use cases like a media server without some caching in front. Note that the individual modules can bring their own limitations too, for instance Kafka is not very well suited to large document messages (and the default limit is 1MB). We've often found it can be convenient to put ~arbitrary sized things in when prototyping, at least. What kind of performance do you need from it? As ever, it's best to benchmark these things around your own data carefully 🙂

stathissideris14:06:26

I don’t have specific needs yet, I was considering crux for a toy project that would involve some document handling and I was wondering whether I should store the files as they are or whether I should just store the paths pointing to a bucket or something

👍 3
refset15:06:35

if it's just a toy then I'd say give it a go. Although perhaps not if you are looking to deploy on a really tiny instance. We would like to add real support for large file handling eventually

2