Fork me on GitHub
#xtdb
<
2021-09-16
>
mac10:09:26

Are there any plans to remove the need to include the same logic vars in all legs of an or clause? They create a lot of noice, and it seems to me that the missing clause [(any? x]) could be auto generated if need be? I might be missing something.

refset10:09:17

Hi, there are no immediate plans to change this behaviour, although I have certainly experienced the same thoughts/questions. The reasoning behind the current behaviour is that it is an accurate implementation of the axioms of Datalog. The [(any? x)] clauses definitely could be inferred/auto-generated, but I imagine there may be occasions during development where, being implicit, that makes things more confusing (despite the reduced line count) - it is probably worth experimenting with something like this in userspace though. Alternatively, an option could be added to the query like :ignore-non-matching-rule-leg-vars true Another thought is that we could add a built-in pred, equivalent to any? but that accepts varargs (which clojure.core/any? sadly doesn't), that way you could at least consolidate everything into a single clause [(any? x y z)]

mac10:09:52

The latter would definitely be a nice bit of sugar. I might just implement my own version of this for now.

🙂 2
refset11:09:55

Cool, I'll be curious to see what you come up with! We should probably create a more permanent issue or GH discussion to track this though - would you like to do the honours? Otherwise I'm happy enough to just briefly copy and paste these notes myself

mac15:09:38

Happy to do it. Done.

🙏 2
❤️ 2
teodorlu13:09:38

Good afternoon 🙂 I'm curious about tradeoffs when choosing storage engine for local indexes. 1. Do you provide docs / discussion / guidelines for choosing storage engine for local indexes? 2. Modules are provided for both RocksDB and LMDB. Are RocksDB/LMDB suitable for different purposes? I'm interested in storage space usage, ingest latency+throughput and read latency+throughput.

refset13:09:50

'Afternoon! We generally recommend RocksDB as the default, for a variety of reasons (stability, support, ingestion performance etc.), unless you really need the fastest possible query performance, in which case you may want to consider LMDB. We don't have any super detailed documentation on the comparison today, unfortunately, but I would always strongly recommend doing some benchmarking against your use-case anyway (and take a look at our bench subproject if you are looking for code to borrow!).

👍 2
teodorlu13:09:44

Thanks for the reply. I'm doing a bit of benchmarking as we speak! RocksDB seems to be quite a bit more space efficient than LMDB. Y axis is the size of the index store on disk in KB. In E1, I've batched inserts into 10,000 PUTs per transactions, in E2 I have one PUT per transaction.

refset15:09:27

> LMDB doesn't have any built in compression support at all. TIL, via http://www.lmdb.tech/bench/inmem/compress/LMDB/ Nice graph 🙂

👍 2
tatut04:09:48

is there some known instability in LMDB?

refset08:09:06

Not as such. It's just a bigger unknown (fewer users, tiny core dev team) even though the code base is technically much smaller & simpler

nivekuil08:09:51

lmdb will almost certainly be substantially more stable for basically the reasons refset listed but it's not as commonly used with XT I gather so it will probably be a worse experience. Though I think the fancy upcoming indexes don't use either so it's probably not worth worrying about?

tatut08:09:46

I've been using it and it has "just worked" though I don't have experience with larger databases or heavier load

👌 2
refset10:09:25

> lmdb will almost certainly be substantially more stable for basically the reasons refset 😄 very fair! > the fancy upcoming indexes don't use either so it's probably not worth worrying about? Kind of, although we can't make promises about the future release plans yet, so anyone looking to put something into prod in the meantime still needs to give this some thought & testing