xtdb 2020-08-27 | Slack Archive

Nejc12:08:13

One question regarding eviction and deletion of (old) data. Let's say, that our complete dataset will evolve through time, is there a way to set some kind of TTL for documents which have new versions. The following rule is what I'm looking at:

delete all invalid documents older than <date>. // invalid documents - documents which have new versions with higher valid date

Or maybe something like this:

automatically delete invalid documents which already have <n> new versions

The thing is, that if "schema" is evolving, the storage required can grow fast and we would like to have a mechanism to handle this. Is this something what can we expect from Crux to offer, or should we manually handle this by our own logic?

refset13:08:39

We did some exploration in this space earlier in the year, driven by client requirements. A PR came very close to being merged into master, but we decided against merging in the end, mainly because client requirements changed and we weren't eager to support the functionality (vs all the other important things on the roadmap). So it's definitely possible to do something sensible, and maybe you could resurrect this without too much trouble (or we could help 🙂 ): https://github.com/juxt/crux/pull/727

Nejc13:08:16

Well, this is definitely something worth taking a look into. If we decide to go with Crux, I could resurrect this PR for sure. We would end up coding it anyways sooner or later.

👍 3

refset13:08:25

Cool, I'd be happy to chat sometime about requirements and what we considered and what we didn't yet try

👍 3

Nejc13:08:18

We are currently in the "proof of concept" phase to be able to use Crux for our project. I'm working on this and at the moment looks very promising. The list requirements is still not final. And I'd be glad to discuss them with you once we have the final picture. I'll keep you posted. In the meantime and I may came up with more (dummy) questions.

👌 3

🙂 3

mauricio.szabo21:08:27

Hi, I've been digging into Crux sources because I wanted to understand more about the KV store (I'm trying to see if I can extend it to MongoDB, maybe just for fun 😄), but I keep finding myself quite lost on the source. Can I send my questions here, on a thread? For example, the API for store receive lots of UnsafeBuffer, but I have no idea what do do with their contents, or even if they should be treated as implementation details, etc...

jarohen21:08:47

Hey @mauricio.szabo - yep, definitely happy to answer these kind of questions 🙂 We've recently opened up a crux-dev channel over on our Zulip for people more interested in how Crux works, and how to extend it - https://juxt-oss.zulipchat.com/#narrow/stream/251593-crux-dev (we tend to find Zulip easier than Slack to keep track when there're a few different longer threads going on!)

2020-08-27

Channels