Fork me on GitHub
#xtdb
<
2021-02-21
>
Luis Santos16:02:29

Hi everyone, I had look at the code and PRs and found some efforts about compacting the log locally but nothing about the main storage. I also read about the idea of creating some compaction policies and get "old" transactions expired. Whats the current state of transaction log compaction both locally and on the main storage? Thanks.

refset23:02:34

Hi Luis 🙂 we don't have any official support for compaction today, but you've correctly spotted that we did do some work last year to prove that truncating the transaction-time history within the node indexes is a viable strategy should that ever be an essential requirement for some user. Removing transactions from the transaction log is not possible today unless you are prepared to never upgrade Crux in future, since some Crux upgrades will always require a full re-index that requires replaying the entire log. What kind of problem are you hoping to solve with compaction? e.g. Are you working with limited hardware / budget? Or a very large data set?

Luis Santos21:02:46

Hi @U899JBRPF , I don't have a problem at the moment. I've been reading and playing with Crux as I'm trying to understand its features, possibilities and limitations. Some of the areas that I have a lot of questions are different deployments and the architecture styles. One could use Crux in traditional client-server architecture using the HTTP API or instead a crux node can deployed as part of your application allowing for very fast local access patterns. The second pattern would not scale well with highly elastic applications as the state would have to be reconstructed every single time the application would come up. These are just some of the ideas I've been thinking about, not real problems. 🙂

refset21:02:01

Ah okay, good to know! On this point: > The second pattern would not scale well with highly elastic applications as the state would have to be reconstructed every single time the application would come up. ...have you seen our checkpointing feature? https://opencrux.com/reference/21.02-1.15.0/checkpointing.html

Luis Santos22:02:34

To be honest I saw it but I didn't pay attention. But It makes a lot of sense. This helps mitigate the problem I described above.

🙂 3
refset22:02:40

Great, I feel like the page must get skipped over quite a bit, maybe there's a better place for it to live, or we should emphasise the checkpointing capability elsewhere with signposts :thinking_face:

👍 3