Fork me on GitHub

Hey folks! Stopped by to say "great work" with Crux and for a question: I was wondering if Crux handles the "after the horizon" storage of transaction and document log. I have to say I have not read the doc in depth but mainly I wanted to know to the data after the Kafka retention period has expired?


Hey - thanks, appreciated!


Crux relies on infinite retention in Kafka 🙂


Historically Kafka hasn't been well suited to this role, but over the past few years this role has been approved as valid usage and I believe new developments have improved support for it. I know Confluent is very keen on this usage too, so there's a lot of hope in the future for tiered storage that makes it slightly cheaper to offload old segments of the log to e.g. much cheaper blob storage


I'm not sure current Confluent Cloud pricing is awesome for infinite data retention... (not a technical issue, and not an issue for on-prem Kafka deployments...)


@U07TTE6RH interesting feedback. I guess for the use-cases Crux is targeting at the moment the general cost of storage (Confluent Cloud or otherwise) is a rounding error. Hopefully someone figures out how to do tiered storage as a service soon. Streamlio Cloud is also heading down this route (though we've not tested Crux with Pulsar yet)


Yes I actually asked because of Pulsar and its abstraction over pure streams + cold data retention. For us that is still one of the most manual thing we can possibly have, especially around event replay. Thanks for answering..

👍 4