Hey - thanks, appreciated!

Crux relies on infinite retention in Kafka 🙂

Historically Kafka hasn't been well suited to this role, but over the past few years this role has been approved as valid usage and I believe new developments have improved support for it. I know Confluent is very keen on this usage too, so there's a lot of hope in the future for tiered storage that makes it slightly cheaper to offload old segments of the log to e.g. much cheaper blob storage

see also https://twitter.com/pyrostoreio

I'm not sure current Confluent Cloud pricing is awesome for infinite data retention... (not a technical issue, and not an issue for on-prem Kafka deployments...)

@U07TTE6RH interesting feedback. I guess for the use-cases Crux is targeting at the moment the general cost of storage (Confluent Cloud or otherwise) is a rounding error. Hopefully someone figures out how to do tiered storage as a service soon. Streamlio Cloud is also heading down this route (though we've not tested Crux with Pulsar yet)

Yes I actually asked because of Pulsar and its abstraction over pure streams + cold data retention. For us that is still one of the most manual thing we can possibly have, especially around event replay. Thanks for answering..