Clojurians Log v2

> a local disk version would work for a single machine deployment so we definitely already have this local-only, filesystem-backend, single-node option, but it's mainly intended for development as we assume most people want HA

ok, so even if a local filesystem is diligently backed up, it isn’t considered production grade?

in reality we could definitely support that setup as production grade, because the code should be production-worthy regardless**, it's more that so far it simply hasn't yet become a priority to consider & review seriously. Always happy to have the conversations though if you have an XT-shaped problem that needs solving 🙂 **although we would definitely want to double-check our fsync's etc. before documenting the option and giving it the green light to deploy, so please talk to us first

roger that

My experience as an SRE, I would be very hesitant to deploy something that’s not HA into production.

Would you consider a S3 compatible locally hosted solution? I've heard good things about https://github.com/seaweedfs/seaweedfs

(coupled with your own kafka cluster)

There’s also a dependency on SNS and SQS as of now. But that would go away as I understand

> Would you consider a S3 compatible locally hosted solution? yes, as long as no code changes are required it should be okay to support 👍

also had a possibly crazy idea, what about something like hazelcast which provides clustered java collections

I had a somewhat related question sometime back: https://clojurians.slack.com/archives/CG3AM2F7V/p1718782037739449 some of the replies are relevant

Hey @tatut 👋 the two main components we need are the tx-log and the object-store - we've implemented these atop each of the cloud providers' services, but the idea (same as for XT1) is that they're also open to extension to whatever you have available on-prem. If you'd like, let us know what that might be (feel free to DM if you'd prefer), and we'll see what we can do 🙂

no concrete plans, just curious if xt2 is something to consider for an on-prem case in the future

what’s the “golden stores” equivalent for xt2 that needs to be available for recovery if everything is lost?

both the tx-log and object-store are still golden stores in XT2. the difference to XT1 is that the tx-log is now transient rather than infinite retention - we essentially mirror it (albeit in indexed form) in the obj-store

managed kafka and S3 are hassle free in a cloud environment, but I wouldn’t want to run kafka myself on-prem and operate some S3 work alike on top of it… but a local disk version would work for a single machine deployment

FYI: we've started work on https://github.com/xtdb/xtdb/issues/3631 today, removing the need for the extra SNS/SQS and their equivalents