Clojurians Log v2
xtdb 2024-06-19
Hello, I'm verifying that my understanding is correct in that in order to deploy a multi node HA setup to a cloud provider, XT2 as of now needs the following at minimum: • Kafka • Object store lke S3, Azure Blob etc • Queuing and notification systems like SQS/SNS, PubSub etc If that's the case its quite a lot more infra, costs and chances of failure compared to the setup in XT1. I'm trying to justify if that would be a good fit for use in a large-ish distributed system which already has its own moving parts and this adds quite a bit of operational overhead to that.
understood, yep 👍 broadly speaking, thinking about the core components: • The tx-log from XT1 has remained relatively unchanged - it no longer needs to be infinite retention, but the infrastructure remains the same. We've discontinued the JDBC tx-log here - even with the highest isolation levels in Postgres, MySQL, etc, there are still issues that mean that they aren't ideal tx-logs for our total ordering requirements. • The doc-store from XT1 has changed purpose a little in the object-store, but is again relatively unchanged in infrastructure. The main difference is that we're storing much larger blobs here rather than individual documents, so again JDBC is less suitable than it was previously, but otherwise it's the same components The queuing and notification are new, yes - these aren't critical to the system, rather an optimisation to save frequent listObjects requests on the object-stores. We're looking into ways to bring this in-house - initially we wanted to delegate this complexity to the cloud-provided products but I'd be surprised if we didn't eventually incorporate it into the nodes, to save this infrastructural complexity, as you say
Thanks a lot for this info! The second aspect of the need for an object store is well understood by me too and it totally makes sense. As well as the rationale for the SNS/SQS. The log store needing Kafka is what I'm more worried about. Being an SRE myself, getting a big thing like Kafka into the mix is quite traumatic (for the lack of a better word 😛) for me, specially for us folks on the operational sides. Could something be done there? Maybe something that could use S3 as the backing store too; i suppose like the datomic transactor to order things? Just some ideas.
I'm imagining a clojure ref like thing as a service using s3 as a backing store 🙂
But probably that's a perf choke point having its own scaling issues; being stateful
yeah, that comes with its own complexities, as you say, and it's a single-point-of-failure
but definitely not ruling anything out on that front
just really that Kafka's easiest for us to get going with and it's something that a lot of folks already have access to (even if they're not the biggest fans 🙂)
dont get me wrong, kafka is possibly the best thing to use but equally worse from the other side of operations and cost. even if its managed.
once we have a working database released and clients using it I'm envisaging us having more time to improve things like this, fwiw - just that there's an awful lot in even getting to that stage 😅
Kafka is definitely a battle we've elected not to pick in that phase 🙂
yeah can totally empathaise. will get along with the single node disk setup for now but will go all in with some simpler infra!
aside: totally a fan of XTQL, just that this is a tad bit more pressing for me 😛
> The tx-log from XT1 has remained relatively unchanged - it no longer needs to be infinite retention, but the infrastructure remains the same. We've discontinued the JDBC tx-log here - even with the highest isolation levels in Postgres, MySQL, etc, there are still issues that mean that they aren't ideal tx-logs for our total ordering requirements. @jarohen is this deprecation something coming up in the new release? We unfortunately have a lot at stake and running multiple JDBC tx-log components all over in production. I know it's all open source, but what is the recommendation for us moving forward?
Hi @richiardiandrea the statement you quoted is exclusively about 2.x, so I can confirm that JDBC components will not be deprecated as part of any future 1.x releases. Note that any migration from 1.x to 2.x will demand broader changes on your end than just the tx-log, but if you have a pressing need/desire to plan a move to 2.x in the near future then it's worth chatting again soon to map out all the considerations & options (and I just saw your messages on Zulip, so let's take it there 🙂)
aware I've linked to this thread from a few others, so: the above is (as of today) no longer the case; we no longer require SQS/SNS etc - we now use a second Kafka topic 🚀