Does the kafka-connect module (sink connector) still work in xtdb2? I only see docs in the v1 documentation and the readme on github's build instructions are out of date (specifies building with clj but no deps.edn)
Hey @noonian no, the published v1 module (the one on Confluent Hub, if that's where you're looking?) isn't compatible with v2. We have a basic v2 KC module here: https://github.com/xtdb/xtdb/tree/main/modules/kafka-connect but also an evolution/reworked v2 module here (pending an upstream consolidation at some point): https://github.com/egg-juxt/xtdb-kafka-connect There's also this recent CDC-like TxSink capability (still WIP): https://github.com/xtdb/xtdb/pull/4986 What is it you're hoping to achieve?
Thank you, the egg-juxt one might work for me. We have an existing system with data in mongodb and a number of connectors consuming messages from kafka connect when collections are updated. I want to make a PoC that replicates data to xtdb and was hoping I could run a sink connector to transact into xtdb automatically from our existing topics. The kafka-connect in the xtdb repo was what I was looking at but I wasnt sure how to build it
Cool, thanks for the context. Happy to help as needed, and I'm definitely keen to hear your feedback :) cc @titogarcia
Yes, we'd love to hear any issues/suggestions. The connector is currently being developed further based on the needs of one of our Design Partners, but is designed for broader general use.