Fork me on GitHub
kosengan06:02:15 Seems like a potential alternative to Kafka for Crux.. any thoughts?

Toyam Cox07:02:44

I like not needing to stand up kafka......

Toyam Cox07:02:13

But I also like not paying for software as a tiny entity.


It seems pretty neat, but we've not experimented with it ourselves. There's an open question about whether or not idempotence=true is critical for Crux to work correctly (I'm not sure whether Redpanda has implemented that yet) which would require some investigation before I think we could say it was a suitable replacement. The performance boost certainly couldn't hurt :)

✔️ 4

They are almost done removing the zookeeper from Kafka. And with docker it's easy to setup anyway. But just for Crux Kafka might be overkill.


it's still 3+ separate jvms to run the coordinators though. rp is single-binary with a very convenient cli tool. it's gratis as well


Not sure if here is the right place to follow up on this discussion, but I am pretty curious about Redpanda. Jock is also interesting but seems more beta status.. I wonder how stable Redpanda is, what parts of Kafka it can actually replace plus how compatible it will be then. I wonder if you can still run Kafka Streams and Kafka Connect on top of Redpanda.. but I suppose it should work?


Guess here is fine, I did see some issues with consumer groups, but they seem supported, unlike jocko. Once you want to scale you really need/want to use use consumer groups. Redpanda even seems to have a schema registry. Further off-topic but one of my ideas was to build something like Redpanda in Rust, but if it already exists is not that usefull..


writing a scylla/cassandra doc store; so it seems that the id gets passed into the doc record as strings? am I going through id->buffer and then converting the agrona thing into bytes, if I want to store the id,doc pairs both as BLOBs?


(crux.memory/->on-heap (c/->id-buffer "123")) I think is what I want?


they get passed in as crux.codec.Id objects - that should work, yep 🙂


argh, now alia wants me to convert a java.nio.HeapByteBuffer to something nippy can thaw. I just want to pass around 0's and 1's!


actually, crux.memory has an impl for ByteBuffer already so I just used that. It works!

🎉 4

inchoate scylla doc store, barely tested, for anyone interested:

👏 8