Fork me on GitHub

I am currently using core async and go-loops to process a queue where a "user" can join the queue, and when I have 10 users in my queue i send an invitation to them and they will respond. I was wondering about an approach in a cluster. Should I rely on something like Kafka or RabbitMq instead? This approach works fine with 1 machine..


Core async is low level (as stated from many different sources) but I need to "know" who is in a channel or not. Right now it's only in-memory on the machine where the jvm runs. I guess it is time to hook up a data-store for this? For the cases where Node A goes down (with xxx users in queue) but I want to be able to get access to them in Node B.


what I've used in the past is a kafka topic where each message has a key that corresponds to a channel


so the kafka / core.async adapter is a simple router


I think manifold can be used to connect network streams to channels in a more elegant way, but I haven't used it in anger


the problem I think you will run in to is, presumably, you want the entire cluster to agree on how users are assigned to groups, and what the membership of each group is. Which implies either some single coordinator or distributed consensus, neither of which I think kafka(rabbitmq definitely doesn't) does.


the off the shelf approach would be to use some database with serializable transactions as the single point of coordination


I guess kafka requires zookeeper these days, so maybe use zookeeper