Fork me on GitHub

is it possible to do something like:

(doseq [m messages*]
        (client/send! c/kafka-producer "topic" id h))


what i mean, is it possible to nest a/go ?, is that a bad habit or something wrong in the example above?


it's possible to nest go blocks, although it may have unexpected results if you don't have the right mental model for what a nested go block implies. the above example might be ok, but it depends on whether messages* blocks when you iterate over it. If iterating over messages* does block, then it probably does not belong in a go block.


otherwise, the above example is fine. however, it probably doesn't have much benefit in practice over:

(doseq [m messages*]
        (client/send! c/kafka-producer "topic" id h)))


why not just do the following?

(doseq [m messages*]
 (client/send! c/kafka-producer "topic" id h))


client/send! is asynchronous ?


i assumed it’s not and use a/go to make it async


why do you want to make it asynchronous?


conceptually, a go block that doesn't interact with any channels is just queuing up some work to do on a thread pool. it doesn't seem like running client/send! from a thread pool versus the current thread makes much of a difference unless there something else going on


also, if client/send! can take long enough to effectively matter, it should be inside thread, as putting it in go will clog up the limited core.async threadpool


@U145X0BPE this is a very common error, it behaves fine in unit tests and simple experiments, and has severe consequences under load


also, given that kafka enforces ordering internally, writing to one topic from N concurrent threads will be slower than doing them one at a time


Perfect, thanks guys 🙂

Ben Sless03:10:44

This is Kafka specific but keep in mind compression happens on your thread, so while send is asynchronous, when the buffer is full you'll block for the duration of compression. What I usually do with a producer is put it in a loop inside a real thread which takes the messages from a channel.

👍 6