Fork me on GitHub
Slack Archive

This page is not created by, affiliated with, or supported by Slack Technologies, Inc.

2023-02-04

Channels

#jackdaw
<
2023-02-04
>
Mathieu Pasquet18:02:36

Hi! I'm trying to write an http -> kafka service. In psuedo-code, for each incomming http request I'm doing this 

(defn send-records! [records]
  (doseq [record records]
    (jc/send! producer record #(async/put! results [%1 %2]))
The idea being that I can start all the sends in my handler thread, and then return that thread to the thread-pool pretty much immeditately. Then, later, I can read from the results chanel and if there are any errors, I can respond to the http request with those errors so the user can try again later. The problem I'm having is that jc/send! is sometimes blocking. If the producer buffer runs out of space (becasue kafka is offline for example), and a user asks to send 50 records, that request will block for 50 * 1m, which is clearly not acceptable. The solution I have in mind is to simply not call send! if the send buffer is full, and instead immediately get back to the user with a 503 or something, but I can't seem to see how to do that. Is there a way to see if the producer send buffer is full?

Mathieu Pasquet18:02:55

So I didn't find a way to check if the buffer is full directly, but I do know the upper bound on how big my messages are. So intead of trying to check if the buffer is full, I just use an atom to keep track of how many messages are in the buffer and if that number * the message size approaches the size of the buffer I avoid calling send!

Paco19:02:07

Soo this wouldn’t be a jackdaw specific thing . I think you would set the specific settings with your supplied Kafka producer config.