Clojurians Log v2

I realize that I somehow need to create an input stream, and I see stuff about flushing. This is all very new and confusing to me https://upload.wikimedia.org/wikipedia/en/2/2d/Pok%C3%A9mon_Psyduck_art.png

There's an Apache Commons IO class that might help you: .input.ReaderInputStream

Alternatively, if your buffered reader is reading lines, you could always iterate over the readLine method..

Thank you! This gets the output correct, but when I curl the endpoint it doesn't stream the response, I just get it in batch:

(defn streaming-response [request]
  (let [reader (create-bufferedreader)]
    (ReaderInputStream. reader)))

(doseq [line (line-seq reader)]
     (println line))

You could just return (line-seq reader) as the body if you wanted. piped-input-stream allows you to turn something that operates on an OutputStream into an InputStream.

(line-seq reader) still only gives me the output when it's all done, at least when I run it as curl --no-buffer I verified that the reader is giving a streaming response using the doseq code above. Is it worth trying ReaderInputStream or do you think that would have the same issue?

It sounds like something is buffering the output too aggressively. What adapter are you using?

httpkit

I'm not familiar with how that handles streaming output. Let me take a look at the source code.

I could try another adapter if it seems easier - I see jetty suggested for example.

Jetty has the same issue.

Curious. Do you only have access to the BufferedReader? Are you certain that isn't overbuffering?

As in, can you get the BufferedReader to output correctly if you just have a loop printing to the terminal?

(let [reader (create-buffferedreader)]
  (loop []
    (when-let [line (.readLine reader)]
      (println line)
      (flush)
      (recur))))

That snippet works great - streams very nicely.

Even

(doseq [line (line-seq (create-bufferedreader))]
  (println line))

The output when using curl is without newlines. I have no idea why that would be, but then I guess it makes sense that line-seq doesn't stream? If everything is just one line. I'm not sure how that would happen though, since it's fine locally.

Sorry for the delay. Another thing you could try is to use either piped-input-stream or the more modern StreamableResponseBody protocol. The Ring Jetty adapter supports the latter, but httpkit doesn't yet (I intend to submit a PR next month about that.

Let me see if I can give you an example:

(extend-type java.io.BufferedReader
  ring.core.protocols/StreamableResponseBody
  (write-body-to-stream [reader response output-stream]
    (let [writer ( output-stream)]
      (loop []
        (when-let [line (.readLine reader)]
          (.write writer line)
          (.flush writer)
          (recur))))))

Then you can just return the BufferedReader as the response body. The write-body-to-stream method will tell the Ring adapter how to write it.

The (.flush writer) will hopefully force the adapter not to buffer the data, which may be what's causing your issue.

You may want to explicitly set a charset on the writer when writing, as well.

You don't need to apologize for anything - answering my questions is not your job. I appreciate all the help you've given. Second, this now works perfectly. Thank you so much! Super duper kind.

No problem! Glad it worked.