Fork me on GitHub

I have a small problem I want to tackle with clojure.async. I have an API I need to get resources from, but the results are paginated and I can only get the next page after receiving a url from the previous page. I want to get each page, then merge the results with the last page into a map.

(defn get-data-async
  (let [in-c (async/chan)
        out-c (async/chan)]
    ;; Blocking HTTP gets are run in a thread, putting results into in-c as they come in
     (loop [current-req req]
       (if (nil? current-req)
         (async/close! in-c)
         (let [result (get-result current-req)]
           (async/>!! in-c result)
           (recur (next-req result))))))
    ;; Process data as it arrives into a single map; result is put into out-c
    (async/go-loop [result (async/<! in-c)
                    collector nil]
      (if (nil? result)
        (async/>! out-c collector)
        (recur (async/<! in-c) (merge-result collector result))))
Was hoping I could get input on how to make this better. Appreciate any help!

Jakub Holý (HolyJak)17:12:18

Why sending :stop instead of closing the channel, which is the standard way to signal end? While reading <!! If nil you know you should send the collected result.

👍 4

@U0522TWDA Thanks, fixing my code to close channel instead


Take unfold from rewrite it to produce items on a channel instead of a seq and to treat the generating function as returning a channel

🆒 4

thanks for sharing this!