asking anyone, but particularly @tbaldridge — I am simulating http-kit's get call, which will return a promise and which can optionally take a callback function. Based on Rich’s async talk, I did some test async code and was wondering which of the following two versions is superior? My guess is the first one, since it seems as if it would create fewer resources. First, the get call:

(defn http-get
  ([url callback]
   (let [sleep-time (+ 1000 (rand-int 1000))
         prom (promise)]
     (future
      (println (str "\nIn http-get's future -- about to sleep for" sleep-time "ms"))
      (Thread/sleep sleep-time)
      (deliver prom sleep-time)
      (when callback
        (callback sleep-time)))
     prom))
  ([url]
   (http-get url nil)))

it sleeps from 1 to 2 seconds to simulate retrieving data

The first version to call this function is as such:

(defn search
  [& requests]
  (let [result-chan (chan)]
    (run! (fn [req] (http-get req #(put! result-chan %)))
          requests)
    (alt!!
      (timeout 1500) "timed out 1500ms!"
      result-chan ([result] result))))

it creates a channel, then for every request it calls the http-get function and gives it a callback, which simply puts the result on the channel. The goal is to get the first reply, or a 1.5s timeout.

The following version seems like the wrong way to go about it, but it was similar to what was given in Rich’s talk as an example, so I want to understand if this would be a common approach:

(defn http-get-to-chan
  [url]
  (go @(http-get url)))

(defn search-v2
  [& requests]
  (let [result-chan (chan)]
    (run! #(go (>! result-chan (<! (http-get-to-chan %))))
          requests)
    (alt!!
      (timeout 1500) "timed out 1500ms!"
      result-chan ([result] result))))

This version is similar, except that it explicitly uses a go block, and has to create an extra channel in the http-get-to-chan function. This version avoids the callback, but introduces complexity via the extra go blocks. Thoughts on either version?

to be blunt; why are you trying to avoid the callback?

there is no reason at all — just wanted to explore options, and perhaps there will be a case where a library does not provide a callback option.

(perhaps it only returns a promise, for example)

One creates one channel for each request, the other creates 3 and just shuffles data around

But more concerning is that the second approach performs a blocking call inside a callback

and go blocks use a fixed sized thread pool.

you mean the http-get-to-chan yes?

yeah, the deref is blocking

this is the slide from Rich’s talk:

https://www.infoq.com/presentations/clojure-core-async at 41:16

yeah, sadly that's a converted example from Go...and go is really wonky

so instead of doing (go …) there, I would do (thread @blocking-call)?

you could, but a better approach is to have each call to your webserver accept a channel

So you can do :

(def c (chan)) (doseq [request requests] (http-call request c)) (alt!! c ([result] result) (timeout 1500) ...)

bleh that looks ugly, but you get the point

but that will block some puts, so make the creation of the channel be (def c (chan (count requests)))

so that there's enough space for all the results

ok — this is almost exactly what I did in the first search version given above, except that instead of taking a callback which uses a closure to capture the channel, it takes a channel itself — right?

right

both are valid though, it all depends on what interface you're given, and how you want to wrap it up.

ok excellent — thanks for the feedback @tbaldridge — I have to become familiar with clojure async idioms, and this was very helpful

@tbaldridge @joshjones Coming into and working with Core.async I can't tell you how much time I lost eventually just to learn that you should never do a blocking call inside of a go block.

It might be a good idea to put somewhere in the docs that you can seize up the threadpool by doing so

because at first it appears that core.aysnc == clojure-flavored go

and the above linked talk seems to be misleading

It's in the docs though: https://clojuredocs.org/clojure.core.async/go

How would you approach working with core async channels if you need two way communication

I am working with kafka, and they have producers and consumers for streams of data and I am trying to model them as channels

But consumers can dynamically subscribe to various streams so I was thinking about having a commands queue that I can use to talk to the consumer

then when the consumer polls the broker and gets the latest data it would put it onto a output channel that someone else could consume

Am I going down the right path, or should I have one two way channel? Or should I be approaching this completely differently?

So why do you need two way communication? One channel for incoming messages, one for outgoing

but don't "ack" messages until they are really done processing, that's just asking for trouble

I would treat the consumer as something like a mult, a think that you can tap for channels that map to streams, not as a single channel