Are you executing them all in their own go blocks?

I have no idea of performance, but if you’re doing lots of requests together, then it reminded me of David’s post from 8 years ago: http://swannodette.github.io/2013/08/02/100000-processes/

That was a follow-on post from one where he described using go blocks for parallel requests: http://swannodette.github.io/2013/07/12/communicating-sequential-processes/

Hi, sort of, initially I read all those values from firebase in just a single go block and take each one with "<p!" but then some of them get passed into another function which does have it's own go block as well.

You're reading in a single request with a big response?

no, no, each item is a call to firebase

If that's the case then I would execute them all together in go blocks, like David showed in those examples. It should reduce your latency

I rewrote the code using channels but it didn't help much performance wise so I ended up taking out go blocks and channels and going with just promises - code ended up being uglier and required me to write data using transactions to avoid overwriting data in firebase but it helped with the performance which was mission critical. I really enjoyed using and learning about core.async however the extra overhead affecting performance was too much for the task but I'll be using it for other things for sure. Perhaps there are some tricks how to optimize and make it faster but I'm still pretty new to clojurescript in general so have yet to learn. Thanks for your help!

Impossible to give much useful feedback in the abstract without some code samples and specifics. core.async should not have significant overhead, esp. when compiled under :advanced I believe

So you are saying if it is compiled under advanced the performance should be faster? I'm using shadow cljs and I don't see any settings for that in my shadow-cljs.edn file, perhaps it happens somewhere behind the scenes when running the production build. I'll see if I can post some code samples because I would rather do this with core.async, the code becomes much easier to reason about.

Ok here it is, I removed some implementation details but it should be enough to give you an idea, sorry about long post. I have a list of couple of thousands ids which need to be shortened and stored in firebase. Shortened version is like a human friendly version of the same, but it must be unique.

(def ids ["someListContainingLong80CharacterStringsContaining2000Total"
          ....
          ....])

(:require
 [cljs.core.async.interop :refer-macros [<p!]])

(go
  (doseq [id (distinct ids)]
    (let [short (<p! (fb/get-long2short id))]
      (when (nil? short)
        (<! (shorten id))))))

(defn shorten [id]
  (go
    (loop [abbr-len 3]
      (let [candidate (subs id 0 abbr-len)
            candidate-id (<p! (fb/get-short2long candidate))]
        (cond
          (empty? candidate-id)
          (do
            (fb/set-short2long candidate id)
            (recur abbr-len))
          (= candidate-id id)
          candidate
          :else
          (recur (inc abbr-len)))))))

(defn get-long2short [id]
  (new js/Promise (fn [resolve _]
                    (.. firebase database (ref (str "long2short/" id))
                        (once "value"
                              (fn success [snapshot]
                                (let [data (-> snapshot .val (js->clj :keywordize-keys true))]
                                  (resolve data))))))))

(defn get-short2long [id]
      ;; same as get-long2short just querying "short2long/" collection
  )

(defn shorten [id]
  (go
    (loop [abbr-len 3]
      (let [candidate (subs id 0 abbr-len)
            c (chan)
            candidate-id (<! (fb/get-short2long candidate c))]
  ...
  )

(put! c data)

I’m on a phone and it's hard to look at right now, but 2 things stand out: • you need to do a round trip for each individual id. There's no way that can be made fast. You either need to figure it out locally, or in batches. • shorten has the potential to make multiple requests per id. I would add a tap to accumulate some stats and see how often re-attempts occur. If hardly ever, then fine. But especially if you run in parallel, then I’d expect that you may see a lot.

Is there anything core.async related that could be slowing things down? When I run this exact same logic with promises alone it finishes in around 10 seconds while the version with core.async (whether using "<p!" in combination with promises or putting values on channels without promises) takes over 200 seconds.

Believe it or not, the whole issue with performance came down putting "go" block inside of "doseq" instead of outside, so this:

(go
  (doseq [id (distinct ids)]
    (let [short (<p! (fb/get-long2short id))]
      (when (nil? short)
        (<! (shorten id))))))

(doseq [id (distinct ids)]
  (go    
    (let [short (<p! (fb/get-long2short id))]
      (when (nil? short)
        (<! (shorten id))))))

The first one is a single go block. Each call to get-long2short waits for a response before proceeding to the next one. The second starts an independent go block for each id. They all make their requests independently at close to the same time, and then collect the responses as they come in. It's the difference between serial and parallel.

You can (update response :body slurp) to get it as a string