Fork me on GitHub
#beginners
<
2020-09-15
>
trung05:09:59

Hi team, I’m trying to make concurrent calls to apis to gather paginated results, I’m thinking of using atom, future and loop for that, my take so far is as follows, but not sure how do I ensure not breaking out of loop before all future calls finish? Any advise please? Thanks

(let [panda-mappings (atom {})
      range 200]
  (loop [offset 0]
    (future (swap! panda-mappings assoc offset "val"))
    (if (> offset range)
      (println @panda-mappings)
      (recur (+ offset 50))))
  (println @panda-mappings))

trung05:09:43

results is

trung05:09:04

I’m trying to get something like this instead…

{100 val, 200 val, 150 val, 250 val, 0 val, 50 val}

dpsutton05:09:12

clojure.core.async has some ideas about how to communicate like this. you could check out clojure.core.async/pipeline-blocking

trung05:09:45

you mean future is not the right tool for this kind of problem?

dpsutton06:09:59

there are many tools for this. but core.async can give you good coordination tools for your problem of waiting until all the requests finish

dpsutton06:09:26

pipeline-blocking will actually use thread underneath it but with async channels to communicate results and pending results

cgrand07:09:29

I second looking at pipeline-blocking. However the way you use swap! and future can’t work. If you want to use futures you must use them to perform the actual fetch and you have to store the future inside the map. Then once all futures started you have to wait.

cgrand07:09:06

(loop [futures {} [offset & offsets] (range 0 250 50)]
  (if offset
    (recur (assoc futures offset (future (DO-FETCH offset))) offsets)
    ; done, let's wait
    (into {} (map (fn [[o f]] [o @f])) futures)))

trung02:09:12

cool @, really like this idiomatic way, thanks. One more question please, when I try doing (map (fn [[k v]] @v)) instead of (map (fn [[k v]] [k @v])) , to get all results without the indices to the map, I get “Vector arg to map conj must be a pair” IllegalArgumentException, I’ve been mucking around without luck, any thoughts?

noisesmith02:09:58

you can only conj a key/value pair to a hash map

noisesmith02:09:47

replace the {} with #{} or [] and it works, if you didn't need the keys