Fork me on GitHub
#core-async
<
2021-03-04
>
pinealan10:03:22

What’s the best way to use clj-http concurrently? I know it supports async out of the box, but it’s callback based and felt a bit clunky. Are there clj-http wrappers that integrates it with core.async API i.e. channels? I feel like it should be quite simple too to write my own wrapper, but I’m new to async clojure so any pointers are welcomed!

jumar10:03:29

What do you want to achieve- it might be as simple as spawning several futures and wait until they are done

pinealan10:03:19

I’m looking to achieve something like JS Promise.all, I have a usecase with hundreds of requests that needs to be fired, and it’s proving to be the bottleneck right now. My idea was that core.async would have such facilities. I might be wrong but where else in clojure should I look?

mccraigmccraig11:03:45

for promise-like things (i.e. single values rather than streams) i generally find promise interfaces easier to use - i've often used funcool/promesa , which is a nice wrapper around java's CompletableFuture (and js/promises in cljs-land) - https://funcool.github.io/promesa/latest/promesa.core.html#var-all mpent/auspex is another option based on CompletableFuture, but i haven't yet used that

pinealan11:03:16

thanks promesa seems to be what I’m looking for

mpenet11:03:34

https://github.com/exoscale/telex works with auspex out of the box fyi

mpenet11:03:52

it's quite easy to parallelise work if that's what you're after

mpenet11:03:55

auspex is quite similar to promesa, promesa attempts to work a bit everywhere (jvm, cljs), auspex is conceptually simpler but under the hood it's using the same java machinery on the jvm

mpenet11:03:23

(auspex is also designed to make porting manifold code very easy, the api is nearly 1-1)

noisesmith19:03:45

in my experience using put! inside the callback to a clj-http call is all it takes to integrate seamlessly with core.async

noisesmith20:03:17

or perhaps >!! instead, depending on whether you'd rather error on heavy load vs. blocking and waiting it out

noisesmith20:03:13

also re: futures, you do not need that at all, clj-http already manages a threadpool it uses for its requests

noisesmith20:03:00

(and core.async/thread integrates better with go blocks when you do need to spawn threads, since it returns a channel you can park on)

Ben Sless08:03:32

You can do a sort of mix of the put! and >!! approaches which is similar to how pipeline-async handles the async function - allocate a channel of size 1 and put! in it, then synchronize on it in whichever way you want

(defn request
  [req]
  (let [ch (chan 1)]
    (http/request
     req
     (fn [resp]
       (put! ch resp)
       (close! ch)))
    ch))

jumar14:03:53

My point is that people seem to reach to core.async without a good reason. I often find that using plain future or something like claypoole lib is more than enough (simpler, easier, more straightforward) @U051SS2EU > also re: futures, you do not need that at all, clj-http already manages a threadpool it uses for its requests Do you mean the connection thread pool? How that helps in not blocking the client until the request is done without using something like future ? Perhaps you meant the :async / :async? option? https://github.com/dakrone/clj-http#async-http-request (but it seems those provide only callback style APIs)

Ben Sless14:03:45

> people seem to reach to core.async without a good reason. That is true

noisesmith18:03:03

@U06BE1L6T I misunderstood you, and yes, I meant the async option (which would be impossible without the thread pool)

noisesmith18:03:51

I articulated it poorly, but my thought was that you are already using a thread pool for the requests, you don't need thread allocation if you are using that lib, you just need management of the results