Fork me on GitHub
#missionary
<
2021-11-06
>
jfntn18:11:27

👋 Hi everyone, I’m trying to interop with an aysnc http client following the callbacks to tasks examples on the README in the wiki, I ended up with:

;; HTTP Client

(defn request
  [{:keys [async?] :as opts}]
  (let [v (m/dfv)]
    (hc/request
      (cond-> opts
        (nil? async?) (assoc :async? true))
      #(v (fn [] %)) #(v (fn [] (throw %))))
    (m/absolve v)))

(tests
  "get"
  (m/? (request {:url "" :method :get})) :<> {:status 200})

jfntn18:11:34

This works, but I don’t think these are the right semantics. The requests kicks off right away, but I’d like to delay those side effects until the task is ran?

Ben Sless19:11:51

You're running the task when you call m/? on it

Ben Sless19:11:14

Does it also run when you just call (request ,,,)?

Ben Sless19:11:34

Looks like it will, perhaps you should comment on this thread? https://github.com/leonoel/missionary/issues/45

leonoel19:11:37

If you don't need cancellation, you can do that :

(defn request
  [{:keys [async?] :as opts}]
  (fn [s f]
    (hc/request
      (cond-> opts
        (nil? async?) (assoc :async? true))
      s f) #()))

leonoel19:11:49

which client is it ?

jfntn19:11:16

@U053XQP4S thank you that works! Will need to mull over this #() though…

jfntn19:11:28

Is it possible to support cancellation here?

leonoel19:11:39

here's an alternative

(defn request
  [{:keys [async?] :as opts}]
  (m/sp
    (let [v (m/dfv)]
      ((m/? (hc/request
              (cond-> opts
                (nil? async?) (assoc :async? true))
              #(v (fn [] %)) #(v (fn [] (throw %)))))))))

jfntn19:11:56

Hmm that one does not seem to work, should it be?

(defn request-leo2
  [{:keys [async?] :as opts}]
  (m/sp
    (let [v (m/dfv)]
      (hc/request
        (cond-> opts
          (nil? async?) (assoc :async? true))
        #(v (fn [] %)) #(v (fn [] (throw %))))
      ((m/? v)))))

leonoel19:11:38

you're right, good catch !

jfntn19:11:00

The synchronous client api seems easier to achieve this:

(defn request2
  [{:keys [async?] :as opts}]
  (m/via m/blk
    (m/?
      (m/sp
        (hc/request opts
          (cond-> opts
            async? (assoc :async? false)))))))
But I’m not sure I understand the resource management tradeoffs? Is it correct that with request2 the sync client method only runs on the missionary blocking pool, but with request the async client decides where the request and callbacks are scheduled?

leonoel19:11:41

it is correct

leonoel19:11:50

but it should be written

(defn request2
  [{:keys [async?] :as opts}]
  (m/via m/blk
    (hc/request opts
      (cond-> opts
        async? (assoc :async? false)))))

👍 1
leonoel19:11:18

don't call blocking code in sp

leonoel19:11:10

async mode of hato relies on futures, therefore you can't have proper cancellation, cf the issue linked by @UK0810AQ2

leonoel19:11:00

the sync mode I'm not sure about, it will depend on how HttpClient reacts to thread interruption

jfntn21:11:35

I’m trying to think about sync vs. async performance from first principles, can someone check my reasoning? I’m assuming it’d be optimal if: • the client runs nothing but i/o operations on m/blk – i.e. bytes in, bytes out • all response processing runs via m/cpu hato by default automatically coerces the response, so this behavior would have to be prevented and manually ran via m/cpu. But if we assume that, isn’t the optimal profile achievable with either apis? • async: use m/blk as the executor when creating the client instance, then create tasks with the m/dfv wrapper above • sync: just call the blocking method inside (m/via m/blk …)

leonoel21:11:39

HttpClient documentation is not very clear about it, as I understand it the executor parameter is not for doing IO but for running callbacks. • async : use m/cpu as executor, optimal thread usage • sync : use (m/via m/blk) to wrap blocking calls, one thread per request, beware of unbounded parallelism

Ben Sless06:11:33

you could invest some effort in turning hato inside out, separate the actual http request from everything else, then wrap those as tasks