Clojurians Log v2
missionary 2025-11-30
i'm playing around with missionarifying the https://github.com/gnarroway/hato http library but am running into a curious hanging issue. in the code below:
• request wraps hato's https://github.com/gnarroway/hato?tab=readme-ov-file#async-requests like i've seen done in https://github.com/leonoel/task?tab=readme-ov-file#examples. hato is a wrapper for jdk's HttpClient which https://stackoverflow.com/a/55258556. i've added some logging to better see when the continuations are called.
• pull executes a number of requests concurrently and collects their results or rethrows any exceptions (which should cancel all concurrent requests). i've added a bunch of logging to see what's going on.
• random-reqs generates a mix of http requests with a 200 or 500 response. note that hato throws an exception for unsuccessful responses like 500. i add sequential ids to the requests for easy tracing.
the problem is that executing (m/? (pull 8 (random-reqs 4 4))) will sometimes correctly throw an exception and sometimes just hang idefinitely (usually with 1 request never having its success or failure continuations called). what could be the issue? is it something that has to do with hato's or HttpClient's internals? did i wrap hato incorrectly? reducing the concurrency to as low as 2 still exhibits the issue. setting it to 1 makes the task correctly throw an exception every time.
for convenience you can easily git clone the https://github.com/hokomo/missionary-lab/blob/1d4e1300aee187c5c481406d232b9c7e112aca46/src/missionary_lab/hato.clj and jack in to play around (ignore the rest of the project).
(ns missionary-lab.hato
(:require
[clojure.tools.logging :as log]
[hato.client :as hato]
[missionary.core :as m])
(:import
java.util.concurrent.CompletableFuture))
(defn request [& {:keys [id] :as req}]
(fn [s! f!]
(let [cf (hato/request
(assoc req :async? true)
#(do (log/info "s!" id) (s! %))
#(do (log/info "f!" id) (f! %)))]
#(.cancel ^CompletableFuture cf true))))
(defn pull [par reqs]
(->> (m/ap
(let [{:keys [id] :as req} (m/?> par (m/seed reqs))]
(log/info "Sending" id req)
(try (m/? (request req))
(log/info "Done" id)
(catch Exception e
(log/info "Crashed" id (type e))
(throw e)))))
(m/reduce conj [])))
(defn random-reqs [ok err]
(shuffle (sequence (comp cat (map-indexed #(assoc %2 :id %1)))
[(repeat ok {:url ""})
(repeat err {:url ""})])))
(comment
(m/? (pull 8 (random-reqs 4 4)))
) 
Thank you for sharing this knowledge. I have not dig this far into the various implementations of CompletableFuture, so I'm glad to learn this is even worse than I thought. I do indeed think tasks and promises are not strictly equivalent. In my understanding, the promise semantics can only be well defined if it's fully decoupled from the process in charge of fulfilling this promise. Under this model, a promise is just a container, there is a natural conversion to a task (complete with eventual value), cancellation just means stop awaiting i.e. de-register the listener. The idea of cancelling a promise (implied - cancelling the process in charge of fulfilling the promise) is an attempt to re-couple producer from consumer, which has never been well defined, and usually implemented on a best-effort basis if implemented at all - as you've seen. The thread abstraction is a bit better in that regard, because it doesn't try to decouple the producer from the consumer. The interruption mechanism has been there since the very beginning, so it's harder to ignore. Also, project loom's focus on structured concurrency gives a strong incentive to implement it correctly.
my pleasure! i agree with everything you've said and your intuition regarding promises matches mine exactly: they're just "thread-safe boxes" with an at-most-once fill policy. as such they're not inherently tied to just one underlying process that might be trying to fill them so it's kind of impossible for cancellation to ever be more than "fill the box with a special cancellation value". a particularly nice producer could always subscribe to its own promise's cancellation and trigger the cancellation of the underlying process (like HttpClient does), but even then it cannot get rid of issues (1) and (2), i.e. it loses the synchronization between the promise being filled and the process terminating
> i assume this is due to HttpClient's better internal handling of thread interruption compared to the whole business of propagating cancellation through futures? not sure.
also just for completeness sake, HttpClient actually https://github.com/AdoptOpenJDK/openjdk-jdk/blob/9d8ad2ed62325bd8d813974d5aa1e031ed8bf8da/src/java.net.http/share/classes/jdk/internal/net/http/HttpClientImpl.java#L540 so there should be no difference there, though it also means that issues (1) and (2) remain regardless of which flavor you use, a consequence of the promise-based internals and something we just have to live with.
when it comes to the curious Can't cancel yet with java.io.IOException: Request cancelled message, immediate cancellation through the synchronous api (such as (((request-hato-sync ok-req) #(log/info :ok %) #(log/info :ko %)))) doesn't usually trigger the message https://github.com/AdoptOpenJDK/openjdk-jdk/blob/9d8ad2ed62325bd8d813974d5aa1e031ed8bf8da/src/java.net.http/share/classes/jdk/internal/net/http/HttpClientImpl.java#L547, but you'll still get it if you delay the cancellation slightly (though not too long), e.g. by spawning the process and quickly calling the canceller on your own in the repl. the message is most likely benign and i'd assume the cancellation does get propagated properly at some point in the future.
ultimately, the issue of the continuations not being called on cancellation in the asynchronous case is i believe an error due to how hato & co. wrap the root future returned by HttpClient. if you were to use HttpClient directly or monkey-patch hato to return the root future, the issue goes away. i'll try and bring it up to the maintainers and see what they think.
hato's behavior is weird regarding cancellation
(((request {:url ""}) prn prn)) ;; future is cancelled, but success is eventually called
(((request {:url ""}) prn prn)) ;; future is cancelled, but neither success nor failure is called 
Oh, that's very strange.
Use this version, which forces the continuation to be called exactly once
(defn request [req]
(fn [s! f!]
(let [race (AtomicBoolean. false)
cf (hato/request (assoc req :async? true)
#(when-not (.getAndSet race true) (s! %))
#(when-not (.getAndSet race true) (f! %)))]
#(when-not (.getAndSet race true)
(.cancel ^CompletableFuture cf true)
(f! (missionary.Cancelled. "Request cancelled."))))))Not great, because the task terminates before the request is actually terminated, without knowing if the request is actually cancelled, but at this point the problem is on hato
in fact, the weird behavior is pre java 16, after the fix it seems more consistent - but you still need to call the continuation manually on cancellation
thanks leo! i've https://github.com/hokomo/missionary-lab/blob/a404892b4608dbee5efd25ee61822082819e1562/src/missionary_lab/http.clj for more testing: https://github.com/babashka/http-client, https://github.com/schmee/java-http-clj and https://github.com/dakrone/clj-http. all but the last seem to have the exact same issue (which is perhaps not too surprising as they're all wrappers around jdk's HttpClient)
indeed, i'm running all of my tests on java 21 and none of the following expressions ever call any of the continuations (contrary to the weird behavior you were seeing above i think)
(((request-hato ok-req) #(log/info :ok %) #(log/info :ko %)))
(((request-hato err-req) #(log/info :ok %) #(log/info :ko %)))
(((request-bb ok-req) #(log/info :ok %) #(log/info :ko %)))
(((request-bb err-req) #(log/info :ok %) #(log/info :ko %)))
(((request-jhc ok-req) #(log/info :ok %) #(log/info :ko %)))
(((request-jhc err-req) #(log/info :ok %) #(log/info :ko %)))in fact, the weird behavior is pre java 16, after the fix it seems more consistent - but you still need to call the continuation manually on cancellationthis will still have the issue you've mentioned above, namely that (1) the task can possibly terminate before the request and (2) the task can terminate with a failure even if the request wasn't successfully cancelled (e.g. it proceeded to complete successfully), right?
Yes, you can still observe a period of time where the task is terminated but the request is still active. However, the cancellation is guaranteed to succeed so this period of time should be reasonably short.
It's probably the best we can get - as far as I know the HttpClient async API doesn't provide a way to get the request state after cancellation
I don't quite understand the scenario (2)
i've done a bit more digging.
i was concerned that the way the 3 HttpClient-based libraries wrap the returned future (https://github.com/gnarroway/hato/blob/08f71bf85c62af819aaac5ea93cf552d8c4d0d1c/src/hato/client.clj#L310, https://github.com/babashka/http-client/blob/d56bc7f86903d09ff3faef1500ad36005dab037f/src/babashka/http_client/internal.clj#L337 and https://github.com/schmee/java-http-clj/blob/72550432db9f146621acabd647db262da865740e/src/java_http_clj/core.clj#L165) might be impeding the cancellation mechanism. however HttpClient's sendAsync introduces the notion of a cancelable future and guarantees that derived futures will propagate the cancellation upstream [https://bugs.openjdk.org/browse/JDK-8252515, https://docs.oracle.com/en/java/javase/21/docs/api/java.net.http/java/net/http/HttpClient.html#sendAsync(java.net.http.HttpRequest,java.net.http.HttpResponse.BodyHandler,java.net.http.HttpResponse.PushPromiseHandler), https://github.com/AdoptOpenJDK/openjdk-jdk/blob/9d8ad2ed62325bd8d813974d5aa1e031ed8bf8da/src/java.net.http/share/classes/jdk/internal/net/http/HttpClientImpl.java#L650, https://github.com/AdoptOpenJDK/openjdk-jdk/blob/9d8ad2ed62325bd8d813974d5aa1e031ed8bf8da/src/java.net.http/share/classes/jdk/internal/net/http/MultiExchange.java#L358], so there should be no issue there.
i then enabled -Djdk.httpclient.HttpClient.log=all and uncovered the following output when immediately cancelling a request such as with (((request-hato ok-req) #(log/info :ok %) #(log/info :ko %))), for any of the 3 HttpClient-based libraries:
Nov 30, 2025 8:34:40 PM jdk.internal.net.http.Exchange checkCancelled
INFO: MISC: Exchange: request [] no impl is set.
Can't cancel yet with java.io.IOException: Request cancelled hato/request) to execute its exceptional handler on cancellation and invoke the failure continuation?! explanations welcome!
finally i decided to https://github.com/hokomo/missionary-lab/blob/0a6a2925622373705aa9e3573fd295e7226f07fa/src/missionary_lab/http.clj#L22 using an m/via with a virtual thread executor. surprisingly, https://github.com/hokomo/missionary-lab/blob/6c2e4019dfece43d136561c3331f96e4fc7f44c8/src/missionary_lab/http.clj#L117 work exactly as expected, even when cancelled immediately as above. i assume this is due to HttpClient's better internal handling of thread interruption compared to the whole business of propagating cancellation through futures? not sure.
this leads me to a question that is perhaps related to one of your past comments on https://old.reddit.com/r/Clojure/comments/k2db8k/leonoelmissionary_a_functional_effect_and/ge0at66/: is wrapping a CompletableFuture api into a missionary task a fundamentally "lossy" operation, in the sense that it gives rise to problems (1) and (2)? in general it seems easier to wrap a cooperatively-interruptible synchronous api (where interruption sets a flag and the process terminates "naturally" via one of its usual code paths, with or without taking the interruption into account) rather than a promisified asynchronous api (where interruption immediately "fills some box" (i.e. exceptionally completes a future) and calls some continuations without regard to the termination of the underlying process)I don't quite understand the scenario (2)i was paraphrasing your statement "without knowing if the request is actually cancelled", though it's possible i misunderstood it. i took it to mean that it's possible to invoke the task's canceller which will unconditionally call the failure continuation, even if e.g. the cancellation ended up being a no-op and the process "just finished" and was about to invoke the success continuation. the near success was effectively "overwritten" by the cancellation.
i would've expected at least the outermost future (the one returned from e.g. hato/request) to execute its exceptional handler on cancellation and invoke the failure continuation?! explanations welcome!>
actually, i forgot for a moment how exceptionally works. when called on a future f1 it returns a new future f2 that will complete with the value of the handler called on f1's exceptional value (once it is available). however, cancelling f2 itself (i.e. completing it exceptionally with a CancellationException) means the handler will never run (as f2 is already complete)!
in other words, is https://github.com/gnarroway/hato/blob/08f71bf85c62af819aaac5ea93cf552d8c4d0d1c/src/hato/client.clj#L310 actually correct? even if futures derived from HttpClient 's sendAsync propagate cancellation, the outermost future returned from that whole expression is the one returned by exceptionally. cancelling it might propagate cancellation upstream, but it will also prevent the exceptional handler from ever running and invoking the failure continuation. would the correct way to write it not be the one below, where we instead return the "root" future?
(let [root (.sendAsync http-client http-request bh)]
(-> root (.thenApply ...) (.exceptionally ...)
root)
request to make that change makes all of my previous tests run correctly. however i don't think it fundamentally changes the situation as we still have issues (1) and (2) as before, right? that is, cancelling the "root" future will still immediately "fill a box" and invoke the exceptionally handler and the failure continuation, regardless of the progress of the underlying process?here are the 2 kinds of traces i'm getting for the smaller sample (m/? (pull 2 (random-reqs 2 2))) . one that hangs:
[nREPL-...] INFO missionary-lab.hato - Sending 1 {:url , :id 1}
[nREPL-...] INFO missionary-lab.hato - Sending 3 {:url , :id 3}
[ForkJoinPool.commonPool-worker-32] INFO missionary-lab.hato - f! 3
[ForkJoinPool.commonPool-worker-32] INFO missionary-lab.hato - Crashed 3 clojure.lang.ExceptionInfo [nREPL-...] INFO missionary-lab.hato - Sending 2 {:url , :id 2}
[nREPL-...] INFO missionary-lab.hato - Sending 0 {:url , :id 0}
[ForkJoinPool.commonPool-worker-41] INFO missionary-lab.hato - s! 0
[ForkJoinPool.commonPool-worker-41] INFO missionary-lab.hato - Done 0
[ForkJoinPool.commonPool-worker-41] INFO missionary-lab.hato - Sending 3 {:url , :id 3}
[ForkJoinPool.commonPool-worker-41] INFO missionary-lab.hato - f! 3
[ForkJoinPool.commonPool-worker-41] INFO missionary-lab.hato - Crashed 3 clojure.lang.ExceptionInfo
[ForkJoinPool.commonPool-worker-32] INFO missionary-lab.hato - f! 2
[ForkJoinPool.commonPool-worker-32] INFO missionary-lab.hato - Crashed 2 java.util.concurrent.CompletionException
Execution error (ExceptionInfo) at hato.middleware/exceptions-response (middleware.clj:153).
status: 500