Clojurians Log v2
ring 2024-07-21
Apologies for the really dumb question - I’m exploring implementing load shedding for our reitit / ring app. At the moment all the middleware / handlers are synchronous, but I want to limit simultaneous concurrent requests on a per-client basis. Is it possible to insert an asynchronous middleware somewhere in the chain that can shed load / put requests in separate queue (e.g. for priveleged clients). Or is it logically impossible to mix and match async and sync?
An asynchronous handler is just a function: the part that makes it asynchronous is how the adapter uses it. So any asynchronous middleware would need to be at the top of the chain. You then have the problem of blocking I/O. In a synchronous handler you're going to be blocking the current thread while you wait for database or file results. This is why asynchronous apps generally need to be designed that way through the whole chain. You do have a few options though. You could use virtual threads - I believe there's a few third-party adapters that can do that. Or you could start with an asynchronous adapter and handler, then implement your own threadpool that you could assign on a per-client basis, assuming the client can be determined from the incoming request.
> you could start with an asynchronous adapter and handler, then implement your own threadpool that you could assign on a per-client basis, assuming the client can be determined from the incoming request. thanks @weavejester, yep that sounds like a great approach. We can indeed figure out the client from the incoming request, so it sounds like you’re saying we can have our jetty adapter in async mode, and then the first middleware can be async and figure out whether to shed the request or to queue it, and the ones we want to queue can be put onto our own threadpool 🙂
Right. The initial async handler can assign the thread and return quickly. Something like:
(defn wrap-threadpool [handler]
(fn [request respond raise]
(send-to-client-threadpool
(find-client request)
#(when-some [resp (try
(handler request)
(catch Exception ex
(raise ex)
nil)]
(respond resp)))))Perfect! While you’re here, would you mind expanding on what the “async-timeout” is measured from and when the “async-timeout-handler” is called? It seems like an important part of the equation and I haven’t quite grokked it
It's the time it takes between receiving the request and getting the response. Say the timeout was 60s - after giving the request over to the async handler, it would wait 60s for a response before giving up and returning the async timeout handler instead.
Got it, thanks so much for your help, and grateful for all your work on ring!!
No problem!
@mourjo I’m following (and enjoying very much, thank you) your guide to load shedding in clojure. I’m wondering how important is the extra key that needed the ring fork. Will the async-timeout and the bounded queue be useful on their own?
So if I understand correctly, there are two queues 1. the jetty queue, which hold requests which wait to be handed off by the async handler. 2. Then the internal queue, connected to a threadpool, which is where the handler implementation is called Tracking the request time of ingestion allows the async handler to reject requests that have been in (1) for too long. The fact that (2) is bounded is what does the concurrency control. Is that the right model?
Yes that is correct 👍
Hi Daniel, thanks for the the feedback and for reading the article on load shedding 🙇 To answer your question, the Ring fork was required at the time of writing the post. The async timeout handler was merged into the original repository, but the timeout value was still left as a potential future improvement. https://github.com/ring-clojure/ring/pull/410 The bounded queue is not related to ring specifically, it is only used to pass async server requests to a pool of worker threads.
Might also be useful to check this section in the readme https://github.com/mourjo/procrustes?tab=readme-ov-file#strategy-for-load-shedding