Clojurians Log v2

I would suggest not trying to recover from an OOM scenario. Most code is not designed to gracefully tolerate exceptions being thrown from an allocation and it can leave programs in an undefined state (like the timeout-daemon dying, leading to downstream memory leaks). Instead, I would exit on OOM (like you implemented) and if it becomes a pattern, HeapDump on OOM and analyze the heap in MAT https://eclipse.dev/mat/ I had to do the hardening of Datomic's transaction processing routines to guard against OOMs and it was very time consuming even when scoped to a 50 line chunk of Clojure. I can't imagine doing it for an entire application without that being an upfront requirement from the start.

Thanks for the insight!

The timeout channels are not really garbage collected even if the go-block creating them is already finished. Are you creating very many timeouts for each request, or very long timeouts that would linger and pile up over time?

> The timeout channels are not really garbage collected even if the go-block creating them is already finished. Are you sure about that? Do you have a source for that?

Lets look at the implemenation of the timeout channel: https://github.com/clojure/core.async/blob/master/src/main/clojure/clojure/core/async/impl/timers.clj There are two datastructures that keps time-out channels, an unbounded DelayQueue timeouts-queue (the jdk implementation is clever: it has a priority queue sorted on wallclock timeout (in nanoseconds) in a blocking .take. The implementation looks for the next upcoming element, and sleeps until that element should be delivered. If a new element is added before the one timeing out most recently should be delivered, the waiting thread is canceled and re-scheduled to the newer most recent element to be delivered. The concurrentskiplist timeouts-map acts a lookup on timeout "wall clock time", so if many elements happens to be timing out within the same timeout gap, defined by TIMEOUT_RESOLUTION_MS, all of them have the same timeout chan. The only way* timeout chans are consumed is within timeout-worker (line 43), which is started by the timeout-daemon (line 52) at the creation of the first timeout chan. *There is one implementation detail here: if you close a returned timeout chan using (a/close! the-timeout-chan) all go-blocks with a scheduled timeout with-in the 10ms gap will be closed prematurely as well. see first of Notes in https://clojuredocs.org/clojure.core.async/timeout Core.async is designed to have a maximum of 1024 channel operations per channel "queued up" (which not be confused with channel buffer capactity, which can be more than 1024 items). It is not that hard to happen to initialize more than 1024 go-blocks that happens to be scheduled to the same timeout chan, within the same 10 ms, especially if one uses it as a crude scheduler and backwards-calculcate many items to trigger at, say 12:00. To generate more than 1024 timeout-channel-"take"-s within 10ms (if the timeout is constant) is a bit less likely, but it could probably happen, especially if each request creates several timeout channels. If you want a scheduler functionality with the possiblity to actually remove items, it is possible to use a https://github.com/clojure/data.priority-map where the value (!) is the nanoseconds timeout, and the key is something unique (a counter or uuid) which can be indexed with other data structures, and re-implement the logic of delay queue but make sure it accepts that the item it was scheduled to trigger by was already removed.