Is there some timeout for how long a connection can wait in queue until it's processed by a worker thread?
Not a direct answer to your question, but maybe you can use the field :aleph/request-arrived in the request map to see how long the request has been in the queue already and to log it if it reaches something close to the idle-timeout
I was not aware of that, gonna give it a try!
I'm seeing some 502 errors (not logged by my application but by the load balancer in front), all around 4.1sec I was wondering if it might be caused by all the worker threads being busy (which is plausible since I observe this under spiking load) and connections are dropped from the queue after 4 seconds However I couldn't find any configuration option for that in netty or aleph
It would likely be the proxy server timing the connections out
I considered that but there are two issues 1. all configured connection timeouts are significantly longer than 4 seconds 2. the load balancer (aws alb) logs the 4 seconds as "upstream response time" while usually if the server is just down it logs upstream time as -1
If the server is so overloaded that it hasn't gotten to processing the request in 4 seconds the proxy will see that as part of the response time
That's the thing. I have to guess (since the request never reaches my application code where logging is) that the physical connection succeeds but no worker thread is available But as I said configured timeouts are way higher (an order of magnitude) than 4 seconds which is why I'm wondering if it's some netty/aleph default I'm running into here
Yes I have an idle-timeout of 30000 (30sec) and a shutdown-timeout of 5 (5sec), though my understanding is that the latter configures how long it should wait when the server gets .close'd