Fork me on GitHub
#aleph
<
2017-02-20
>
stijn08:02:02

does anyone know what might be the reason for:

stijn08:02:04

2017-02-20T08:40:28.953+0000 cantona-api-383746225-whp97 ERROR [aleph.http.server:0] - error in HTTP handler
                               java.lang.Thread.run    Thread.java: 745
                                                ...
           manifold.executor/thread-factory/reify/f   executor.clj:  44
           io.aleph.dirigiste.Executor$Worker$1.run  Executor.java:  62
                                                ...
      aleph.http.server/handle-request/fn/f--auto--     server.clj: 156
                        bidi.vhosts/make-handler/fn     vhosts.clj: 199
                                     bidi.ring/fn/G       ring.clj:  12
                                       yada.bidi/fn       bidi.clj:  75
                        yada.handler/handle-request    handler.clj: 169
yada.handler/handle-request-with-maybe-subresources    handler.clj: 120
                                 clojure.core/apply       core.clj: 648
                                                ...
                            manifold.deferred/chain   deferred.clj: 909
                            manifold.deferred/chain   deferred.clj: 933
                                                ...
                                 clojure.core/apply       core.clj: 641
                                 clojure.core/apply       core.clj: 654
                                                ...
                        manifold.deferred/fn/chain-   deferred.clj: 888
              manifold.deferred.Deferred/onRealized   deferred.clj: 417
                    manifold.deferred/add-listener!   deferred.clj: 262
             manifold.deferred.Deferred/addListener   deferred.clj: 384
                io.aleph.dirigiste.Executor.execute  Executor.java: 332
java.util.concurrent.RejectedExecutionException:

dm308:02:52

your thread pool doesn’t accept new tasks

dm308:02:06

it’s full

stijn08:02:23

ok, I should probably setup a stats logger

stijn08:02:51

it started happening when the server was under heavy load, but it doesn't 'repair' itself when the load lowers

stijn08:02:07

I'm using the default executor

dm308:02:58

I’m not sure what’s the default in your stack but the default Manifold execute-pool allows an Integer/MAX_VALUE number of threads

dm308:02:37

so either the pool you’re using has lower limits, or something was hogging resources for a long time

dm308:02:22

also I’m not sure what’s the behaviour in case the pool can’t create a new Thread - can this only happen due to OOM - then an OOM would’ve been thrown I think

dm308:02:27

also might have happened if the executor was in process of being shut down

dm308:02:38

@stijn could that be the case?

dm308:02:31

I’d bet you got this exception during the shutdown process

stijn08:02:10

I will scan the logs to try and find the root cause

stijn08:02:31

it might be that resources were exhausted

residentsummer09:02:50

well, I’ve seen this too. Do you, by chance, make requests to other services with aleph.http?

stijn09:02:10

no, we've deliberately swapped this out for http-kit because of https://github.com/ztellman/aleph/issues/217

residentsummer09:02:01

Yes, that what I had in mind