Fork me on GitHub

hello cljdocists! On a fresh pull of cljdoc from github, when I run clojure -A:test all tests pass, but then I get RejectedExcutionException from kaocha.

unit:   100% [==================================================] 18/18
Exception in thread "main" java.util.concurrent.RejectedExecutionException: Task [email protected] rejected from [email protected][Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 8]
	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(
	at java.util.concurrent.ThreadPoolExecutor.reject(
	at java.util.concurrent.ThreadPoolExecutor.execute(
	at java.util.concurrent.AbstractExecutorService.submit(
	at clojure.core$future_call.invokeStatic(core.clj:6893)
	at clojure.lang.RestFn.applyTo(
	at clojure.core$apply.invokeStatic(core.clj:657)
	at clojure.core$apply.invoke(core.clj:652)
	at kaocha.plugin.notifier$run_command.invokeStatic(notifier.clj:80)
	at kaocha.plugin.notifier$run_command.invoke(notifier.clj:70)
	at kaocha.plugin.notifier$notifier_post_run_hook.invokeStatic(notifier.clj:113)
	at kaocha.plugin.notifier$notifier_post_run_hook.invoke(notifier.clj:91)
	at clojure.lang.AFn.applyToHelper(
	at clojure.lang.AFn.applyTo(
	at clojure.core$apply.invokeStatic(core.clj:659)
	at clojure.core$apply.invoke(core.clj:652)
	at kaocha.plugin$run_hook_STAR_$fn__1233.invoke(plugin.clj:42)
	at clojure.lang.PersistentVector.reduce(
	at clojure.core$reduce.invokeStatic(core.clj:6747)
	at clojure.core$reduce.invoke(core.clj:6730)
	at kaocha.plugin$run_hook_STAR_.invokeStatic(plugin.clj:40)
	at kaocha.plugin$run_hook_STAR_.doInvoke(plugin.clj:39)
	at clojure.lang.RestFn.invoke(
	at clojure.lang.AFn.applyToHelper(
	at clojure.lang.RestFn.applyTo(
	at clojure.core$apply.invokeStatic(core.clj:663)
	at clojure.core$apply.invoke(core.clj:652)
	at kaocha.plugin$run_hook.invokeStatic(plugin.clj:51)
	at kaocha.plugin$run_hook.doInvoke(plugin.clj:50)
	at clojure.lang.RestFn.invoke(
	at kaocha.api$run$fn__2929.invoke(api.clj:97)
	at clojure.core$with_redefs_fn.invokeStatic(core.clj:7434)
	at clojure.core$with_redefs_fn.invoke(core.clj:7418)
	at kaocha.api$run.invokeStatic(api.clj:88)
	at kaocha.api$run.invoke(api.clj:71)
	at kaocha.runner$run.invokeStatic(runner.clj:118)
	at kaocha.runner$run.invoke(runner.clj:68)
	at kaocha.runner$_main_STAR_.invokeStatic(runner.clj:136)
	at kaocha.runner$_main_STAR_.doInvoke(runner.clj:122)
	at clojure.lang.RestFn.invoke(
	at clojure.lang.AFn.applyToHelper(
	at clojure.lang.RestFn.applyTo(
	at clojure.core$apply.invokeStatic(core.clj:657)
	at clojure.core$apply.invoke(core.clj:652)
	at kaocha.runner$_main.invokeStatic(runner.clj:147)
	at kaocha.runner$_main.doInvoke(runner.clj:145)
	at clojure.lang.RestFn.invoke(
	at clojure.lang.AFn.applyToHelper(
	at clojure.lang.RestFn.applyTo(
	at clojure.lang.Var.applyTo(
	at clojure.core$apply.invokeStatic(core.clj:657)
	at clojure.main$main_opt.invokeStatic(main.clj:317)
	at clojure.main$main_opt.invoke(main.clj:313)
	at clojure.main$main.invokeStatic(main.clj:424)
	at clojure.main$main.doInvoke(main.clj:387)
	at clojure.lang.RestFn.applyTo(
	at clojure.lang.Var.applyTo(
	at clojure.main.main(
If I delete tests.edn (turfs koacha reporter and plugin notifier), and try again, I see no problems:
18 tests, 70 assertions, 0 failures.


I dug up a recent build on circleci and don’t see evidence of this issue:


Any other cljdoc developers seeing this exception?


I think I recently had a similar thing in a different codebase and the issue was that shutdown-agents was called at some point when the process was not about to exit


thanks for the lead, I shall poke around.


yup, there is a call to shutdown-agents in cljdoc tests, which kaocha is just not happy about!


Should be back up. I’ve spent some more time looking into this problem recently and it seems it has little to do with Consul and/or Nomad and instead it’s just the instance that’s crashing completely. It also then fails to restart properly but I guess these are two different problems.


I’m not super well-versed in Linux server admin stuff so definitely winging it a little bit over here 🙂 Looking at the files in /var/crash seems to indicate that the instance runs out of memory but the DigitalOcean monitoring agent pretty consistently reports 50% memory utilization :thinking_face:

[  768.241566] SLUB: Unable to allocate memory on node -1 (gfp=0x20)
[  768.243343]   cache: kmalloc-64(1:65d047087234dff92ea3990405519ae3b3d48b4417448cccced03e009003d1d9), object size: 64, buffer size: 64, default order: 0, min order: 0
[  768.248041]   node 0: slabs: 207, objs: 13248, free: 0
[  809.694213] BUG: unable to handle kernel NULL pointer dereference at 0000000000000010


It’s a CentOS server btw, if anyone has experience debugging this kind of stuff, “welcome to my crib” 😄


This is why we have managed servers 🙂

🙃 4