Clojurians Log v2

Thanks for the responses! I "fixed" the BLOCKED issue by replacing SimpleDateFormat with DateTimeFormatter - if this turns out to be a good change in testing, I'll submit a PR to aleph for consideration. But, this doesn't fix the CPU spinlock, which I still have not identified. Nothing more to do, but to continue digging and removing variables (it takes many hours to force the issue, but it does seem to be reproducible). 🤞

My guess would be your code is spinning out threads in an uncontrolled manner (without limiting the number of threads, or waiting for them to complete), which is causing memory pressure which is causing the gc to run all the time, which is the source of your high load

Notice the SingleThreadEventExecutor and the thread is BLOCKED on a lock.

There are 2 more threads that are also BLOCKED waiting for the lock release.

Any idea what could cause a lockup on java.util.ListResourceBundle? Is this a filesystem corruption (this is running in Docker and k8s)? A memory issue? (I was initially debugging high thread count, so could this be a symptom of not enough memory to allocate new threads)?

Correcting screenshot:

What about the other threads? I wouldn't expect threads that are BLOCKED to consume a lot of CPU. If you didn't have enough memory to allocate new threads I would expect to see OutOfMemoryError.

yes, that thread is likely blocked for a very short time there, but it happens a lot, every http needs to parse the date string, and it needs some locale information to do it, and that code is loading the locale information and there is a lock somewhere to make sure two threads don't race loading it

you can see the method here https://code.yawk.at/java/14/java.base/java/util/ListResourceBundle.java#191 it is synchronized, which is the lock that is being held, but it immediately returns if lookup is not null