Clojurians Log v2

It's possible you're performing actions that pin virtual threads to O/S carrier threads -- that would cause thread starvation and blocking.

e.g., MySQL JDBC driver prior to 9.0.0 or something else that still uses synchronized.

Is there a way we can see what is causing threads to be pinned? We don’t use MySQL JDBC but we do use the Postgres driver

Virtual threads is well-suited to I/O heavy actions, rather than CPU heavy actions. But if your I/O is potentially a bottleneck -- such as a connection pool -- you can overrun it pretty easily with vthreads and you'll get timeouts etc.

Add the JVM option -Djdk.tracePinnedThreads=short (or full) and you'll see if there are pinned threads.

I'm still on the fence about vthreads. I was very excited about the idea of using them, but in practice it has been hard to find places where they would be useful, given the constraints of stuff like connection pools and HTTP throttling on external services.

Ok thanks. And yeah we just wanted to compare the performance in our load tests and see if how it compared.

And you're using Jetty 12 with which adapter?

[ring/ring "1.12.2"] [ring/ring-core "1.12.2"] [ring/ring-jetty-adapter "1.12.2"] [ring/ring-defaults "0.5.0"] [ring-cors "0.1.13"] [org.postgresql/postgresql "42.7.3"]

Ah, so that's Jetty 11 then.

Jetty 12 has a different architecture that might be more conducive to vthreads (with all the caveats above still in play) but I don't think I would want to try running Jetty 11 with a vthread per request...

Ah ok that could explain it then what other adapters could we test out?

This has a Jetty 12 version (despite its name): https://github.com/sunng87/ring-jetty9-adapter

Ok is that what your team uses? Is there any thing else to consider compared to the ring jetty adapter?

We tried it -- we'd been using the sunng87 adapter for quite a while to leverage Jetty 10/11 while Ring was still stuck on Jetty 9 -- but the real obstacle for us was a loss of observability in Jetty 12 as it was not fully supported by New Relic (we lost all "transaction" metrics).

We went back to the standard Ring adapter once it updated to Jetty 11 and had WebSockets built-in (another reason we had previously switched to sunng87 was its WebSocket support).

(New Relic has since improved Jetty 12 support so we may try it again once the default Ring adapter supports it)

Ok thanks I appreciate it. We will test it out and compare

Just remember that your database is probably going to be your bottleneck and load testing a regular thread pool based server will have inherent throttling to avoid overwhelming your DB (with appropriate tuning) but you lose that throttling with virtual threads so the behavior will be different and your load test may fail because of that...