Also relevant is that it dies while idle (not being loaded by requests) so this would suggest some thread activity from house-keeping etc although I have no facts to support this

what version of datomic cloud

Compute stack is 8772

Storage is 470 (not sure if this is important)

i would first try upgrading to the latest (8794)

ok. I’ll try that right away. thanks

I presume you mean compute only upgrade?

correct

ok. now on 8794. I’ll load it with requests and will let it sit to see if it happens again. normally takes an hour or so of idle time. I’ll report back here either way

:thumbsup: Also check your system logs when/if you see the behavior

if possible, can you paste the full stack trace of the error you saw previously?>

sure….

{ “Msg”: “Uncaught Exception: unable to create new native thread”, “Ex”: { “Via”: [ { “Type”: “java.lang.OutOfMemoryError”, “Message”: “unable to create new native thread”, “At”: [ “java.lang.Thread”, “start0”, “Thread.java”, -2 ] } ], “Trace”: [ [ “java.lang.Thread”, “start0”, “Thread.java”, -2 ], [ “java.lang.Thread”, “start”, “Thread.java”, 717 ], [ “java.util.concurrent.ThreadPoolExecutor”, “addWorker”, “ThreadPoolExecutor.java”, 957 ], [ “java.util.concurrent.ThreadPoolExecutor”, “processWorkerExit”, “ThreadPoolExecutor.java”, 1025 ], [ “java.util.concurrent.ThreadPoolExecutor”, “runWorker”, “ThreadPoolExecutor.java”, 1167 ], [ “java.util.concurrent.ThreadPoolExecutor$Worker”, “run”, “ThreadPoolExecutor.java”, 624 ], [ “java.lang.Thread”, “run”, “Thread.java”, 748 ] ], “Cause”: “unable to create new native thread” }, “Type”: “Alert”, “Tid”: 18, “Timestamp”: 1567951027119 }

not sure what you mean by “system logs”

cloudwatch logs

for your datomic system

where are you seeing that ^ error?

ah ok. that’s where this stack trace is from

ok

all other events at that time look normal

very unscientifically, it seems to tolerate clj-gatling load a bit better on this new version. I’ll have to wait now to see if it dies. Good timing as gotta cook dinner (NL time) but will check in later

Does your ion webapp do any async work?

anything that might be spawning threads?

yes, it uses http-kit client in async mode to call an ECS service. async calls via pedestal interceptor/handlers. that said, this problem occurred before I was using async mode

also using jarohen/chime to periodically report metrics i.e. cron like. again, prior to using chime, this instability was present

full stack is lacinia-pedestal / pedestal / resolvers making http calls returning a core.async channel (to allow pedestal to park/async)

most api endpoints are sync/blocking but I suspect the http callouts so I focus on those to reproduce the error.

so far, no hang so will keep waiting on it

prior to async http-kit, was using blocking http-kit calls. I think that is using async underneath so there was probably async machinery being used when this originally manifested

that would definitely be my suspicion for where to look; that error generally indicates that the process is creating unbounded numbers of threads and the OS is out of resources to allocate

despite it being called a “memory” error - it is evidently more commonly a thread resource issue

I suspect the same. I don’t have much experience in finding “captured” threads but I’ll start by using a profiler on my localhost and see if I can find anything