You may enable gc logging or profile native memory allocations (Native Memory Tracking might be useful) to understand what's going on during startup. Is it correct that the app won't even finish the startup phase?

Now I remember reading about some issues with Virtual memory size - this isn't usually an issue because virtual memory space can be huge with much of it not being allocated at all; therefore what you're typically interested in is the RSS which should more-or-less correspond to the real memory usage (virtual memory pages in RAM). However, some virtualized environments (maybe docker on Google cloud?) empose limits on the total virtual memory size of a process/container. If that's the case your container might be punished although not really using that much memory. But I have no idea if that's applicable to your case. If it runs with 512 MB it might be something completely different

Yes, it even doesn’t run first println line in main function. So even not finish load.

RSS?

Resident Set Size - that's what ps and other tools show you

@kwladyka This is a pure speculation but the "virtual memory" thing I was referring to can be caused by "ulimit" - similar to this: https://stackoverflow.com/questions/44273737/how-to-use-ulimit-with-java-correctly The thing is that ulimit can (on linux) limit only total virtual memory address space. Maybe the google cloud machine has some default for this setting? (it's usually unlimited)

If you have ssh access to the machine you could verify that via :

prlimit --pid PID -v

I don’t have ssh access

it works with ulimit for me

What do you mean by "it works with ulimit for me"? What exactly you did and where? Did you try gc logging and Native Memory Tracking as I suggested above? What exactly can you do there and how come you don't have access to that machine? I'm afraid that unless you're going to precisely and concisely describe what do you do and what you're seeing you won't get much help. It might also be better suited for general Java/JVM question on StackOverflow. I'm sort running out of ideas based on the limited information I have.

ulimit -Sv 131072 to run java -jar foo.jar unless I misunderstood ulimit

> What exactly can you do there and how come you don’t have access to that machine? cloud run is very limited. I can only upload docker image. That all what I can do. And observe logs in panel which say nothing in this case.

yeah, I was trying to solve this also with guys in google cloud slack and others and nobody can solve this

One guy said google cloud use gVisor so it can make a difference. I didn’t have time to try this so far.

Alternatively I can also try cloud run Anthos which will add more information maybe

these are conclusions which I got from all kinds of slack / community

because in theory this shouldn’t happen

but for some reason it is happening in cloud run

but I have to find time to make tests mentioned above, so far after a few days of debugging I decided to set 512M. It is cheaper than, fix this :/

You tried to run that ulimit on your local machine? If you tried that on your laptop (especiall if it's running Mac OS X) then it might behave differently. I got this error on my Linux server

root@ubuntu-18:/codescene# ulimit -v 131072
root@ubuntu-18:/codescene# java -jar slack-pom.jar
Error occurred during initialization of VM
Could not reserve enough space for code cache

yes, I was trying this on OS X

That might not have the effect you think it has so it's not really testing anything. Anyway, we would need to know what's the configuration on target machine it doesn't help to compare that with your laptop. And it would likely fail with different error if it was a real problem

What's the java version on used on the server?

here is “hello world” example https://github.com/kwladyka/memtest - doesn’t work other branches work

Dockerfile has a answer about anvironment

I was trying also with v11

with buster and scratch

https://github.com/kwladyka/memtest/pulls here are examples of changes which make the app work

one guy even described process of deployment by code trying to solve this

this is not full app, but hello world, but probably the issue is the same

Interesting, it might indeed by an issue with Metaspace sizing - given that when you don't load some of the dependencies it seems to work. You can try adding -XX:+PrintFlagsFinal flag and examine the output

I think I was trying but logs didn’t show me that 😞

but maybe I am wrong, I was trying to solve this so long, then I can’t be sure

I mean the level of tired the topic

but it should work with app which is runnable

I will try in 10 minutes

https://pastebin.com/yp0PYtbG

sorry for the format, it is so hard to copy logs from google

hmm MaxMetaSpaceSize looks huge, but I don’t understand why

and even if so it should happen, because it shouldn’t take so much memory and works in docker on localhost

-XX:MaxMetaspaceSize=30M didn’t solve this

Did you try to set -Xmx explicitly (let's say to 64m)?

yes

If you can you could try to run some monitoring tool alongside the java process, e.g. • _*jstat*_ to monitor heap: `jstat -gc &lt;PID&gt; &lt;repeat-interval-ms&gt;` • _*jcmd &lt;PID&gt; <http://VM.info|VM.info>*_ - very useful (and verbose) info is generated but it takes a while to create; • _*jcmd &lt;PID&gt; VM.native_memory*_ - assuming you start JVM with `-XX:NativeMemoryTracking=summary` or `-XX:NativeMemoryTracking=detail` • _ps_ to monitor RSS • _pmap_ to get much more details about process memory mappings

I will try but this is not so easy, there is no way to log into shell in cloud run. I was trying to run some commands from Java before.

Could you run multiple processes in your docker container?

i can do whatever container can do normally, so AFAIK it is possible but very not recommended

but for test purpose should be fine

Because otherwise you'll get only 1/4 of RAM as MaxHeap ("Xmx")

Do you mean 1/4 if RAM is default?

What is recommended % as default to start with for clojure?

It's my preference but I think 70% is usually ok - but I haven't used it in environments with such restricted RAM capacity; For me it's always 1+ GB of ram available on the machine.

Do you know how to run my Java app throw some tool which will give me full debug about memory consumption before memory exceed? I mean I want to see something useful. Running app directly gives me nothing, so I could run app by another app (tool) which will log constantly memory usage. But immediately without delay, because memory exceed kill the container. Any idea ?

What is the best solution which you know?