Fork me on GitHub

@kwladyka I'd try to restrict docker container running on your local host to 60 MB or something like that and see if it still works. When I tried a simple app I couldn't get it work on less than ~130 MB. JVM process consumes much more than just heap memory (see my SO answer with very useful links: - e.g. during startup Clojure apps tend to load lots of classes and thus Metaspace consumption will be likely at least 20-30 MB. I didn't notice if you mentioned what JDK you're using and if you tune heap size - see XX:MaxRAMPercentage in particular (70% as mentioned by @lukaszkorecki is a good value in my experience) To sum up: to run anything in prod I'd give it at least 256 MB of ram

👍 3

yes, I wanted to try also Metaspace limits, BUT I can’t understand why the same image work on my localhost with 128M and not in google cloud run with 128M or even 256M, it needs 512M


@jumar if this wouldn’t work also in my docker, then this will be very clear. I am not able to recreate this.


You may enable gc logging or profile native memory allocations (Native Memory Tracking might be useful) to understand what's going on during startup. Is it correct that the app won't even finish the startup phase?


Now I remember reading about some issues with Virtual memory size - this isn't usually an issue because virtual memory space can be huge with much of it not being allocated at all; therefore what you're typically interested in is the RSS which should more-or-less correspond to the real memory usage (virtual memory pages in RAM). However, some virtualized environments (maybe docker on Google cloud?) empose limits on the total virtual memory size of a process/container. If that's the case your container might be punished although not really using that much memory. But I have no idea if that's applicable to your case. If it runs with 512 MB it might be something completely different


Yes, it even doesn’t run first println line in main function. So even not finish load.


Resident Set Size - that's what ps and other tools show you


@kwladyka This is a pure speculation but the "virtual memory" thing I was referring to can be caused by "ulimit" - similar to this: The thing is that ulimit can (on linux) limit only total virtual memory address space. Maybe the google cloud machine has some default for this setting? (it's usually unlimited)


If you have ssh access to the machine you could verify that via :

prlimit --pid PID -v


I don’t have ssh access


it works with ulimit for me


What do you mean by "it works with ulimit for me"? What exactly you did and where? Did you try gc logging and Native Memory Tracking as I suggested above? What exactly can you do there and how come you don't have access to that machine? I'm afraid that unless you're going to precisely and concisely describe what do you do and what you're seeing you won't get much help. It might also be better suited for general Java/JVM question on StackOverflow. I'm sort running out of ideas based on the limited information I have.


ulimit -Sv 131072 to run java -jar foo.jar unless I misunderstood ulimit


> What exactly can you do there and how come you don’t have access to that machine? cloud run is very limited. I can only upload docker image. That all what I can do. And observe logs in panel which say nothing in this case.


yeah, I was trying to solve this also with guys in google cloud slack and others and nobody can solve this


One guy said google cloud use gVisor so it can make a difference. I didn’t have time to try this so far.


Alternatively I can also try cloud run Anthos which will add more information maybe


these are conclusions which I got from all kinds of slack / community


because in theory this shouldn’t happen


but for some reason it is happening in cloud run


but I have to find time to make tests mentioned above, so far after a few days of debugging I decided to set 512M. It is cheaper than, fix this :/


You tried to run that ulimit on your local machine? If you tried that on your laptop (especiall if it's running Mac OS X) then it might behave differently. I got this error on my Linux server

root@ubuntu-18:/codescene# ulimit -v 131072
root@ubuntu-18:/codescene# java -jar slack-pom.jar
Error occurred during initialization of VM
Could not reserve enough space for code cache
(it's a simple app - look here:


yes, I was trying this on OS X


That might not have the effect you think it has so it's not really testing anything. Anyway, we would need to know what's the configuration on target machine it doesn't help to compare that with your laptop. And it would likely fail with different error if it was a real problem


What's the java version on used on the server?


here is “hello world” example - doesn’t work other branches work


Dockerfile has a answer about anvironment


I was trying also with v11


with buster and scratch

kwladyka20:06:49 here are examples of changes which make the app work


one guy even described process of deployment by code trying to solve this


this is not full app, but hello world, but probably the issue is the same


Interesting, it might indeed by an issue with Metaspace sizing - given that when you don't load some of the dependencies it seems to work. You can try adding -XX:+PrintFlagsFinal flag and examine the output


I think I was trying but logs didn’t show me that 😞


but maybe I am wrong, I was trying to solve this so long, then I can’t be sure


I mean the level of tired the topic


but it should work with app which is runnable


I will try in 10 minutes


sorry for the format, it is so hard to copy logs from google


hmm MaxMetaSpaceSize looks huge, but I don’t understand why


and even if so it should happen, because it shouldn’t take so much memory and works in docker on localhost


-XX:MaxMetaspaceSize=30M didn’t solve this


Did you try to set -Xmx explicitly (let's say to 64m)?


If you can you could try to run some monitoring tool alongside the java process, e.g. • _*jstat*_ to monitor heap: `jstat -gc &lt;PID&gt; &lt;repeat-interval-ms&gt;` • _*jcmd &lt;PID&gt; <|>*_ - very useful (and verbose) info is generated but it takes a while to create; • _*jcmd &lt;PID&gt; VM.native_memory*_ - assuming you start JVM with `-XX:NativeMemoryTracking=summary` or `-XX:NativeMemoryTracking=detail` • _ps_ to monitor RSS • _pmap_ to get much more details about process memory mappings

👍 3

I will try but this is not so easy, there is no way to log into shell in cloud run. I was trying to run some commands from Java before.


Could you run multiple processes in your docker container?


i can do whatever container can do normally, so AFAIK it is possible but very not recommended


but for test purpose should be fine


Why use XX:MaxRAMPercentage vs let java (new ver. like 11 / 14 / 15) do what it want?


Because otherwise you'll get only 1/4 of RAM as MaxHeap ("Xmx")


Do you mean 1/4 if RAM is default?


What is recommended % as default to start with for clojure?


It's my preference but I think 70% is usually ok - but I haven't used it in environments with such restricted RAM capacity; For me it's always 1+ GB of ram available on the machine.


Do you know how to run my Java app throw some tool which will give me full debug about memory consumption before memory exceed? I mean I want to see something useful. Running app directly gives me nothing, so I could run app by another app (tool) which will log constantly memory usage. But immediately without delay, because memory exceed kill the container. Any idea ?


What is the best solution which you know?