This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2020-06-24
Channels
- # announcements (12)
- # aws (11)
- # babashka (1)
- # beginners (73)
- # cider (4)
- # clj-kondo (2)
- # cljsrn (4)
- # clojars (2)
- # clojure (68)
- # clojure-europe (8)
- # clojure-nl (5)
- # clojure-spec (6)
- # clojure-sweden (1)
- # clojure-uk (29)
- # clojurescript (41)
- # conjure (22)
- # datomic (33)
- # docker (58)
- # duct (3)
- # emacs (8)
- # events (1)
- # expound (3)
- # figwheel-main (5)
- # fulcro (33)
- # graphql (2)
- # kaocha (2)
- # lambdaisland (39)
- # leiningen (1)
- # nrepl (49)
- # nyc (1)
- # off-topic (77)
- # pathom (1)
- # re-frame (33)
- # reagent (28)
- # reitit (1)
- # rewrite-clj (2)
- # shadow-cljs (195)
- # spacemacs (1)
- # sql (60)
- # tools-deps (13)
- # vim (18)
- # xtdb (46)
@kwladyka I'd try to restrict docker container running on your local host to 60 MB or something like that and see if it still works.
When I tried a simple app I couldn't get it work on less than ~130 MB.
JVM process consumes much more than just heap memory (see my SO answer with very useful links: https://stackoverflow.com/questions/62303368/accounting-for-java-memory-consumption/62307787#62307787) - e.g. during startup Clojure apps tend to load lots of classes and thus Metaspace consumption will be likely at least 20-30 MB.
I didn't notice if you mentioned what JDK you're using and if you tune heap size - see XX:MaxRAMPercentage
in particular (70% as mentioned by @lukaszkorecki is a good value in my experience)
To sum up: to run anything in prod I'd give it at least 256 MB of ram
yes, I wanted to try also Metaspace limits, BUT I can’t understand why the same image work on my localhost with 128M and not in google cloud run with 128M or even 256M, it needs 512M
@jumar if this wouldn’t work also in my docker, then this will be very clear. I am not able to recreate this.
You may enable gc logging or profile native memory allocations (Native Memory Tracking might be useful) to understand what's going on during startup. Is it correct that the app won't even finish the startup phase?
Now I remember reading about some issues with Virtual memory size - this isn't usually an issue because virtual memory space can be huge with much of it not being allocated at all; therefore what you're typically interested in is the RSS which should more-or-less correspond to the real memory usage (virtual memory pages in RAM). However, some virtualized environments (maybe docker on Google cloud?) empose limits on the total virtual memory size of a process/container. If that's the case your container might be punished although not really using that much memory. But I have no idea if that's applicable to your case. If it runs with 512 MB it might be something completely different
Yes, it even doesn’t run first println line in main function. So even not finish load.
@kwladyka This is a pure speculation but the "virtual memory" thing I was referring to can be caused by "ulimit" - similar to this: https://stackoverflow.com/questions/44273737/how-to-use-ulimit-with-java-correctly The thing is that ulimit can (on linux) limit only total virtual memory address space. Maybe the google cloud machine has some default for this setting? (it's usually unlimited)
What do you mean by "it works with ulimit for me"? What exactly you did and where? Did you try gc logging and Native Memory Tracking as I suggested above? What exactly can you do there and how come you don't have access to that machine? I'm afraid that unless you're going to precisely and concisely describe what do you do and what you're seeing you won't get much help. It might also be better suited for general Java/JVM question on StackOverflow. I'm sort running out of ideas based on the limited information I have.
> What exactly can you do there and how come you don’t have access to that machine? cloud run is very limited. I can only upload docker image. That all what I can do. And observe logs in panel which say nothing in this case.
yeah, I was trying to solve this also with guys in google cloud slack and others and nobody can solve this
One guy said google cloud use gVisor
so it can make a difference. I didn’t have time to try this so far.
but I have to find time to make tests mentioned above, so far after a few days of debugging I decided to set 512M. It is cheaper than, fix this :/
You tried to run that ulimit on your local machine? If you tried that on your laptop (especiall if it's running Mac OS X) then it might behave differently. I got this error on my Linux server
root@ubuntu-18:/codescene# ulimit -v 131072
root@ubuntu-18:/codescene# java -jar slack-pom.jar
Error occurred during initialization of VM
Could not reserve enough space for code cache
(it's a simple app - look here: https://github.com/curiousprogrammer-net/slack-pom)That might not have the effect you think it has so it's not really testing anything. Anyway, we would need to know what's the configuration on target machine it doesn't help to compare that with your laptop. And it would likely fail with different error if it was a real problem
here is “hello world” example https://github.com/kwladyka/memtest - doesn’t work other branches work
https://github.com/kwladyka/memtest/pulls here are examples of changes which make the app work
Interesting, it might indeed by an issue with Metaspace sizing - given that when you don't load some of the dependencies it seems to work.
You can try adding -XX:+PrintFlagsFinal
flag and examine the output
and even if so it should happen, because it shouldn’t take so much memory and works in docker on localhost
If you can you could try to run some monitoring tool alongside the java process, e.g. • _*jstat*_ to monitor heap: `jstat -gc <PID> <repeat-interval-ms>` • _*jcmd <PID> <http://VM.info|VM.info>*_ - very useful (and verbose) info is generated but it takes a while to create; • _*jcmd <PID> VM.native_memory*_ - assuming you start JVM with `-XX:NativeMemoryTracking=summary` or `-XX:NativeMemoryTracking=detail` • _ps_ to monitor RSS • _pmap_ to get much more details about process memory mappings
I will try but this is not so easy, there is no way to log into shell in cloud run. I was trying to run some commands from Java before.
i can do whatever container can do normally, so AFAIK it is possible but very not recommended
Why use XX:MaxRAMPercentage
vs let java (new ver. like 11 / 14 / 15) do what it want?
It's my preference but I think 70% is usually ok - but I haven't used it in environments with such restricted RAM capacity; For me it's always 1+ GB of ram available on the machine.
Do you know how to run my Java app throw some tool which will give me full debug about memory consumption before memory exceed? I mean I want to see something useful. Running app directly gives me nothing, so I could run app by another app (tool) which will log constantly memory usage. But immediately without delay, because memory exceed kill the container. Any idea ?