Fork me on GitHub
#clojure
<
2020-05-06
>
Setzer2210:05:10

Hi! I'm trying to debug a weird memory issue when running clojure inside Docker containers. I have multiple clojure services (each on a different container, JVM, and so on), set up with sensible memory limits (i.e. ~600MB per process). The thing is, I've recently upgraded my dev machine to 32GB of RAM. And code that worked just fine previously is now crashing with OOM errors! What I mean is that the JVM inside the container reaches its cgroups memory limit, and docker kills it, not that I'm seeing OutOfMemoryError exceptions. I'm using OpenJDK13 and the container-related flags are properly set up using deps.edn:

{:jvm-opts ["-XX:+UseContainerSupport"
             "-XX:InitialRAMPercentage=20"
             "-XX:MaxRAMPercentage=80"]}
What could be causing this? Any ideas? :S

vemv13:05:42

are you 100% certain the flags are being actually applied? ps aux can be a way to check it, JMX being a more hardcore way

Setzer2213:05:28

@U45T93RA6 I thought I'd "made sure" by checking the JVM args from whithin a running repl (with this: https://stackoverflow.com/a/5317220). The flags were there. Could it happen that the flags are ignored despite being parsed by the JVM?

Setzer2213:05:51

Also, what should I look for in ps aux?

Setzer2213:05:54

(btw, thanks!)

vemv13:05:31

Yes, that SO answer describes a JMX way. Then your deps.edn is correct

vemv13:05:16

in ps aux | grep java you an see all the flags/args that were passed to the final Java invocation running your program

vemv16:05:51

Aren't you running multiple JVM instances (one per service), each with "-XX:MaxRAMPercentage=80"? That'd surpass 100% for n>=2

Setzer2222:05:32

@U45T93RA6 yes, that's what I do, but if you set up a memory limit with docker (uses cgroups internally) the JVM will use that limit and treat it as the total RAM, that's why you can set such a high value

👍 4
Setzer2222:05:45

and it seems to work as advertised (at least in some cases?). If any service tried to take an initial portion of 16GB (50%) of the system memory, docker would immediately kill it, but that's not what's happening :S

vemv23:05:51

got it. you might have luck by asking this in stackoverflow as this can be posted as a generic jvm question

Setzer2205:05:29

thanks! I'll try

grounded_sage10:05:38

Is there an example of running clojure using tools-deps from the Clojure image on Docker? https://hub.docker.com/_/clojure

Setzer2210:05:15

I'm not aware of any example, but it's not very different from running it locally. If you get the tools-deps version of the clojure docker image, you basically get a system with the clj command.

grounded_sage10:05:51

Yea I just noticed

grounded_sage10:05:19

I guess a related question is. How does one ensure dependencies are only installed once?

Setzer2210:05:41

there are two ways you can do it

Setzer2210:05:10

the first one will work on any docker version, you need a dockerfile that pretty much does: 1. Copy the deps.edn file (only! not the sources) 2. Run clj , which will fetch the deps declared in deps.edn 3. Copy the source code 4. Builds the project (e.g. uberjar with depstar), runs a REPL, or whatever you need

grounded_sage10:05:36

This is what I have so far

FROM clojure:tools-deps-alpine
RUN mkdir -p /usr/src/app 
WORKDIR /usr/src/app
RUN clojure -A:depstar -m hf.depstar.uberjar MyProject.jar
COPY . /usr/src/app
CMD java -cp MyProject.jar clojure.main -m core 

grounded_sage10:05:53

But I get an error.

Step 1/6 : FROM clojure:tools-deps-alpine
 ---> b8908541da81
Step 2/6 : RUN mkdir -p /usr/src/app
 ---> Running in 0fbcfb5eee31
Removing intermediate container 0fbcfb5eee31
 ---> 91928ecb9d2b
Step 3/6 : WORKDIR /usr/src/app
 ---> Running in 67e70492e8c1
Removing intermediate container 67e70492e8c1
 ---> f087ebf44efd
Step 4/6 : RUN clojure -A:depstar -m hf.depstar.uberjar MyProject.jar
 ---> Running in 9417b55e83cd
Error building classpath. Specified aliases are undeclared: [:depstar]

Setzer2210:05:13

With this first method, you create a base image with the dependencies, and that image only needs to be rebuilt if you change your deps.edn file, because it's the only thing you copied. Then changes in the sources will not trigger to redownload the dependencies

Setzer2210:05:17

Hmmm.. from what I can see, your dockerfile never copies a deps.edn file. Even if you have a ~/.clojure/deps.edn in your home folder, that won't be available inside the container, so the depstar alias will not be defined

grounded_sage10:05:04

Is it too much to ask for the script for the Dockerfile? 😅 I’m new to this Docker stuff

grounded_sage10:05:12

I’m trying :thinking_face:

FROM clojure:tools-deps-alpine
RUN mkdir -p /usr/src/app 
WORKDIR /usr/src/app
COPY deps.edn /usr/src/app/
COPY src/ /usr/src/app/
RUN clojure -A:depstar -m hf.depstar.uberjar MyProject.jar
COPY . /usr/src/app
CMD java -cp MyProject.jar clojure.main -m core 

Setzer2210:05:16

Something like this. But I can't check this works right now. Our actual Dockerfile is a bit more complex than that and I had to strip it down to a simple version:

FROM clojure:openjdk-13-tools-deps-1.10.1.483-buster AS base

WORKDIR /usr/src/app
COPY deps.edn .

# Download all dependencies
RUN clojure -A:uberjar-deps -A:cider-deps -e '(println "Downloaded all dependencies")'

# Copy the source code
COPY src src
COPY test test

# AOT-compile
RUN mkdir classes
RUN clojure -A:cider-deps -e "(compile 'my.main.namespace)"

# Generate uberjar
# NOTE: It is very important to add any aliases that contain build dependencies in the --aliases flag, not before
RUN clojure -A:uberjar-deps -A:uberjar \
  --main-class my.main.namespace \

FROM openjdk-13-slim-buster as prod

WORKDIR /usr/src/app
COPY --from=builder /usr/src/app/target/app.jar .
CMD ["java", "-XX:InitialRAMPercentage=50", "-XX:MaxRAMPercentage=80", "-jar", "app.jar"]

❤️ 4
grounded_sage10:05:21

Thanks. Will take me some time to reverse engineer what is going on 🙂

Setzer2210:05:05

Yes, it took us a while to get to this 😅. The real one is actually 3x longer (but none of it is relevant to the basic use case). Note that this assumes: • A :cider-deps alias that declares dependencies for cider/nrepl (that's not necessary if you don't need to run a REPL inside the container) • A :uberjar-deps dependency that declares the dependencies for depstar, to avoid redownloading for every build • A :uberjar profile that sets the right main class to point to depstar. I've taken extra care to avoid redownloading dependencies as much as possible, but if you really need to avoid redownloading dependencies in a more fine-grained way you'll need to use buildkit, and particularly what's called a buildkit build cache. It's all "experimental" for now, so if you don't build that often and are fine with redownloading dependencies every time your deps.edn file changes, I recommend this approach

Setzer2210:05:54

Also relevant: There are many gotchas when it comes to running a JVM inside docker (see my previous message). Google that for more info. This Dockerfile script already takes care of the basics by: • Using a recent JDK version, with container support enabled by default • Specifying an Initial and Maximum ram percentage You should also make sure to limit the memory of your container with docker. Although I'm having issues with that right now (see my previous message). So take all this with a grain of salt 😅. It's mostly been working so far.

grounded_sage10:05:51

So :uberjar-deps is for the dependencies I need in production. Can you please explain the :uberjar profile a little more?

grounded_sage10:05:15

That’s fine haha. I really appreciate your input! @U70027S0N Very little people use Docker containers with Clojure so the information available online for it is slimmer than normal haha

Setzer2210:05:16

@U05095F2K yeah, I think it makes more sense to just show you the profiles:

:uberjar-deps {:extra-deps {uberdeps {:mvn/version "0.1.8"}}}
:uberjar {:main-opts ["-m" "uberdeps.uberjar"]}

Setzer2210:05:41

so basically I split uberdeps into two profiles, that's the trick I mentioned to avoid redownloading uberdeps for every build

Setzer2210:05:59

Yeah, It's weird how very few people have tried this. Maybe it's just that people are doing it but not talking about it? :man-shrugging:

grounded_sage10:05:30

Maybe it’s so painful and each setup seems unique to their use case that the pain of simplifying it and putting it into a blog post is too much?

Setzer2211:05:30

Yeah, perhaps.. It's probably my case as well. I like sharing knowledge but I don't really have time to put all this knowledge in a public blog

grounded_sage21:05:01

Everything works well for me except for this.

Step 8/12 : RUN clojure -A:uberjar-deps -A:uberjar   --main-class app.core
 ---> Running in 9363d035db23
[uberdeps] Packaging target/app.jar...
+ classes/**
+ src/**
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See  for further details.
+ org.clojure/clojure 1.10.1
.   org.clojure/core.specs.alpha 0.2.44
.   org.clojure/spec.alpha 0.2.176
[uberdeps] Packaged target/app.jar in 712 ms
Removing intermediate container 9363d035db23
 ---> e272737d87e2
Step 9/12 : FROM openjdk:14-slim-buster as prod
14-slim-buster: Pulling from library/openjdk
54fec2fa59d0: Pull complete 
b7dd01647a92: Pull complete 
e4cd17a79ec4: Pull complete 
10aaecc16bec: Pull complete 
Digest: sha256:4190b9aa1ba25ee0082234db30930bcb37ba3eafafbbb937986be24528d8e6ec
Status: Downloaded newer image for openjdk:14-slim-buster
 ---> 3b9a9e451fca
Step 10/12 : WORKDIR /usr/src/app
 ---> Running in bf3fa1b8dbaa
Removing intermediate container bf3fa1b8dbaa
 ---> e63bd7f4f079
Step 11/12 : COPY --from=builder /usr/src/app/target/app.jar .
invalid from flag value builder: pull access denied for builder, repository does not exist or may require 'docker login': denied: requested access to the resource is denied

grounded_sage00:05:18

I got it working. Ended up reverting to depstar

Shuai Lin15:05:49

somehow uncommon question: how could I def a var for another ns?

🙃 4
Shuai Lin15:05:17

the background is I'm using clojure with spark, and try to let the workers execute commands like (defn foo [x] (* x x)) (this gets foo defined on the worker), so later I can use (spark/map rdd foo)

Shuai Lin15:05:04

but the worker always define foo in clojure.core ns, not my own

bronsa15:05:29

intern

👍 4
Shuai Lin15:05:48

nailed it

(defn spark-eval [ns form]
  (let [evaled-form `(binding [*ns* (find-ns (quote ~ns))]
                       (eval (quote ~form)))]
    (println evaled-form)
    (->> (spark/parallelize @_sc (repeat 1 evaled-form))
         (spark/map eval)
         spark/collect)))

(defmacro defnk [name args & body]
  (let [fdef `(defn ~name ~args ~@body)
        ns (symbol (str *ns*))]
    (spark-eval ns fdef)
    fdef))

noisesmith15:05:40

why (repeat 1 evaled-form) rather than [evaled-form] ?

noisesmith15:05:13

or I guess (list evaled-form) if you need lists to get the right behavior

Shuai Lin15:05:39

Because later it'll be (repeat 10 evaled-form) when I have 10 workers

Shuai Lin15:05:24

spark would distribute it to run on every one of them

Shuai Lin15:05:24

one thing that tricks me is this part

`(binding [*ns* (find-ns (quote ~ns))]
                       (eval (quote ~form)))]
for me it shall be the same with this one because eval and quote cancels each other
`(binding [*ns* (find-ns (quote ~ns))]
                ~form)]
but I find the latter does not work - the ns in effect would still be clojure.core

Shuai Lin15:05:04

so looks like *ns* is only useful if it's set before the evaluator kicks in

noisesmith15:05:27

you could also use something like (intern (quote ns) (quote sym) (eval (quote ~form)))` if you limited yourself to def and passed a symbol to bind to

Shuai Lin15:05:02

yeah, that shall work too

noisesmith16:05:27

yesterday I discovered the clojure.test/assert-expr multimethod - it introduces a new syntax that works inside is is any clojure ide / editor integration smart enough that it could find the implementation of a syntax introduced via this sort of indirect extension without hard-coding each known usage of the pattern?

noisesmith16:05:08

that is, fireplace couldn't find it via [d or [C-d , but I wonder if any environment would be able to do that lookup?

pppaul17:05:13

since it's a multi method, you should be able to query it

noisesmith17:05:10

@pppaul sure, but how would I even get from (is (in? x y)) to (defmethod clojure.test/assert-expr 'in? ...) in the defining ns?

noisesmith17:05:36

without special casing the fact that the is macro uses a multimethod over symbols to create assertions

pppaul17:05:31

you query assert-expr first

noisesmith17:05:54

why would I be looking for assert-expr when what I see is (in? x y)

andy.fingerhut18:05:29

clojure.test/is seems like a fairly special flower to me in this regard. I wrote special handling code for it in Eastwood years ago, I know, and was somewhat surprised how special of a flower it truly is.

noisesmith18:05:20

haha, yeah I understand that now... definitely not a pattern I want to use in my own code or extend...

pppaul17:05:03

sorry, thought you were able to see the "is" in your context

noisesmith17:05:23

I can, how would I get from is to assert-expr without special casing is?

noisesmith17:05:58

I suspect I can't and this is a limitation of this sort of extension style, but maybe I'm missing something

pppaul17:05:25

in the macro expand I only see do-report

pppaul17:05:02

I think a special case is needed

noisesmith17:05:08

yeah, somewhere down the stack assert-expr is used, but it's not easy to find / understand

noisesmith17:05:27

so I think I should avoid this style of exension, as it obfuscates things

pppaul17:05:52

personally, I've found the is multi method to be less useful that making macros that produce is statements

Jakub Holý (HolyJak)18:05:39

Hello folks! Anyone experienced with transit? I would like to get it to send change floats to bigdecimals before sending them (the JS side handles big decimals just fine but floats are displayed as [TaggedValue: f, <value>] which is not optimal. Thanks!

noisesmith18:05:53

transit write takes an optional argument mapping types to writers, and the reader takes an optional argument mapping tags to construction functions

Jakub Holý (HolyJak)18:05:32

I see it also takes this option: :transform - a function of one argument that will transform values before they are written.

grounded_sage21:05:10

I’m trying to build a jar with uberdeps and getting stuff here when I try and run it.

Exception in thread "main" java.lang.NoClassDefFoundError: clojure/lang/Var
	at app.core.<clinit>(Unknown Source)
Caused by: java.lang.ClassNotFoundException: clojure.lang.Var
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:602)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
	... 1 more

Alex Miller (Clojure team)21:05:54

some info you will probably need to provide for anyone to help - how did you build it? how are your running it? what's in the jar?

grounded_sage21:05:35

Fair point. deps.edn

{:deps {}
 :paths ["src" "classes"]
 :aliases {:uberjar-deps {:extra-deps {uberdeps {:mvn/version "0.1.8"}}}
           :uberjar {:main-opts ["-m" "uberdeps.uberjar"]}}}
src/app/core.clj
(ns app.core
  (:gen-class))

(defn -main [& args]
  (println "Hello world"))
command line
clojure -A:uberjar-deps -e '(println "Downloaded all dependencies")'
mkdir classes
clojure -e "(compile 'app.core)"
clojure -A:uberjar-deps -A:uberjar --main-class app.core
java -jar target/my-project.jar 

Alex Miller (Clojure team)21:05:12

if you jar -tf target/my-project.jar | grep Var do you see anything?

grounded_sage21:05:01

Ah I seem to have fixed it. Needed Clojure as an explicit dependency

grounded_sage21:05:13

I only realised after pasting in the code blocks lol

Alex Miller (Clojure team)21:05:24

it should be a dependency already via the install deps.edn

Alex Miller (Clojure team)21:05:39

so that implies to me that uberdeps is not doing the right thing

grounded_sage21:05:47

Okay.. yea I usually ignore it.

Alex Miller (Clojure team)21:05:37

I looked at the code and it's ignoring the install deps.edn and user deps.edn, so that's a bug in uberdeps imo

dominicm21:05:12

Install: agree. User: disagree. Building an uberjar is production-sensitive and should ignore the user deps.edn.

Alex Miller (Clojure team)21:05:20

well, that's uberdeps' decision to make. it's currently manuallly merging a subset of install deps.edn

👍 4
seancorfield22:05:42

Yet another case where a tools.deps.alpha-based tool doesn't follow the "expected" behavior regarding the install/user/project deps.edn files 😐 It's why several of those tools drive me crazy!

grounded_sage00:05:43

I decided to go with depstar. Managed to get it working with a docker image build. Though I am confused with this warning? What does it mean?

Step 6/11 : RUN clj -A:depstar -m hf.depstar.uberjar MyProject.jar
 ---> Running in 394071736dfb
Building uber jar: MyProject.jar
{:warning "clashing jar item", :path "about.html", :strategy :noop}
{:warning "clashing jar item", :path "about.html", :strategy :noop}
{:warning "clashing jar item", :path "about.html", :strategy :noop}

seancorfield01:05:17

@U05095F2K It means exactly what it says: It found about.html in multiple artifacts as it was building the JAR and it ignores the duplicates (`:noop`).

seancorfield01:05:48

If you want more detail on exactly what files it is looking at, you'll need to specify the verbose flag, either -vv or -vvv depending on the level of detail you need. That should show you where it is finding multiple about.html files.

grounded_sage03:05:37

The weird thing is this is just a hello world print. Nothing more.

seancorfield03:05:00

Like I said, run depstar with -vvv and see where those files are coming from.

seancorfield03:05:48

RUN clj -A:depstar -m hf.depstar.uberjar MyProject.jar -vvv should do it.

seancorfield03:05:54

BTW, when you're running the CLI in non-interactive mode, you probably want clojure instead of clj.

seancorfield22:05:59

They should not directly read deps.edn and manipulate it! They should rely on t.d.a. and respect the normal merge order and aliases and so on.

4