Shutdown hooks basically only get called in you call System/exit

If your process is being killed by some external thing there is no guarantee they will run

So kubernetes gives it a kill signal, which seems to last some 30-60 seconds

SIGTERM

I have this expression in a -main function of a Clojure web service, which was working with docker and Aws

(.addShutdownHook (Runtime/getRuntime) (Thread. ^Runnable #(stop system))))

Right. How do you stop your docker ?

locally I mostly use docker compose down as I'm often running more than one service. I dont think I've needed to check what AWS does specifically (I dont have access to any system at the moment). I havent tried a hook shutdown on Kubernettes as of yet

So... I tried SSH'ing onto the kubernetes pod and called kill -15 on my process ID ... .i can see the shutdown behaivor actually works as expected

PID TTY      STAT   TIME COMMAND
    1 ?        Ss     0:00 sh -c java -Xms1000m -Xmx4900m  -Ddatomic.metricsCallback=lms.db/stats-handler -Ddatomic.memcachedServers=clj-memcached:80 -jar target/lms.jar
    7 ?        Sl     0:31 java -Xms1000m -Xmx4900m -Ddatomic.metricsCallback=lms.db/stats-handler -Ddatomic.memcachedServers=clj-memcached:80 -jar target/lms.jar

I have a suspiscioun that kubernetes is calling kill -15 on PID = 1

but the shutdown only works on pid = 7

Perhaps use a tool like https://github.com/Yelp/dumb-init which ensures signals get to the relevant processes. dumb-init runs on PID 1 and the clojure app is run as a child and should receive the shutdown signal. Install dumb-init and use the dumb-init command as the ENTRYPOINT command and CMD to pass the java command to start the Clojure service as an argument. dumb-init ensures TERM signals are sent to the Java process and all child processes are cleaned up on shutdown.

ENTRYPOINT ["/usr/bin/dumb-init", "--"]
CMD ["java", "-jar", "/service/practicalli-service.jar"]

To my knowledge nothing goes as deep as Structure and Interpretation of Computer Programs, which many people have adapted to Clojure from Scheme. I enjoyed the perspective in Elements of Clojure but I wouldn't say it goes deep into PL theory

+1 for SICP, it's readily adaptable to Clojure

Not really Clojure related. There are extensions for browsers that let you save the current web page as an eBook. Although I don't know how you'd read it without X - epub also has formatting, it's not just text.

PID TTY      STAT   TIME COMMAND
    1 ?        Ss     0:00 sh -c java -Xms1000m -Xmx4900m  -Ddatomic.metricsCallback=lms.db/stats-handler -Ddatomic.memcachedServers=clj-memcached:80 -jar target/lms.jar
    7 ?        Sl     0:31 java -Xms1000m -Xmx4900m -Ddatomic.metricsCallback=lms.db/stats-handler -Ddatomic.memcachedServers=clj-memcached:80 -jar target/lms.jar

What do you mean by "call"? It seems that that PID 1 is actually the parent process of PID 7, so PID 1 is the thing that executes that java command.

Also, it should've been asked in #off-topic or some other channel - not in #clojure.

the first process is sh and that is forking the java process. maybe you have sh -c 'java -jar ...' as the entrypoint in the image? if so, this is expected.

Yes it's the entrypoint of the image

only because when i try without sh -c i get this error

Message:      failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "java -Xms1000m -Xmx1000m  -Ddatomic.metricsCallback=lms.db/stats-handler -Ddatomic.memcachedServers=clj- │
│ memcached:80 -jar /usr/src/app/target/lms.jar": stat java -Xms1000m -Xmx1000m  -Ddatomic.metricsCallback=lms.db/stats-handler -Ddatomic.memcachedServers=clj-memcached:80 -jar /usr/src/app/target/lms.jar: no such file or directory: unknown    │

can you show what the entry point is in the dockerfile? this seems to be taking the entire string java -jar ... as the excutable and not java and rest as args

command = ["java ${var.settings[terraform.workspace].api.xms} ${var.settings[terraform.workspace].api.xmx}  -Ddatomic.metricsCallback=lms.db/stats-handler -Ddatomic.memcachedServers=clj-memcached:80 -jar /usr/src/app/target/lms.jar"]

I'm not using CMD from dockerfile

i'm using command in kubernetes / terraform

can you try command = ["java", "...", "..."]

ok !

specify each as a list item and not one full string. this is how the standard cmd syntax is

also like its mentioned before, we can handle such question better on #C0PME9N9X for instance 😄

Wow that was totally it

thanks!

id recommend looking into the cmd and entrypoint specs, quite fundamental but confusing too 😅

https://github.com/Yelp/dumb-init is also a useful tool for managing service processes running in Docker, an example can be found in https://practical.li/blog/posts/build-and-run-clojure-with-multistage-dockerfile/

This also happens when you have a shell script wrapper for you java command. You can use exec to make java be the pid 1 but bear in mind that pid 1 has responsibilities like reaping zombies. I like to use tini https://github.com/krallin/tini See also my blog post https://curiousprogrammer.net/posts/2022-04-23-java-in-docker-zombies