Fork me on GitHub
#tools-deps
<
2019-07-18
>
kenny21:07:10

@seancorfield I updated my CI to openjdk-11 from openjdk-8 and my uberjar step using depstar has started failing with some sort of memory issue. The uberjar is built with clojure -J-Xmx3g -A:uberjar -v and the command exits with Killed and the status code 137, which typically indicates a memory issue. I'm using the latest version 0.2.4. Any thoughts on why depstar would have a memory issue on Java 11?

kenny21:07:24

I am using the -v option for depstar. It's interesting that the build always hangs on this file /home/circleci/.m2/repository/http-kit/http-kit/2.3.0/http-kit-2.3.0.jar for a few seconds right before the Killed message.

ghadi21:07:49

Did you set a max heap limit for depstar

ghadi21:07:23

Did you look up what status 137 means?

ghadi21:07:39

Ah you did set Max heap

kenny21:07:03

It's happening on CI and according to CircleCI that usually means a mem issue: https://circleci.com/docs/2.0/java-oom/#debugging-java-oom-errors

kenny21:07:20

It's strange that it's happening at all. I can't imagine it using close to 3g of mem.

kenny21:07:57

It almost makes me think the -Xmx is getting ignored for some reason.

ghadi21:07:42

Set it to 1GB

kenny21:07:15

Same issue

ghadi21:07:15

I think there was a change where depstar now unzips in memory

kenny21:07:39

Ah, that sounds relevant. I'll try an older version.

kenny21:07:42

It works with 0.1.7:

clojure -J-Xmx3g -Sdeps '{:deps {seancorfield/depstar {:mvn/version "0.1.7"}}}' -m hf.depstar.uberjar MyProject.jar

kenny21:07:40

It's far slower though

kenny21:07:09

The code of depstar is small, should be fairly easy to find the issue. Must have something to do with Java 8->11.

ghadi21:07:06

It was smaller before Sean fixed all my bugs

kenny21:07:07

This doesn't sound relevant but I found this https://bugs.openjdk.java.net/browse/JDK-8222532 which is resolved in Java 11.0.4. I'm running 11.0.3.

kenny21:07:52

It's not obvious how to update to 11.0.4 for some reason. Maybe the debian packages haven't updated or something.

ghadi21:07:56

dunno I use adoptopenjdk

kenny22:07:10

I don't think 11.0.4 is released yet.

kenny22:07:14

It's possible that issue is relevant but it's proving hard to validate given 11.0.4 isn't out yet. That issue may not even be the problem. Going to roll back to Java 8 for now.

ghadi22:07:11

adoptopenjdk has it

kenny22:07:14

There doesn't appear to be a debian package yet. Also couldn't find a command to force update to 11.0.4. All guides seem to suggest using an install path that doesn't let you select your version. I don't see a 11.0.4 version on their docker page yet: https://hub.docker.com/r/adoptopenjdk/openjdk11

seancorfield22:07:20

The change with depstar was from expanding everything out to a temporary folder and then copying it all back into a ZIP (JAR) to just copying everything into a ZIP (JAR) directly -- which is why 0.1.7 is so much slower than 0.2.x. It now uses a zip filesystem directly. I wasn't aware that changed the memory usage but it's certainly possible. All our uberjars are 30-40MB each and we haven't seen any memory issues using the default heap settings running clj...

kenny22:07:52

The uberjar for the service with this issue is 60mb.

kenny22:07:41

It definitely has to do with java 11. I reverted our build and prod images to java 8 and everything builds as expected - no mem issue.

seancorfield22:07:43

The change also served as a solution for Windows-related issues with reserved filenames (so you couldn't run depstar 0.1.7 at all on Windows when you depended on libraries that contained certain filenames!).

seancorfield22:07:22

Good to know. We're on Adopt OpenJDK8 everywhere because some stuff we rely on won't run on JDK11.

kenny22:07:40

We've been using CircleCI's pre-built images for building uberjars. Specifically circleci/clojure:openjdk-11-tools-deps-1.10.0.442 is the one that produces the mem issue. Not sure if they use adopt openjdk or not.

just.sultanov08:07:01

@U083D6HK9 we use the latest versions and the project works on openjdk 8,9,11,12:

http-kit                    {:mvn/version "2.4.0-alpha4"} ;; fixes java 11
javax.xml.bind/jaxb-api     {: mvn / version "2.3.1"}     ;; support java 9+
org.clojure/core.rrb-vector {: mvn / version "0.0.14"}    ;; support java 11+
see more info about http-kit: https://github.com/http-kit/http-kit/releases/tag/v2.4.0-alpha3

kenny15:07:24

@U1EQNSHL4 Not sure I understand what you mean. This issue was about building an uberjar with depstar, not an issue with http-kit.