Fork me on GitHub
#tools-deps
<
2019-12-12
>
scottlowe17:12:06

Hi there. I’m working on a project that has a dependency on a large repo of test data, which we don’t want comitted in the main project repo. Once retrieved, the directory is 295 MB uncompressed on disk. We are seeing OOMEs when tools.deps is retrieving via tools.gitlibs:

scottlowe17:12:16

Cloning: [email protected]:MyUser/large-test-files.git
Error building classpath. Java heap space
java.lang.OutOfMemoryError: Java heap space
	at org.eclipse.jgit.internal.storage.pack.BinaryDelta.apply(BinaryDelta.java:163)
	at org.eclipse.jgit.internal.storage.pack.BinaryDelta.apply(BinaryDelta.java:118)
	at org.eclipse.jgit.transport.PackParser.resolveDeltas(PackParser.java:697)
	at org.eclipse.jgit.transport.PackParser.resolveDeltas(PackParser.java:673)
	at org.eclipse.jgit.transport.PackParser.resolveDeltas(PackParser.java:636)
	at org.eclipse.jgit.transport.PackParser.processDeltas(PackParser.java:613)
	at org.eclipse.jgit.transport.PackParser.parse(PackParser.java:584)
	at org.eclipse.jgit.internal.storage.file.ObjectDirectoryPackParser.parse(ObjectDirectoryPackParser.java:197)
	at org.eclipse.jgit.transport.PackParser.parse(PackParser.java:529)
	at org.eclipse.jgit.transport.BasePackFetchConnection.receivePack(BasePackFetchConnection.java:776)
	at org.eclipse.jgit.transport.BasePackFetchConnection.doFetch(BasePackFetchConnection.java:372)
	at org.eclipse.jgit.transport.BasePackFetchConnection.fetch(BasePackFetchConnection.java:302)
	at org.eclipse.jgit.transport.BasePackFetchConnection.fetch(BasePackFetchConnection.java:293)
	at org.eclipse.jgit.transport.FetchProcess.fetchObjects(FetchProcess.java:246)
	at org.eclipse.jgit.transport.FetchProcess.executeImp(FetchProcess.java:162)
	at org.eclipse.jgit.transport.FetchProcess.execute(FetchProcess.java:123)
	at org.eclipse.jgit.transport.Transport.fetch(Transport.java:1269)
	at org.eclipse.jgit.api.FetchCommand.call(FetchCommand.java:237)
	at org.eclipse.jgit.api.CloneCommand.fetch(CloneCommand.java:306)
	at org.eclipse.jgit.api.CloneCommand.call(CloneCommand.java:200)
	at org.eclipse.jgit.api.CloneCommand.call(CloneCommand.java:89)
	at clojure.tools.gitlibs.impl$call_with_auth.invokeStatic(impl.clj:49)
	at clojure.tools.gitlibs.impl$call_with_auth.invoke(impl.clj:41)
	at clojure.tools.gitlibs.impl$git_clone_bare.invokeStatic(impl.clj:71)
	at clojure.tools.gitlibs.impl$git_clone_bare.invoke(impl.clj:68)
	at clojure.tools.gitlibs.impl$ensure_git_dir.invokeStatic(impl.clj:110)
	at clojure.tools.gitlibs.impl$ensure_git_dir.invoke(impl.clj:100)
	at clojure.tools.gitlibs$resolve.invokeStatic(gitlibs.clj:33)
	at clojure.tools.gitlibs$resolve.invoke(gitlibs.clj:29)
	at clojure.tools.gitlibs$procure.invokeStatic(gitlibs.clj:47)
	at clojure.tools.gitlibs$procure.invoke(gitlibs.clj:41)
	at clojure.tools.deps.alpha.extensions.git$eval925$fn__927.invoke(git.clj:41)

scottlowe17:12:30

This error never occurs locally on macos dev machines, but when running on a Travis CI Linux VM. Ubuntu Bionic. I note that tools.gitlibs “0.2.64” is using jgit “4.10.0.201712302008-r”.

scottlowe17:12:46

Digging a bit futher it looks like jGit is on v5.x series and tools.gitlibs is on the v4.x series. Notice that some recent releases of jGit have memory leak fixes. Hard to say if this relates directly to our issues, though.

scottlowe17:12:00

Is this a bug that I should create a ticket for? Or are we using tool.deps outside of its parameters of intended use?

scottlowe17:12:51

I’ve tried giving the JVM a good deal of memory, and when that didn’t work, I’ve tried restricting memory to a smaller amount (1.2 GB) so that the JVM can better plan its memory use… but neither approach has worked.

ghadi17:12:37

@scottlowe how specifically did you give the JVM more memory?

scottlowe17:12:54

Hi Ghadi. I used an env var e.g.: JAVA_OPTS=-Xmx1280m. Less than was available on the Linux instance. I wanted to avoid the Linux OOM-Killer… which may or may not have been related.

ghadi17:12:05

clearing up a couple misconceptions: 1) clj launches two JVMs, one to clone the repos, then another for your project. To modify the JVM for the first one, you need to change this line https://github.com/clojure/brew-install/blob/1.10.1/src/main/resources/clojure#L314 in /usr/local/bin/clojure or wherever your script lives 2) JAVA_OPTS isn't a thing, but a convention in some scripts -- not honored by clj

👍 4
4
scottlowe17:12:15

Wow. I didn’t know that. Thanks for the education 🙂

ghadi17:12:02

try bumping up from 256m and your problem may (?) go away -- definitely worth a issue though, right @alexmiller?

scottlowe17:12:31

I will go and try some different memory allocations. Could you just confirm that tools.gitlibs should be able to handle this size of dependency, if conditions are right?

ghadi17:12:18

not sure what you want me to confirm 🙂 your analysis of the issue seemed right

ghadi17:12:09

bumping JGit is on the radar, as well as maybe making that whole subsystem more resilient

dominicm17:12:01

Seems like loading the code into memory would be a mistake somewhere in the pipeline.

dominicm17:12:12

It can be written straight to disk.

ghadi17:12:51

you're making an assumption that it's loading into memory

scottlowe17:12:54

Yeah. I was wondering about that… should be able to stream any amount of data straight through. Thanks. I’ll go away and experiment, then. Will report back either way… but it may take a few days to be sure that the memory tweaks have worked because we don’t see it every time, because of CI dependency caching etc.

ghadi17:12:05

it might be a large git pack file being decompressed

👍 4
Alex Miller (Clojure team)17:12:56

I have some tabs and editors open working on validating the latest versions of jgit, but it's a notch or two down the stack right now. Kind of an interim step towards comparing latest jgit vs shelling out to git for https/ssh access

Alex Miller (Clojure team)17:12:41

generally, I'd expect the jvm heap on the cp-making jvm to grow quite a bit larger, but it depends what jvm and version you're using

scottlowe18:12:12

Ahh. JVM versions... that’s another longer story. In our dev and CI environment we’re currently restricted to using OpenJDK 8 because of a DB dependency. Our Clojure code should be fine running under version 11 and we hope to move to OpenJDK 11 early next year. It’s possible we could use multiple JVM versions in the meantime (to start the DB under a seperate JVM), using a version switcher or something, but that might be asking for trouble as well as more complexity!

scottlowe18:12:13

Thanks for the comments Alex. I had heard about the shelling out approach for future versions of tools.gitlibs, and I’m hopeful about that approach. I look forward to your findings. 🤞