This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
- # adventofcode (67)
- # announcements (8)
- # babashka (46)
- # beginners (154)
- # calva (5)
- # cider (9)
- # clara (5)
- # clj-kondo (34)
- # cljdoc (31)
- # cljsrn (4)
- # clojure (146)
- # clojure-europe (5)
- # clojure-italy (3)
- # clojure-losangeles (2)
- # clojure-nl (149)
- # clojure-spec (22)
- # clojure-uk (73)
- # clojured (6)
- # clojurescript (95)
- # clojureverse-ops (3)
- # cryogen (7)
- # cursive (12)
- # data-science (1)
- # datomic (9)
- # docker (1)
- # emacs (1)
- # figwheel-main (1)
- # hyperfiddle (1)
- # jobs (3)
- # malli (29)
- # nrepl (2)
- # off-topic (61)
- # pathom (6)
- # pedestal (1)
- # planck (1)
- # reitit (19)
- # shadow-cljs (52)
- # spacemacs (5)
- # tools-deps (24)
- # vim (30)
- # yada (6)
Hi there. I’m working on a project that has a dependency on a large repo of test data, which we don’t want comitted in the main project repo.
Once retrieved, the directory is 295 MB uncompressed on disk. We are seeing OOMEs when
tools.deps is retrieving via
Cloning: [email protected]:MyUser/large-test-files.git Error building classpath. Java heap space java.lang.OutOfMemoryError: Java heap space at org.eclipse.jgit.internal.storage.pack.BinaryDelta.apply(BinaryDelta.java:163) at org.eclipse.jgit.internal.storage.pack.BinaryDelta.apply(BinaryDelta.java:118) at org.eclipse.jgit.transport.PackParser.resolveDeltas(PackParser.java:697) at org.eclipse.jgit.transport.PackParser.resolveDeltas(PackParser.java:673) at org.eclipse.jgit.transport.PackParser.resolveDeltas(PackParser.java:636) at org.eclipse.jgit.transport.PackParser.processDeltas(PackParser.java:613) at org.eclipse.jgit.transport.PackParser.parse(PackParser.java:584) at org.eclipse.jgit.internal.storage.file.ObjectDirectoryPackParser.parse(ObjectDirectoryPackParser.java:197) at org.eclipse.jgit.transport.PackParser.parse(PackParser.java:529) at org.eclipse.jgit.transport.BasePackFetchConnection.receivePack(BasePackFetchConnection.java:776) at org.eclipse.jgit.transport.BasePackFetchConnection.doFetch(BasePackFetchConnection.java:372) at org.eclipse.jgit.transport.BasePackFetchConnection.fetch(BasePackFetchConnection.java:302) at org.eclipse.jgit.transport.BasePackFetchConnection.fetch(BasePackFetchConnection.java:293) at org.eclipse.jgit.transport.FetchProcess.fetchObjects(FetchProcess.java:246) at org.eclipse.jgit.transport.FetchProcess.executeImp(FetchProcess.java:162) at org.eclipse.jgit.transport.FetchProcess.execute(FetchProcess.java:123) at org.eclipse.jgit.transport.Transport.fetch(Transport.java:1269) at org.eclipse.jgit.api.FetchCommand.call(FetchCommand.java:237) at org.eclipse.jgit.api.CloneCommand.fetch(CloneCommand.java:306) at org.eclipse.jgit.api.CloneCommand.call(CloneCommand.java:200) at org.eclipse.jgit.api.CloneCommand.call(CloneCommand.java:89) at clojure.tools.gitlibs.impl$call_with_auth.invokeStatic(impl.clj:49) at clojure.tools.gitlibs.impl$call_with_auth.invoke(impl.clj:41) at clojure.tools.gitlibs.impl$git_clone_bare.invokeStatic(impl.clj:71) at clojure.tools.gitlibs.impl$git_clone_bare.invoke(impl.clj:68) at clojure.tools.gitlibs.impl$ensure_git_dir.invokeStatic(impl.clj:110) at clojure.tools.gitlibs.impl$ensure_git_dir.invoke(impl.clj:100) at clojure.tools.gitlibs$resolve.invokeStatic(gitlibs.clj:33) at clojure.tools.gitlibs$resolve.invoke(gitlibs.clj:29) at clojure.tools.gitlibs$procure.invokeStatic(gitlibs.clj:47) at clojure.tools.gitlibs$procure.invoke(gitlibs.clj:41) at clojure.tools.deps.alpha.extensions.git$eval925$fn__927.invoke(git.clj:41)
This error never occurs locally on macos dev machines, but when running on a Travis CI Linux VM. Ubuntu Bionic. I note that tools.gitlibs “0.2.64” is using jgit “18.104.22.168712302008-r”.
Digging a bit futher it looks like jGit is on v5.x series and tools.gitlibs is on the v4.x series. Notice that some recent releases of jGit have memory leak fixes. Hard to say if this relates directly to our issues, though.
Is this a bug that I should create a ticket for? Or are we using
tool.deps outside of its parameters of intended use?
I’ve tried giving the JVM a good deal of memory, and when that didn’t work, I’ve tried restricting memory to a smaller amount (1.2 GB) so that the JVM can better plan its memory use… but neither approach has worked.
Hi Ghadi. I used an env var e.g.:
JAVA_OPTS=-Xmx1280m. Less than was available on the Linux instance. I wanted to avoid the Linux OOM-Killer… which may or may not have been related.
clearing up a couple misconceptions: 1) clj launches two JVMs, one to clone the repos, then another for your project. To modify the JVM for the first one, you need to change this line https://github.com/clojure/brew-install/blob/1.10.1/src/main/resources/clojure#L314 in /usr/local/bin/clojure or wherever your script lives 2) JAVA_OPTS isn't a thing, but a convention in some scripts -- not honored by clj
try bumping up from 256m and your problem may (?) go away -- definitely worth a issue though, right @alexmiller?
I will go and try some different memory allocations. Could you just confirm that
tools.gitlibs should be able to handle this size of dependency, if conditions are right?
bumping JGit is on the radar, as well as maybe making that whole subsystem more resilient
Seems like loading the code into memory would be a mistake somewhere in the pipeline.
Yeah. I was wondering about that… should be able to stream any amount of data straight through. Thanks. I’ll go away and experiment, then. Will report back either way… but it may take a few days to be sure that the memory tweaks have worked because we don’t see it every time, because of CI dependency caching etc.
I have some tabs and editors open working on validating the latest versions of jgit, but it's a notch or two down the stack right now. Kind of an interim step towards comparing latest jgit vs shelling out to git for https/ssh access
generally, I'd expect the jvm heap on the cp-making jvm to grow quite a bit larger, but it depends what jvm and version you're using
Ahh. JVM versions... that’s another longer story. In our dev and CI environment we’re currently restricted to using OpenJDK 8 because of a DB dependency. Our Clojure code should be fine running under version 11 and we hope to move to OpenJDK 11 early next year. It’s possible we could use multiple JVM versions in the meantime (to start the DB under a seperate JVM), using a version switcher or something, but that might be asking for trouble as well as more complexity!