Fork me on GitHub

Is there anything like an offline mode for tools.deps? I'm finding my CI builds take 2-2.5 minutes per alias per build step just downloading maven-metadata.xml files from my S3 repo and clojars / maven-central, and I can't seem to cache the results


This is separate from the actual .jar / .pom files which I'm able to cache with no problem (in ~/.m2)


We use s3 wagon private in boot and I notice it downloads metadata more often than it should, which also slows down things there


Also, these specific packages have become my arch-enemies as every time I run clojure I see this block of text:

Downloading: com/amazonaws/aws-java-sdk-cloudwatch/maven-metadata.xml from 
Downloading: com/amazonaws/aws-java-sdk-cloudwatch/maven-metadata.xml from 
Downloading: com/amazonaws/aws-java-sdk-dynamodb/maven-metadata.xml from 
Downloading: com/amazonaws/aws-java-sdk-dynamodb/maven-metadata.xml from 
Downloading: com/amazonaws/aws-java-sdk-kinesis/maven-metadata.xml from 
Downloading: com/amazonaws/aws-java-sdk-kinesis/maven-metadata.xml from 


I don't know if there's a TTL set to a ridiculously low value by AWS or what, but I just cannot avoid seeing that every single time if I've got those dpes

Alex Miller (Clojure team)18:02:48

file a ticket with as much info as possible

👍 5

I suspect the answer is no, but just to confirm: is there any way to have tools.deps compile Java code when consuming a Clojure library via git that also includes some Java code?


Alrighty then, thanks!


Is there a way to see why a classpath refresh takes a long time with clj?

Alex Miller (Clojure team)23:02:28

there is a super secret verbose knob you could try

Alex Miller (Clojure team)23:02:03

clj -Sdeps '{:aliases {:v {:verbose true}}}' -A:v

Alex Miller (Clojure team)23:02:19

the most likely culprit is slow I/O of course, which you can really spot only by observing where that pauses


The weird part is it only occurs in one particular project.

Alex Miller (Clojure team)23:02:09

the next most likely is that tools.deps is doing more work than it should, which you may also be able to spot by seeing it redoing work


Hard to tell what's expected and whats not. Expanding git deps takes a good 3-5s for this project.


This project also has a fairly large dependency tree. The ~20s refresh gets a bit painful though.

Alex Miller (Clojure team)23:02:12

git deps are stored in two parts - the git objects, and the working tree (per sha)


It wouldn't be checking out anything new - everything is all on disk. Just changing the SHAs back and forth.

Alex Miller (Clojure team)23:02:19

the git objects are mostly append-only so a fetch should only need to be new stuff, small diffs

Alex Miller (Clojure team)23:02:34

working tree could be slower, but it’s all in the objects

Alex Miller (Clojure team)23:02:16

I guess I would compare doing a git clone or fetch separately and see if it’s substantially different

Alex Miller (Clojure team)23:02:02

there’s caching in many places here - git objects, git working tree per sha, maven deps in maven repo, and classpath in cpcache

Alex Miller (Clojure team)23:02:02

so I guess I’m curious what part of that isn’t cached when it’s slow (and why)


For example, each one of these steps takes 3-5s:

Expanding compute/logger {:git/url [email protected]:ComputeSoftware/logger.git, :sha 9f9fa87011e8b0dabc83f067e2e513d2946aa2e9}
	=>  current is newer{:git/url "[email protected]:ComputeSoftware/logger.git", :sha "319e398132ba440981254466079242efed193d56"}
There are several of those.

Alex Miller (Clojure team)23:02:07

is it correct that there is a newer sha?


Yes, 319 is newer than 9f9f. Just ran time on the -Spath command. It actually took 52s for a CP refresh.


I should clarify - that is one example of the messages that takes a while. AFAICT, there are no duplicates in the log.

Alex Miller (Clojure team)23:02:46

so it should be fetching the git repo, then cloning the working tree. Are there maybe big binary objects in the working tree? One idea I had with gitlibs was to have the ability to narrow the working tree to just a subset of the full dir (so you could exclude like an images directory or something was irrelevant to the Clojure source)

Alex Miller (Clojure team)23:02:49

still not entirely sure that’s the slow bit from your description but maybe a testable hypothesis