Is there anything like an offline mode for tools.deps? I'm finding my CI builds take 2-2.5 minutes per alias per build step just downloading maven-metadata.xml files from my S3 repo and clojars / maven-central, and I can't seem to cache the results

This is separate from the actual .jar / .pom files which I'm able to cache with no problem (in ~/.m2)

We use s3 wagon private in boot and I notice it downloads metadata more often than it should, which also slows down things there

Also, these specific packages have become my arch-enemies as every time I run clojure I see this block of text:

Downloading: com/amazonaws/aws-java-sdk-cloudwatch/maven-metadata.xml from 
Downloading: com/amazonaws/aws-java-sdk-cloudwatch/maven-metadata.xml from 
Downloading: com/amazonaws/aws-java-sdk-dynamodb/maven-metadata.xml from 
Downloading: com/amazonaws/aws-java-sdk-dynamodb/maven-metadata.xml from 
Downloading: com/amazonaws/aws-java-sdk-kinesis/maven-metadata.xml from 
Downloading: com/amazonaws/aws-java-sdk-kinesis/maven-metadata.xml from 

I don't know if there's a TTL set to a ridiculously low value by AWS or what, but I just cannot avoid seeing that every single time if I've got those dpes

file a ticket with as much info as possible

I suspect the answer is no, but just to confirm: is there any way to have tools.deps compile Java code when consuming a Clojure library via git that also includes some Java code?

no

Alrighty then, thanks!

Is there a way to see why a classpath refresh takes a long time with clj?

there is a super secret verbose knob you could try

clj -Sdeps '{:aliases {:v {:verbose true}}}' -A:v

the most likely culprit is slow I/O of course, which you can really spot only by observing where that pauses

The weird part is it only occurs in one particular project.

the next most likely is that tools.deps is doing more work than it should, which you may also be able to spot by seeing it redoing work

Hard to tell what's expected and whats not. Expanding git deps takes a good 3-5s for this project.

This project also has a fairly large dependency tree. The ~20s refresh gets a bit painful though.

git deps are stored in two parts - the git objects, and the working tree (per sha)

It wouldn't be checking out anything new - everything is all on disk. Just changing the SHAs back and forth.

the git objects are mostly append-only so a fetch should only need to be new stuff, small diffs

working tree could be slower, but it’s all in the objects

I guess I would compare doing a git clone or fetch separately and see if it’s substantially different

there’s caching in many places here - git objects, git working tree per sha, maven deps in maven repo, and classpath in cpcache

so I guess I’m curious what part of that isn’t cached when it’s slow (and why)

For example, each one of these steps takes 3-5s:

Expanding compute/logger {:git/url [email protected]:ComputeSoftware/logger.git, :sha 9f9fa87011e8b0dabc83f067e2e513d2946aa2e9}
	=>  current is newer{:git/url "[email protected]:ComputeSoftware/logger.git", :sha "319e398132ba440981254466079242efed193d56"}

is it correct that there is a newer sha?

Yes, 319 is newer than 9f9f. Just ran time on the -Spath command. It actually took 52s for a CP refresh.

I should clarify - that is one example of the messages that takes a while. AFAICT, there are no duplicates in the log.

so it should be fetching the git repo, then cloning the working tree. Are there maybe big binary objects in the working tree? One idea I had with gitlibs was to have the ability to narrow the working tree to just a subset of the full dir (so you could exclude like an images directory or something was irrelevant to the Clojure source)

still not entirely sure that’s the slow bit from your description but maybe a testable hypothesis