Fork me on GitHub
#tools-deps
<
2019-02-06
>
timgilbert18:02:13

Is there anything like an offline mode for tools.deps? I'm finding my CI builds take 2-2.5 minutes per alias per build step just downloading maven-metadata.xml files from my S3 repo and clojars / maven-central, and I can't seem to cache the results

timgilbert18:02:12

This is separate from the actual .jar / .pom files which I'm able to cache with no problem (in ~/.m2)

borkdude18:02:50

We use s3 wagon private in boot and I notice it downloads metadata more often than it should, which also slows down things there

timgilbert18:02:31

Also, these specific packages have become my arch-enemies as every time I run clojure I see this block of text:

Downloading: com/amazonaws/aws-java-sdk-cloudwatch/maven-metadata.xml from 
Downloading: com/amazonaws/aws-java-sdk-cloudwatch/maven-metadata.xml from 
Downloading: com/amazonaws/aws-java-sdk-dynamodb/maven-metadata.xml from 
Downloading: com/amazonaws/aws-java-sdk-dynamodb/maven-metadata.xml from 
Downloading: com/amazonaws/aws-java-sdk-kinesis/maven-metadata.xml from 
Downloading: com/amazonaws/aws-java-sdk-kinesis/maven-metadata.xml from 

timgilbert18:02:00

I don't know if there's a TTL set to a ridiculously low value by AWS or what, but I just cannot avoid seeing that every single time if I've got those dpes

Alex Miller (Clojure team)18:02:48

file a ticket with as much info as possible

👍 5
avi18:02:43

I suspect the answer is no, but just to confirm: is there any way to have tools.deps compile Java code when consuming a Clojure library via git that also includes some Java code?

avi18:02:44

Alrighty then, thanks!

kenny23:02:49

Is there a way to see why a classpath refresh takes a long time with clj?

Alex Miller (Clojure team)23:02:28

there is a super secret verbose knob you could try

Alex Miller (Clojure team)23:02:03

clj -Sdeps '{:aliases {:v {:verbose true}}}' -A:v

Alex Miller (Clojure team)23:02:19

the most likely culprit is slow I/O of course, which you can really spot only by observing where that pauses

kenny23:02:55

The weird part is it only occurs in one particular project.

Alex Miller (Clojure team)23:02:09

the next most likely is that tools.deps is doing more work than it should, which you may also be able to spot by seeing it redoing work

kenny23:02:14

Hard to tell what's expected and whats not. Expanding git deps takes a good 3-5s for this project.

kenny23:02:44

This project also has a fairly large dependency tree. The ~20s refresh gets a bit painful though.

Alex Miller (Clojure team)23:02:12

git deps are stored in two parts - the git objects, and the working tree (per sha)

kenny23:02:02

It wouldn't be checking out anything new - everything is all on disk. Just changing the SHAs back and forth.

Alex Miller (Clojure team)23:02:19

the git objects are mostly append-only so a fetch should only need to be new stuff, small diffs

Alex Miller (Clojure team)23:02:34

working tree could be slower, but it’s all in the objects

Alex Miller (Clojure team)23:02:16

I guess I would compare doing a git clone or fetch separately and see if it’s substantially different

Alex Miller (Clojure team)23:02:02

there’s caching in many places here - git objects, git working tree per sha, maven deps in maven repo, and classpath in cpcache

Alex Miller (Clojure team)23:02:02

so I guess I’m curious what part of that isn’t cached when it’s slow (and why)

kenny23:02:34

For example, each one of these steps takes 3-5s:

Expanding compute/logger {:git/url [email protected]:ComputeSoftware/logger.git, :sha 9f9fa87011e8b0dabc83f067e2e513d2946aa2e9}
	=>  current is newer{:git/url "[email protected]:ComputeSoftware/logger.git", :sha "319e398132ba440981254466079242efed193d56"}
There are several of those.

Alex Miller (Clojure team)23:02:07

is it correct that there is a newer sha?

kenny23:02:49

Yes, 319 is newer than 9f9f. Just ran time on the -Spath command. It actually took 52s for a CP refresh.

kenny23:02:20

I should clarify - that is one example of the messages that takes a while. AFAICT, there are no duplicates in the log.

Alex Miller (Clojure team)23:02:46

so it should be fetching the git repo, then cloning the working tree. Are there maybe big binary objects in the working tree? One idea I had with gitlibs was to have the ability to narrow the working tree to just a subset of the full dir (so you could exclude like an images directory or something was irrelevant to the Clojure source)

Alex Miller (Clojure team)23:02:49

still not entirely sure that’s the slow bit from your description but maybe a testable hypothesis