I’m using seancorfield/build-clj (deprecated, I know…). This is what I see downloaded at invocation time:

Downloading: org/clojure/tools.cli/0.3.5/tools.cli-0.3.5.pom from central
Downloading: org/clojure/data.xml/0.0.8/data.xml-0.0.8.pom from central
Downloading: riddley/riddley/0.1.12/riddley-0.1.12.pom from clojars
Downloading: org/clojure/tools.cli/0.3.5/tools.cli-0.3.5.jar from central
Downloading: org/clojure/data.xml/0.0.8/data.xml-0.0.8.jar from central
Downloading: riddley/riddley/0.1.12/riddley-0.1.12.jar from clojars

can someone remind me what use -T has inside a project compared to -X?

Try using the -X execution flag, -T might be excluding dependencies

(Btw, https://github.com/babashka/babashka/blob/master/examples/download-aliases.clj is a small babashka script to download all deps from all aliases)

@U04V15CAJ I believe the distinction is that -T “will use only the :paths and :deps from the :build alias”

@U05254DQM Thanks. Pretty sure I’ve tried that also but I will confirm now.

Oh right, I guess you can't "prepare" all of those combinations in one go since they are exclusive wrt dependencies and paths

Yeah. And I’m ok with multiple prep commands, I just want a combo that gets me there…

I assume a cache is already added to the workflow to cache the libraries

Still I don't get the difference between -T and -X the more I think about it. Doesn't -X + :paths + :deps have the same effect as -T? Since :extra-paths would add, but :paths would replace... right?

@U05254DQM Yes, that is correct. But this is literally on the same machine in the same build run. It’s simply not downloading the libraries the first time. (It doesn’t show the download messages there either).

note the things being downloaded above are poms, used for dependency analysis (not the jars). Also, -X and -T do almost the same thing (and literally share the same code by the time you get to execution)

@U064X3EF3 I see POMs and JARs in the output. Am I misreading?

Maybe -T always wipes the top level deps, unlike -X? that must have been it

oh sorry, I didn't read all of that, yes

-T excludes the project deps (unlike -X)

The code for build-clj is pretty simple…I don’t see any dynamic dependency magic happening, so I think it must be in some transitive dependency, but I can’t figure out how to get the transitive dependency tree for a tool… that would be really helpful.

you can

tool by name or by alias?

By alias

clj -X:deps tree :project nil :aliases '[:toolalias]'

should be same I think

I had to amend that to clj -X:deps tree :project nil :aliases '[:uberjar]' (hope that’s right), but that gave me this:

org.clojure/clojure 1.11.1
  . org.clojure/spec.alpha 0.3.218
  . org.clojure/core.specs.alpha 0.2.62

So those dependencies don’t seem to appear. I doubt riddley, e.g., is a transitive dependencies of those.

@U064X3EF3 Thanks. I would not have come up with that incantation myself!

At this point I suspect it would be helpful to have a repro deps.edn

Starting to feel that way.

Ok, I’ll circle back with one. Thanks for the help, gents.

well, not sure that will work - removing the project deps.edn also removes the aliases in it so if the tool is defined in the current project, that may not work

Ah, fair point

if you move it to your ~/.clojure/deps.edn, that would work. or you could re-introduce that alias with :extra '{:aliases {:uberjar {...}}}'

Good thinking. Moved it to my ~/.clojure/deps.edn. More reasonable output now, but I still don’t see (all of) the dependencies that get downloaded at run time.

Sooo…something is adding dependencies dynamically when the tool runs?

Working on a clean repro repo.

assuming you are running uberjar as a tool, and that is using build-clj, which is using tools.build, which is almost surely creating a basis, which will download the project deps

so you would need to prepare both:

clojure -P -T:tool-name
clojure -P

Oh… I’m not doing the latter. I wonder if that is the issue.

I think it's definitely the issue :)

Indeed it was. 🎉 Thanks a ton.

So the (bare) clojure -P does what, exactly? Preps tools.deps dependencies? I guess I assumed that would happen automatically by virtue of preparing anything else.

clojure does two things - 1) run a process that computes the dependency for the program to run (optionally, guarded by a cache) and 2) run the program using the computed classpath based on #1. -P does just #1.

I (believe) I knew that much but I’m unsure how to square that with clojure -P -T:uberjar not (to my mind) fully computing the dependencies needed to run clojure -T:uberjar.

the first runs a program that uses tools.deps to download project deps, but clojure -P -T:uberjar does not actually run the program (#2)

so the first line downloads the deps to run the uberjar program. the second line downloads the deps that will be used by that program

Ahhh. I may need to process that a little to update my mental model. 🙂 But, to restate what I think I’m learning, dependencies like riddley are not used directly by the :uberjar (-aliased) tool (hence my trouble finding them in its code) but are needed for clojure to invoke that tool.

they could easily be transitive deps, don't know

I had been thinking of -P as “prepare everything we need to run this program”, but it’s more like “prepare everything this program needs to run”. Subtle distinction.

@U064X3EF3 @U04V15CAJ @U05254DQM Thanks a ton. Been pulling my hair out over this for a while. CI times are now back below five minutes and some generous soul is saving some bandwidth. 🙂

@U0CV48L87 I'm happy you solved your problem but I'm not sure how you solved it? By understanding that -P didn't do what you thought it did?

More or less. I didn’t realize it was necessary to run clojure -P in addition to clojure -P -T:uberjar in order to prepare everything needed to run clojure -T:uberjar. I suppose I’ll always add a clojure -P line to my CI builds from here on.

@U04V15CAJ Does that make sense? I’m still a little fuzzy on the why. It would be clearer to me if I were just using -A , in which case I can imagine that dependencies needed directly by the alias are not necessarily everything needed to run the program in that configuration (i.e. something more globally-scoped within the project). With -T I figured we were totally ignoring the surrounding context and letting the tool completely take over the dependency list, but perhaps some things are loaded in the process of bootstrapping the tool? I really don’t know.

Wait, maybe I totally misunderstood that. I’m building an uberjar here. It’s probably simply that the full set of project dependencies needed for the uberjar itself were never fully downloaded.

And I was mistaking that output for the uberjar tooling downloading its own dependencies.

right. for creating an uberjar you need tools.build and its dependencies. but when executing build.clj dependencies are pulled while calculating the basis. to "prepare" that you should just do clojure -P combined with all the aliases that you're using in your basis (if any)

Ok, yes, I have now confirmed what I said above. Yeah… I actually understood all of those things and my mental model was correct, I was just misreading the dependency fetches during uberjarring time, thinking they were the dependencies of the tool itself, which I thought I had somehow failed to prepare. In actuality they were the handful of (general) project dependencies that hadn’t been pulled in by previous steps in the CI process (including testing, hence most dependencies already needing to be pulled). Had I seen a large number of dependencies get pulled at that time then it would have been obvious to me what was going on. I think they might specifically be dependencies which happen to get shadowed by test dependencies (I do run clojure -P -M:test early in the build, FWIW).

Not sure if this helps, but I've used -X along with docker build cache in the build and haven't noticed issues

COPY deps.edn /build/
RUN clojure -P -X:env/test:package/uberjar

Thanks. I’ve done it with a multi-stage Docker build as well, and it’s a very cool approach. In this case I was using Circle CI’s native caching, but it’s all working great now. Ultimately just a misread on what problem I actually needed to be solving. Classic debugging story.