This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2023-02-23
Channels
- # announcements (1)
- # babashka (68)
- # babashka-sci-dev (12)
- # beginners (36)
- # biff (22)
- # calva (20)
- # clerk (1)
- # clj-on-windows (7)
- # clojure (27)
- # clojure-conj (8)
- # clojure-denmark (2)
- # clojure-europe (141)
- # clojure-france (1)
- # clojure-italy (4)
- # clojure-nl (1)
- # clojure-uk (2)
- # clojurescript (7)
- # conjure (8)
- # core-async (111)
- # cursive (3)
- # datahike (4)
- # datalevin (18)
- # events (9)
- # gratitude (4)
- # guix (2)
- # helix (3)
- # hyperfiddle (62)
- # introduce-yourself (2)
- # kaocha (4)
- # london-clojurians (3)
- # lsp (7)
- # malli (34)
- # membrane (1)
- # nbb (9)
- # polylith (4)
- # portal (6)
- # reagent (4)
- # releases (2)
- # remote-jobs (4)
- # shadow-cljs (30)
- # sql (2)
- # tools-deps (58)
- # xtdb (9)
Hey all, I’m trying to tighten up our CI process so all dependencies are fetched ahead of time (with -P
) and cached. But even with clojure -P -T:tool-name
, dependencies still get downloaded when I invoke the tool (later). I can’t seem to figure out how to solve it. Any tips?
It feels like there may be some dependencies getting added at runtime (though I don’t see where).
I’m using seancorfield/build-clj
(deprecated, I know…). This is what I see downloaded at invocation time:
Downloading: org/clojure/tools.cli/0.3.5/tools.cli-0.3.5.pom from central
Downloading: org/clojure/data.xml/0.0.8/data.xml-0.0.8.pom from central
Downloading: riddley/riddley/0.1.12/riddley-0.1.12.pom from clojars
Downloading: org/clojure/tools.cli/0.3.5/tools.cli-0.3.5.jar from central
Downloading: org/clojure/data.xml/0.0.8/data.xml-0.0.8.jar from central
Downloading: riddley/riddley/0.1.12/riddley-0.1.12.jar from clojars
Try using the -X execution flag, -T might be excluding dependencies
(Btw, https://github.com/babashka/babashka/blob/master/examples/download-aliases.clj is a small babashka script to download all deps from all aliases)
@U04V15CAJ I believe the distinction is that -T “will use only the :paths
and :deps
from the :build
alias”
@U05254DQM Thanks. Pretty sure I’ve tried that also but I will confirm now.
Oh right, I guess you can't "prepare" all of those combinations in one go since they are exclusive wrt dependencies and paths
I assume a cache is already added to the workflow to cache the libraries
Still I don't get the difference between -T and -X the more I think about it. Doesn't -X + :paths
+ :deps
have the same effect as -T? Since :extra-paths
would add, but :paths
would replace... right?
@U05254DQM Yes, that is correct. But this is literally on the same machine in the same build run. It’s simply not downloading the libraries the first time. (It doesn’t show the download messages there either).
note the things being downloaded above are poms, used for dependency analysis (not the jars). Also, -X and -T do almost the same thing (and literally share the same code by the time you get to execution)
@U064X3EF3 I see POMs and JARs in the output. Am I misreading?
oh sorry, I didn't read all of that, yes
The code for build-clj is pretty simple…I don’t see any dynamic dependency magic happening, so I think it must be in some transitive dependency, but I can’t figure out how to get the transitive dependency tree for a tool… that would be really helpful.
tool by name or by alias?
clj -X:deps tree :project nil :aliases '[:toolalias]'
should be same I think
I had to amend that to clj -X:deps tree :project nil :aliases '[:uberjar]'
(hope that’s right), but that gave me this:
org.clojure/clojure 1.11.1
. org.clojure/spec.alpha 0.3.218
. org.clojure/core.specs.alpha 0.2.62
So those dependencies don’t seem to appear. I doubt riddley, e.g., is a transitive dependencies of those.
@U064X3EF3 Thanks. I would not have come up with that incantation myself!
well, not sure that will work - removing the project deps.edn also removes the aliases in it so if the tool is defined in the current project, that may not work
if you move it to your ~/.clojure/deps.edn, that would work. or you could re-introduce that alias with :extra '{:aliases {:uberjar {...}}}'
Good thinking. Moved it to my ~/.clojure/deps.edn
. More reasonable output now, but I still don’t see (all of) the dependencies that get downloaded at run time.
assuming you are running uberjar as a tool, and that is using build-clj, which is using tools.build, which is almost surely creating a basis, which will download the project deps
so you would need to prepare both:
clojure -P -T:tool-name
clojure -P
I think it's definitely the issue :)
So the (bare) clojure -P
does what, exactly? Preps tools.deps
dependencies? I guess I assumed that would happen automatically by virtue of preparing anything else.
clojure
does two things - 1) run a process that computes the dependency for the program to run (optionally, guarded by a cache) and 2) run the program using the computed classpath based on #1. -P does just #1.
I (believe) I knew that much but I’m unsure how to square that with clojure -P -T:uberjar
not (to my mind) fully computing the dependencies needed to run clojure -T:uberjar
.
the first runs a program that uses tools.deps to download project deps, but clojure -P -T:uberjar
does not actually run the program (#2)
so the first line downloads the deps to run the uberjar program. the second line downloads the deps that will be used by that program
Ahhh. I may need to process that a little to update my mental model. 🙂
But, to restate what I think I’m learning, dependencies like riddley are not used directly by the :uberjar
(-aliased) tool (hence my trouble finding them in its code) but are needed for clojure
to invoke that tool.
I had been thinking of -P
as “prepare everything we need to run this program”, but it’s more like “prepare everything this program needs to run”. Subtle distinction.
@U064X3EF3 @U04V15CAJ @U05254DQM Thanks a ton. Been pulling my hair out over this for a while. CI times are now back below five minutes and some generous soul is saving some bandwidth. 🙂
@U0CV48L87 I'm happy you solved your problem but I'm not sure how you solved it? By understanding that -P didn't do what you thought it did?
More or less.
I didn’t realize it was necessary to run clojure -P
in addition to clojure -P -T:uberjar
in order to prepare everything needed to run clojure -T:uberjar
.
I suppose I’ll always add a clojure -P
line to my CI builds from here on.
@U04V15CAJ Does that make sense?
I’m still a little fuzzy on the why. It would be clearer to me if I were just using -A
, in which case I can imagine that dependencies needed directly by the alias are not necessarily everything needed to run the program in that configuration (i.e. something more globally-scoped within the project).
With -T
I figured we were totally ignoring the surrounding context and letting the tool completely take over the dependency list, but perhaps some things are loaded in the process of bootstrapping the tool? I really don’t know.
Wait, maybe I totally misunderstood that. I’m building an uberjar here. It’s probably simply that the full set of project dependencies needed for the uberjar itself were never fully downloaded.
And I was mistaking that output for the uberjar tooling downloading its own dependencies.
right. for creating an uberjar you need tools.build and its dependencies. but when executing build.clj
dependencies are pulled while calculating the basis. to "prepare" that you should just do clojure -P
combined with all the aliases that you're using in your basis (if any)
Ok, yes, I have now confirmed what I said above.
Yeah… I actually understood all of those things and my mental model was correct, I was just misreading the dependency fetches during uberjarring time, thinking they were the dependencies of the tool itself, which I thought I had somehow failed to prepare.
In actuality they were the handful of (general) project dependencies that hadn’t been pulled in by previous steps in the CI process (including testing, hence most dependencies already needing to be pulled). Had I seen a large number of dependencies get pulled at that time then it would have been obvious to me what was going on.
I think they might specifically be dependencies which happen to get shadowed by test dependencies (I do run clojure -P -M:test
early in the build, FWIW).
Not sure if this helps, but I've used -X along with docker build cache in the build and haven't noticed issues
COPY deps.edn /build/
RUN clojure -P -X:env/test:package/uberjar
https://practical.li/blog/posts/build-and-run-clojure-with-multistage-dockerfile/
I haven't used tools.bould yet, but assume it should be very similar (it's on my to-do list)