This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
- # announcements (3)
- # aws (7)
- # babashka (11)
- # beginners (96)
- # clojure (15)
- # clojure-europe (9)
- # clojure-germany (2)
- # clojure-italy (1)
- # clojure-nl (3)
- # clojure-poland (3)
- # clojure-serbia (1)
- # clojurescript (13)
- # depstar (28)
- # fulcro (34)
- # graalvm (5)
- # honeysql (5)
- # malli (11)
- # off-topic (27)
- # pathom (9)
- # polylith (74)
- # portal (10)
- # re-frame (13)
- # releases (1)
- # ring (8)
- # shadow-cljs (3)
- # spacemacs (10)
- # tools-deps (8)
Hi, @seancorfield — I used depstar for the first time last week, and it’s so fantastic. I have no idea why I waited so long to try it! 🙂 (I had used lein for this before…)
When creating an uberjar, depstar seems to re-download all the external dependencies from maven central, even though in an earlier Docker build step, I ran
clj -Spath to pre-download them (presumably into .m2 directory?)
Is my understanding of what is happening correct? And if so, is there a way to have depstar use .m2 for jars, instead of re-downloading? Thx!!!
tools.deps.alpha under the hood so it’s behavior should be exactly the same as the Clojure CLI.
So I’d need more information about exactly what commands you are running and the project setup.
(and I’ve never seen it download deps that are already cached locally in
~/.m2 — which is what all of the underlying t.d.a stuff uses for both the Clojure CLI and for
What you might be seeing is
tools.deps.alpha itself being downloaded (along with all of its transitive deps) since the CLI bundles all that into its uberjar.
…so strange… I’m staring at the build steps right now… Here’s what has me puzzled… Annotated build output:
> Step #1: Step 15/17 : WORKDIR /src/pubsub-web
> Step #1: ---> Running in 35dbc5d9db0f
*clj -Spath* happened here... nothing happened, because deps.edn didn't change, so all jars are already downloaded...
> Step #1: Removing intermediate container 35dbc5d9db0f
> Step #1: ---> 9781d01f38bc
> Step #1: Step 16/17 : RUN make uberjar
> Step #1: ---> Running in 4cbf3018663a
uberjar generation happens here… note that all jars are downloaded again. Why?
> Step #1: clojure -X:uberjar :aot true :jar ./target/pubsub-web-standalone.jar \
> Step #1: :main-class web.server
> Step #1: Downloading: com/github/seancorfield/depstar/2.0.206/depstar-2.0.206.pom from clojars
> Step #1: Downloading: org/clojure/tools.deps.alpha/0.9.863/tools.deps.alpha-0.9.863.pom from central
> Step #1: Downloading: org/clojure/tools.logging/1.1.0/tools.logging-1.1.0.pom from central
> Step #1: Downloading: org/clojure/pom.contrib/1.0.0/pom.contrib-1.0.0.pom from central
In other words, you are not pre-downloading the dependencies that
depstar itself needs.
In other words, the exact same command you would use to build the uberjar except with
-P at the beginning.
The CLI prepare option is designed for exactly this purpose (instead of using
-Spath or similar).
OH RIGHT!!! I read about -P a couple of days ago, but I couldn’t find it this morning! (I spent an hour googling for “what is equivalent to ‘lein deps’ for clj”. 🙂
I’ll add a note to the
depstar README about CI usage and
-P for future souls on this journey.
For the second time ever, I’m going to write up a gist to remind myself of an interaction we’ve had on Clojurians, in the hopes of helping something struggling with the same problem — the first time was for jdbc and mysql. 🙂
Search for stuff on Zulip chat — it’s all mirrored there and that has no limit on search history 🙂
There’s a subtle issue there: if you run
depstar and pass in aliases to it (perhaps to pin library versions, or some other “production” aliases for the JAR contents), you need to also have those aliases in the
clojure -P command (as well as
:uberjar or whatever your
depstar alias is).
Yes indeed! That did the trick, and indeed, that was exactly what was required to eliminate the need to download jars at uberjar generation time:
I’m a little curious about why 14 jars are still being downloaded at uberjar generation time… but it’s almost not worth the time to dig any further. 🙂 THANK YOU!!!
# to pre-download dependencies, only done if deps.edn changes RUN mkdir /tmp/pubsub-web COPY ./pubsub-web/deps.edn /tmp/pubsub-web WORKDIR /tmp/pubsub-web RUN clj -P RUN clj -A:depstar -P
Step #1: clojure -X:uberjar :aot true :jar ./target/pubsub-web-standalone.jar \ Step #1: :main-class web.server Step #1: Downloading: com/cognitect/transit-clj/1.0.324/transit-clj-1.0.324.pom from central Step #1: Downloading: com/cognitect/transit-java/1.0.343/transit-java-1.0.343.pom from central Step #1: Downloading: javax/xml/bind/jaxb-api/2.3.0/jaxb-api-2.3.0.pom from central Step #1: Downloading: org/msgpack/msgpack/0.6.12/msgpack-0.6.12.pom from central Step #1: Downloading: javax/xml/bind/jaxb-api-parent/2.3.0/jaxb-api-parent-2.3.0.pom from central Step #1: Downloading: net/java/jvnet-parent/5/jvnet-parent-5.pom from central Step #1: Downloading: com/googlecode/json-simple/json-simple/1.1.1/json-simple-1.1.1.pom from central Step #1: Downloading: org/javassist/javassist/3.18.1-GA/javassist-3.18.1-GA.pom from central Step #1: Downloading: com/cognitect/transit-java/1.0.343/transit-java-1.0.343.jar from central Step #1: Downloading: com/googlecode/json-simple/json-simple/1.1.1/json-simple-1.1.1.jar from central Step #1: Downloading: org/javassist/javassist/3.18.1-GA/javassist-3.18.1-GA.jar from central Step #1: Downloading: org/msgpack/msgpack/0.6.12/msgpack-0.6.12.jar from central Step #1: Downloading: javax/xml/bind/jaxb-api/2.3.0/jaxb-api-2.3.0.jar from central Step #1: Downloading: com/cognitect/transit-clj/1.0.324/transit-clj-1.0.324.jar from central Step #1: [main] INFO hf.depstar.uberjar - Compiling web.server ... Step #1: 2021-03-27 18:01:20.801:INFO::main: Logging initialized @6780ms to org.eclipse.jetty.util.log.StdErrLog
You’d have to dig into the dependency trees of things to figure that out I suspect… I don’t know what would bring those in… they are not from
I ran this in
depstar’s repo folder:
so the only Cognitect deps in the tree are the AWS stuff — not transit etc.
(! 503)-> clojure -Stree|fgrep cognitect . com.cognitect.aws/api 0.8.505 . com.cognitect/http-client 0.1.106 . com.cognitect.aws/endpoints 184.108.40.2069 . com.cognitect.aws/s3 811.2.858.0
Is it possible you have dynamically required/resolved dependencies in your own code?
Huh. Maybe middleware is dynamically resolved? (This is definitely challenging my very shaky mental model of how dependency resolution actually works.) Filing this under “things to better understand someday.” 🙂
Thanks for your help, @seancorfield! Keep up the amazing work!!!
. metosin/reitit-middleware 0.5.12 . metosin/reitit-ring 0.5.12 . lambdaisland/deep-diff 0.0-47 . mvxcvi/puget 1.1.2 . mvxcvi/arrangement 1.2.0 X fipp/fipp 0.6.17 :older-version X fipp/fipp 0.6.17 :older-version . org.clojure/core.rrb-vector 0.0.14 . tech.droit/clj-diff 1.0.1 . mvxcvi/arrangement 1.2.0 . metosin/muuntaja 0.6.8 . metosin/jsonista 0.3.1 . com.cognitect/transit-clj 1.0.324 . com.cognitect/transit-java 1.0.343 X com.fasterxml.jackson.core/jackson-core 2.8.7 :older-version . org.msgpack/msgpack 0.6.12 . com.googlecode.json-simple/json-simple 1.1.1 . org.javassist/javassist 3.18.1-GA X commons-codec/commons-codec 1.10 :older-version . javax.xml.bind/jaxb-api 2.3.0