Fork me on GitHub
#depstar
<
2021-09-03
>
kirill.salykin09:09:26

hi, Please advice on what I am doing wrong:

- echo "prepare"
    - clojure -Sdeps '{:mvn/local-repo ".m2"}' -Spath -A:prod
    - echo "-----------------------------------------------"
    - ls -lah .m2
    - clojure -Sdeps '{:mvn/local-repo ".m2"}' -X:uberjar :classpath '"'$(clojure -Sdeps '{:mvn/local-repo ".m2"}' -Spath -A:prod)'"'
On second command (when building uberjar) all deps were downloaded again (But I do expect them already be in the local .m2 directory) thanks!

seancorfield16:09:50

Are you sure it's your :prod deps that are re-downloaded? I would expect the depstar deps to be downloaded in that second command because you had not downloaded them before. You need clojure -Sdeps '{:mvn/local-repo ".m2"}' -Spath -A:prod:uberjar for that.

kirill.salykin19:09:19

Actually I think you are right, sorry for bothering

kirill.salykin20:09:00

It works locally but still loads all deps on CI

clojure -Sdescribe
clojure -Sdeps '{:mvn/local-repo ".m2"}' -A:prod -Spath
echo "-----------------------------------------------"
clojure -Sdeps '{:mvn/local-repo ".m2"}' -A:uberjar -Spath
echo "-----------------------------------------------"
clojure -Sdeps '{:mvn/local-repo ".m2"}' -X:uberjar :classpath '"'$(clojure -Sdeps '{:mvn/local-repo ".m2"}' -A:prod -Spath)'"'

kirill.salykin20:09:06

{:deps {;; Platform and system logic
        aero/aero                           {:mvn/version "1.1.6"}
        buddy/buddy-sign                    {:mvn/version "3.2.0"}
        ch.qos.logback/logback-classic      {:mvn/version "1.2.3"}
        clj-htmltopdf/clj-htmltopdf         {:mvn/version "0.1-alpha7"}
        clojure.java-time/clojure.java-time {:mvn/version "0.3.2"}
        com.novemberain/pantomime           {:mvn/version "2.11.0"}
        cryptohash-clj/cryptohash-clj       {:mvn/version "0.1.6"}
        expound/expound                     {:mvn/version "0.8.7"}
        haversine/haversine                 {:mvn/version "0.1.1"}
        integrant/repl                      {:mvn/version "0.3.2"}
        integrant/integrant                 {:mvn/version "0.8.0"}
        metosin/spec-tools                  {:mvn/version "0.10.4"}
        org.apache.pdfbox/pdfbox            {:mvn/version "2.0.22"}
        org.apache.pdfbox/pdfbox-tools      {:mvn/version "2.0.22"}
        org.bouncycastle/bcprov-jdk15on     {:mvn/version "1.67"}
        org.clojure/clojure                 {:mvn/version "1.10.1"}
        org.clojure/core.async              {:mvn/version "1.3.610"}
        org.clojure/core.cache              {:mvn/version "1.0.207"}
        org.clojure/data.csv                {:mvn/version "1.0.0"}
        org.clojure/tools.cli               {:mvn/version "1.0.194"}

        ;; Database and SQL
        hikari-cp/hikari-cp       {:mvn/version "2.13.0"}
        honeysql/honeysql         {:mvn/version "1.0.444"}
        lambdaisland/uri          {:mvn/version "1.4.54"}
        migratus/migratus         {:mvn/version "1.3.3"}
        nilenso/honeysql-postgres {:mvn/version "0.4.112"}
        org.clojure/java.jdbc     {:mvn/version "0.7.11"}
        org.postgresql/postgresql {:mvn/version "42.2.18"}

        ;; HTTP and web
        buddy/buddy-auth      {:mvn/version "2.2.0"}
        clj-http/clj-http     {:mvn/version "3.10.3"}
        hiccup/hiccup         {:mvn/version "1.0.5"}
        metosin/reitit        {:mvn/version "0.5.10"}
        org.clojure/data.json {:mvn/version "1.0.0"}
        ring/ring-defaults    {:mvn/version "0.3.2"}
        ring/ring-json        {:mvn/version "0.5.0"}
        ring/ring             {:mvn/version "1.8.2"}

        ;; Integration with 3rd parties
        circleci/rollcage       {:mvn/version "1.0.208"}
        clj-commons/iapetos     {:mvn/version "0.1.11"}
        io.prometheus/simpleclient_hotspot {:mvn/version "0.11.0"}
        clj-commons/clj-ssh     {:mvn/version "0.5.15"}
        com.sun.mail/javax.mail {:mvn/version "1.6.2"}
        ovotech/clj-gcp         {:mvn/version "0.6.11"}}

 :paths ["src/clj" "src/cljc" "resources"]

 :aliases
 {:dev {:extra-paths ["env/dev/clj"]
        :extra-deps  {clj-commons/clj-yaml        {:mvn/version "0.7.0"}
                      criterium/criterium         {:mvn/version "0.4.6"}
                      pogonos/pogonos             {:mvn/version "0.1.1"}
                      ubergraph/ubergraph         {:mvn/version "0.8.2"}
                      vlaaad/reveal               {:mvn/version "1.3.196"}

                      ;; to be deleted?
                      binaryage/devtools {:mvn/version "0.9.10"}}}

  :prod {:extra-paths ["env/prod/clj"]}

  :repl {:extra-paths ["env/dev/resources"]
         :ns-default  cockpit.repl
         :jvm-opts    ["-Dvlaaad.reveal.prefs={:font-size 30 :font-family \"Roboto\"}"]}

  :test {:extra-paths ["test" "test-resources"]
         :extra-deps {clj-http-fake/clj-http-fake {:mvn/version "1.0.3"}
                      nubank/matcher-combinators  {:mvn/version "3.1.4"}
                      cloverage/cloverage           {:mvn/version "1.2.2"}
                      lambdaisland/kaocha {:mvn/version "1.0.887"}
                      lambdaisland/kaocha-cloverage {:mvn/version "1.0.75"}
                      lambdaisland/kaocha-junit-xml {:mvn/version "0.0.76"}}
         :main-opts  ["-m" "kaocha.runner"]}

  :uberjar {:replace-deps {com.github.seancorfield/depstar {:mvn/version "2.1.278"}}
            :exec-fn      hf.depstar/uberjar
            :exec-args    {:jar             "cockpit.jar"
                           :aot             true
                           :main-class      cockpit.server}}

  :frontend {:replace-paths ["src/cljs" "src/cljc"]
             :deps  {thheller/shadow-cljs        {:mvn/version "2.9.10"}
                     camel-snake-kebab           {:mvn/version "0.4.1"}
                     cljs-ajax                   {:mvn/version "0.8.0"}
                     cljs-http                   {:mvn/version "0.1.45"}
                     com.andrewmcveigh/cljs-time {:mvn/version "0.5.2"}
                     day8.re-frame/http-fx       {:mvn/version "0.1.6"}
                     metosin/reitit              {:mvn/version "0.3.10"}
                     re-frame                    {:mvn/version "1.2.0"}
                     reagent                     {:mvn/version "1.0.0"}
                     tongue                      {:mvn/version "0.3.0"}
                     fork                        {:mvn/version "2.2.4"}
                     phrase                      {:mvn/version "0.3-alpha4"}
                     binaryage/oops              {:mvn/version "0.7.0"}}
             :main-opts  ["-m" "shadow.cljs.devtools.cli"]}}}

kirill.salykin20:09:52

please advice what iI am doing wrong

kirill.salykin20:09:57

:version "1.10.3.933"

seancorfield20:09:20

Without access to your systems, I have no idea. This is really a tools.deps issue and I saw you'd posted on http://ask.clojure.org and Alex couldn't repro based on the information you have provided so far.

seancorfield20:09:28

One thing I'll observe about your deps.edn file is that -A:prod does almost nothing: it simply adds one extra path.

kirill.salykin20:09:40

Indeed I asked Alex, but thats something different

kirill.salykin20:09:58

> One thing I’ll observe about your `deps.edn` file is that `-A:prod` does almost nothing: it simply adds one extra path. indeed

kirill.salykin20:09:48

> Without access to your systems, I have no idea. This is really a `tools.deps` issue and I saw you’d posted on http://ask.clojure.org and Alex couldn’t repro based on the information you have provided so far. is there a way to debug what is happening? maybe some verbose mode so I can ask Alex with more input?

seancorfield21:09:59

OK, I ran these commands locally -- deliberately using two different local repos -- and the depstar invocation did not download anything extra:

543  clojure -Sdeps '{:mvn/local-repo "foo"}' -P -A:prod
  544  clojure -Sdeps '{:mvn/local-repo "bar"}' -P -A:uberjar
  545  clojure -Sdeps '{:mvn/local-repo "bar"}' -X:uberjar :classpath '"'$(clojure -Sdeps '{:mvn/local-repo "foo"}' -Spath -A:prod)'"' 2>&1 > /tmp/x
Note that I have -Spath before -A:prod, not after it.

seancorfield21:09:18

I'm also using -P for "prepare" and, again, before -A:...

kirill.salykin21:09:23

thanks for trying

seancorfield21:09:57

As several people have said, -P / -Spath should be before -A:... so I would suggest making that change to double-check the behavior.

kirill.salykin21:09:52

already checking this as we speak

kirill.salykin21:09:54

thank you for your advices, but ci still downloads it which tools-deps version you have locally? 1.10.3.943 or 1.10.3.967?

seancorfield21:09:47

I'm using 1.10.3.967.

seancorfield21:09:57

(I'm always on the latest version)

kirill.salykin21:09:19

maybe it version specific will try on monday Sean, thank you so much for your help! 🙏

seancorfield21:09:56

I just repeated my tests with 1.10.3.943 (I have lots of versions installed) and it still worked.

kirill.salykin21:09:09

That drives me crazy

kirill.salykin21:09:33

Then indeed it is my ci setup

seancorfield21:09:17

That made me go back over the depstar code and I think there is a path through that would create a basis without respecting the :mvn/local-repo.

seancorfield21:09:57

I realized my tests were "working" because I still have my ~/.m2/repository and I wouldn't notice if it was reaching into that for deps.

seancorfield21:09:20

I've never noticed this before because people normally have :mvn/local-repo in their deps.edn file rather than on the command-line.

seancorfield21:09:50

The issue is that it needs a basis for synchronizing the pom.xml file and it has to compute that -- the :classpath alone is not enough. But it preemptively calculates it rather than only calculating it if it needs it -- and it doesn't need it in your case because you are not sync'ing the pom file.

seancorfield21:09:35

Ah, that's not sufficient. AOT needs a basis as well, so it would still have to calculate for that.

seancorfield21:09:25

The only way this is going to work is if you have :mvn/local-repo in your deps.edn file -- until I figure out a way to let you tell depstar about it so it can be incorporated into the basis calculations it has to do for pom and AOT...

kirill.salykin21:09:35

Ah, i can create a ci alias for this

seancorfield21:09:58

I don't think that setting can be in an alias -- it has to be at the top-level.

seancorfield21:09:36

I just realized that t.d.a/create-basis can be passed :extra as a simulation of what -Sdeps provides so that would be the escape hatch.

seancorfield22:09:49

If I expose that in depstar, you wouldn't need the :classpath stuff -- you could just pass :aliases '[:prod]' :mvn/local-repo '".m2"' and let depstar handle it all.

seancorfield22:09:00

OK change your :uberjar alias to depend on com.github.seancorfield/depstar {:sha "40f2bd719a026a89d6a37b273ca72efc5d52c93b"} and then the following should work as you expect:

clojure -Sdeps '{:mvn/local-repo ".m2"}' -X:uberjar :mvn/local-repo '".m2"' :aliases '[:prod]'
I'll leave the ticket open until a) you've confirmed it actually works and b) I've properly documented this new option.

seancorfield22:09:31

(you'll still want the two -P commands to prepare the deps for the cache)

kirill.salykin22:09:56

I will! Thanks! But ut is almost 1am now, will try it tomorrow or monday

2
kirill.salykin06:09:01

it worked!!! thank you so much!

seancorfield17:09:38

Cool. I'll get it fully documented and a new release out, probably early next week.