Clojurians Log v2

There is no 1.12 version of the Clojure CLI. You need the latest version of the CLI (1.11.1.1273)

Any version of the CLI can use any version of Clojure - that depends on the Clojure version in your deps.edn

Thanks! Can I assume that any stable Clojure release has a corresponding CLI?

That’s not a meaningful question

So, no

The CLI version is actually meaningfully only the last part. The first 3 parts indicate the Clojure version it was built with and the Clojure version you’ll get from the root deps.edn if you don’t say otherwise

Haha, thanks. Is there a way I can programmatically figure out the latest CLI download path?

Yes

There is a file https://download.clojure.org/install/stable.properties

That has the version and the sha of the download

Also if you use the url at the top but without the version, that’s always the latest

https://download.clojure.org/install/linux-install.sh

Awesome! Exactly what I need.

-Srepro will omit the user deps.edn

You always need a Clojure dep, the default is in the root deps.edn, but overrides by project deps.edn

But maybe it’s more useful to tell me what you’re actually trying to do

I am trying to feed ClojureCLR a classpath based on a deps.edn file. This is what I have now:

#!/usr/bin/env bb

(require '[clojure.string :as string])
(require '[babashka.process :refer [sh]])
(require '[babashka.fs :as fs])

(sh "clojure" "-P" "-A:dev")

(def original-classpath (string/trim (:out (sh "clojure" "-Spath" "-A:dev"))))

(def skip-deps ["org/clojure/clojure" 
                "org/clojure/core.specs.alpha"
                "org/clojure/spec.alpha"])
(def skipped-deps-re (re-pattern (str ".*/\\.m2/repository/("
                                      (string/join "|" skip-deps)
                                      ")/.*")))

(defn skip-path? [path]
  (re-matches skipped-deps-re path))

(defn process-classpath [classpath]
  (let [paths (string/split classpath #":")]
    (->> paths
         (remove skip-path?)
         (map (fn [path]
                (if (.endsWith path ".jar")
                  (let [deps-subdir (string/replace (fs/file-name path) #".jar$" "")
                        deps-dir (str "/app/dependencies/" deps-subdir)]
                    (sh "mkdir" "-p" deps-dir)
                    (sh "unzip" "-q" "-o" path "-d" deps-dir)
                    deps-dir)
                  path)))
         (string/join ":"))))

(when (= *file* (System/getProperty "babashka.file"))
  (-> original-classpath
      process-classpath
      println))

I’ll add the -Srepro, even if it isn’t strictly needed because this is run in a Docker container that does not have a user deps.edn.

You can just call tools.deps directly

Instead of shelling out to use the CLI which uses tools.deps

Hmmm, not sure how. Babashka has tools.deps in there somewhere?

Call clojure.tools.deps/create-basis and pass it :root nil :user nil. That gives you a basis with a :classpath-roots, then call join-classpath

I think it does, but prob a question for the borkmeister

Alternately you can call the create-basis variant provided in the :deps alias

clj -X:deps create-basis :root nil :user nil

Something like that, on my phone

I’ll experiment with this a bit.

Oh wait, that’s not there anymore but there is prob some path to doing it

But really, tools.deps is a library to make classpaths. Ideally you can just call it to do that

@pez babashka's classpath is empty by default and if you use (babashka.deps/add-deps '{:deps {}}) it will only add those deps to the classpath, not even clojure itself

You can retrieve the bb classpath with (babashka.classpath/get-classpath)

This is done via the :classpath-overrides option similar to this:

clj -Sdeps '{:deps {} :aliases {:rm-clojure {:classpath-overrides {org.clojure/clojure nil}}}}' -A:rm-clojure -Spath

Can I get add-deps to use my deps.edn file?

@pez Yes: (add-deps (edn/read-string (slurp "my-deps.edn")))

How would I tell it to use the :dev alias in there?

@pez

(add-deps {:deps ..} {:aliases [:dev]})

Awesome, works! Does it also download dependencies?

yes

This is great! Now I have this:

#!/usr/bin/env bb

(require '[clojure.string :as string])
(require '[clojure.edn :as edn])
(require '[babashka.process :refer [sh]])
(require '[babashka.deps :as deps])
(require '[babashka.classpath :as classpath])
(require '[babashka.fs :as fs])

(defn process-classpath [classpath]
  (let [paths (string/split classpath #":")]
    (->> paths
         (map (fn [path]
                (if (.endsWith path ".jar")
                  (let [deps-subdir (string/replace (fs/file-name path) #".jar$" "")
                        deps-dir (str "/app/dependencies/" deps-subdir)]
                    (sh "mkdir" "-p" deps-dir)
                    (sh "unzip" "-q" "-o" path "-d" deps-dir)
                    deps-dir)
                  path)))
         (string/join ":"))))

(when (= *file* (System/getProperty "babashka.file"))
  (deps/add-deps (edn/read-string (slurp "deps.edn")) {:aliases [:dev]})
  (-> (classpath/get-classpath)
      process-classpath
      println))

root@d278348ba283:/app# output=$(docker/cheap-deps.clj) 
Downloading: org/clojure/clojure/1.11.1/clojure-1.11.1.pom from central
Downloading: org/clojure/spec.alpha/0.3.218/spec.alpha-0.3.218.pom from central
Downloading: org/clojure/core.specs.alpha/0.2.62/core.specs.alpha-0.2.62.pom from central
Downloading: org/clojure/pom.contrib/1.1.0/pom.contrib-1.1.0.pom from central
Downloading: org/clojure/clr/tools.nrepl/0.1.0-alpha1/tools.nrepl-0.1.0-alpha1.pom from clojars
Downloading: org/clojure/clr/tools.reader/1.3.7/tools.reader-1.3.7.pom from clojars
Downloading: org/clojure/core.specs.alpha/0.2.62/core.specs.alpha-0.2.62.jar from central
Downloading: org/clojure/spec.alpha/0.3.218/spec.alpha-0.3.218.jar from central
Downloading: org/clojure/clojure/1.11.1/clojure-1.11.1.jar from central
Downloading: org/clojure/clr/tools.reader/1.3.7/tools.reader-1.3.7.jar from clojars
Downloading: org/clojure/clr/tools.nrepl/0.1.0-alpha1/tools.nrepl-0.1.0-alpha1.jar from clojars
root@d278348ba283:/app# echo $output
src:dev:/app/dependencies/tools.nrepl-0.1.0-alpha1:/app/dependencies/tools.reader-1.3.7

> What’s short command line to tell Babashka to do this initial work? I'm not sure what the question is

btw instead of (sh "mkdir" "-p") , you could use (fs/create-dirs deps-dir) and for (sh "unzip") you could possibly use fs/unzip

Also instead of #":" I recommend using fs/path-separator for better cross-platform compatibility

The question. I want to populate the maven repo with the stuff that babashka needs at container build time. So that running my script doesn’t pay that overhead.

bb already has babashka.classpath/split-classpath which can split the classpath for you in a cross-platform manner

> I want to populate the maven repo with the stuff that babashka needs at container build time Do you mean the "installation" of deps.clj, i.e. downloading the tools jar?

Yes, if that’s what causes all those downloads, then that’s what I mean. 😃

The script now looks like

(defn process-classpath [classpath]
  (->> (classpath/split-classpath classpath)
       (map (fn [path]
              (if (.endsWith path ".jar")
                (let [deps-subdir (string/replace (fs/file-name path) #".jar$" "")
                      deps-dir (str "/app/dependencies/" deps-subdir)]
                  (fs/create-dirs deps-dir)
                  (fs/unzip path deps-dir {:replace-existing true})
                  deps-dir)
                path)))
       (string/join ":")))

(when (= *file* (System/getProperty "babashka.file"))
  (deps/add-deps (edn/read-string (slurp "deps.edn")) {:aliases [:dev]})
  (-> (classpath/get-classpath)
      process-classpath
      println))

> Yes, if that’s what causes all those downloads, then that’s what I mean. I don't know what you mean by "all those downloads", can you be more specific? If you download deps using this script, then it's expected that ... there are "all those downloads"?

(.endsWith path ".jar") => (str/ends-with? path ".jar") would be my preference here

Sorry for being unclear. Of those downloads there, only the last two are from my deps.edn (one directly, and one transiently). The rest is something triggered by my script.

are you sure? e.g.:

Downloading: org/clojure/clr/tools.nrepl/0.1.0-alpha1/tools.nrepl-0.1.0-alpha1.pom from clojars
Downloading: org/clojure/clr/tools.reader/1.3.7/tools.reader-1.3.7.pom from clojars

I'd do an -Stree or -X:deps tree to double check

Those are the last two. My deps.edn looks like so:

{:paths ["src"]
 :aliases {:dev {:extra-paths ["dev"]
                 :extra-deps {org.clojure.clr/tools.nrepl {:mvn/version "0.1.0-alpha1"}}}}}

# clojure -Stree 
org.clojure/clojure 1.11.1
  . org.clojure/spec.alpha 0.3.218
  . org.clojure/core.specs.alpha 0.2.62

Well, those other deps aren't needed by bb so I think it's just the result of tools.deps downloading those as well

Is there a way I can trigger the download of those dependencies without running my script?

This isn’t enough:

RUN curl -sLO  \
    && chmod +x install \
    && ./install \
    && bb -e '(require (quote [babashka.deps :as deps]))'

I think you could even prevent downloading those dependencies with a hack:

clj -Sdeps '{:deps {org.clojure/clojure {:local/root "src" :deps/manifest :deps} org.clojure/core.specs.alpha {:local/root "src" :deps/manifest :deps}}}}' -Spath
src:/private/tmp/src/src:/private/tmp/src/src

but note that those libs will end up on the classpath, but combined with :classpath-overrides they will then go away again, I think

You could force downloading those dependencies by just pre-fetching an arbitrary dependency with add-deps in the image build, probably

I don’t have clj in the container any longer. Can I do this with Babashka?

yes, with add-deps

bb -e '(babashka.deps/add-deps ...)'

$ bb -e '(babashka.deps/add-deps (quote {:mvn/local-repo "/tmp/src" :deps {medley/medley {:mvn/version "1.0.0"}}}))'
Downloading: org/clojure/clojure/1.11.1/clojure-1.11.1.pom from central
Downloading: org/clojure/spec.alpha/0.3.218/spec.alpha-0.3.218.pom from central
Downloading: org/clojure/core.specs.alpha/0.2.62/core.specs.alpha-0.2.62.pom from central
Downloading: org/clojure/pom.contrib/1.1.0/pom.contrib-1.1.0.pom from central
Downloading: medley/medley/1.0.0/medley-1.0.0.pom from clojars
Downloading: org/clojure/core.specs.alpha/0.2.62/core.specs.alpha-0.2.62.jar from central
Downloading: org/clojure/spec.alpha/0.3.218/spec.alpha-0.3.218.jar from central
Downloading: org/clojure/clojure/1.11.1/clojure-1.11.1.jar from central
Downloading: medley/medley/1.0.0/medley-1.0.0.jar from clojars

so yeah, it seems to download clojure + spec etc even though they are cancelled out on the classpath with classpath-overrides 🤷

$ bb -e '(babashka.deps/add-deps (quote {:mvn/local-repo "/tmp/src" :deps {medley/medley {:mvn/version "1.0.0"}}})) (prn (babashka.classpath/get-classpath))'
"/tmp/src/medley/medley/1.0.0/medley-1.0.0.jar"