Fork me on GitHub
#tools-deps
<
2023-05-14
>
pez08:05:32

What’s the best way to get 1.12.0-alpha3 installed in an apt-get based Docker container? This doesn’t work:

root@da127afd0308:/app# curl -O 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   243    0   243    0     0    371      0 --:--:-- --:--:-- --:--:--   371
root@da127afd0308:/app# tail -3 linux-install-1.12.0-alpha3.sh 
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>QHDZR8BX0JE4XQS0</RequestId><HostId>6kINnasq/o/x2gOdNFw0J3HxPPk1Owa081QnyKtTj9yPEHsOB6G/5o2HbcX5iBlYQ07TZ/AkFBo=</HostId></Error>root@da127afd0308:/app# 

Alex Miller (Clojure team)11:05:29

There is no 1.12 version of the Clojure CLI. You need the latest version of the CLI (1.11.1.1273)

Alex Miller (Clojure team)11:05:39

Any version of the CLI can use any version of Clojure - that depends on the Clojure version in your deps.edn

pez12:05:02

Thanks! Can I assume that any stable Clojure release has a corresponding CLI?

Alex Miller (Clojure team)12:05:55

That’s not a meaningful question

Alex Miller (Clojure team)12:05:09

The CLI version is actually meaningfully only the last part. The first 3 parts indicate the Clojure version it was built with and the Clojure version you’ll get from the root deps.edn if you don’t say otherwise

pez12:05:27

Haha, thanks. Is there a way I can programmatically figure out the latest CLI download path?

Alex Miller (Clojure team)12:05:41

That has the version and the sha of the download

Alex Miller (Clojure team)12:05:48

Also if you use the url at the top but without the version, that’s always the latest

pez12:05:29

Awesome! Exactly what I need.

pez15:05:19

Is there a way to tell clojure to only consider the classpath that is specified in the local deps.edn file? So, nothing from user deps.edn and nothing that is not specified (like no implicit clojure dependency)?

Alex Miller (Clojure team)15:05:49

-Srepro will omit the user deps.edn

Alex Miller (Clojure team)15:05:44

You always need a Clojure dep, the default is in the root deps.edn, but overrides by project deps.edn

Alex Miller (Clojure team)15:05:13

But maybe it’s more useful to tell me what you’re actually trying to do

pez15:05:55

I am trying to feed ClojureCLR a classpath based on a deps.edn file. This is what I have now:

#!/usr/bin/env bb

(require '[clojure.string :as string])
(require '[babashka.process :refer [sh]])
(require '[babashka.fs :as fs])

(sh "clojure" "-P" "-A:dev")

(def original-classpath (string/trim (:out (sh "clojure" "-Spath" "-A:dev"))))

(def skip-deps ["org/clojure/clojure" 
                "org/clojure/core.specs.alpha"
                "org/clojure/spec.alpha"])
(def skipped-deps-re (re-pattern (str ".*/\\.m2/repository/("
                                      (string/join "|" skip-deps)
                                      ")/.*")))

(defn skip-path? [path]
  (re-matches skipped-deps-re path))

(defn process-classpath [classpath]
  (let [paths (string/split classpath #":")]
    (->> paths
         (remove skip-path?)
         (map (fn [path]
                (if (.endsWith path ".jar")
                  (let [deps-subdir (string/replace (fs/file-name path) #".jar$" "")
                        deps-dir (str "/app/dependencies/" deps-subdir)]
                    (sh "mkdir" "-p" deps-dir)
                    (sh "unzip" "-q" "-o" path "-d" deps-dir)
                    deps-dir)
                  path)))
         (string/join ":"))))

(when (= *file* (System/getProperty "babashka.file"))
  (-> original-classpath
      process-classpath
      println))

pez15:05:16

I’ll add the -Srepro, even if it isn’t strictly needed because this is run in a Docker container that does not have a user deps.edn.

Alex Miller (Clojure team)15:05:32

You can just call tools.deps directly

👀 2
Alex Miller (Clojure team)15:05:59

Instead of shelling out to use the CLI which uses tools.deps

pez15:05:43

Hmmm, not sure how. Babashka has tools.deps in there somewhere?

Alex Miller (Clojure team)15:05:02

Call clojure.tools.deps/create-basis and pass it :root nil :user nil. That gives you a basis with a :classpath-roots, then call join-classpath

Alex Miller (Clojure team)15:05:26

I think it does, but prob a question for the borkmeister

Alex Miller (Clojure team)15:05:04

Alternately you can call the create-basis variant provided in the :deps alias

Alex Miller (Clojure team)15:05:48

clj -X:deps create-basis :root nil :user nil

Alex Miller (Clojure team)15:05:02

Something like that, on my phone

pez15:05:56

I’ll experiment with this a bit.

Alex Miller (Clojure team)15:05:43

Oh wait, that’s not there anymore but there is prob some path to doing it

Alex Miller (Clojure team)15:05:27

But really, tools.deps is a library to make classpaths. Ideally you can just call it to do that

borkdude15:05:30

@U0ETXRFEW babashka's classpath is empty by default and if you use (babashka.deps/add-deps '{:deps {}}) it will only add those deps to the classpath, not even clojure itself

borkdude15:05:36

You can retrieve the bb classpath with (babashka.classpath/get-classpath)

borkdude15:05:09

This is done via the :classpath-overrides option similar to this:

clj -Sdeps '{:deps {} :aliases {:rm-clojure {:classpath-overrides {org.clojure/clojure nil}}}}' -A:rm-clojure -Spath

pez15:05:44

Can I get add-deps to use my deps.edn file?

borkdude15:05:14

@U0ETXRFEW Yes: (add-deps (edn/read-string (slurp "my-deps.edn")))

pez15:05:19

How would I tell it to use the :dev alias in there?

borkdude15:05:28

@U0ETXRFEW

(add-deps {:deps ..} {:aliases [:dev]})

pez15:05:32

Awesome, works! Does it also download dependencies?

pez16:05:50

This is great! Now I have this:

#!/usr/bin/env bb

(require '[clojure.string :as string])
(require '[clojure.edn :as edn])
(require '[babashka.process :refer [sh]])
(require '[babashka.deps :as deps])
(require '[babashka.classpath :as classpath])
(require '[babashka.fs :as fs])

(defn process-classpath [classpath]
  (let [paths (string/split classpath #":")]
    (->> paths
         (map (fn [path]
                (if (.endsWith path ".jar")
                  (let [deps-subdir (string/replace (fs/file-name path) #".jar$" "")
                        deps-dir (str "/app/dependencies/" deps-subdir)]
                    (sh "mkdir" "-p" deps-dir)
                    (sh "unzip" "-q" "-o" path "-d" deps-dir)
                    deps-dir)
                  path)))
         (string/join ":"))))

(when (= *file* (System/getProperty "babashka.file"))
  (deps/add-deps (edn/read-string (slurp "deps.edn")) {:aliases [:dev]})
  (-> (classpath/get-classpath)
      process-classpath
      println))
It does some work when running the script that I might want to have it do as part of building the container. Like downloading Clojure and stuff.
root@d278348ba283:/app# output=$(docker/cheap-deps.clj) 
Downloading: org/clojure/clojure/1.11.1/clojure-1.11.1.pom from central
Downloading: org/clojure/spec.alpha/0.3.218/spec.alpha-0.3.218.pom from central
Downloading: org/clojure/core.specs.alpha/0.2.62/core.specs.alpha-0.2.62.pom from central
Downloading: org/clojure/pom.contrib/1.1.0/pom.contrib-1.1.0.pom from central
Downloading: org/clojure/clr/tools.nrepl/0.1.0-alpha1/tools.nrepl-0.1.0-alpha1.pom from clojars
Downloading: org/clojure/clr/tools.reader/1.3.7/tools.reader-1.3.7.pom from clojars
Downloading: org/clojure/core.specs.alpha/0.2.62/core.specs.alpha-0.2.62.jar from central
Downloading: org/clojure/spec.alpha/0.3.218/spec.alpha-0.3.218.jar from central
Downloading: org/clojure/clojure/1.11.1/clojure-1.11.1.jar from central
Downloading: org/clojure/clr/tools.reader/1.3.7/tools.reader-1.3.7.jar from clojars
Downloading: org/clojure/clr/tools.nrepl/0.1.0-alpha1/tools.nrepl-0.1.0-alpha1.jar from clojars
root@d278348ba283:/app# echo $output
src:dev:/app/dependencies/tools.nrepl-0.1.0-alpha1:/app/dependencies/tools.reader-1.3.7
What’s short command line to tell Babashka to do this initial work?

borkdude16:05:25

> What’s short command line to tell Babashka to do this initial work? I'm not sure what the question is

borkdude16:05:26

btw instead of (sh "mkdir" "-p") , you could use (fs/create-dirs deps-dir) and for (sh "unzip") you could possibly use fs/unzip

🙏 2
borkdude16:05:39

Also instead of #":" I recommend using fs/path-separator for better cross-platform compatibility

🙏 2
pez16:05:41

The question. I want to populate the maven repo with the stuff that babashka needs at container build time. So that running my script doesn’t pay that overhead.

borkdude16:05:07

bb already has babashka.classpath/split-classpath which can split the classpath for you in a cross-platform manner

🙏 2
borkdude16:05:53

> I want to populate the maven repo with the stuff that babashka needs at container build time Do you mean the "installation" of deps.clj, i.e. downloading the tools jar?

pez16:05:18

Yes, if that’s what causes all those downloads, then that’s what I mean. 😃

pez16:05:52

The script now looks like

(defn process-classpath [classpath]
  (->> (classpath/split-classpath classpath)
       (map (fn [path]
              (if (.endsWith path ".jar")
                (let [deps-subdir (string/replace (fs/file-name path) #".jar$" "")
                      deps-dir (str "/app/dependencies/" deps-subdir)]
                  (fs/create-dirs deps-dir)
                  (fs/unzip path deps-dir {:replace-existing true})
                  deps-dir)
                path)))
       (string/join ":")))

(when (= *file* (System/getProperty "babashka.file"))
  (deps/add-deps (edn/read-string (slurp "deps.edn")) {:aliases [:dev]})
  (-> (classpath/get-classpath)
      process-classpath
      println))
And runs crazy fast compared to what I started with!

borkdude16:05:41

> Yes, if that’s what causes all those downloads, then that’s what I mean. I don't know what you mean by "all those downloads", can you be more specific? If you download deps using this script, then it's expected that ... there are "all those downloads"?

borkdude16:05:22

(.endsWith path ".jar") => (str/ends-with? path ".jar") would be my preference here

🙏 2
pez17:05:17

Sorry for being unclear. Of those downloads there, only the last two are from my deps.edn (one directly, and one transiently). The rest is something triggered by my script.

borkdude17:05:53

are you sure? e.g.:

Downloading: org/clojure/clr/tools.nrepl/0.1.0-alpha1/tools.nrepl-0.1.0-alpha1.pom from clojars
Downloading: org/clojure/clr/tools.reader/1.3.7/tools.reader-1.3.7.pom from clojars

borkdude17:05:50

I'd do an -Stree or -X:deps tree to double check

pez18:05:42

Those are the last two. My deps.edn looks like so:

{:paths ["src"]
 :aliases {:dev {:extra-paths ["dev"]
                 :extra-deps {org.clojure.clr/tools.nrepl {:mvn/version "0.1.0-alpha1"}}}}}

pez18:05:13

# clojure -Stree 
org.clojure/clojure 1.11.1
  . org.clojure/spec.alpha 0.3.218
  . org.clojure/core.specs.alpha 0.2.62

borkdude18:05:24

Well, those other deps aren't needed by bb so I think it's just the result of tools.deps downloading those as well

pez18:05:06

Is there a way I can trigger the download of those dependencies without running my script?

pez19:05:35

This isn’t enough:

RUN curl -sLO  \
    && chmod +x install \
    && ./install \
    && bb -e '(require (quote [babashka.deps :as deps]))'

borkdude19:05:39

I think you could even prevent downloading those dependencies with a hack:

clj -Sdeps '{:deps {org.clojure/clojure {:local/root "src" :deps/manifest :deps} org.clojure/core.specs.alpha {:local/root "src" :deps/manifest :deps}}}}' -Spath
src:/private/tmp/src/src:/private/tmp/src/src
So by setting clojure + core.specs.alpha to a local (existing!) dir it won't download those

borkdude19:05:09

but note that those libs will end up on the classpath, but combined with :classpath-overrides they will then go away again, I think

borkdude19:05:16

You could force downloading those dependencies by just pre-fetching an arbitrary dependency with add-deps in the image build, probably

pez19:05:58

I don’t have clj in the container any longer. Can I do this with Babashka?

borkdude19:05:08

yes, with add-deps

borkdude19:05:19

bb -e '(babashka.deps/add-deps ...)'

borkdude19:05:15

$ bb -e '(babashka.deps/add-deps (quote {:mvn/local-repo "/tmp/src" :deps {medley/medley {:mvn/version "1.0.0"}}}))'
Downloading: org/clojure/clojure/1.11.1/clojure-1.11.1.pom from central
Downloading: org/clojure/spec.alpha/0.3.218/spec.alpha-0.3.218.pom from central
Downloading: org/clojure/core.specs.alpha/0.2.62/core.specs.alpha-0.2.62.pom from central
Downloading: org/clojure/pom.contrib/1.1.0/pom.contrib-1.1.0.pom from central
Downloading: medley/medley/1.0.0/medley-1.0.0.pom from clojars
Downloading: org/clojure/core.specs.alpha/0.2.62/core.specs.alpha-0.2.62.jar from central
Downloading: org/clojure/spec.alpha/0.3.218/spec.alpha-0.3.218.jar from central
Downloading: org/clojure/clojure/1.11.1/clojure-1.11.1.jar from central
Downloading: medley/medley/1.0.0/medley-1.0.0.jar from clojars

borkdude19:05:41

so yeah, it seems to download clojure + spec etc even though they are cancelled out on the classpath with classpath-overrides 🤷

borkdude19:05:23

$ bb -e '(babashka.deps/add-deps (quote {:mvn/local-repo "/tmp/src" :deps {medley/medley {:mvn/version "1.0.0"}}})) (prn (babashka.classpath/get-classpath))'
"/tmp/src/medley/medley/1.0.0/medley-1.0.0.jar"