Fork me on GitHub
#tools-deps
<
2021-11-19
>
seancorfield02:11:52

I'm trying to debug a tools issue in CI. I'm seeing

clojure -Ttool list
Error building classpath. Bad coordinate for library , expected map: nil
The CLI is 1.10.3.933 The deps.edn file in the project seems valid (`clojure -Stree` works). I've also tried updating to 1.10.3.1029. The failed CI is https://github.com/rm-hull/nvd-clojure/runs/4259660700?check_suite_focus=true -- you can see the output of clojure -Sdescribe and the root deps.edn just as a sanity check...

Alex Miller (Clojure team)03:11:58

I think that should be -Ttools ?

Alex Miller (Clojure team)03:11:15

I added a check for unknown tool name with a better error message in a couple places

seancorfield04:11:26

Er, yes, OK, that doesn't really explain what we were seeing, but I'll go back to the PR and see what's up.

seancorfield04:11:59

Here's the earlier failure:

clojure -Ttools install nvd-clojure/nvd-clojure '{:mvn/version "RELEASE"}' :as nvd
Error building classpath. Bad coordinate for library , expected map: nil
and that definitely has -Ttools @alexmiller

seancorfield04:11:44

I ended up "solving" it by writing the tools/tools.edn file manually -- so it's something to do with clojure not laying that down when it first runs...

Alex Miller (Clojure team)04:11:40

so there was no tools.edn there?

seancorfield04:11:37

Correct, here's another run:

+ clojure -Ttools list
Error building classpath. Bad coordinate for library , expected map: nil
+ ls -l /home/runner/.config/clojure/
total 4
-rw-r--r-- 1 runner docker 1490 Nov 19 04:07 deps.edn
+ ls -l /home/runner/.config/clojure/tools/
ls: cannot access '/home/runner/.config/clojure/tools/': No such file or directory
+ mkdir /home/runner/.config/clojure/tools
+ echo '{:lib io.github.clojure/tools.tools :coord {:git/tag "v0.2.2" :git/sha "e1febed7ddaa5be15721255c13eb68e11bbbb398"}}'
+ clojure -Ttools list
(works now)

seancorfield04:11:04

TOOL   LIB                            TYPE  VERSION
tools  io.github.clojure/tools.tools  :git  v0.2.2
+ clojure -Ttools install nvd-clojure/nvd-clojure '{:mvn/version "RELEASE"}' :as nvd
Installed nvd
+ clojure -Ttools list
TOOL   LIB                            TYPE  VERSION
nvd    nvd-clojure/nvd-clojure        :mvn  1.9.0
tools  io.github.clojure/tools.tools  :git  v0.2.2

Alex Miller (Clojure team)04:11:23

hmm, maybe something with using a special config dir

seancorfield04:11:31

What makes clojure decide whether to lay down the tools stuff?

Alex Miller (Clojure team)04:11:54

if it's missing or the one in the install is newer than the one there

seancorfield04:11:31

Any suggestions for debugging why it isn't doing it (above)?

Alex Miller (Clojure team)04:11:25

the logic is in the clojure script, so hack away

seancorfield04:11:59

Hmm, OK. Will take a deeper look tomorrow, when I'm back at my main system...

Alex Miller (Clojure team)04:11:06

what's your clj -Sdescribe?

Alex Miller (Clojure team)04:11:31

if [ "$install_dir/tools.edn" -nt "$config_dir/tools/tools.edn" ]; then
  mkdir -p "$config_dir/tools"
  cp "$install_dir/tools.edn" "$config_dir/tools/tools.edn"
fi
is the whole of the relevant code

Alex Miller (Clojure team)04:11:05

maybe that condition needs to explicitly check for missing, although I know I've tested this several ways

seancorfield04:11:59

Just a sec... I had clojure -Sdescribe in an earlier CI script...

seancorfield04:11:56

Here's what appeared for 933 (which failed the same way):

{:version "1.10.3.933"
 :config-files ["/opt/hostedtoolcache/ClojureToolsDeps/1.10.3-933/x64/lib/clojure/deps.edn" "/home/runner/.config/clojure/deps.edn" "deps.edn" ]
 :config-user "/home/runner/.config/clojure/deps.edn"
 :config-project "deps.edn"
 :install-dir "/opt/hostedtoolcache/ClojureToolsDeps/1.10.3-933/x64/lib/clojure"
 :config-dir "/home/runner/.config/clojure"
 :cache-dir ".cpcache"
 :force false
 :repro false
 :main-aliases ""
 :repl-aliases ""}

seancorfield04:11:47

Here's the output from 1029:

{:version "1.10.3.1029"
 :config-files ["/opt/hostedtoolcache/ClojureToolsDeps/1.10.3-1029/x64/lib/clojure/deps.edn" "/home/runner/.config/clojure/deps.edn" "deps.edn" ]
 :config-user "/home/runner/.config/clojure/deps.edn"
 :config-project "deps.edn"
 :install-dir "/opt/hostedtoolcache/ClojureToolsDeps/1.10.3-1029/x64/lib/clojure"
 :config-dir "/home/runner/.config/clojure"
 :cache-dir ".cpcache"
 :force false
 :repro false
 :main-aliases ""
 :repl-aliases ""}

seancorfield04:11:15

This is on Ubuntu 20.04.3 LTS according to that CI log.

Alex Miller (Clojure team)04:11:05

I repro'ed it when there is no tools dir

Alex Miller (Clojure team)04:11:45

actually, no I didn't, it's fine

Alex Miller (Clojure team)04:11:07

well I'm sure it's something in that area, can't really be much else

seancorfield14:11:38

@alexmiller Should I create a JIRA for this or are you already on it?

Alex Miller (Clojure team)14:11:43

if you have a repro for it....

Alex Miller (Clojure team)14:11:07

otherwise not sure what I would do with it

Alex Miller (Clojure team)14:11:15

I tested a few things, haven't been able to repro

seancorfield14:11:23

Yeah, it's a weird one. I'd never seen it before and the only repro I have is that CI pipeline...

Alex Miller (Clojure team)14:11:43

does it fail like that occasionally or always?

seancorfield14:11:32

Always. And I think I know why:

+ ls -lR /opt/hostedtoolcache/ClojureToolsDeps/1.10.3-1029/x64/lib/clojure
/opt/hostedtoolcache/ClojureToolsDeps/1.10.3-1029/x64/lib/clojure:
total 12
-rw-r--r--+ 1 runner docker  512 Nov 19 14:17 deps.edn
-rw-r--r--+ 1 runner docker 1490 Nov 19 14:17 example-deps.edn
drwxr-xr-x+ 2 runner docker 4096 Nov 19 14:17 libexec

/opt/hostedtoolcache/ClojureToolsDeps/1.10.3-1029/x64/lib/clojure/libexec:
total 15752
-rw-r--r--+ 1 runner docker 16125828 Nov 19 14:17 clojure-tools-1.10.3.1029.jar
-rw-r--r--+ 1 runner docker     3636 Nov 19 14:17 exec.jar
There is no tools.edn in the installation. So this is a bug with the DeLaGuardo/setup-clojure Action I think? @delaguardo?

seancorfield14:11:59

Looks like https://github.com/DeLaGuardo/setup-clojure/blob/master/src/cli.ts doesn't copy the tools.edn file into the tool cache.

delaguardo14:11:46

Right, this is a problem of the action. Will fix it later today

seancorfield14:11:51

@delaguardo Given the caching of the clojure setup, will you be able to fix it for existing versions, or will this only affect new CLI versions going forward?

delaguardo14:11:53

Should not be a problem. I will check it as well

seancorfield14:11:26

Cool. I just read the issue that mentioned caching which was what made me think about it...

seancorfield14:11:52

setup-clojure is awesome BTW -- thank you for your work on it!

bananadance 1
borkdude14:11:33

why isn't this using the bash script on linux and macos but it is using the powershell script on Windows?

delaguardo19:11:10

I don't have much experience with windows and rely heavenly on premade script. For other platforms it seems like a better approach to control the resources because of few reasons: • Better control what should be cached (unfortunately this hit the problem of very strange cache behavior of GitHub actions, main reason why I don't yet reply in the corresponding issue) • I don't like the way how installation script works in general. It follows some "best practices" which i don't understand personally. Maybe it is the time to reconsider, but before that i would like to have a reply to the patch i proposed for packaging scripts.

seancorfield14:11:30

(and I guess it will need to be conditional on tools.edn existing -- that only came in with 1.10.3.933 right @alexmiller?)

rickmoynihan15:11:13

@dominicm I have a project which still uses pack.alpha to create build artifacts; unfortunately the project has now acquired a dependency that uses the new :git/sha key, and not the old :sha key. This causes pack.alpha to choke… I tried bumping the tools.deps.alpha version it uses to the latest; but it seems there have been some breaking changes for you in that dep. Specifically clojure.tools.deps.alpha.reader/merge-deps install-deps and slurp-deps have now moved/vanished.

Alex Miller (Clojure team)15:11:40

they're in clojure.tools.deps.alpha (that change happened long long ago)

rickmoynihan15:11:34

I guess I’m probably better migrating this to use tools.build uber instead; though I was using the skinny jar functionality to preserve the true classpath; rather than munge it

rickmoynihan15:11:53

@alexmiller: so they’ve just moved ns? Or are they removed entirely now?

rickmoynihan15:11:27

just trying to figure out what the quicker fix will be — migrating or patching a fork of pack.alpha

Alex Miller (Clojure team)15:11:29

moved, and I think the install one changed names to root-deps

Alex Miller (Clojure team)15:11:51

the merge stuff might be factored slightly differently too, don't remember now

Alex Miller (Clojure team)15:11:10

the same stuff is all there, just might be slightly different api

rickmoynihan15:11:01

yeah — I’ll see if I can patch it

rickmoynihan15:11:45

thanks again for pointing me in the right direction 🙇

rickmoynihan15:11:09

perfect that’s the commit I’m looking for 👌

rickmoynihan16:11:17

@alexmiller thanks, that was exactly what I was looking for… I’ve patched up the skinny feature of pack.alpha here: https://github.com/juxt/pack.alpha/pull/94

mkvlr19:11:59

when generating a jar, do all deps need to be available on maven as well and I can’t depend on :git/sha deps? Seeing the following warning and indeed valuehash is missing from the jar:

$ clj -T:build jar
Producing jar: target/clerk-0.3.265.jar
Skipping coordinate: {:git/sha ff1d4b7f1260daf41c786a61cb45d02871b7baf9, :git/url , :deps/manifest :deps, :deps/root /Users/mk/.gitlibs/libs/io.github.arachne-framework/valuehash/ff1d4b7f1260daf41c786a61cb45d02871b7baf9, :parents #{[]}, :paths [/Users/mk/.gitlibs/libs/io.github.arachne-framework/valuehash/ff1d4b7f1260daf41c786a61cb45d02871b7baf9/src]}
I guess it makes sense that this doesn’t work. Any recommended way to bring this in (I can’t find a jar on clojars or central)? I guess https://jitpack.io should work.

seancorfield19:11:07

@mkvlr That message comes from tools.deps when it is building the pom.xml file -- and, yes, you can't have a JAR on Maven/Clojars that depends on git deps because consumers generally couldn't resolve that.

mkvlr19:11:53

@seancorfield makes sense, would copying the sources into my jar be a bad idea?

seancorfield19:11:48

Potentially a bad idea, yes. Users of your JAR would "unexpectedly" get all those other nses and that might cause conflicts.

mkvlr19:11:07

so better to fork & publish under a different name with different namespace root?

mkvlr19:11:55

jitpack doesn’t work, or only partially, master seems to work but the lastest sha which master points to doesn’t. Seems to be bad to depend on a mutable name for sure.

seancorfield19:11:10

This was a possibility with depstar -- by default it copied local and git deps into the JAR (since consumers of JARs couldn't consume those as deps in the pom) but there were caveats in the docs about it, and an option to not do that. tools.build by default does the latter -- not copying them (which is the better default behavior). Hard to say what you should do without knowing more about your project...

seancorfield19:11:24

You're depending on Arachne but that doesn't have published artifacts?

seancorfield19:11:14

In which case, anyone using your library would also have to depend on Arachne via git anyway -- so why publish your library at all? Why not let people depend on it via git, like they would have to do for Arachne?

seancorfield19:11:02

(git deps are "contagious" in that respect: once you depend on them, your "customers" have to depend on them as well...)

mkvlr19:11:30

I want consumers of our lib (http://github.com/nextjournal/clerk) to use it via maven as well (if they’re using lein or whatever). In general I’d like to make consumption easy. Can’t find a published artifact of https://github.com/arachne-framework/valuehash which I’d like to depend on, I’ll open a ticket with them.

Alex Miller (Clojure team)19:11:05

I'm not sure Luke is still doing anything with that

Alex Miller (Clojure team)19:11:12

so you might need to fork and publish if you want it out there

mkvlr19:11:50

@alexmiller I’ll ask and do that if I don’t hear back, thank you. Might have been once published to http://maven.arachne-framework.org/artifactory/arachne-dev which shows up in some repos but http only is probably also not a good idea?

Alex Miller (Clojure team)19:11:12

not even supported by most recent maven

👍 1
Alex Miller (Clojure team)19:11:33

subject to MITM attacks