Fork me on GitHub
#tools-deps
<
2021-09-15
>
vemv00:09:45

Can these be silenced?

Downloading: io/grpc/grpc-core/maven-metadata.xml from central
Downloading: io/grpc/grpc-api/maven-metadata.xml from central
Downloading: org/clojure/clojure/maven-metadata.xml from clojars
Downloading: io/grpc/grpc-netty-shaded/maven-metadata.xml from central
Downloading: org/clojure/clojure/maven-metadata.xml from central
(sorry if FAQ)

seancorfield01:09:48

@vemv That indicates something is using either a version range or a symbolic version (LATEST or RELEASE).

vemv01:09:14

TIL, thanks! makes sense. we're not even using the one dep that is specified at RELEASE so yeah... will zap it

Carsten Behring09:09:55

Sometimes running "clj" downloads a large amount of dependencies. There is a progress report, but per download only. But if any single download is very large, we do not see any progress, and clj seems to hang. "mvn install" does this better, it has a per file donload progress. Maybe this it could be added to "clj" ? Sometimes I do a "clj -Spom; mvn install" just to avoid this apparent hanging

Alex Miller (Clojure team)12:09:33

We have a ticket for this, can’t say it’s very high on my priority list

rickmoynihan15:09:45

Are there any known issues with tools.deps and :mvn/repos where tools deps won’t fetch artifacts from a 3rd party repo when you have 2 private s3 repos for your corp (in ~/.clojure/deps.edn ) and then add an extra 3rd party repo for a supplier in a specific projects deps.edn?

rickmoynihan15:09:38

Basically we’re finding that we can’t resolve a bunch of deps that one of our suppliers is hosting on their own maven repo when we include our s3 repos too

borkdude15:09:11

Is this the s3 concurrency thing again?

rickmoynihan15:09:25

I was wondering that actually

rickmoynihan15:09:00

I’ll try setting -Sthreads 1

Alex Miller (Clojure team)15:09:12

how are you providing creds? ambient AWS or via repo settings?

Alex Miller (Clojure team)15:09:37

doesn't sound like a concurrency thing, sounds like an access thing

rickmoynihan15:09:55

For the s3 repos I think it’s done via settings.xml — for the supplier repo — there are no creds (it’s all public)

rickmoynihan15:09:18

the s3 repos are our in-house stuff, and they work in other projects etc fine — it’s only the project which adds a new supplier repo in the project deps.edn that struggles. It also gets our deps fine; it’s the suppliers deps it doesn’t resolve.

rickmoynihan15:09:39

but if I remove our s3 repos from ~/.clojure/deps.edn and remove the deps we need from there from the project; so just the suppliers :mvn/repos are active in the projects deps.edn with their dependencies it works.

borkdude15:09:14

@rickmoynihan you can use -Srepro for this

👌 2
rickmoynihan15:09:42

ahh I think I see what @alexmiller is saying — that perhaps the s3 creds are providing creds for the 3rd party repo!? Which it’s then refusing?!

rickmoynihan15:09:16

A colleague has also managed to recreate the same findings (after blowing away her .m2)

rickmoynihan15:09:25

-Sthreads 1 seems to be helping… it looks like it has got further anyway (just taking a long time to fetch)

rickmoynihan15:09:48

ok -Sthreads 1 seems to have actually worked — though it took forever

rickmoynihan15:09:48

I’m not sure if the supplier have some dodgy deps either — as literally 1/2 of maven central came down… more than I’d normally expect… e.g. even apache ant! 😕

rickmoynihan15:09:22

at one point I thought it might have been fetching the same deps in a loop -- though could have been deps with similar names

rickmoynihan15:09:31

checking -Stree output now to see if anything odd is going on

rickmoynihan16:09:51

hmm ok I guess it’s legit though does seem a bit odd:

com.cemerick/url 0.1.1
  . pathetic/pathetic 0.5.0
    . com.cemerick/clojurescript.test 0.0.4
      . org.clojure/clojurescript 0.0-1586
        . com.google.javascript/closure-compiler r2180
          . args4j/args4j 2.0.16
          X com.google.guava/guava 13.0.1 :older-version
          X com.google.protobuf/protobuf-java 2.4.1 :superseded
          . org.json/json 20090211
          . org.apache.ant/ant 1.8.2
            . org.apache.ant/ant-launcher 1.8.2
          X com.google.code.findbugs/jsr305 1.3.9 :older-version
          . com.googlecode.jarjar/jarjar 1.1
        . org.clojure/google-closure-library 0.0-2029-2
          . org.clojure/google-closure-library-third-party 0.0-2029-2
        . org.mozilla/rhino 1.7R4

souenzzo19:09:33

Hello I'm having this error and I can't understand why Error building classpath. Could not find artifact com.datomic:ion:jar:0.9.50 in central () This error says that i can't find com.datomic/ion, but it exists in my fs

ls ~/.m2/repository/com/datomic/ion/0.9.50/ 
ion-0.9.50.jar  ion-0.9.50.jar.sha1  ion-0.9.50.pom  ion-0.9.50.pom.sha1  _remote.repositories
Also, the same deps.edn, in exacly the same commit works on other machines (CI, other devs, and my machine if I use a time machine to be back in yesterday) cognitect-dev-tools credentials is present in ~/.m2/settings.xml and :mvn/repos configured in ~/.clojure/deps.edn

souenzzo19:09:43

clj -Strace do not write trade.edn

souenzzo19:09:44

clj -Srepro -Sforce -Sdeps '{}' -Strace do the same error as clj -Spath (in the same directory that contains my deps.edn) clj -Strace in a empty dir works

Alex Miller (Clojure team)19:09:52

Do you have ambient aws creds with s3 read?

souenzzo19:09:05

i can do aws s3 ls and it list my S3 buckets

souenzzo19:09:15

there is something that I can do to get more debug info? why trace do not work? why -Sforce -Srepro -Sdeps {} do not work?

Alex Miller (Clojure team)19:09:07

why are you doing -Sdeps {} ?

souenzzo19:09:28

-Sdeps do not "overwrite" deps.edn?

souenzzo19:09:01

oh. ok. And about trace? Why it not work? Why/which package is requiring datomic/ion? (that is a huge project, with a lot of local/root) Why it is trying to find datomic/ion if it exists on disk?

Alex Miller (Clojure team)19:09:04

trace is written at the end, so wouldn't work if you are erroring out

Alex Miller (Clojure team)19:09:16

not sure I can answer why/which is requiring datomic/ion - presumably something that uses datomic

Alex Miller (Clojure team)19:09:45

it might be looking for a different version of datomic/ion than you have

Alex Miller (Clojure team)19:09:05

is this a deps.edn you can share?

souenzzo19:09:45

nope 😕 It contians many local/root and git deps.

Alex Miller (Clojure team)19:09:48

I guess it looks like the same version above

Alex Miller (Clojure team)19:09:22

I would certainly try re-running your original command with -Sforce (no -Sdeps)

Alex Miller (Clojure team)19:09:47

other than that, not sure what to suggest other than selectively bisecting your deps to get to something shareable/traceable

Alex Miller (Clojure team)19:09:06

that doesn't look like the right maven repo

Alex Miller (Clojure team)19:09:11

should be "datomic-cloud" {:url ""} I think?

Alex Miller (Clojure team)19:09:59

clj -Sdeps '{:deps {com.datomic/ion {:mvn/version "0.9.50"}} :mvn/repos {"datomic-cloud" {:url ""}}}

Alex Miller (Clojure team)19:09:33

as to why it's downloading when it exists, I'm honestly not sure about that one

Alex Miller (Clojure team)19:09:10

my suspicion is that some of the update checker stuff in maven relies on metadata files the s3 repo does not provide and it's redoing something it shouldn't

Alex Miller (Clojure team)19:09:48

certainly that would make sense to me for SNAPSHOT or RELEASE versions, but in this case it's neither so that seems weird (unless you did have that somewhere)

souenzzo19:09:17

clj -Sdeps '{:deps {com.datomic/ion {:mvn/version "0.9.50"}}}' -Spath
src:....

[0 ~/empty-dir 16:52:23]
vim ~/.clojure/deps.edn  ## remove datomic-cloud

[0 ~/empty-dir 16:52:35]
clj -Sdeps '{:deps {com.datomic/ion {:mvn/version "0.9.50"}}}' -Spath
Error building classpath. Could not find artifact com.datomic:ion:jar:0.9.50 in central ()

Alex Miller (Clojure team)19:09:23

so in that first case, it's not re-downloading

Alex Miller (Clojure team)19:09:46

you're only running into this when it can't resolve from anywhere

Alex Miller (Clojure team)20:09:20

I logged here https://clojure.atlassian.net/browse/TDEPS-208 and I will take a closer look at this when I have some time

souenzzo20:09:10

Should be related with this file, that as far i know, is a new file

cat  ~/.m2/repository/com/datomic/ion/0.9.50/_remote.repositories 
#NOTE: This is a Maven Resolver internal implementation file, its format can be changed without prior notice.
#Wed Sep 15 16:44:43 BRT 2021
ion-0.9.50.jar>datomic-cloud=
ion-0.9.50.pom>datomic-cloud=

Alex Miller (Clojure team)20:09:29

yeah, this is related to the metadata files stuff I mentioned earlier

Alex Miller (Clojure team)20:09:04

I may not be able to "fix" this if it's under the api layer, but need to look at it more closely

Santiago19:09:37

are there templates for tools.deps like there are for leinigen? wondering if there is a way to bootstrap a pedestal project faster

souenzzo20:09:11

It is "deps-new"

Santiago20:09:27

nice thanks!

seancorfield20:09:55

https://github.com/seancorfield/deps-new will let you write a template for a Pedestal project and then reuse that template to create new Pedestal projects faster in future, but I'm not sure if that's what you're actually asking @UFPEDL1LY?

seancorfield20:09:17

There's also https://github.com/seancorfield/clj-new which can use Leiningen, Boot, and "clj-templates" to create new projects -- but a lein-template is going to create a lein-based project, not a CLI-based one.

seancorfield20:09:38

I don't know if there's a Pedestal clj-template...

Santiago20:09:43

there are some, but they’re all lein -based. I guess I’ll make one for CLI