Fork me on GitHub
#tools-deps
<
2022-02-28
>
Eugen11:02:55

hi, I was using this function to get the local path for a jar (zip file). But now resolve-deps (latest version) does not fill the path no more .

(defn jar-path
  "Find the path on disk where resolve-deps downloads the file.

   Example: (jar-path res ')"
  [resolve-deps-result lib]
  (first (:paths (get resolve-deps-result lib))))
I get :
deps (tools/resolve-deps (:ontolog-ui-deps basis) {:trace true})

#:{ontolog-ui {:mvn/version 0.1.13174, :extension zip, :deps/manifest :mvn, :parents #{[]}, :paths nil}}

Eugen12:02:55

seems like I don't have the zip locally. I suspect this is the cause. However I am expecting that resolve-deps should download the dependency. The zip is in the maven repo.

Eugen12:02:56

reverting back to version 0.12.1036 works

Eugen12:02:04

I found the issue for this in rlease 0.12.1058 - TDEPS-209 Include only jar files in classpath from Maven artifacts

Eugen12:02:57

@alexmiller: I am affected by this change. I am using the functionality to download a zip file (clojurescript UI app) from maven repository . Any way I can download a sepcific maven artifacts to a directory ?

Alex Miller (Clojure team)13:02:09

tools.deps is a library whose primary focus is building classpaths from deps, not to download arbitrary artifacts from maven

Alex Miller (Clojure team)13:02:05

It is possible to use some of the more internal functions that do this, rather than the top level apis

Eugen16:02:08

thanks, I managed to find a lower level function and make it work. Hopefully it will remain stable ๐Ÿ™‚ . It think it would be beneficial to have a way to set the download

;; give a dep, download just that dep (not transitive - that's handled by the core algorithm)
  (let [repos (maven/active-repositories (maven/read-settings) {:additional-profiles ["drevidence"]})]
    (ext/coord-paths 'com.drevidence.docsearch/ontolog-ui {:mvn/version "RELEASE"
                                                           :extension "zip"} :mvn {:mvn/repos repos}))

Eugen16:02:33

I'm using deps-deploy/maven-settings :as maven

Alex Miller (Clojure team)17:02:03

I'd be surprised if that works given the check in that method

Alex Miller (Clojure team)17:02:45

(when (contains? #{"jar"} extension) ...

Eugen18:02:24

yeah, you are right. I tested with the old version - ahh

Eugen18:02:15

get-artifact is private so not much I can do there, will check deeper

Eugen18:02:33

maven/coord->artifact + ArtifactRequest. might do the trick

Alex Miller (Clojure team)18:02:04

at some point, just using pomegranate might be easier for this

Eugen18:02:04

thanks, I will take a look at that as well

seancorfield18:02:19

@alexmiller Are there still some known race conditions in downloading dependencies in the very latest t.d.a.? (details in ๐Ÿงต )

seancorfield18:02:52

I just updated Component to 1.1.0 and we use it in a lot of local subproject deps.edn files and got this:

Downloading: com/stuartsierra/component/1.1.0/component-1.1.0.pom from clojars
Downloading: com/stuartsierra/component/1.1.0/component-1.1.0.pom from clojars
Downloading: com/stuartsierra/component/1.1.0/component-1.1.0.pom from clojars
Downloading: com/stuartsierra/component/1.1.0/component-1.1.0.pom from clojars
Error building classpath. Failed to read artifact descriptor for com.stuartsierra:component:jar:1.1.0
org.eclipse.aether.resolution.ArtifactDescriptorException: Failed to read artifact descriptor for com.stuartsierra:component:jar:1.1.0
	at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.loadPom(DefaultArtifactDescriptorReader.java:259)
	at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.readArtifactDescriptor(DefaultArtifactDescriptorReader.java:175)
	at org.eclipse.aether.internal.impl.DefaultRepositorySystem.readArtifactDescriptor(DefaultRepositorySystem.java:255)
	at clojure.tools.deps.alpha.extensions.maven$read_descriptor.invokeStatic(maven.clj:115)
	at clojure.tools.deps.alpha.extensions.maven$fn__1127.invokeStatic(maven.clj:143)
	at clojure.tools.deps.alpha.extensions.maven$fn__1127.invoke(maven.clj:143)
	at clojure.lang.MultiFn.invoke(MultiFn.java:244)
	at clojure.tools.deps.alpha$expand_deps$children_task__770$fn__772$fn__773.invoke(alpha.clj:405)
	at clojure.tools.deps.alpha.util.concurrent$submit_task$task__479.invoke(concurrent.clj:35)
	at clojure.lang.AFn.call(AFn.java:18)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.eclipse.aether.resolution.ArtifactResolutionException: Could not transfer artifact com.stuartsierra:component:pom:1.1.0 from/to clojars (): /Users/sean/.m2/repository/com/stuartsierra/component/1.1.0/component-1.1.0.pom.part (No such file or directory)
	at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:425)
	at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifacts(DefaultArtifactResolver.java:229)
	at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifact(DefaultArtifactResolver.java:207)
	at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.loadPom(DefaultArtifactDescriptorReader.java:244)
	... 13 more
Caused by: org.eclipse.aether.transfer.ArtifactTransferException: Could not transfer artifact com.stuartsierra:component:pom:1.1.0 from/to clojars (): /Users/sean/.m2/repository/com/stuartsierra/component/1.1.0/component-1.1.0.pom.part (No such file or directory)
	at org.eclipse.aether.connector.basic.ArtifactTransportListener.transferFailed(ArtifactTransportListener.java:52)
	at org.eclipse.aether.connector.basic.BasicRepositoryConnector$TaskRunner.run(BasicRepositoryConnector.java:369)
	at org.eclipse.aether.util.concurrency.RunnableErrorForwarder$1.run(RunnableErrorForwarder.java:75)
	at org.eclipse.aether.connector.basic.BasicRepositoryConnector$DirectExecutor.execute(BasicRepositoryConnector.java:628)
	at org.eclipse.aether.connector.basic.BasicRepositoryConnector.get(BasicRepositoryConnector.java:262)
	at org.eclipse.aether.internal.impl.DefaultArtifactResolver.performDownloads(DefaultArtifactResolver.java:514)
	at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:402)
	... 16 more
Caused by: java.io.FileNotFoundException: /Users/sean/.m2/repository/com/stuartsierra/component/1.1.0/component-1.1.0.pom.part (No such file or directory)
	at java.base/java.io.FileInputStream.open0(Native Method)
	at java.base/java.io.FileInputStream.open(FileInputStream.java:216)
	at java.base/java.io.FileInputStream.<init>(FileInputStream.java:157)
	at org.eclipse.aether.internal.impl.DefaultFileProcessor.copy(DefaultFileProcessor.java:163)
	at org.eclipse.aether.internal.impl.DefaultFileProcessor.copy(DefaultFileProcessor.java:151)
	at org.eclipse.aether.internal.impl.DefaultFileProcessor.move(DefaultFileProcessor.java:252)
	at org.eclipse.aether.connector.basic.BasicRepositoryConnector$GetTaskRunner.runTask(BasicRepositoryConnector.java:482)
	at org.eclipse.aether.connector.basic.BasicRepositoryConnector$TaskRunner.run(BasicRepositoryConnector.java:364)
	... 21 more

seancorfield18:02:42

Four concurrent downloads of the POM, from the look of it, and one of them seems to have deleted a partial download out from one of the others maybe?

seancorfield18:02:14

(running the CLI command again worked just fine the next time)

Alex Miller (Clojure team)18:02:42

yes, still known issues

Alex Miller (Clojure team)18:02:06

really, the problems are in Maven land and are being worked on for maven 4, but it's not done yet

Alex Miller (Clojure team)18:02:43

"fixing" it in tda means sharing less stuff, and probably being slower, but that's probably what I'll have to do

Alex Miller (Clojure team)18:02:44

when you build your stack on a pile of mutable stuff, well ... these things are hard :)

seancorfield18:02:19

Nice to know the central pillar of the Java library ecosystem is so solid ๐Ÿ˜‰

๐Ÿ™ƒ 1
Alex Miller (Clojure team)19:02:54

given how much I've already jettisoned from the Maven stack, I sometimes dream about getting rid of the whole thing

๐Ÿ˜„ 2
Alexander Kouznetsov23:11:29

Was this problem solved in recent versions? Iโ€™m seeing the exact same issue running on org.clojure/tools.deps.alpha "0.14.1178" Also seeing this discussion in https://clojurians.slack.com/archives/C6QH853H8/p1642027973048500?thread_ts=1642027964.048400&amp;cid=C6QH853H8.

Alexander Kouznetsov23:11:24

One interesting note is that we have an extra custom maven repo configured with :mvn/repos in our deps.edn file and every time the issue happens, I see 3 Downloading: from and usually two of them show clojars for repository and one - our custom one even though it has no such artifact:

Downloading: clj-yaml/clj-yaml/0.4.0/clj-yaml-0.4.0.pom from clojars
...
Downloading: clj-yaml/clj-yaml/0.4.0/clj-yaml-0.4.0.pom from clojars
...
Downloading: clj-yaml/clj-yaml/0.4.0/clj-yaml-0.4.0.pom from liftoff
...
Error building classpath. Failed to read artifact descriptor for clj-yaml:clj-yaml:jar:0.4.0
...
When it succeeds, then there are also sometimes several lines for the same file but they all are ending in from clojars.