Fork me on GitHub
#tools-deps
<
2022-01-12
>
weavejester16:01:12

Can a tool dependency easily access its own version data? As git deps appear to need a SHA, I can’t hardcode the version.

weavejester16:01:57

For context, I’m building a tool that bootstraps a bb.edn file, which itself needs a dependency.

Alex Miller (Clojure team)17:01:38

as in, you are running code from the tool and want to know what version of itself that is?

weavejester13:01:11

Yep, that’s right. It’s so I can tell Babashka via bb.edn that this is the version of the library it should reference.

Alex Miller (Clojure team)17:01:16

there is not a public api (yet!) but this hack will work: (-> (System/getProperty "clojure.basis") slurp clojure.edn/read-string :libs (get 'org.clojure/clojure) :mvn/version)

weavejester13:01:03

Thanks! That appears to work!

Alex Miller (Clojure team)17:01:33

or at the end, check :git/tag , :git/sha etc

seancorfield22:01:44

Not sure if this is "known issue" or not: in CI (and pretty much only in CI), we are seeing intermittent "Failed to read artifact" errors from the aether stuff in the CLI (stacktrace in thread). Is this the multi-threading/race condition issue?

🙌 1
seancorfield22:01:53

Error building classpath. Failed to read artifact descriptor for ring:ring:jar:1.9.4
org.eclipse.aether.resolution.ArtifactDescriptorException: Failed to read artifact descriptor for ring:ring:jar:1.9.4
	at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.loadPom(DefaultArtifactDescriptorReader.java:259)
	at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.readArtifactDescriptor(DefaultArtifactDescriptorReader.java:175)
	at org.eclipse.aether.internal.impl.DefaultRepositorySystem.readArtifactDescriptor(DefaultRepositorySystem.java:255)
	at clojure.tools.deps.alpha.extensions.maven$read_descriptor.invokeStatic(maven.clj:115)
	at clojure.tools.deps.alpha.extensions.maven$fn__1127.invokeStatic(maven.clj:143)
	at clojure.tools.deps.alpha.extensions.maven$fn__1127.invoke(maven.clj:143)
	at clojure.lang.MultiFn.invoke(MultiFn.java:244)
	at clojure.tools.deps.alpha$expand_deps$children_task__770$fn__772$fn__773.invoke(alpha.clj:406)
	at clojure.tools.deps.alpha.util.concurrent$submit_task$task__479.invoke(concurrent.clj:35)
	at clojure.lang.AFn.call(AFn.java:18)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.eclipse.aether.resolution.ArtifactResolutionException: Could not transfer artifact ring:ring:pom:1.9.4 from/to clojars (): /root/.m2/repository/ring/ring/1.9.4/ring-1.9.4.pom.part (No such file or directory)
	at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:425)
	at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifacts(DefaultArtifactResolver.java:229)
	at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifact(DefaultArtifactResolver.java:207)
	at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.loadPom(DefaultArtifactDescriptorReader.java:244)
	... 13 more
Caused by: org.eclipse.aether.transfer.ArtifactTransferException: Could not transfer artifact ring:ring:pom:1.9.4 from/to clojars (): /root/.m2/repository/ring/ring/1.9.4/ring-1.9.4.pom.part (No such file or directory)
	at org.eclipse.aether.connector.basic.ArtifactTransportListener.transferFailed(ArtifactTransportListener.java:52)
	at org.eclipse.aether.connector.basic.BasicRepositoryConnector$TaskRunner.run(BasicRepositoryConnector.java:369)
	at org.eclipse.aether.util.concurrency.RunnableErrorForwarder$1.run(RunnableErrorForwarder.java:75)
	at org.eclipse.aether.connector.basic.BasicRepositoryConnector$DirectExecutor.execute(BasicRepositoryConnector.java:628)
	at org.eclipse.aether.connector.basic.BasicRepositoryConnector.get(BasicRepositoryConnector.java:262)
	at org.eclipse.aether.internal.impl.DefaultArtifactResolver.performDownloads(DefaultArtifactResolver.java:514)
	at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:402)
	... 16 more
Caused by: java.io.FileNotFoundException: /root/.m2/repository/ring/ring/1.9.4/ring-1.9.4.pom.part (No such file or directory)
	at java.base/java.io.FileInputStream.open0(Native Method)
	at java.base/java.io.FileInputStream.open(FileInputStream.java:219)
	at java.base/java.io.FileInputStream.<init>(FileInputStream.java:157)
	at org.eclipse.aether.internal.impl.DefaultFileProcessor.copy(DefaultFileProcessor.java:163)
	at org.eclipse.aether.internal.impl.DefaultFileProcessor.copy(DefaultFileProcessor.java:151)
	at org.eclipse.aether.internal.impl.DefaultFileProcessor.move(DefaultFileProcessor.java:252)
	at org.eclipse.aether.connector.basic.BasicRepositoryConnector$GetTaskRunner.runTask(BasicRepositoryConnector.java:482)
	at org.eclipse.aether.connector.basic.BasicRepositoryConnector$TaskRunner.run(BasicRepositoryConnector.java:364)
	... 21 more

seancorfield22:01:37

Re-running the failed step nearly always succeeds (and the "failed" artifact is random from build to build).

hiredman22:01:03

Is our m2 saved between builds?

seancorfield22:01:43

Mostly, yes, but it drifts as we update deps, and then BitBucket deletes and recreates the cache about every two weeks (to make it fresh again).

Alex Miller (Clojure team)23:01:10

Don't think that matches anything I've seen

Alex Miller (Clojure team)23:01:49

ring stuff will be coming from clojars right?

Alex Miller (Clojure team)23:01:12

It looks like that is in the maven stuff that downloads then copies into place and it is missing the download. Is it possible bitbucket deletes out from under you?

seancorfield23:01:02

No. This all happens in a single step. I don't know that it is only Clojars deps -- I think I've seen it from Maven deps too -- but I'll keep a closer eye on that in future.

seancorfield23:01:01

Good to know this isn't a known-but-supposedly-fixed issue 🙂

Alex Miller (Clojure team)23:01:16

there are general issues with Maven concurrent downloads that they've been working on for Maven 4

Alex Miller (Clojure team)23:01:44

is there any chance of concurrent processes hitting this same repo or it for sure just 1?

seancorfield23:01:56

I'm not using -Sthreads 1 but otherwise it is just this one process.

hiredman23:01:51

do we know how bitbucket shares the m2 cache? if all the builds should have access to m2, what happens when multiple builds are running at once?

hiredman23:01:28

(found the docs, directory is just and restored after a successful build, no shared filesystem or anything like that)

Alexander Kouznetsov23:11:55

Was this problem solved in recent versions? I’m seeing the exact same issue running on org.clojure/tools.deps.alpha "0.14.1178" Also seeing this discussion in https://clojurians.slack.com/archives/C6QH853H8/p1646073532171879?thread_ts=1646073499.713759&amp;cid=C6QH853H8.

seancorfield23:11:27

It's tools.deps now -- the alpha version is quite old.

👍 1
seancorfield23:11:42

What version is your CLI? clojure -version

seancorfield23:11:50

Looks like you're using https://clojure.org/releases/tools#v1.11.1.1113 which was April 2022. The latest is 1.11.1413, which uses tools.deps 0.18.1354 @UHS6PHL31

🙌 1
Alexander Kouznetsov23:11:24

Thanks! I will try bumping the version just wanted to make sure the problem is fixed there.

Alexander Kouznetsov23:11:05

And yes, you’re correct, we’re using that version of clojure CLI.

seancorfield22:01:11

We're on the latest CLI (1058).