Fork me on GitHub
#tools-deps
<
2021-09-16
>
Carsten Behring19:09:38

Running by repl, I get repeatedly the same error message about a certain jar:

Error building classpath. Could not acquire write lock for 'artifact:org.bytedeco:mkl:2020.3-1.5.4'
java.lang.IllegalStateException: Could not acquire write lock for 'artifact:org.bytedeco:mkl:2020.3-1.5.4'

borkdude19:09:48

concurrency issue? -Sthreads 1?

Carsten Behring19:09:09

Indeed, this seems to fix it. I remmber that the jars are big, so individaul download takes long. rw-r--r-- 1 carsten carsten 219M Sep 3 21:35 mkl-2020.3-1.5.4-linux-x86_64-redist.jar -rw-r--r-- 1 carsten carsten 172M Sep 3 21:28 mkl-2020.3-1.5.4-windows-x86_64-redist.jar -rw-r--r-- 1 carsten carsten 147M Sep 3 21:34 mkl-2020.3-1.5.4-macosx-x86_64-redist.jar -rw-r--r-- 1 carsten carsten 137M Sep 3 21:34 mkl-2020.3-1.5.4-linux-x86-redist.jar -rw-r--r-- 1 carsten carsten 112M Sep 3 21:33 mkl-2020.3-1.5.4-windows-x86-redist.jar -rw-r--r-- 1 carsten carsten 4.4M Sep 3 21:25 mkl-2020.3-1.5.4-linux-x86_64.jar -rw-r--r-- 1 carsten carsten 4.1M Sep 3 21:28 mkl-2020.3-1.5.4-linux-x86.jar -rw-r--r-- 1 carsten carsten 4.1M Sep 3 21:25 mkl-2020.3-1.5.4-macosx-x86_64.jar -rw-r--r-- 1 carsten carsten 3.3M Sep 3 21:25 mkl-2020.3-1.5.4-windows-x86_64.jar -rw-r--r-- 1 carsten carsten 2.0M Sep 3 21:28 mkl-2020.3-1.5.4-windows-x86.jar

Carsten Behring19:09:39

So it seems that clj has a reproducible issue with those.

Carsten Behring19:09:04

I have it inside a Docker, so on an empty maven folder.

Carsten Behring19:09:18

and it happens on every run

Carsten Behring19:09:44

(or docker build, better said)

Alex Miller (Clojure team)19:09:40

I'm going to be looking at some of this stuff tomorrow

Carsten Behring20:09:15

I have it on every run against an empty .m2 folder with this deps.edn

{
 :mvn/repos {"clojars" {:url ""}
             "central" {:url ""}
             "bedatadriven" {:url ""}}

 :paths ["src" "resources"]

 :deps {
        org.clojure/clojure      {:mvn/version "1.10.3"}
        clj-python/libpython-clj {:mvn/version "2.000"}
        scicloj/clojisr          {:mvn/version "1.0.0-BETA19"}
        scicloj/tablecloth       {:mvn/version "6.012"}
        scicloj/notespace        {:mvn/version "3-beta9"}
        scicloj/scicloj.ml       {:mvn/version "0.1.0-beta4"}}}

Alex Miller (Clojure team)21:09:41

Hey Carsten, I tried a lot of stuff today and I was able to reproduce this exactly once. The message you were seeing comes out of Maven itself. In a recent lib update I moved the maven resolver libs up to 1.7.x but those versions are not necessarily guaranteed to work with Maven core 3.8.x (they are expecting Maven core 4.0 alphas). Resolver 1.7 has been working on replacing the locking/synchronization infrastructure which is directly in this area. Anyhow, I have fallen back to 1.6.x which should work with the core libs I'm using. If you wanted to try with Clojure CLI 1.10.3.981, I'd be interested to know if that helped or not.

Alex Miller (Clojure team)16:09:18

one thing that I think contributes here is a lot of the deps under http://scicloj.ml are really big native libs

Alex Miller (Clojure team)16:09:30

I suspect the size == longer download time which exacerbates the problem

Carsten Behring16:09:46

Iyes, the MKL libs are very big

Carsten Behring16:09:58

> several jars of > 100 MB

Carsten Behring16:09:24

error building classpath. Could not acquire write lock for 'artifact:org.bytedeco:mkl:2020.3-1.5.4' java.lang.IllegalStateException: Could not acquire write lock for 'artifact:org.bytedeco:mkl:2020.3-1.5.4'

Alex Miller (Clojure team)16:09:52

yeah, that exception is in Maven

Carsten Behring16:09:57

It works with this clj -P -Sthreads 1

Carsten Behring16:09:26

i will quickly try with 1.10.3.981,

Alex Miller (Clojure team)16:09:26

they have completely rewritten this area in the newest Maven resolver, but it does not work with the current released version of Maven core

Alex Miller (Clojure team)16:09:41

in the latest I've fallen back to prior Maven resolver version

Carsten Behring16:09:42

docker build running, it takes some minutes

Carsten Behring16:09:21

It works with 1.10.3.981

Alex Miller (Clojure team)16:09:33

well that took their new code out of the loop, so that's good to know