Fork me on GitHub
#tools-deps
<
2019-03-04
>
rschmukler21:03:48

Is anyone using https://github.com/clojure/tools.deps.alpha/tree/add-lib effectively? I'm getting a NullPointerException on clojure.tools.deps.alpha.util.maven/make-session (maven.clj:141) - seemingly since around the time the zero-day vulnerabilities in the JVM got fixed (but I might be conflating things). No worries if the answer is "this branch isn't supported, don't use it"

Alex Miller (Clojure team)21:03:20

could you be more specific about what you're doing?

rschmukler21:03:45

Calling add-lib causes that to be thrown

rschmukler21:03:31

Hmm actually though, not all the time

Alex Miller (Clojure team)21:03:54

so, that's a branch, what commit are you using, what is the full call, etc. give me a repro.

rschmukler21:03:16

calling it with clojure.java-time {:mvn/version 0.3.2} seems to be the issue.

rschmukler21:03:29

@alexmiller will do - give me a moment to try and create a minimal repro repo

Alex Miller (Clojure team)21:03:08

you don't need a repo - you can put it all in an clj -Sdeps call to start, then whatever expression you're doing

Alex Miller (Clojure team)21:03:40

$ clj -Sdeps "{:deps
>                {org.clojure/tools.deps.alpha
>                 {:git/url \""
>                  :sha \"e160f184f051f120014244679831a9bccb37c9de\"}}}"
Checking out:  at e160f184f051f120014244679831a9bccb37c9de
Clojure 1.10.0
user=> (use 'clojure.tools.deps.alpha.repl)
nil
user=> (add-lib 'clojure.java-time {:mvn/version "0.3.2"})
true
user=> (require 'java-time)
Syntax error (IllegalAccessError) compiling at (temporal.clj:324:1).
class clojure.lang.PersistentUnrolledVector$Card1 tried to access field clojure.lang.APersistentVector._hash (clojure.lang.PersistentUnrolledVector$Card1 is in unnamed module of loader clojure.lang.DynamicClassLoader @7a0e1b5e; clojure.lang.APersistentVector is in unnamed module of loader 'app')

rschmukler21:03:43

@alexmiller I'm not even getting that far, although I can't seem to reproduce the issue in a new repo so now I'm trying to recreate more and more of the state in my user.clj as to what's happening before I invoke add-lib.

Alex Miller (Clojure team)21:03:19

I get that defaulting to Clojure 1.10. Not sure what clojure.java-time or the dependency clj-tuple assume with clj or java version

rschmukler21:03:14

Interesting. I've got clojure.java-time working w/ a 1.10 repo, although I added it via deps.edn, not dynamically via add-lib - so I haven't tested it.

Alex Miller (Clojure team)21:03:12

clj-tuple seems to be compiled Java classes, certainly possible that doesn't play nice with add-lib

Alex Miller (Clojure team)21:03:39

that error doesn't match what you described though re NPE

rschmukler21:03:45

I'm seeing it w/ other libs as well - but I'm hitting the error while invoking add-lib, not after (and trying to require it)

rschmukler21:03:02

While I try and reproduce, here's a stack trace if it helps: https://gist.github.com/rschmukler/8ae09c46ede6bab8d003dc4fe8cccef8

Alex Miller (Clojure team)21:03:08

what sha are you using for tools.deps?

rschmukler21:03:21

e160f184f051f120014244679831a9bccb37c9de

rschmukler21:03:30

Also it seems to raise the error w/ about 50% of the deps.

rschmukler21:03:55

ie. I wrote a wrapper that calls add-lib on my project for everything in deps.edn

rschmukler21:03:59

(defn sync-libs
  "Dynamically refreshes all libs in `deps.edn` and installs any missing"
  ([] (sync-libs nil))
  ([?alias]
   (let [edn  (-> "deps.edn" slurp read-string)
         libs (if ?alias
                (get-in edn [:aliases ?alias :extra-deps])
                (get edn :deps))]
     (doseq [[lib-sym lib-spec] libs]
       (try
         (add-lib lib-sym lib-spec)
         (catch NullPointerException e
             (println "Error raised for:" lib-sym lib-spec)))))))

rschmukler21:03:46

But again, this doesn't appear to be an issue in a different project. Still trying to build back up to it happening.

Alex Miller (Clojure team)21:03:17

add-lib doesn't re-add any deps it already has btw

rschmukler21:03:45

That's fine, I was presuming it to be a no-op

rschmukler21:03:17

(ie. my workflow is I have a file watcher watching deps.edn and it syncs the libs on change. I'm not doing any diffing)

Alex Miller (Clojure team)21:03:43

the error seems to imply you can't make a maven system, but not sure why you wouldn't get an error at that point

Alex Miller (Clojure team)21:03:04

(use 'clojure.tools.deps.alpha.util.maven)
(make-system)

rschmukler21:03:38

That worked (ie. no exception)

Alex Miller (Clojure team)21:03:31

well, not really sure. you're doing this under nrepl and it's possible the classloader hierarchy there is having some impact, don't really know.

Alex Miller (Clojure team)21:03:45

I haven't seen this reported before (although obviously not many people using this)

rschmukler21:03:00

I'm actually seeing it even if I don't use nrepl (ie. if I just use a plain ol' clojure repl)

Alex Miller (Clojure team)21:03:19

well, that's useful in ruling things out

rschmukler21:03:41

But yeah, I appreciate that I'm beyond the bleeding edge w/ this. I'll see if I can reproduce it...

Alex Miller (Clojure team)21:03:47

is there any chance you have this happening in multiple threads a the same time?

rschmukler21:03:23

I don't think so. I thought it might be some sort of race condition but it seems to be consistent in the dependencies that it fails on

rschmukler21:03:54

(and since it's just a clojure repl, I can't imagine anything spawning threads)

Alex Miller (Clojure team)21:03:41

you could try (dotimes [_ 20] (assert (not (nil? (make-session (make-system) default-local-repo)))))

Alex Miller (Clojure team)21:03:01

see if you get any nils

rschmukler21:03:46

Alright, will try momentarily

Alex Miller (Clojure team)21:03:14

I guess would also be good to know which java version you are on

rschmukler21:03:31

I found the library that causes it

rschmukler21:03:10

So, if you have an existing deps.edn w/ metasoarous/oz {:mvn/version "1.5.6"} and then try and invoke add-lib the errors happen

rschmukler21:03:20

confirming in my other repo

rschmukler21:03:20

Note that the sync-libs is commented out. Digging as to why that lib causes it.

rschmukler21:03:50

λ java --version
openjdk 11.0.2 2018-10-16
OpenJDK Runtime Environment (build 11.0.2+7)
OpenJDK 64-Bit Server VM (build 11.0.2+7, mixed mode)

rschmukler21:03:48

Yep, this looks like it

rschmukler21:03:15

> Another problem might happen if you mix lazy-require with some other code that does dynamic namespace loading. lazy-require itself makes sure to serialize the loading of namespaces so that nothing breaks because of concurrent loading (e.g., see clojure-emacs/cider#2092). However, if other code also loads namespaces in the background, something might happen. If you ever experience such issues, please file a ticket.

rschmukler21:03:49

Although, it's not like I'm requiring that lib, or even using it. ie. simply having it in the project (oz, not confirmed that it's this library specifically) causes the issue.

Alex Miller (Clojure team)21:03:03

well, certainly sounds likely

Alex Miller (Clojure team)21:03:16

oz is using in it's notebook namespaces

rschmukler21:03:15

Heh, we are doing exactly the same thing 😛

Alex Miller (Clojure team)21:03:22

actually that loads it but doesn't use it

rschmukler21:03:29

Although, I don't think it's actually using it. If you look at the lib itself, it's pure

rschmukler21:03:58

ie. it's not like it's doing anything to the run-time or anything