Fork me on GitHub
#clojure-dev
<
2020-06-03
>
mikerod17:06:33

Tried this out in both java8 and java11 with clojure 1.10.1 and [clj-commons/pomegranate “1.2.0”] - and find the behavior hard to explain - in the repl - using lein 2.9.3. Wonder if anyone has any deeper insight on the classloader trickery happening: ---- These DO NOT work. hickory isn’t found on classpath. (1)

(p/add-dependencies
 :coordinates '[[hickory "0.7.1"]]
 :repositories (merge {"clojars" ""}
                      pa/maven-central))

(require 'hickory.core)
(2)
(p/add-dependencies
 :classloader (clojure.lang.RT/baseLoader)
 :coordinates '[[hickory "0.7.1"]]
 :repositories (merge {"clojars" ""}
                      pa/maven-central))

(require 'hickory.core)
(3)
(def cl (clojure.lang.RT/baseLoader))

(p/add-dependencies
 :classloader cl
 :coordinates '[[hickory "0.7.1"]]
 :repositories (merge {"clojars" ""}
                      pa/maven-central))

(require 'hickory.core)
(4)
(let [cl (clojure.lang.RT/baseLoader)]
  (p/add-dependencies
   :classloader cl
   :coordinates '[[hickory "0.7.1"]]
   :repositories (merge {"clojars" ""}
                        pa/maven-central)))

;; NOTE: NOT INSIDE the `let`
(require 'hickory.core)
However, this DOES work: (5)
(let [cl (clojure.lang.RT/baseLoader)]
  (p/add-dependencies
   :classloader cl
   :coordinates '[[hickory "0.7.1"]]
   :repositories (merge {"clojars" ""}
                        pa/maven-central))
  ;; NOTE: INSIDE the `let`.
  (require 'hickory.core))
Also, after (5) happens in the repl, you can just use the new dependencies with no further let-bindings or references to that classloader. I think I somewhat understand why at this point since the dynamic classloaders are likely just sharing state of these clj loaded classes.

mikerod17:06:27

it’s pretty weird to me. The case where the classloader is in a let-binding seems to matter. It makes it “used by require

mikerod17:06:41

but no other time. Notably, not even a var holding the loader matters (case (3))

hiredman17:06:37

I would try #2 with both expressions in a let with no bindings

hiredman17:06:44

(let [] ...)

hiredman17:06:21

I forget the exact details but when evaluating/compiling code the compiler will often push a new dynamic classloader binding, which is sometimes what RT/baseLoader returns

hiredman17:06:22

so my guess is what you are seeing is differences in behavior based on if the add-dependencies expression is compiled and run with the same classloader on that stack as the require expression

mikerod17:06:55

I think what your saying makes sense

mikerod17:06:08

I wasn’t thinking enough about clojure.lang.RT/baseLoader returning different things

mikerod17:06:45

you’r probably rigth I’m getting the wrong case

hiredman17:06:12

not 100% sure, wrapping in a let like that will make the two part of the same compilation unit, without all the explicit classloader stuff, so if that fixes the issue it is likely what is happening

mikerod17:06:48

yeah, and perhaps I should use clojure.lang.RT/makeClassLoader as well to test

mikerod17:06:54

which would always give me a dynamic loader

mikerod17:06:56

regardless of baseloader

hiredman17:06:13

I doubt that will do anything

mikerod17:06:20

but the compilation unit binding Compiler.LOADER is a prime suspect

hiredman17:06:54

the issue isn't likely with what you are explicitly passing to pomegranate, but what the compiler is implicitly doing behind the scenes

hiredman17:06:50

I've had to fiddle with requires with pomegranate in the past(the behavior changed around clojure 1.8 I think), but never dug into why

mikerod18:06:09

so doing case (2) in a let-binding as you said works

mikerod18:06:12

as expected

mikerod18:06:22

eg:

(let []
  (p/add-dependencies
   :classloader (clojure.lang.RT/baseLoader)
   :coordinates '[[hickory "0.7.1"]]
   :repositories (merge {"clojars" ""}
                        pa/maven-central))

  (require 'hickory.core))

mikerod18:06:32

Then my repl loader permanently can refer to these new classes after too

mikerod18:06:35

(as before)

mikerod18:06:05

and

(p/add-dependencies
 :classloader (clojure.lang.RT/makeClassLoader)
 :coordinates '[[hickory "0.7.1"]]
 :repositories (merge {"clojars" ""}
                      pa/maven-central))
(require 'hickory.core)
does not work; as you assumed above - I’m still not sure I can reason why it doesn’t work

mikerod18:06:17

oh yes, I think I can

mikerod18:06:32

require is just not occurring in a compilation unit that would know of this new one

mikerod18:06:32

So basically, if we can get “loading” (eg. require) to happen in the same compilation unit as using clojure.lang.RT/baseLoader to add the deps; we’ll be dealing using that same mutated loader from pomegranate

mikerod18:06:54

I think once we do the load; it is safe to just use the stuff after that point within the repl

mikerod18:06:21

meaning, I don’t know of a reason I’d need to hold onto the loader and keep binding it

mikerod18:06:39

to close it out

mikerod18:06:41

you can also do

mikerod18:06:46

(def cl (clojure.lang.RT/makeClassLoader))
(with-bindings {clojure.lang.Compiler/LOADER cl}
  (p/add-dependencies
   :classloader cl
   :coordinates '[[hickory "0.7.1"]]
   :repositories (merge {"clojars" ""}
                        pa/maven-central))

  (require 'hickory.core))

mikerod18:06:19

then again, using with-bindings may be making them both in the same compilation unit still

mikerod18:06:40

so probably should break all the way into push thread bindings around multiple separate forms to see it in most raw

hiredman18:06:22

no, I doubt you need to do any explicit classloader stuff at all

mikerod18:06:31

(def cl (clojure.lang.RT/makeClassLoader))
(with-bindings {}
  (p/add-dependencies
   :classloader cl
   :coordinates '[[hickory "0.7.1"]]
   :repositories (merge {"clojars" ""}
                        pa/maven-central))

  (require 'hickory.core))

mikerod18:06:38

so it does get compiled in different context

hiredman18:06:51

stop passing in a classloader

mikerod18:06:13

(with-bindings {}
  (p/add-dependencies
   :coordinates '[[hickory "0.7.1"]]
   :repositories (merge {"clojars" ""}
                        pa/maven-central))

  (require 'hickory.core))

hiredman18:06:16

wrap it in let instead

mikerod18:06:20

I get let works

mikerod18:06:28

I’m trying to show the underlying mechanism of why let works

mikerod18:06:44

the sharing of the clojure.lang.Compiler/LOADER

mikerod18:06:59

(let []
  (p/add-dependencies
   :coordinates '[[hickory "0.7.1"]]
   :repositories (merge {"clojars" ""}
                        pa/maven-central))

  (require 'hickory.core))
doesn’t work though

mikerod18:06:02

you have to refer to baseloader

mikerod18:06:06

don’t think pomegranate does (by default)

seancorfield19:06:44

@mikerod This seems to be all accidental complexity introduced by pomegranate -- using add-lib for this sort of thing "just works".

mikerod20:06:12

@seancorfield add-lib being an unreleased fn right?

seancorfield20:06:27

(! 504)-> clj -A:deps -r
user=> (require '[clojure.tools.deps.alpha.repl :refer [add-lib]])
nil
user=> (add-lib 'hickory {:mvn/version "0.7.1"})
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See  for further details.
true
user=> (require 'hickory.core)
nil
user=> 

seancorfield20:06:03

:deps ; to get access to clojure.tools.deps.alpha.repl/add-lib
  {:extra-deps {org.clojure/tools.deps.alpha
                {:git/url ""
                 :sha "19d197ab221d37db750423eb970880cb87a91100"}}

dominicm20:06:06

@seancorfield they use similar strategys. This doesn't just work with add-lib when using lein/add-lib

mikerod20:06:26

yeah pomegranate isn’t really doing much fancy here either

dominicm20:06:28

There's a long outstanding cider issue about this.

dominicm20:06:37

I think it recently got closed

mikerod20:06:40

just trying to add to a classloader when it has the ability to be added to - via a protocol

dominicm20:06:56

I'd love to know what the deal is with COMPILER, because that stumped me.

dominicm20:06:22

Sorry, LOADER. hazy memory

mikerod20:06:29

It makes sense above

mikerod20:06:33

but it’s sort of digging

mikerod20:06:39

you can do similar with the context classloader instead

mikerod20:06:53

but you are somewhat relying on the fact that clj require (`load`) will use a RT/baseLoader

seancorfield20:06:04

So the problem here is... what exactly? I'm confused. You show something related to Leiningen / Pomegranate that doesn't work in situations you think it should... and that's a bug/problem in... what? I'm not following...

mikerod20:06:08

which will in turn use the bound LOADER or context loader if you set that etc

dominicm20:06:14

But why does lein have particular issues with LOADER. I couldn't get it to trigger under clojure.

seancorfield20:06:37

Right, this is feeling like a lein/CIDER problem, not a Clojure problem.

mikerod20:06:41

It’s just that in some repl’s the classloaders are new each eval

mikerod20:06:43

or something to that extent

mikerod20:06:53

so if you use something like pomegranate incorreclty - you may mutate the wrong loader

mikerod20:06:05

and not really have ability to use that loader for something like require subsequently

mikerod20:06:11

above there is outlined approaches for that to not be the case

mikerod20:06:25

it’s something that happens in lein, not really a “problem”

seancorfield20:06:31

So that's... a bug in lein?

mikerod20:06:35

just something you sort of have to understand if mutating a loader

dominicm20:06:38

Hmm. I fixed that in nrepl though

mikerod20:06:44

you have to be sure you are using that mutated loader

dominicm20:06:44

I gave them a common base.

seancorfield20:06:32

(sorry, I'm not trying to be difficult, I'm just trying to see how this relates to #clojure-dev )

mikerod20:06:12

I wanted to understand the underlying mechanism for why a given compile-context seemed to change my result

mikerod20:06:17

I don’t care that I was using a lein repl to get to that state

mikerod20:06:22

so to me - relates directly

mikerod20:06:44

you can have different repl impl’s some may or may not end up the same - depends on how they manage classloaders etc

mikerod20:06:57

I’m not saying there is even a problem, was just baffled by how let seemingly caused a difference for no reason

mikerod20:06:13

and conclusion was: it formed a “single compilation unit”

mikerod20:06:18

where the Compiler.LOADER was bound

mikerod20:06:28

which affects the clojure.lang.RT/baseLoader return val

mikerod20:06:55

could perhaps have been find in #clojure channel

mikerod20:06:06

hard to say when it’s pretty internal-digging

dominicm20:06:10

Why does that change the base loader? That's been my question for years :)

mikerod20:06:24

static public ClassLoader baseLoader(){
	if(Compiler.LOADER.isBound())
		return (ClassLoader) Compiler.LOADER.deref();
	else if(booleanCast(USE_CONTEXT_CLASSLOADER.deref()))
		return Thread.currentThread().getContextClassLoader();
	return Compiler.class.getClassLoader();
}

mikerod20:06:32

then you can see in clojure.lang.Compiler.eval() how this LOADER is bound

mikerod20:06:11

so if you do some mutation to it - via pomegranate and also end up doing a load while still in that same eval - you are working against same loader at the right time

hiredman20:06:13

likely the reason lein's repl behaves different is the clojure.main/repl sets the context class loader to be a dynamic classloader

mikerod20:06:38

don’t have to rely on let though - you can just use with-bindings ; you also don’t even need LOADER; you can set the USE_CONTEXT_CLASSLOADER and achieve similar

mikerod20:06:03

Yeah, never looked into why lein does whatever it does with loaders

hiredman20:06:15

lein likely does nothing

hiredman20:06:33

and it isn't really on lein

hiredman20:06:47

the issue is with the server side of the nrepl it creates

seancorfield20:06:57

FWIW, of your original five cases, (1) does work in the regular clj REPL (and it also worked for me in a plain lein repl) (2), (3), and (4) do not work, and (5) does work. -- @mikerod Does (1) not work for you in lein repl?

mikerod20:06:33

yeah, the server is really from nrepl

mikerod20:06:49

and it does have middleware setting the ccl and LOADER etc - so things are being done

hiredman20:06:04

it likely all depends on the mix of nrepl and clojure versions being used

seancorfield20:06:13

If (1) works for you in lein repl, then I'm much clearer about what you're asking -- which is about the (2), (3), and (4) cases which don't work in either REPL and so that points to an isolated class loader issue (am I close / caught up now?)

mikerod20:06:27

@seancorfield all above was for lein repl

mikerod20:06:38

and (1) was in the “doesn’t work” list

mikerod20:06:51

so whatever is happening in the repl there - which really relates to nrepl - didn’t end up sharing the loader across forms

mikerod20:06:13

perhaps do could work, but think that may be split

seancorfield20:06:13

Ah, so you have some middleware/plugin that is breaking (1) in lein repl (as well as the three common cases)...

mikerod20:06:18

so have to use let or something to keep it grouped

mikerod20:06:37

yes, many variables involved concerning what repl is used and what that repl does

mikerod20:06:07

And again: I posted the examples trying to just get an idea of what mechanism was causing the the weird difference I saw when using let.

mikerod20:06:15

I’m not reporting like a bug or issue - to be clear

mikerod20:06:30

was exploratory

1
mikerod20:06:49

good news is - it makes sense how to make pomegranate work now in lein repl’s (to me at least)

mikerod20:06:08

I think add-lib will hit similar situations in lein

mikerod20:06:18

looks to do something quite similar with how it mutates the loader

seancorfield20:06:27

I haven't used lein for much of anything since late 2015, so I was just very puzzled when I tried your example (1) in a plain lein repl and it worked... hence my confusion! 🙂

mikerod20:06:45

hmm weird if it worked

mikerod20:06:54

but yeah, I know you seem to prefer the alt stack

seancorfield20:06:26

(! 513)-> lein version
Leiningen 2.9.3 on Java 14 OpenJDK 64-Bit Server VM
(! 514)-> lein repl
nREPL server started on port 52427 on host 127.0.0.1 - 
REPL-y 0.4.4, nREPL 0.6.0
Clojure 1.10.0
OpenJDK 64-Bit Server VM 14+36-1461
Just so it's clear what version(s) I tried it on. No project.clj present.

dominicm20:06:40

@mikerod next version of nrepl has a workaround for this

mikerod20:06:38

Same as me @seancorfield besides I tried java8 & 11 (but those shouldn’t change the loader setup of the nrepl server)

mikerod20:06:47

@dominicm interesting; I’ll have to see - the current repl still can work as is - just have to be more careful about things, but would be nice to not have to

mikerod20:06:16

still no idea why Sean’s is working haha

mikerod20:06:56

not for this case:

(p/add-dependencies
 :coordinates '[[hickory "0.7.1"]]
 :repositories (merge {"clojars" ""}
                      pa/maven-central))

(require 'hickory.core)
with that version of lein

hiredman20:06:45

the issue is the version of nrepl, the lein version is a red herring

mikerod20:06:52

oh, guess I somewhat cut off too [cemerick.pomegranate.aether :as pa] for the maven repo

mikerod20:06:12

@hiredman lein version directly chooses the nrepl version though?

mikerod20:06:15

it’s part of it’s deps

mikerod20:06:22

and I’m using 2.9.3

hiredman20:06:29

it is overridable

mikerod20:06:53

I’m not overridding

hiredman20:06:01

I would double check

mikerod20:06:03

unless Sean is or something (doubt it since he doesn’t use lein)

hiredman20:06:56

I am not sure how one would do that though, I try deps :tree, but the nrepl version might be part of some profile thing, dunno

mikerod20:06:15

yeah, I’m using nrepl "0.6.0" as well

mikerod20:06:22

in deps tree