Fork me on GitHub
#tools-deps
<
2021-07-27
>
dpsutton15:07:03

I've narrowed the problem down. One jar has an entry license/LICENSE and another jar has a LICENSE. When tools.build attempts to copy the contents of the jars it blows up. osx is case insensitive so there cannot be a file LICENSE and a folder license. But removing this issue just leaves how to resolve when a jar has a file named X and another jar has a folder named X.

dominicm15:07:56

Oh, because tools.deps just makes a folder and converts that to a jar. @seancorfield I'm not sure how depstar works, and your code is probably more straightforward than mine (mine got a bit fancy), but in pack.alpha jars are created using outputstreams and there's no intermediate filesystem required. That would help with having consistent behaviour for case sensitivity that matches the underlying format.

dominicm15:07:23

Pack got fancy because it supports generating jars within jars (or jars within zips for lambdas) and it will parallelize that as much as possible.

dpsutton15:07:07

i think depstar just consumes directly from the jar as a source without the intermediate local filesystem copy

dpsutton15:07:35

(read before typing. sorry)

seancorfield16:07:05

Right, depstar copies everything directly from wherever it is (directory or JAR) into the target JAR (as a ZipFileSystem, as I recall). I don't know what depstar would do if it finds license (file) and license/LICENSE (folder containing file), but I think the ZipFileSystem is case-sensitive even on macOS, in which case a LICENSE file won't conflict with a license/LICENSE directory?

didibus16:07:24

On compile task what exactly does :filter-nses options do? Will it make it so only classes with the prefix are included in the class-dir? Or will it make it so only namespaces with the prefix are compiled which means the resulting class-dir will also include all transient classes which might not start with the prefix?

seancorfield16:07:13

I dunno. I'd have to look at the source code 🙂

seancorfield16:07:37

Or maybe try it out on a project to see what it does... 🙂

seancorfield16:07:26

Does it actually filter on the namespace or the class name?

Alex Miller (Clojure team)16:07:00

Namespace prefix is what you specify

jjttjj20:07:23

Any tips for what I should look for in my clj -Stree output when I have some dependency issue? When I add tech.ml.dataset to my project, when I'm also using aleph as a webserver and make a web request, I get the error

16:16:39.778 [manifold-pool-2-1] ERROR aleph.http.server - error in HTTP handler
java.lang.IllegalArgumentException: contains? not supported on type: manifold.deferred.Deferred
There is no immediate connection in the deps tree between tech.ml.dataset and aleph or manifold

seancorfield20:07:09

@jjttjj What makes you thing it's a dependency problem?

seancorfield20:07:52

Is it purely the addition of that dependency with no code changes at all?

jjttjj20:07:19

If I remove the dependency it works again

seancorfield20:07:40

OK, and what's in your deps.edn file for the dependencies?

jjttjj20:07:04

It's a lot of stuff

jjttjj20:07:27

Wondering if there's a good way to narrow it down a bit

seancorfield20:07:47

Can you just share the tech.ml.dataset dep and the aleph and manifold deps? Then I can try to repro

jjttjj20:07:10

Sure one sec I'll throw that together

hiredman20:07:49

easier just to share the rest of the stacktrace

jjttjj21:07:31

Oh yeah, here it is:

17:06:57.893 [manifold-pool-2-3] ERROR aleph.http.server - error in HTTP handler
java.lang.IllegalArgumentException: contains? not supported on type: manifold.deferred.Deferred
	at clojure.lang.RT.contains(RT.java:853)
	at clojure.core$contains_QMARK_.invokeStatic(core.clj:1494)
	at clojure.core$contains_QMARK_.invoke(core.clj:1486)
	at ring.middleware.flash$flash_response.invokeStatic(flash.clj:21)
	at ring.middleware.flash$flash_response.invoke(flash.clj:14)
	at ring.middleware.flash$wrap_flash$fn__100872.invoke(flash.clj:39)
	at ring.middleware.session$wrap_session$fn__101193.invoke(session.clj:108)
	at ring.middleware.keyword_params$wrap_keyword_params$fn__101239.invoke(keyword_params.clj:53)
	at ring.middleware.nested_params$wrap_nested_params$fn__101297.invoke(nested_params.clj:89)
	at ring.middleware.multipart_params$wrap_multipart_params$fn__101595.invoke(multipart_params.clj:171)
	at ring.middleware.params$wrap_params$fn__101619.invoke(params.clj:67)
	at ring.middleware.cookies$wrap_cookies$fn__101072.invoke(cookies.clj:214)
	at ring.middleware.absolute_redirects$wrap_absolute_redirects$fn__101792.invoke(absolute_redirects.clj:47)
	at ring.middleware.content_type$wrap_content_type$fn__101740.invoke(content_type.clj:34)
	at ring.middleware.default_charset$wrap_default_charset$fn__101764.invoke(default_charset.clj:31)
	at ring.middleware.not_modified$wrap_not_modified$fn__101721.invoke(not_modified.clj:61)
	at ring.middleware.x_headers$wrap_x_header$fn__100835.invoke(x_headers.clj:22)
	at ring.middleware.x_headers$wrap_x_header$fn__100835.invoke(x_headers.clj:22)
	at ring.middleware.x_headers$wrap_x_header$fn__100835.invoke(x_headers.clj:22)
	at clojure.lang.AFn.applyToHelper(AFn.java:154)
	at clojure.lang.AFn.applyTo(AFn.java:144)
	at clojure.lang.AFunction$1.doInvoke(AFunction.java:31)
	at clojure.lang.RestFn.invoke(RestFn.java:408)
	at clojure.lang.Var.invoke(Var.java:384)
	at aleph.http.server$handle_request$fn__55977$f__48214__auto____55978.invoke(server.clj:170)
	at clojure.lang.AFn.run(AFn.java:22)
	at io.aleph.dirigiste.Executor$Worker$1.run(Executor.java:62)
	at manifold.executor$thread_factory$reify__48096$f__48097.invoke(executor.clj:47)
	at clojure.lang.AFn.run(AFn.java:22)
	at java.base/java.lang.Thread.run(Thread.java:832)
Having trouble getting the minimal repro to fail this way at the moment

hiredman21:07:52

the flash and session stuff is all tied to browser state

hiredman21:07:28

so my guess is the http://tech.ml depedency has nothing to do with it, what triggers it is if your browser has a session cookie set and tries to send it or something

hiredman21:07:48

looks like ring-session has(had?) incompatibilities with aleph in the past

jjttjj23:07:56

I hope this isn't too off topic, but here's a minimal example of the error. It does have to do with the ring session stuff it seems, I'm just confused by why the dataset repo would cause this https://github.com/jjttjj/aleph-dataset-bug-repro1 Here's a diff of the -Stree with and without the dataset dep https://www.diffchecker.com/JdvUqlE7 But excluding the ones that the dataset diff pulls in didn't change anything

seancorfield23:07:16

It brings in org.ow2.asm/asm 9.0 instead of 5.2 -- the latter is a dependency of core.async

hiredman20:07:03

it will show you exactly what is happening

💯 2
seancorfield20:07:04

Sure, but that won't answer the original Q which was what to look for in clj -Stree.

hiredman20:07:40

it likely isn't anything to do with deps like that

seancorfield20:07:23

If adding a dep without changing any code causes this... and removing the dep again solves the problem... without changing any code...

hiredman20:07:00

well, the rest of the stacktrace will tell us

hiredman20:07:08

I sort of suspect code is changing

seancorfield20:07:45

Is static init in a class/jar invoked if the new dep is never referenced in code? I thought it wasn't until the class was referenced...

hiredman20:07:45

aleph can be a little cagey about when it is handling you a fully realized value, and when you are getting a something deferred, which can trip people up when they treat like "just ring"

hiredman20:07:53

which that error looks like a text book case of

hiredman20:07:21

the rest of the stracktrace would tell us right away

didibus20:07:57

Ya, I think we need a way to discourage that, and when its needed, to encourage people to add a step in their Jar where they only include class files that are required, so it doesn’t pull in all the dependencies as well and the classes that might be better of as source.

didibus20:07:25

By “a way”. I’m thinking more through education

hiredman20:07:29

you are talking about filtering the output of compile?

didibus20:07:52

Yes, like go compile -> class-dir -> filter -> jar

hiredman20:07:11

that is also a bad idea, compile generates the classes it needs and there is no guarantee the same classes will get generated again later

seancorfield20:07:24

That's for :uberjar though, not lein jar right?

didibus20:07:35

Well, really it goes: src ------------------------------> jar compile -> class-dir -> filter -> jar

hiredman20:07:51

lein had(has?) a feature to do that

hiredman20:07:56

it is gross

didibus20:07:14

Hum, no it doesn’t exactly, but I think it had something called excision which you could use for it

didibus20:07:32

What’s gross about it?

hiredman20:07:34

do you want me to go find the commit for it?

hiredman20:07:01

when the compiler generates classes those classes have dependencies on each other based on the names of the classes

hiredman20:07:25

a second run of the compiler in some other time and place will likely not generate the same names for classes

hiredman20:07:06

so when you delete or filter out classes, you run the risk of next time your code is loaded, the compiler won't generate the same class names, which will result in classes your compiled code needs being missing

didibus20:07:21

That’s true, its a prefix filter, and it relies on the fact that class names are always prefixed the same even if the part after changes

hiredman20:07:21

https://github.com/technomancy/leiningen/commit/c4271c07c697c948716c2dd1af86f77cef3f9a4c is the lein feature that was default to on at one point being default to off because it broke things

didibus20:07:47

Hum, I mean that might just be because it failed at doing so. But I’m talking about something more explicit, well you need to specify what to include, which is normally very little. So you’re left with mostly source files, and a few select class files.

didibus20:07:16

Ok ya, so I looked up how I do it, I use namespace-munge to find the class name from a namespace.

didibus20:07:24

And I have another logic for gen-class that takes the gen-class :name And then in the build you need to explicitly list the namespaces and gen-class names you want to include their compiled classes in the Jar.

didibus21:07:09

And so what I do is that I include everything that in: <namespace-munge>*.class from the compiled dir into the Jar, for each explicitly listed namespaces. And for gen-class, you can just take the full :name from the gen-class and add .class after, since its guaranteed to have that name always

hiredman21:07:17

you will notice that you completely left out the case that the mailing list thread calls out (protocols)

didibus21:07:28

Hum, wouldn’t the protocol also be prefixed with the namespace-munge ?

hiredman21:07:21

compilation is transitive

hiredman21:07:12

so if I am namespace N and require and use a protocol from namespace A, compiling N will generate a classfile for A's protocol

jjttjj21:07:31

Oh yeah, here it is:

17:06:57.893 [manifold-pool-2-3] ERROR aleph.http.server - error in HTTP handler
java.lang.IllegalArgumentException: contains? not supported on type: manifold.deferred.Deferred
	at clojure.lang.RT.contains(RT.java:853)
	at clojure.core$contains_QMARK_.invokeStatic(core.clj:1494)
	at clojure.core$contains_QMARK_.invoke(core.clj:1486)
	at ring.middleware.flash$flash_response.invokeStatic(flash.clj:21)
	at ring.middleware.flash$flash_response.invoke(flash.clj:14)
	at ring.middleware.flash$wrap_flash$fn__100872.invoke(flash.clj:39)
	at ring.middleware.session$wrap_session$fn__101193.invoke(session.clj:108)
	at ring.middleware.keyword_params$wrap_keyword_params$fn__101239.invoke(keyword_params.clj:53)
	at ring.middleware.nested_params$wrap_nested_params$fn__101297.invoke(nested_params.clj:89)
	at ring.middleware.multipart_params$wrap_multipart_params$fn__101595.invoke(multipart_params.clj:171)
	at ring.middleware.params$wrap_params$fn__101619.invoke(params.clj:67)
	at ring.middleware.cookies$wrap_cookies$fn__101072.invoke(cookies.clj:214)
	at ring.middleware.absolute_redirects$wrap_absolute_redirects$fn__101792.invoke(absolute_redirects.clj:47)
	at ring.middleware.content_type$wrap_content_type$fn__101740.invoke(content_type.clj:34)
	at ring.middleware.default_charset$wrap_default_charset$fn__101764.invoke(default_charset.clj:31)
	at ring.middleware.not_modified$wrap_not_modified$fn__101721.invoke(not_modified.clj:61)
	at ring.middleware.x_headers$wrap_x_header$fn__100835.invoke(x_headers.clj:22)
	at ring.middleware.x_headers$wrap_x_header$fn__100835.invoke(x_headers.clj:22)
	at ring.middleware.x_headers$wrap_x_header$fn__100835.invoke(x_headers.clj:22)
	at clojure.lang.AFn.applyToHelper(AFn.java:154)
	at clojure.lang.AFn.applyTo(AFn.java:144)
	at clojure.lang.AFunction$1.doInvoke(AFunction.java:31)
	at clojure.lang.RestFn.invoke(RestFn.java:408)
	at clojure.lang.Var.invoke(Var.java:384)
	at aleph.http.server$handle_request$fn__55977$f__48214__auto____55978.invoke(server.clj:170)
	at clojure.lang.AFn.run(AFn.java:22)
	at io.aleph.dirigiste.Executor$Worker$1.run(Executor.java:62)
	at manifold.executor$thread_factory$reify__48096$f__48097.invoke(executor.clj:47)
	at clojure.lang.AFn.run(AFn.java:22)
	at java.base/java.lang.Thread.run(Thread.java:832)
Having trouble getting the minimal repro to fail this way at the moment

hiredman21:07:48

like, the classfile format is not crazy, just parse the classfiles, figure out which ones reference each other, instead of reinventing this completely broken worse is better 80% solution stuff over and over

hiredman21:07:08

yeah, that has nothing to do with the ml library

hiredman21:07:45

that is the flash middleware not being able to handle what aleph is passing it

didibus21:07:12

Well, the point is to not include transitive .class files, and so only include what you need to be locked down and made accessible from Java. But I had not looked into protocols I don’t think. So you can’t use a compiled protocols from some non compiled namespace?

hiredman21:07:34

it is complicated

hiredman21:07:16

aot compiled code can get loaded from a different classloader, so you end up with visibility issues

hiredman21:07:36

so if you have a defrecord that inline satisfies some protocol, it implements the interface behind that protocol under the hood

didibus21:07:47

Hum, I’ve only ever used it for gen-class and gen-interface when I write a Java lib in Clojure. So I guess I haven’t encountered the weirdness around protocols yet.

hiredman21:07:18

so when that defrecord's class file is loaded the jvm needs to be able to find that interface class file in the classloader defrecord is loaded from or a parent

hiredman21:07:19

but in the aot case the defrecord maybe loaded in the app or system (or whatever its callled) classloader, where the defprotocol when it is run defined the class in the compilers dynamic classloader which is a child of the app or system

didibus21:07:17

Ya I see, so the compiled interface for the protocol is loaded in the parent classloader, and than the dynamicclassloader when it compiles the inline implementation for it that will fail to find the class.

hiredman21:07:32

no, the reverse

didibus21:07:47

Oh, reverse, oh so if you don’t AOT the defprotocol, but AOT your call site? Wait so is a AOT protocol works with a non-aot namespace that extends it?

hiredman21:07:03

same outcome, but in this case the defrecord's class (created via aot) would be loaded in the parent, but the defprotocol would create the interface it needs in the child (the aot'ed interface having been deleted by filtering)

didibus21:07:43

I see, so it fails both ways.

hiredman21:07:06

if in your project you have a namespace N, and you use some library A, and that library defines a protocol P, and in N I created a record R that inline satisfies P, and then aot N, and prefix filter out all the classfiles that don't start with N, the interface for P gets deleted

didibus21:07:10

Ok ya I get it.

didibus21:07:43

Wouldn’t gen-class have this issue as well? Like if the gen-class is compiled, but its implementing namespace is not. Then wouldn’t the gen-class be loaded in the parent classloader and when trying to import the corresponding implementing clojure namespace it would fail to find it? Since that one is compiled at runtime by the child class loader?

hiredman21:07:19

I forget but I believe gen-class refers to clojure functions via vars through the namespace, so no name based classfile links

didibus21:07:04

Hum, ya might be that’s why it works

didibus21:07:13

It uses loadWithClass

hiredman21:07:16

it has been approximately a billion years since I used gen-class for anything other than just getting a main class to launch, and maybe half a billion since I've done that

didibus21:07:48

It seems the most reliable way is still to just have Java shims

didibus21:07:33

But even for protocols, then if you want a lib that works with Java and Clojure, you’re looking at not making it a protocol, but instead making it an Interface in Java.

hiredman21:07:54

or just don't aot libraries

didibus21:07:13

You need too for them to work as a Java lib though, that’s my use case

didibus21:07:54

And not force people into using a Clojure Java API interface, but make them believe they are just dealing with a real java lib

hiredman21:07:28

I have been known to just use clojure.asm.* to generate the exact bytecode I want, if I had to deliver a java api like that I would be very tempted (likely foolishly) to do that.

didibus21:07:33

I remember seeing a lib that was basically S-expr to jvm bytecode for Clojure… can’t remember, but that could be a simpler way of doing the same

didibus21:07:50

Oh, JiSE, hum but it does not support interfaces 😞 I guess though, even with clojure.asm.* I’m not sure how I would generate something that is both a valid Protocol when consumed from Clojure, and a valid Interface when consumed from Java

hiredman21:07:24

I've always thought it would be interesting to attach a protocol definition to an existing interface instead of generating a new one

didibus21:07:12

What do you mean by attach?

hiredman21:07:09

a protocol is sort of a pair of dynamically extendable state and a java interface, satisfying inline in a deftype or defrecord uses the interface, extending to an existing type uses the dynamically extendable state

hiredman21:07:38

so when you defprotocol you get both those things, because it creates some mutable state and defines the interface

hiredman21:07:58

but I dunno, it seems like you could just pass the interface in

didibus21:07:01

Like if you already have an interface… Ya, so just go like this interface is also now this protocol which just adds that state machinery to it. That would kind of allow all interfaces in Clojure core to become protocols as well without breakage in a way.

didibus21:07:49

And make all Java interfaces a protocol too. It would be neat.

seancorfield22:07:43

(ns my.entry.point
  (:gen-class))

(defn -main [& args] ((requiring-resolve 'my.impl/main) args))
🙂 Problem solved!

seancorfield22:07:38

AOT will only compile my.entry.point and it only depends on clojure.core stuff which is already compiled. No transitive compilation issues.

dpsutton22:07:30

quick check: what's the difference between specifying an alias with :deps vs :extra-deps? Does :deps clobber the top level and extra-deps is additive?

seancorfield22:07:21

:deps and :replace-deps are "the same" I think, but I've noticed that when I use -A or -M it seems that those dependencies still combine with other aliases which sort of surprised me so I'm not sure whether something else changed in the latest t.d.a around that...

seancorfield22:07:18

Looking at the t.d.a code, it seems like :replace-deps generally merge but if you use a "tool" context, you only get :replace-deps -- and it looks like :deps is treated as a legacy alias for :replace-deps (and in one case it seems like t.d.a will merge them if both are present so together they replace :extra-deps etc)

seancorfield23:07:47

(defn tool
  "Transform project edn for tool by applying tool args (keys = :paths, :deps) and
  returning an updated project edn."
  [project-edn tool-args]
  (let [{:keys [replace-deps replace-paths deps paths]} tool-args]
    (cond-> project-edn
      (or deps replace-deps) (merge {:deps (merge deps replace-deps)})
      (or paths replace-paths) (merge {:paths (vec (concat paths replace-paths))}))))
So the output is just what was in the tool-args and that will override the top-level :deps (and :paths) from the project deps.edn file.

jjttjj23:07:56

I hope this isn't too off topic, but here's a minimal example of the error. It does have to do with the ring session stuff it seems, I'm just confused by why the dataset repo would cause this https://github.com/jjttjj/aleph-dataset-bug-repro1 Here's a diff of the -Stree with and without the dataset dep https://www.diffchecker.com/JdvUqlE7 But excluding the ones that the dataset diff pulls in didn't change anything