Fork me on GitHub
#tools-deps
<
2020-06-28
>
borkdude10:06:46

What does this mean?

Exception in thread "main" clojure.lang.ExceptionInfo: Manifest type not detected when finding deps for babashka.curl/babashka.curl in coordinate {:git/url "", :sha "2dc7f53271de3c2edc3e1474a000b9dfa7324eaf"} {:lib babashka.curl/babashka.curl, :coord {:git/url "", :sha "2dc7f53271de3c2edc3e1474a000b9dfa7324eaf"}}, compiling:(/tmp/prog.clj:3:1)

borkdude10:06:57

as a result of:

(require '[clojure.tools.deps.alpha :as deps])

(-> {:deps '{babashka.curl
             {:git/url ""
              :sha "2dc7f53271de3c2edc3e1474a000b9dfa7324eaf"}}
     :mvn/repos {"central" {:url ""}
                 "clojars" {:url ""}}}
    (deps/resolve-deps nil)
    (deps/make-classpath nil nil))

borkdude10:06:44

I get the same result when using this coordinate with -Sdeps from the command line

Alex Miller (Clojure team)12:06:46

this should not affect anything

lread13:06:05

oh, looks like I misdiagnosed! thanks for the info!

Alex Miller (Clojure team)13:06:55

it's just map merges and stuff in the code, so I would not expect it to violate any Clojure-ish expectations here

borkdude11:06:44

That appears to be it. Weird.

lread11:06:06

You are most welcome, by chance I hit that one yesterday.

Alex Miller (Clojure team)12:06:02

do you have a repro of that?

borkdude11:06:38

@lee Hmm, now it's coming back:

$ clj -Sdeps "{:deps {babashka.curl
              {:git/url \"\"
               :sha \"cbf8429addac1d2f71ac8ddfb847657223fdf0f7\"}}}" -Spath
Error building classpath. Manifest type not detected when finding deps for babashka.curl/babashka.curl in coordinate {:git/url "", :sha "cbf8429addac1d2f71ac8ddfb847657223fdf0f7"}

borkdude11:06:23

it did work for:

$ clj -Sdeps "{:deps {babashka.curl
              {:git/url \"\"
               :sha \"1dc0f61d7baabee1f009543d60bfb79ee4a7101c\"}}}" -Spath

borkdude11:06:30

while the commit cbf8 didn't do anything relevant compared to 1dc0with the deps.edn file....

borkdude11:06:15

Adding :deps/manifest :deps seems to fix it.

borkdude11:06:18

Added this to the issue

borkdude11:06:18

I'm also getting this for a local/root dep:

$ clj -Sdeps "{:deps {org.clojure/tda {:local/root \"/Users/borkdude/Dropbox/dev/temp/tools.deps.alpha\" }}}" -Spath
Error building classpath. Manifest type not detected when finding deps for org.clojure/tda in coordinate #:local{:root "/Users/borkdude/Dropbox/dev/temp/tools.deps.alpha"}

Alex Miller (Clojure team)12:06:41

Are they reproducible with -Sforce

borkdude12:06:47

same result

borkdude12:06:20

/tmp $ clj -Sdeps "{:deps {babashka.curl
              {:git/url \"\"
               :sha \"cbf8429addac1d2f71ac8ddfb847657223fdf0f7\"}}}" -Spath -Sforce -Srepro
Error building classpath. Manifest type not detected when finding deps for babashka.curl/babashka.curl in coordinate {:git/url "", :sha "cbf8429addac1d2f71ac8ddfb847657223fdf0f7"}

Alex Miller (Clojure team)12:06:05

@borkdude I can't repro any of your failing cases

borkdude12:06:34

Hah. Worked after a rm -rf ~/.gitlibs

borkdude12:06:44

Not for

clj -Sdeps "{:deps {org.clojure/tda {:local/root \"/Users/borkdude/Dropbox/dev/temp/tools.deps.alpha\" }}}" -Spath
though

borkdude12:06:04

That's probably another problem. It does work with a vanilla checkout of tools.deps:

clj -Sdeps "{:deps {org.clojure/tda {:local/root \"tools.deps.alpha\" }}}" -Spath

borkdude12:06:36

I'll close the issue.

Alex Miller (Clojure team)12:06:12

tools.deps.alpha has a couple build steps (java compilation, resource replacement) so it's not a good example lib for use as local/git dep

Alex Miller (Clojure team)12:06:44

btw, in the near future, using unqualified lib symbols is going to be deprecated and will complain, so "babashka.curl" should be changed to "babashka.curl/babashka.curl" etc

Alex Miller (Clojure team)12:06:48

(and this is a step towards actually removing support for that)

borkdude13:06:09

@alexmiller I know. I had already compiled the java and added target/classes to the paths

borkdude13:06:20

But still didn’t work probably due to the resource replacement

borkdude13:06:18

I’m trying to get the native tools deps working using Ghadi’s git shell branch of the git libs library, but this stuff got me distracted for now

borkdude13:06:06

Should probably push the compiled assets to Clojars and work with that instead of tools deps deps

borkdude13:06:33

Btw, the url of a git dep is already a unique identifier. Why should a user have to make up some fully symbol keyword in the first place?

Alex Miller (Clojure team)13:06:17

well you can just mvn clean install and use that local snapshot artifact too - sometimes that's eaesiest for maven built things

Alex Miller (Clojure team)13:06:57

re unique id, for the exact same reasons namespaces are useful for global differentiation, group-ids are a good idea for qualified names. using same symbol group+artifact was always a limited idea. lein/clojars made it too easy and common. we're not going to perpetuate making it implicit.

borkdude13:06:46

But why have a name for a git dep at all?

Alex Miller (Clojure team)13:06:45

because names are the identity of the lib, which may exist at different coordinates and you want to avoid putting the "same" lib on the classpath from multiple locations

Alex Miller (Clojure team)13:06:55

(admittedly still some work to do on cross-coordinate type version selection, but the hooks are there)

borkdude13:06:17

Good point. The only use case for doing that would be when you would mount different dirs from the same git dep at different SHAs, which seems far fetched. Still if you would use the URL as the identifier you would be able to detect conflicts more easily since different people are going to use different qualified syms for the same lib?

rickmoynihan08:06:50

Not that far fetched as we actually do this in one or two places. One of our apps supports different customer configurations via t.d.a aliases, and there is essentially a :customer/all alias that pulls together a combined classpath for all customers, for working at the REPL (which lets you switch between customers). One of the deps in that alias contains fixture data for all customers with a :deps/root for each customer, and it’s possible for the sha’s for customers to drift slightly from each other (even though they’re all in the same repo). So this t.d.a feature is very handy, as otherwise we’d need a fixture data repo per customer, which would make updating fixture harder across customers harder. Essentially it just means the artifact id in this becomes git url + deps-root. It makes a lot of sense when you use :deps/root. Normally they all reference the same SHA, but sometimes it’s pragmatic to apply changes to one customer first, before rolling out wider. This can also happen when you want a branch to built repeatably by C.I. (before you’ve applied the change across all customers).

rickmoynihan08:06:24

We then coin a local dep-id of something like: swirrl/fixtures.customer-name

borkdude08:06:53

Yeah, that does make sense.

rickmoynihan08:06:25

It’s essentially just a lighter weight way to manage separate dependencies that you often make cross-cutting changes over.

rickmoynihan08:06:11

Though I do suspect t.d.a might have some bugs resolving these sometimes — I’ve seen a few cases where when fetching these deps it seems to either blow up or mix up the commits, e.g. it would resolve customer-b to customer-a’s sha. I suspect it might be related to the parallel resolution stuff. cc @alexmiller

rickmoynihan08:06:28

It’s quite hard to reproduce though

Alex Miller (Clojure team)13:06:11

yes, I am almost certain that it's parallel resolution of git deps

👍 3
Alex Miller (Clojure team)13:06:39

we do not currently do that level of detection but it is something I would like to do

Alex Miller (Clojure team)13:06:06

ultimately the library has to determine the identifier by which it should be known

Alex Miller (Clojure team)14:06:19

the organization should be something you own or register, like domain name, trademark. third party sites like github can also confer identity - we recommend in https://clojure.org/guides/deps_and_cli to combine them by creating a group-id like github-borkdude

borkdude14:06:19

Which the URL is already an instance of

Alex Miller (Clojure team)14:06:15

url is really a combination of site (github), org, and repo. github-org/repo combines those

Alex Miller (Clojure team)14:06:21

but it does not account for forks

borkdude14:06:51

Interesting :thinking_face:

Alex Miller (Clojure team)14:06:02

to use a fork, you need to use the same lib, but different url, and then they diverge

Alex Miller (Clojure team)14:06:50

your goal with using a fork really is to replace (identity) the lib

Alex Miller (Clojure team)14:06:12

but then you have interesting questions of ownership over the lib identity

borkdude14:06:13

If I want to maintain a fork as a real fork, and not just for PRs, I would not use the original identifier anyway?

Alex Miller (Clojure team)14:06:33

well that's the question

borkdude14:06:45

So then the URL still is the most identifying thing for it

Alex Miller (Clojure team)14:06:09

if you didn't use the original, then dep resolution will give you both libs if you have a transitive dep using the original

Alex Miller (Clojure team)14:06:38

and in that case, it would be good to have deps tell you that you've done this

borkdude14:06:43

It’s the same in maven world right now

Alex Miller (Clojure team)14:06:13

although probably much harder to automatically detect

Alex Miller (Clojure team)14:06:11

(note that git urls are also not unique - there are multiple url formats to refer to the same repo, but I think that's a solvable problem)

borkdude16:06:14

I have got something compiling now with tda + tools.gitlibs@shell-git branch One issue I stumbled upon:

Error: com.oracle.graal.pointsto.constraints.UnresolvedElementException: Discovered unresolved type during parsing: org.apache.maven.project.ReactorModelPool. To diagnose the issue you can use the --allow-incomplete-classpath option. The missing type is then reported at run time when it is accessed the first time.
Detailed message:
Trace:
	at parsing clojure.tools.deps.alpha.extensions.pom$model_resolver.invokeStatic(pom.clj:51)
Call path from entry point to clojure.tools.deps.alpha.extensions.pom$model_resolver.invokeStatic(Object):
	at clojure.tools.deps.alpha.extensions.pom$model_resolver.invokeStatic(pom.clj:41)
	at clojure.tools.deps.alpha.extensions.pom$model_resolver.invoke(pom.clj:41)
	at clojure.core.proxy$clojure.lang.APersistentMap$ff19274a.hashCode(Unknown Source)
	at java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:936)
	at java.util.Properties.getProperty(Properties.java:1125)
	at com.oracle.svm.core.jdk.SystemPropertiesSupport.getProperty(SystemPropertiesSupport.java:144)
	at com.oracle.svm.core.jdk.Target_java_lang_System.getProperty(JavaLangSubstitutions.java:345)
	at com.oracle.svm.jni.JNIJavaCallWrappers.jniInvoke_VA_LIST:Ljava_lang_System_2_0002egetProperty_00028Ljava_lang_String_2_00029Ljava_lang_String_2(generated:0)
I now use --allow-incomplete-classpath just to see how far it gets. I now get a NPE in util/maven.clj on the line local-repo-mgr (.newLocalRepositoryManager system session local-repo) where system appears to be null. Might be related to the --allow-incomplete-classpath but I'll keep digging.

Alex Miller (Clojure team)16:06:39

It is almost certainly related

Alex Miller (Clojure team)16:06:36

There is a similar thing like this logged where an error occurs while setting up the service locator

borkdude16:06:58

So ditching jgit for shell git works. When are you going to ditch org.apache.maven libs? 😉

borkdude16:06:19

I wonder what's the issue with this class

ghadi18:06:01

before using a solution like shelling out to git, I think it’s wise to re-examine the problem

ghadi18:06:28

I know I sound like a broken record

borkdude18:06:04

Alex pointed me at that shell-git branch when I told him jgit didn't work with GraalVM. Just examining + listing all the problems. Shelling out to git allows me to explore the next problem without being stuck on it. There might not be only one "the problem", there might be many.

ghadi19:06:11

what part of jgit didn't work?

borkdude19:06:09

I think the output I got was similar to this one: https://bugs.eclipse.org/bugs/show_bug.cgi?id=546175

borkdude19:06:14

I think that problem should be solvable using the approach quarkus takes: https://quarkus.io/guides/jgit

borkdude19:06:27

so if there's need, it can be made to work, I'm pretty sure