I see, so local deps always wins, and in the case of conflicting local deps it would error.

But maven and git can both use the commit sha to find out which one is newest

I'm not seeing anything scm related or git related in the manifest spec: https://docs.oracle.com/javase/8/docs/technotes/guides/jar/jar.html#Manifest_Specification there's a version-number field, but that assumes digits

@didibus it would be in pom.xml not the manifest

for example

<scm>
    <url></url>
    <connection>scm:git:</connection>
    <developerConnection>scm:git:@github.com/seancorfield/next-jdbc.git</developerConnection>
    <tag>05a6d8a47fcea7dc2928e5e2c176310fb7205e0c</tag>
  </scm>

I thought it was common to put tag to HEAD ?

Also, what does maven do with that? Is the pom.xml uploded to Clojars as well?

Yes, the pom.xml is what tells Clojars how to link to the commit that is tied to the release.

It can either be a specific tag or a SHA.

I see

(I happen to use SHAs, just because I have a deploy script that does that automatically)

Sounds like days worth of fun debugging conflicts to be honest 😛 But hopefully it mostly works itself out in practice

Why would you think that?

Clojars shows which commit is referenced in pom.xml

Well, maybe I'm wrong. But I can see the tag not mapping to the correct version, and issues incurring. Though I guess there's nothing to do about that

Sure, you can "lie" in pom.xml if you don't have some automated release process...

And what does happen if you set HEAD as the tag?

Is that automatically replaced by the HEAD sha at time of deploy?

Deployment is independent of Git (so, no).

(at least, that is my understanding)

Certainly, you can mess up the <scm> part of pom.xml -- when I first started publishing to Clojars via clojure-generated POM files, I messed up several times. Hence, automation 🙂

cljdoc also relies on pom.xml being accurate so it can pull the correct version of your repo from GitHub to generate docs that match the release artifact...

I mean, the example on Maven shows HEAD as tag: https://maven.apache.org/pom.html#SCM

It says: tag: Specifies the tag that this project lives under. HEAD (meaning, the SCM root) should be the default.

Anyways, it does seem more standard then I thought. It should be a pretty nice way to handle cross resolver conflicts.

I guess a local deps could also use the pom.xml found in the directory for conflict resolution ?

it could if there is one

although I think the chances that value is meaningful+correct in a local dep is low

vs the rev in an artifact, which is almost always updated during deployment

open question what to do for a local dep that also happens to be a git repo

I would need to think about that for a while and make some tables

there are requests for git deps on the file system too (which we don't support right now) so that intersects

The world of dependency management is hard. I've lost so many hours of my life managing dependencies 😛 At least Clojure libs rarely break anything, so which version ends up chosen doesn't matter as much 😛

yeah, I've been doing this stuff for 22 years in the Java universe. those early years pre-Maven were kinda weird.

I do love how tools.deps resolves conflicts by picking latest though. That's a huge improvement for me with regards to Maven.

Somehow I managed to completely skip Maven until I got to Clojure... 👀

managing your deps manually is a very instructive process (not very efficient in the long run, but doing it a little really gives you an appreciation)

I used (and wrote) a lot of bespoke tools pre-Maven

I never lived in the pre-maven world, for the better I guess

adding libs on the fly in the repl? If I’m not mistaking the commit pointed to in http://insideclojure.org/2018/05/04/add-lib/ is gone, and github says there is no add-lib function or a repl ns in the repo. Any ideas on how people do this nowadays?

@niclasnilsson If you look at this: https://github.com/seancorfield/dot-clojure/blob/master/deps.edn

and in particular line 139

you'll see how it is used 🙂

@dharrigan, thanks (I wonder why github didn’t find add-lib in that very repo though?) . But, I don’t get it work though. Installing the deps worked, but I get a NullPointerException when trying to add a lib. Anything you recognize?

user=> (require '[clojure.tools.deps.alpha.repl :refer [add-lib]])
nil
user=> (add-lib 'honeysql {:mvn/version "0.9.8"})
Execution error (NullPointerException) at clojure.tools.deps.alpha.util.maven/make-session (maven.clj:173).
null

add-lib exists only on a branch, has never been integrated on the main line

@alexmiller, yep, understood. I used the sha in Sean’s deps.edn right now though

(when I get the NPE)

You should never see that error but I have seen people report this. I’ve never been able to repro

Interesting. Any special investigation I can undertake on my env?

It should be impossible by my understanding of the code so clearly some assumption I have is wrong

Even more interesting 🙂

@niclasnilsson seems to work for me!

user=> (require '[clojure.tools.deps.alpha.repl :refer [add-lib]])
nil
user=> (add-lib 'honeysql {:mvn/version "0.9.8"})
true

Try deleting your .cpcache?

Good idea. Will try. (But I’ll save it some place else)

Should t be using cpcache

k

That’s for caching classpaths but you’re not making those here, you’re dynamically updating runtime

What repl env are you in?

:thumbsup:

Not all repls have the right classloader setup for this to work

Clearing .cpcaches didn’t work

I’d be shocked if it did :)

repl: Vanilla clj

(no socket, no nrepl)

Well, don’t know then

The docs say “Note that for successful use, you must be in a REPL environment where a valid parent DynamicClassLoader can be found in which to add the new lib urls.“, but I suppose that’s the case when I just fire up clj?

Yes

Ok

Hmm… It works if I have a deps.edn only contaning

{:deps
 {org.clojure/tools.deps.alpha {:git/url ""
                                :sha "148cab8f154711f8495bd632ebdc53a14dff7f09"}}}

But not in my project, so likely something in there messes it up. I’ll reduce my deps.edn and see if I can find it.

you can do clojure -Stree to get a list

If you have something that doesn’t work now, I’d take that

I had a repro for this involving dependency conflicts

https://github.com/juxt/pack.alpha/issues/64 oz transitively brings in a different maven package.

The session lookup fails because its the wrong class

It's one from a different dependency

It's not specific to the add lib branch, it's just the only time users interact with maven outside of the clj internal classpath

{:deps        
 {metasoarous/oz                {:mvn/version "1.6.0-alpha5"}
  org.clojure/tools.deps.alpha  {:git/url ""
                                 :sha "148cab8f154711f8495bd632ebdc53a14dff7f09"}}}

(require '[clojure.tools.deps.alpha.repl :refer [add-lib]])
(add-lib 'honeysql       {:mvn/version "0.9.8"})

Ah, I see @dominicm already pointed out oz and the cause.

@alexmiller let me know if there's anything I can do to help with explaining or documenting this. I spent a lot of effort debugging it, and I hate for that to be a waste.

Oh interesting, I had not seen all that debugging before

Do you have jira access?

You helped me with it 😜

I do, yes.

If you could file a tdeps ticket, that would be great for tracking

I don’t think there is one

Certainly oz should not be publishing an uberjar, that’s evil

I’d like to be able to fail better in tdeps though

oz shouldn't be doing an uberjar, but this is unrelated to that.

the problem is that maven-aether-resolver and maven-resolver-provider conflict. I think oz gets the former through pomegranate, but icr right now, will double check before I file the jira.

@alexmiller I think I did mention the resolution in this channel, is there a better place to mention that kind of thing to make sure you don't miss it next time?

As a total aside, on my wishlist if for maven to support "provides", as in "juxt/tools.namespace" provides "org.clojure/tools.namespace", so there should only be one of them in the classpath (same for things where a -complete artifact is provided such as antlr. There's N artifacts and a "bundled" artifact too.)

I'm having a deps issue with (drumroll, please), jackson. I have com.fasterxml.jackson.core/jackson-core {:mvn/version "2.10.1"} in my :deps map, but when I do an ion-dev push it reports a conflict of com.fasterxml.jackson.core/jackson-core #:mvn{:version "2.9.8"}. This manifests itself when my ns gets loaded with the error "com.fasterxml.jackson.core.exc.InputCoercionException," which is a 2.10 addition. I tried adding the library to :override-deps instead, without success. Any tips on what I am doing wrong (besides needing jackson)?

can you share your ion-dev alias?

specifically, is it using :extra-deps or :deps?

and if it's using :extra-deps, can you switch it to :deps? (need newish version of clj for that to work)

Sure, latest attempt:

{:aliases
                {:ion-dev
                 {:deps      {com.datomic/ion-dev {:mvn/version "0.9.247"}}
                  :main-opts ["-m" "datomic.ion.dev"]}}

 :paths         ["src" "resources"]

 :deps          {com.cognitect/anomalies    {:mvn/version "0.1.12"}
                 com.datomic/client-cloud   {:mvn/version "0.8.78"}
                 com.datomic/ion            {:mvn/version "0.9.35"}
                 org.clojure/clojure        {:mvn/version "1.10.1"}
                 org.clojure/data.json      {:mvn/version "0.2.6"}
                 metosin/ring-http-response {:mvn/version "0.9.1"}
                 ring/ring-defaults         {:mvn/version "0.3.2"}
                 ring/ring-json             {:mvn/version "0.5.0"}
                 metosin/jsonista           {:mvn/version "0.2.5"}
                 metosin/reitit             {:mvn/version "0.3.10"}
                 javax.xml.bind/jaxb-api    {:mvn/version "2.4.0-b180830.0359"}
                 }

 :override-deps {com.fasterxml.jackson.core/jackson-core     {:mvn/version "2.10.1"}
                 com.fasterxml.jackson.core/jackson-databind {:mvn/version "2.10.1"}}

 :mvn/repos     {"datomic-cloud" {:url ""}}}

so the error is not during push, it's when your code is run in the ion

right?

that :override-deps in your deps.edn isn't doing anything - that only works in an alias

but you can just add those to your main :deps

Right, I get a warning when I push stating that I have the dependency conflict on 2.9.8 and the error is at runtime. I am assuming this is because the ion is using the older version without the InputCoercionException class.

Yeah, that override-deps was just my attempt to get the override to work. It was in deps before.

{:aliases
                {:ion-dev
                 {:deps      {com.datomic/ion-dev {:mvn/version "0.9.247"}}
                  :main-opts ["-m" "datomic.ion.dev"]}}
 :paths         ["src" "resources"]
 :deps          {com.cognitect/anomalies    {:mvn/version "0.1.12"}
                 com.datomic/client-cloud   {:mvn/version "0.8.78"}
                 com.datomic/ion            {:mvn/version "0.9.35"}
                 org.clojure/clojure        {:mvn/version "1.10.1"}
                 org.clojure/data.json      {:mvn/version "0.2.6"}
                 metosin/ring-http-response {:mvn/version "0.9.1"}
                 ring/ring-defaults         {:mvn/version "0.3.2"}
                 ring/ring-json             {:mvn/version "0.5.0"}
                 metosin/jsonista           {:mvn/version "0.2.5"}
                 metosin/reitit             {:mvn/version "0.3.10"}
                 javax.xml.bind/jaxb-api    {:mvn/version "2.4.0-b180830.0359"}
                 com.fasterxml.jackson.core/jackson-core     {:mvn/version "2.10.1"}
                 com.fasterxml.jackson.core/jackson-databind {:mvn/version "2.10.1"}}

:mvn/repos     {"datomic-cloud" {:url ""}}}

ok, well then I would send you to #datomic - this is not the first time they've seen something like this, but I don't know the recommended path forward. I think you're right on with what the problem is.

I'll give that shot and report the output. I think I had done that before. BTW, the jaxb addition was because of this warning when I do the ion push: [s3-transfer-manager-worker-6] WARN com.amazonaws.util.Base64 - JAXB is unavailable. Will fallback to SDK implementation which may be less performant

Cool, let me give this a shot and I'll let you know what happened then bounce over to datomic.

yeah, that's likely due to being on java 9+

it's included in java 8

but they've been shedding all those j2ee libs from the jdk

Is the datomic runtime 8 or 9?

dunno

I mean, it must be 8 or 11

9 and 10 are dead and I can't imagine they're using those, so I'd guess 8

Full output:

Downloading: com/datomic/java-io/0.1.11/java-io-0.1.11.pom from 
(cognitect.s3-libs.s3/upload "datomic-code-d70d7189-fa46-4cd5-8004-423b714e486b" [{:local-zip ".datomic-ions/datomic/apps/ion-repl/stable/ff0ed784b940e81fa02f8a0808a3fc06719bfc1b.zip", :s3-zip "datomic/apps/ion-repl/stable/ff0ed784b940e81fa02f8a0808a3fc06719bfc1b.zip"}] {:op :push})
[s3-transfer-manager-worker-6] WARN com.amazonaws.util.Base64 - JAXB is unavailable. Will fallback to SDK implementation which may be less performant
{:rev "ff0ed784b940e81fa02f8a0808a3fc06719bfc1b",
 :deploy-groups (ion-repl-Compute-1TPY2S14U7NT9),
 :dependency-conflicts
 {:deps
  {com.cognitect/transit-java #:mvn{:version "0.8.311"},
   org.clojure/clojure #:mvn{:version "1.10.0"},
   commons-codec/commons-codec #:mvn{:version "1.12"},
   com.datomic/client-api #:mvn{:version "0.8.37"},
   com.fasterxml.jackson.core/jackson-core #:mvn{:version "2.9.8"},
   com.datomic/client #:mvn{:version "0.8.86"},
   org.msgpack/msgpack #:mvn{:version "0.6.10"},
   com.datomic/client-impl-shared #:mvn{:version "0.8.69"},
   com.cognitect/transit-clj #:mvn{:version "0.8.285"},
   com.cognitect/s3-creds #:mvn{:version "0.1.23"},
   com.datomic/client-cloud #:mvn{:version "0.8.80"},
   com.amazonaws/aws-java-sdk-kms #:mvn{:version "1.11.479"},
   com.amazonaws/aws-java-sdk-s3 #:mvn{:version "1.11.479"}},
  :doc
  "The :push operation overrode these dependencies to match versions already running in Datomic Cloud. To test locally, add these explicit deps to your deps.edn."},
 :deploy-command
 "clojure -A:ion-dev '{:op :deploy, :group ion-repl-Compute-1TPY2S14U7NT9, :rev \"ff0ed784b940e81fa02f8a0808a3fc06719bfc1b\"}'",
 :doc
 "To deploy to ion-repl-Compute-1TPY2S14U7NT9, issue the :deploy-command"}

I'll try the deploy, but I think it's going to be broken. I'll try #datomic. Thanks!

yeah, I mean it's telling you that in the result there

Right. "We're using this library that hasn't got a class you're going to need."

@alexmiller Hi there Alex, thanks for building/maintaining deps. And all the rest you do for the clj community. Much appreciation! Had a couple of questions. 1. I’m trying to use tools.deps as a lib. But have an issue using resolve-deps . One of the deps.edn files uses a package from a mvn repo, where the repo is declared in the deps.edn file, and where the cli picks it ub correctly. But it blows up for me when using the lib with:

Could not find artifact io.confluent:kafka-schema-registry-client:jar:5.3.0
   in central ()

im calling it using

(-> {:deps {'root/root {:local/root "<path to root project>"}}
       :mvn/repos mvn/standard-repos}
      (deps/resolve-deps nil
                         {:trace true})))

the required file includes the mvn repo as

:mvn/repos
 {"confluent" {:url ""}}

2. I have a case where that file adds a jar to its paths. But when using juxt’s pack, it thinks the jar lives at one of the transative local/root dependencies dirs. I dont know if this is a pack problem or something with tools.deps. But im guessing deps.

How do I best give you info to help fix the bug? I’ve managed to work around it for now using depstar