Clojurians Log v2

you are cooking chef_kiss

Impressive how it evolved through the years! Cheers and thanks for putting it out in the open!

Most users of tools.deps as a library use the public API namespaces or parts of the implementation that did not change and should be unaffected. You may be affected if you use clojure.tools.deps.util.maven and specifically the functions make-settings, make-system, make-session, remote-repos, remote-repo - in general there is a new flow for setting up the Maven state under MIMA that pokes through these functions. Some known users of these functions include clojure/tools.build (me), liquidz/antq, Olical/depot (@olical), and babashka/tools-deps-native (@borkdude). When these repos update to the latest tools.deps, they should update their code to the new API (use make-context, make-system (new arity), make-system-session, and new arities of remote-repos or remote-repo - see usage in tools.deps for examples). I was able to test most of these even without update and they still pass the existing test suites (but you're silently missing some features). If anyone needs help, ping me. I also reviewed the code in liquidz/build.edn, jlesquembre/clj-nix, kepler16/kmono, clj-holmes/clj-watson @seancorfield, polyfy/polylith, and clojure-emacs/refactor-nrepl @bozhidar and I believe all those are unaffected, can just update tools.deps version as is.

wow, thorough review!

I've also added ns docstrings more clearly marking what I consider API / SPI / implementation in tools.deps

clj-watson/src/clj_watson/diplomat/dependency.clj
clj-watson/src/clj_watson/controller/deps.clj
clj-watson/src/clj_watson/controller/remediate.clj

I don't recall if that's easy or not - there are some things buried down there that probably should be more public. and I am generally fine with tools (which control their transitive dep version) using some of these lower bits as long as you understand that impl may change.

I think you probably just borrow the standard-repos in those, which is fine. should probably just be somewhere marked as API

I'll have a look over the weekend. In the meantime, I'll update our work repo to the latest CLI and blow away our BitBucket m2 cache and see how the download process goes...

instead of "blow away", I would recommend renaming ~/.m2/repository so you still have it just in case :)

Updated Clojure CLI version number to fix a bad version in :deps alias

First test -- worked great! Downloaded the entire world with no errors. Updating to that new CLI version and doing it again 🙂

Yup, 1654 works just fine too -- blew away all the CI caches and let it download the world 🙂 No errors. (sorry for slow response -- was in a meeting)

@alexmiller Confirmed the only thing Watson uses in util.maven is standard-repos -- is there a supported, API-level way to get that?

I would just leave as is for now

deps.clj released

I've promoted 1.12.5.1654 to stable

Just noticed this (new) warning during our CI run:

org.apache.maven.repository.internal.DefaultArtifactDescriptorReader] [] {} 1 problem was encountered while building the effective model for org.javassist:javassist:jar:3.18.1-GA during n/a
Problem
* 'dependencies.dependency.systemPath' for com.sun:tools:jar refers to a non-existing file /Developer/jdk-26/../lib/tools.jar. Please verify that you run Maven using a JDK and not just a JRE. @ org.javassist:javassist:3.18.1-GA

(and, yes, it's JDK 26... I looked for tools.jar in my myriad JDK installs and only JDK 8 seems to contain that)

We have 1.12.5.1654 on all our servers now. Thank you for this update!

are you saying that warning is related to the new CLI?

I assume the updated Maven perhaps? But I haven't dug into it yet. Just wondered if it rang any bells for anyone in this thread.

afaik, the CLI / tools.deps does not depend on javassist

No, I know.

(and I mean that transitively too)

This is a transitive dependency in one of our projects when we run uber but it wasn't showing the warning before we updated the CLI (and thus the underlying Maven libraries). So the earlier version of Maven didn't complain about that it seems...

ah, ok

. com.cognitect/transit-clj 1.0.333
    . com.cognitect/transit-java 1.0.371
      . com.fasterxml.jackson.core/jackson-core 2.14.2
      . org.msgpack/msgpack 0.6.12
        . com.googlecode.json-simple/json-simple 1.1.1
        . org.javassist/javassist 3.18.1-GA
      . javax.xml.bind/jaxb-api 2.4.0-b180830.0359
        . javax.activation/javax.activation-api 1.2.0

That's coming in from Portal.

the newest version of transit-java updates the msgpack dep and javassist is no longer a transitive dep, fyi

Yup, confirmed that if I exclude transit from portal and add 1.1.357 explicitly at the top-level, the warning goes away.

New deps:

com.cognitect/transit-clj 1.1.357
  . com.cognitect/transit-java 1.1.403
    . com.fasterxml.jackson.core/jackson-core 2.21.1
    . org.msgpack/msgpack-core 0.9.11
    . javax.xml.bind/jaxb-api 2.4.0-b180830.0359
      . javax.activation/javax.activation-api 1.2.0

Created an issue against #C0185BFLLSE to update Transit.

Thanks, @alexmiller

@slipset I don't think deps-deploy is compatible with the latest tools.deps / tools.build with the updated Maven deps... deps-deploy has

:deps {org.clojure/clojure {:mvn/version "1.12.4"}
        clj-commons/pomegranate {:mvn/version "1.3.27"}
        s3-wagon-private/s3-wagon-private {:mvn/version "1.3.5"}
        org.clojure/tools.deps {:mvn/version "0.18.1354"}
        org.apache.maven/maven-settings {:mvn/version "3.9.4"}
        org.apache.maven/maven-settings-builder {:mvn/version "3.9.4"}
        org.slf4j/slf4j-nop {:mvn/version "RELEASE"}
        org.sonatype.plexus/plexus-sec-dispatcher {:mvn/version "1.4"}}

Execution error (ClassNotFoundException) at java.net.URLClassLoader/findClass (URLClassLoader.java:445).
org.apache.maven.settings.DefaultMavenSettingsBuilder

@alexmiller Does that sound like a Maven version conflict?

(background: I updated both next.jdbc and HoneySQL to the latest CLI and the latest tools.build and the deploy part of CI failed as above)

@seancorfield are there even newer maven libs? @tcrawley submitted some upgrade PRs very recently

could be a version issue, that version of tools.deps is much older than latest (0.31.1620)

that vintage probably relied on the maven 3.8.x libs, which are not compatible with 3.9.x (but the new version of tools.deps is)

Hmm, I'm calling deps-deploy in the build.clj process so that would already have the latest tools.deps -- and since it references DefaultMavenSettingsBuilder, I thought it might be the maven-settings(-builder) deps but it sounds like 3.9.x is already the latest there?

I'll fork deps-deploy and update all its deps and see if the problem goes away...

Latest tools.deps should be transitively pulling in the 3.9.16 level of maven-resolver libs (which I think is the latest)

@slipset If I update just tools.deps in the deps-deploy project, the tests fail with that DefaultMavenSettingsBuilder error.

the stuff deps-deploy uses from tools.deps is all now available in tools.deps.edn so that could be used instead and it would cut all of the maven stuff from tools.deps out of the tree and use pomegranates

the api functions are slightly different so it requires a couple code changes but it's just a couple lines I think

I switched it to tools.deps.edn and updated the changed call (and keys), but the deps-deploy code explicitly uses that missing class:

(defn- set-settings-builder
  "Copied from clojure.tools.deps.alpha"
  [^DefaultMavenSettingsBuilder default-builder settings-builder]
  (doto (.. default-builder getClass (getDeclaredField "settingsBuilder"))
    (.setAccessible true)
    (.set default-builder settings-builder)))

So, presumably it needs to have the updated versions copied in from the new tools.deps code

not being used in calls to the tools.deps stuff so doesn't matter

I think read-edn-files is the only function that needs to be updated, then you could just swap from tools.deps to tools.deps.edn in deps and namespace definition

Right. I did that (as I said above). But that's not the blocker here.

(defn- read-edn-files
  "Given as options map, use tools.deps.edn to read and merge the
  applicable `deps.edn` files"
  [{:keys [repro] :or {repro true}}]
  (let [{:keys [root user project]} (t/create-edn-maps nil)]
    (t/merge-edns (if repro
                    [root project]
                    [root user project]))))

Right, yes, I already made that change locally 🙂

what's left has nothing to do with tools.deps - that's just calls directly to maven and that needs to match what Pomegranate requires

you could try the equivalent from latest tools.deps.util.maven

oh, that doesn't exist anymore as it wasn't needed

Yup... I'm chatting with Erik via DM about it now. Thanks for the suggestion to switch to tools.deps.edn -- that will be an improvement.

maybe try

(defn get-settings
  ^Settings []
  (let [^SettingsBuilder builder (.newInstance (DefaultSettingsBuilderFactory.))
        request (DefaultSettingsBuildingRequest.)
        user-settings (jio/file (System/getProperty "user.home") ".m2" "settings.xml")]
    (when (.exists user-settings)
      (.setUserSettingsFile request user-settings))
    (.getEffectiveSettings (.build builder request))))

that's what I did in tools.deps

so you can just delete set-settings-builder

(also, if any more things come up, let's make a new thread :)

I have a PR in to deps-deploy for the changes. I think, right now, any projects that try to run deps-deploy via a direct call inside build.clj will fail due to the Maven changes (in the latest tools.build, due to the latest tools.deps), so I can't deploy next.jdbc or HoneySQL unless I back off the tools.build version 🙂

(and, yeah, copied that get-settings from clojure.tools.deps.util.maven into deps-deploy to replace the older Maven settings code Erik had)

seems like this is really due to deps-deploy updates to latest pomegranate, not tools.deps

The current deps-deploy works with tools.build 0.10.13 but not 0.10.14 -- that was the change I made in my project that triggered the problem.

but deps-deploy doesn't use tools.deps or maven stuff from it - the maven libs its using are coming from pomegranate

I updated tools.build (and the CLI version) in next.jdbc and HoneySQL -- the only changes -- and the CI builds failed. I think you're missing an important bit of context: I invoked deps-deploy directly inside build.clj so it is subject to the classpath for running builds.

If you run deps-deploy from the command-line directly, via a deps.edn alias, you won't hit this. Only if you call it directly from build.clj.

so it's really the combination of the two

Yeah, all the projects deps-new creates take this approach -- :build alias has both tools.build and deps-deploy and then build.clj has a deploy fn that invokes it (deps-deploy) directly in that process (rather than via a subprocess as a separate command). Glad I didn't go ahead and update deps-new yet 🙂

Dependency Hell 😄

this is a perfect example of why I created tools.deps.edn :)

needing to do some deps.edn reading/merging but not needing all the dep resolution or libs

Yup, it was a really good move!

I'm trying to catch up here. Is the fix to have pomegranate on 3.9.x? I tried moving to 3.9.x as part of the updates to use HTTPTransporter, but ran into missing transitive dependencies (https://github.com/clj-commons/pomegranate/pull/233#issuecomment-4178241538). I could take another look though. Or maybe it is time for clojure.tools.deps.deploy? :)

@tcrawley No changes needed for Pomegranate here. This was a weird dependency issue caused by an outdated tools.deps.alpha dep combined with more recent Maven deps (`deps-deploy` already depended on maven-settings 3.9.x), used in the context of tools.build 0.10.14, which depends on the latest tools.deps (not alpha) which depends on Maven 3.9.x 🙂

But, yeah, it sure would be nice to have Maven-style deployment as part of tools.build 🤔

it is on my list :)

Ah, gotcha. The issue I ran into was with maven-resolver-provider 3.9.x, not maven-settings. Thanks for the clarification @seancorfield!

@tcrawley do you (or @alexmiller) happen to know what the org.sonatype.plexus/plexus-sec-dispatcher dep might be for in deps-deploy? It's not compatible with something in the Maven 3.9.x stuff I think, but deps-deploy doesn't directly use it (so I assume it's for some sort of "provider" on the classpath)?

I don't know what it is, but pomegranate doesn't bring it in. Should we start a new thread somewhere else for that?

Already handled via DM with Erik and included in the PR (to remove it, unless he has a good reason to keep it).

don't know

deps-deploy-0.2.5 is out with @seancorfield’s fixes

Confirmed both next.jdbc and HoneySQL build and deploy cleanly with 0.2.5. Thank you!

Thank you!

Thanks for the heads up! https://github.com/cljdoc/cljdoc-analyzer/blob/master/src/cljdoc_analyzer/deps.clj, but seems unaffected by this latest release.

Can confirm that Clojure CLI 1.12.5.1654 works for us with an authenticated Maven repo (https://maven.pkg.github.com/...).