Fork me on GitHub
#clojars
<
2023-05-23
>
emccue16:05:48

Silly q - but for artifacts published on clojars is it possible for them to depend on artifacts that are not on clojars? I'm back on my resolver implementation train

Alex Miller (Clojure team)16:05:01

For example, every Clojure library in clojars depends on Clojure in Maven Central

Alex Miller (Clojure team)16:05:40

“Where” is intentionally orthogonal from coordinates

emccue16:05:59

When I run a fetch for ring/ring I do find the clojure artifacts surprisingly

emccue16:05:39

which kinda threw me

emccue16:05:08

public class ClojureFetch {
    public static void main(String[] args) {
        var libraries = new Fetch()
                .addDependency(Dependency.maven(
                        "ring:ring:1.10.0",
                        MavenRepository.remote("")
                ))
                .withCache(Cache.standard(Path.of("./local")))
                .run()
                .libraries();

        libraries.forEach((library, path) ->
                System.out.println(library + " = " + path)
        );
    }
}

emccue16:05:17

Library[group=crypto-random, artifact=crypto-random] = ./local/https/repo.clojars.org/crypto-random/crypto-random/1.2.1/crypto-random-1.2.1.jar
Library[group=commons-codec, artifact=commons-codec] = ./local/https/repo.clojars.org/commons-codec/commons-codec/1.15/commons-codec-1.15.jar
Library[group=ring, artifact=ring-devel] = ./local/https/repo.clojars.org/ring/ring-devel/1.10.0/ring-devel-1.10.0.jar
Library[group=org.clojure, artifact=tools.namespace] = ./local/https/repo.clojars.org/org/clojure/tools.namespace/0.2.11/tools.namespace-0.2.11.jar
Library[group=org.clojure, artifact=java.classpath] = ./local/https/repo.clojars.org/org/clojure/java.classpath/0.3.0/java.classpath-0.3.0.jar
Library[group=ring, artifact=ring-codec] = ./local/https/repo.clojars.org/ring/ring-codec/1.2.0/ring-codec-1.2.0.jar
Library[group=commons-fileupload, artifact=commons-fileupload] = ./local/https/repo.clojars.org/commons-fileupload/commons-fileupload/1.5/commons-fileupload-1.5.jar
Library[group=hiccup, artifact=hiccup] = ./local/https/repo.clojars.org/hiccup/hiccup/1.0.5/hiccup-1.0.5.jar
Library[group=ns-tracker, artifact=ns-tracker] = ./local/https/repo.clojars.org/ns-tracker/ns-tracker/0.4.0/ns-tracker-0.4.0.jar
Library[group=ring, artifact=ring] = ./local/https/repo.clojars.org/ring/ring/1.10.0/ring-1.10.0.jar
Library[group=clj-stacktrace, artifact=clj-stacktrace] = ./local/https/repo.clojars.org/clj-stacktrace/clj-stacktrace/0.2.8/clj-stacktrace-0.2.8.jar
Library[group=org.eclipse.jetty, artifact=jetty-server] = ./local/https/repo.clojars.org/org/eclipse/jetty/jetty-server/9.4.51.v20230217/jetty-server-9.4.51.v20230217.jar
Library[group=ring, artifact=ring-servlet] = ./local/https/repo.clojars.org/ring/ring-servlet/1.10.0/ring-servlet-1.10.0.jar
Library[group=org.clojure, artifact=clojure] = ./local/https/repo.clojars.org/org/clojure/clojure/1.7.0/clojure-1.7.0.jar
Library[group=commons-io, artifact=commons-io] = ./local/https/repo.clojars.org/commons-io/commons-io/2.11.0/commons-io-2.11.0.jar
Library[group=ring, artifact=ring-jetty-adapter] = ./local/https/repo.clojars.org/ring/ring-jetty-adapter/1.10.0/ring-jetty-adapter-1.10.0.jar
Library[group=ring, artifact=ring-core] = ./local/https/repo.clojars.org/ring/ring-core/1.10.0/ring-core-1.10.0.jar
Library[group=crypto-equality, artifact=crypto-equality] = ./local/https/repo.clojars.org/crypto-equality/crypto-equality/1.0.1/crypto-equality-1.0.1.jar

emccue16:05:02

but my coordinates are not orthogonal to where

emccue16:05:45

so when I fetch the manifest for a dependency, I am looking at that pom file and the coordinate I spit out will contain the repositories to check

emccue16:05:46

conceptually the problem now is that I want to be able to say "only check clojars for this dep"

Dependency.maven(
    "ring:ring:1.10.0",
    MavenRepository.remote("")
);
But "what repos to check for dependencies" is tricky

emccue16:05:17

i can say "just propagate the list you checked for this dependency"

emccue16:05:00

or "read the pom of the dependency, pull out repository declarations there + add central, use that" - but that feels wretched

emccue16:05:42

public record MavenCoordinate(
        Version version,
        List<MavenRepository> repositories,
        List<Scope> scopes,
        @ToolsDeps(
                value = "",
                details = "TDeps makes classifier part of the artifact group/artifact[$classifier]"
        )
        Classifier classifier,
        Classifier sourceClassifier,
        Classifier documentationClassifier
) 
relevant model

emccue16:05:06

and then this is where i just propagate the repository that found the artifact as the repository to check for dependencies

emccue16:05:08

final PomManifest getManifest(Library library, Version version, Cache cache, List<Scope> scopes) {
    var effectivePom = EffectivePomInfo.from(getAllPoms(library, version, cache));
    return PomManifest.from(
            effectivePom,
            scopes,
            (depVersion, depExclusions) -> new MavenCoordinate(depVersion, this)
    ).normalize(cache);
}

emccue17:05:26

maybe im just modeling this poorly

emccue17:05:39

nevermind! should have looked closer

emccue17:05:21

there we are. twas goobering it

quoll22:05:04

What about something in Clojars depending on a library released as a git coordinate?

emccue23:05:44

I guess it comes down to this

emccue23:05:04

For a library A, I am going to check some list of repos to find it

emccue23:05:32

That library is going to declare dependencies on B, C, and D

emccue23:05:59

For those should I search • The same list of repositories I searched for A • The repository I found A in • The same list of repositories I searched for A + repositories declared in A's pon • The repositories I found A in + repositories declared in A's pom • Some other option?

emccue23:05:20

And how should I handle if a repository is not accessible anymore or...idk

emccue23:05:22

@U051N6TTC I don't think POMs as a manifest format have the ability to support dependencies that aren't taken from a maven-like directory structure somewhere

Alex Miller (Clojure team)23:05:42

Re the search question, I believe Maven will do #3. tools.deps will do #1 (it always uses the list of repos defined at the top deps.edn + Maven central +Clojars)

Alex Miller (Clojure team)23:05:31

Maven checks all repos, skipping those that fail, and erroring if no one has it

Alex Miller (Clojure team)23:05:11

Poms can have extensions that know how to handle new url types. Leiningen uses that infrastructure to add s3 support via s3p:// for example (tdeps does this a bit differently). It would be possible to create a Clojure lib git extension that worked in maven poms

emccue23:05:30

I think Coursier does #1 too

emccue00:05:26

So I'm fine doing #1 for now, but with the "per coordinate" thing. I can add knobs later

quoll05:05:38

Just this evening I happened to add in a dependency to a git-coordinate project for something I have on Clojars, and then when I went to push to Clojars I realized my problem. Oops 🙂 Hence, my question this evening. Anyway, I own the lib I’m depending on, so that’s now been promoted to Clojars as well

emccue04:05:55

org.clojure/clojure 1.11.0
  . org.clojure/spec.alpha 0.3.218
  . org.clojure/core.specs.alpha 0.2.62
ring/ring 1.9.3
  . ring/ring-core 1.9.3
    X org.clojure/clojure 1.7.0 USE_TOP
    . ring/ring-codec 1.1.3
      X org.clojure/clojure 1.5.1 USE_TOP
      . commons-codec/commons-codec 1.15
    . commons-io/commons-io 2.6
    . commons-fileupload/commons-fileupload 1.4
      X commons-io/commons-io 2.2 OLDER_VERSION
    . crypto-random/crypto-random 1.2.0
      X org.clojure/clojure 1.2.1 EXCLUDED
      X commons-codec/commons-codec 1.6 OLDER_VERSION
    . crypto-equality/crypto-equality 1.0.0
      X org.clojure/clojure 1.2.1 EXCLUDED
  . ring/ring-devel 1.9.3
    X org.clojure/clojure 1.7.0 USE_TOP
    . ring/ring-core 1.9.3
    . hiccup/hiccup 1.0.5
      X org.clojure/clojure 1.2.1 EXCLUDED
    . clj-stacktrace/clj-stacktrace 0.2.8
      X org.clojure/clojure 1.4.0 EXCLUDED
    . ns-tracker/ns-tracker 0.4.0
      X org.clojure/clojure 1.5.1 USE_TOP
      . org.clojure/tools.namespace 0.2.11
      . org.clojure/java.classpath 0.3.0
        X org.clojure/clojure 1.4.0 USE_TOP
  . ring/ring-jetty-adapter 1.9.3
    X org.clojure/clojure 1.7.0 USE_TOP
    . ring/ring-core 1.9.3
    . ring/ring-servlet 1.9.3
    . org.eclipse.jetty/jetty-server 9.4.40.v20210413
      . javax.servlet/javax.servlet-api 3.1.0
      . org.eclipse.jetty/jetty-http 9.4.40.v20210413
        . org.eclipse.jetty/jetty-util 9.4.40.v20210413
        . org.eclipse.jetty/jetty-io 9.4.40.v20210413
      . org.eclipse.jetty/jetty-io 9.4.40.v20210413
        . org.eclipse.jetty/jetty-util 9.4.40.v20210413
  . ring/ring-servlet 1.9.3
    X org.clojure/clojure 1.7.0 USE_TOP
    . ring/ring-core 1.9.3

emccue04:05:02

clj -Stree -Sdeps '{:deps {org.clojure/clojure {:mvn/version "1.11.0"} ring/ring {:mvn/version "1.9.3"}}}'
org.clojure/clojure 1.11.0
  . org.clojure/spec.alpha 0.3.218
  . org.clojure/core.specs.alpha 0.2.62
ring/ring 1.9.3
  . ring/ring-core 1.9.3
    . ring/ring-codec 1.1.3
      . commons-codec/commons-codec 1.15
    . commons-io/commons-io 2.6
    . commons-fileupload/commons-fileupload 1.4
      X commons-io/commons-io 2.2 :older-version
    . crypto-random/crypto-random 1.2.0
      X commons-codec/commons-codec 1.6 :older-version
    . crypto-equality/crypto-equality 1.0.0
  . ring/ring-devel 1.9.3
    . ring/ring-core 1.9.3
    . hiccup/hiccup 1.0.5
    . clj-stacktrace/clj-stacktrace 0.2.8
    . ns-tracker/ns-tracker 0.4.0
      . org.clojure/tools.namespace 0.2.11
      . org.clojure/java.classpath 0.3.0
  . ring/ring-jetty-adapter 1.9.3
    . ring/ring-core 1.9.3
    . ring/ring-servlet 1.9.3
    . org.eclipse.jetty/jetty-server 9.4.40.v20210413
      . javax.servlet/javax.servlet-api 3.1.0
      . org.eclipse.jetty/jetty-http 9.4.40.v20210413
        . org.eclipse.jetty/jetty-util 9.4.40.v20210413
        . org.eclipse.jetty/jetty-io 9.4.40.v20210413
      . org.eclipse.jetty/jetty-io 9.4.40.v20210413
        . org.eclipse.jetty/jetty-util 9.4.40.v20210413
  . ring/ring-servlet 1.9.3
    . ring/ring-core 1.9.3

emccue04:05:20

i'm not hiding the clojure stuff but other than that - getting the right results