Clojurians Log v2

I was moving over one of my projects from depstar to the latest build-clj. I ran into the following issue that was not there when I used depstar. Not sure what the error means exactly and what to do about it.

Dorabs-iMac:recj dorab$ clj -T:build uber

Writing pom.xml...
Skipping coordinate: {:local/root /Users/dorab/Projects/gadj, :deps/manifest :deps, :deps/root /Users/dorab/Projects/gadj, :parents #{[]}, :paths [/Users/dorab/Projects/gadj/src /Users/dorab/Projects/gadj/resources]}
Skipping coordinate: {:local/root /Users/dorab/Projects/gsheetj, :deps/manifest :deps, :deps/root /Users/dorab/Projects/gsheetj, :parents #{[]}, :paths [/Users/dorab/Projects/gsheetj/src /Users/dorab/Projects/gsheetj/resources]}
Copying src, resources...
Skipping compilation because :main, :ns-compile, and :sort were omitted...
Building uberjar target/recj-0.2.97.jar...
Execution error (ExceptionInfo) at clojure.tools.build.tasks.uber/explode (uber.clj:172).
Cannot write META-INF/license/LICENSE.tomcat-native.txt from io.grpc/grpc-netty-shaded as parent dir is a file from another lib. One of them must be excluded.

Full report at:
/var/folders/lk/mf8f6ghs0zqcw70nmy739khm0000gn/T/clojure-7423998483693171531.edn

@dorab Yup, this bit me too. depstar writes everything directly into a temporary zip file which is case-sensitive. tools.build writes everything to disk first, then bundles it up into a JAR. You're on a case-insensitive O/S. Most libs have META-INF/LICENSE as a text file. The grpc-netty-shaded lib unfortunately has the path shown above which conflicts on case-insensitive O/S.

You can solve this by explicitly depending on an alternative to that lib. We depended on grpc-okhttp as a top-level dep and that gets picked up as a good implementation for what grpc-netty-shaded provides.

we dropped in depstar because it handled this case. i'm trying the latest tools.build but i suspect we will have to work around this. I was surprised to see depstar archived so quickly

Here's where we ran into this:

com.google.cloud/google-cloud-vision {:mvn/version "1.103.7"
                                        :exclusions [io.grpc/grpc-netty-shaded]}
  io.grpc/grpc-okhttp {:mvn/version "1.39.0"}

@dpsutton depstar 1.0 actually had this exact same problem because it worked the same way as tools.build does. 2.0 only avoided it because a user ran into a problem with a library that contained aux.clj which you can't write to the filesystem on Windows (because AUX is a reserved filename(!) along with NUL, PRN, and a few other things).

ha, didn't know that was the cause. i figured it would have been the license problem

2.0 was also substantially faster by writing to ZipFileSystem and then just moving it to the final location but it's not how other JAR builders work as far as I know? What does Leiningen do here?

@seancorfield Thanks. I will try that workaround. In my case, the dependency is coming from one of the Google APIs as well (Google Ads or Google Sheets - not sure which at the moment).

FWIW, the okhttp lib is a lot smaller than the netty-shaded version -- saved about 8MB in our JAR I think.

We also ran into a problem with a lib that depended on log4j2 2.13.3 as a pom which doesn't work properly with deps.edn but that problem is hidden by the fact it also depended on all the jar artifacts as well. However, when you try to build an uberjar, tools.build choked on finding something that wasn't either a .jar it could expand or a directory it could expand.

depstar explicitly ignored such files (and would tell you about it if you asked, as I recall), and I just got a complaint about depstar ignoring .tar.gz files which come from Oracle's GraalVM JARs I think and it expects them as top-level deps -- which doesn't work really in a JVM world. I gather it's an Oracle-specific issue.

As for the swift archiving of depstar, I think it's a problem that we have a bunch of different JAR builders for deps.edn instead of everyone pitching in to make one solution "the best" and I really want the official core team's offering to be the solution.

Thanks for the explanations. Agree with having one solution.

I've been working with Alex on getting tools.build to a point where I can stop maintaining depstar and start using the official core tool at work -- and I'd been using tools.build for my OSS projects for a while, switching away from depstar.

yeah good point.

removed depstar. thanks for it in the meantime sean and ghadi

@seancorfield Your suggestion about the exclusion and grpc-okhttp worked in the sense that I was able to create the uberjar. When I tried to run the uberjar using java -jar target/recj-0.2.98.jar --help I get no main manifest attribute, in target/recj-0.2.98.jar. I presume that means I need to supply some other parameters to the uber task that sets the main class?

:main

@dorab Just FYI, since you were talking about what deps-new produces by way of build.clj:

seanc@Sean-win-11-laptop:~/clojure$ clojure -Tnew app :name foo/bar
Creating project from org.corfield.new/app in bar
seanc@Sean-win-11-laptop:~/clojure$ cat bar/build.clj
(ns build
  (:refer-clojure :exclude [test])
  (:require [org.corfield.build :as bb]))

(def lib 'foo/bar)
(def version "0.1.0-SNAPSHOT")
(def main 'foo.bar)

(defn test "Run the tests." [opts]
  (bb/run-tests opts))

(defn ci "Run the CI pipeline of tests (and build the uberjar)." [opts]
  (-> opts
      (assoc :lib lib :version version :main main)
      (bb/run-tests)
      (bb/clean)
      (bb/uber)))

Thanks. Seems like the value of :main could be figured out from the value of :lib (by default, if not specified)?

Got the project transferred over from using depstar to build-clj. Thanks for your help. Now, on to moving thenext project over to build-clj. 🙂

@dorab Well, lib/`version` are used (if provided) by write-pom for the group/artifact (and version), and lib is used by default for the JAR filename, but main is a namespace that isn't necessarily related to the lib at all -- although in the simple case of a new app they are related. Although I'm tempted to remove lib/`version` there for a minimal app project since they're not really needed (an app doesn't need a pom.xml)... but that's a tough call.

Can the build.clj (for example) live in another directory other than the root of the project?

I continue my happy migration to tools.build and am finding myself a bit confused by API docs for https://clojure.github.io/tools.build/clojure.tools.build.api.html#var-install. Perhaps this is because I am coming from https://github.com/slipset/deps-deploy#install-locally. The tools.build install API docs say Generate pom file and install pom and jar to local Maven repo but then later :class-dir - required, used to find the pom file, which leaves me asking myself, “wait, does this create the pom.xml or read the pom.xml”? And then, because it takes :version and :lib as required params, I ask myself, “wait, can’t it get these from the jar I created, or maybe… does it create the jar?“.

@slipset has joined the channel

In other news, (as I’m reading through some of the backlog here) I’ll make deps-deploy use tools-build for building jars in the very near future. And, as mentioned elsewhere, I’d be more than happy to follow @seancorfield’s suite and archive deps-deploy as soon as tools-build has the same capabilities, ie install to local (which it seems tools-build has) and deploy (which I understand may or may not become a part of tools-build ?)

I would still plan to use deps-deploy for that part