Fork me on GitHub
#tools-build
<
2022-05-02
>
Chris Lester05:05:06

Hi, I have a noobie question around java compilation using tools.build (assuming this is the right place to ask). I’m trying to get a combined java/clojure component to compile/build and allow me to import the java result into the clj side of that component (polylith .. with nested deps.edn containing the deps). Where should the java dependencies be provided if not in a deps.edn? Do I need to write a pom (at the component level?) for those?

delaguardo05:05:23

Both types, java and clojure dependencies, should go to deps.edn

delaguardo05:05:29

I assume those java dependencies are reachable as a maven library.

seancorfield05:05:11

Aim to compile the Java code first and make sure classes is on the path for the Clojure code to execute.

Alex Miller (Clojure team)06:05:34

Be sure to check out the “Mixed Java / Clojure build” example at https://clojure.org/guides/tools_build too

Alex Miller (Clojure team)06:05:22

When compiling the Java classes you need a basis, which defines the classpath based on a deps.edn, but there are lots of ways to determine what basis is calculated (using aliases, or even dynamically in the build script). But as Sean said, you will emit the classes to a class dir and then it's just important that class dir is included in the classpath that the Clojure code uses

Chris Lester17:05:30

Thx, I started with the mixed clj/java build example, passing the java src paths in (pulled from aliases in the nested deps). I’ve been using the existing basis created from the polylith project (in the sample poly build.clj). Will look at the basis classpath and see what is getting calculated. The deps are apache, javax, and jnr dependencies (so published in maven public repos). The clj code using some of those imports them ok.

tcrawley14:05:38

Howdy! I'm trying to track down an issue with tools.build and how it determines the version of Clojure to include in an uberjar. I have the following structure (this is from a reproducer that is a simplification of our production build): • projectA does not have a direct dependency on clojure, but does have an :overrides alias that has a :default-deps that specifies 1.11.1 • projectA depends on projectB (as {:deps/manifest :deps} • projectB has a dependency on clojure of org.clojure/clojure nil • projectA has a build.clj that builds an uberjar, using a basis that includes :aliases [:overrides] • projectA has a build.clj that is basically the example from https://clojure.org/guides/tools_build#_compiled_uberjar_application_build with the basis modified to include aliases When in projectA, clojure -Stree -A:overrides shows clojure 1.11.1, but building an uberjar with clojure -T:build uber puts clojure 1.10.3 in the jar instead of 1.11.1. 1.10.3 is the version that the tools.build version I am using depends on, so I suspect that is where it comes from. If I add org.clojure/clojure nil as a direct dependency of projectA, the uberjar has 1.11.1 in it. Is this a known issue?

seancorfield15:05:36

I would use :override-deps for that. I don't think :default-deps is intended to affect transitive dependencies?

seancorfield15:05:18

(caveat: I've never really understood what :default-deps is for -- it's never done what I expect, based on the docs)

tcrawley16:05:32

The above is a simplification of our repo structure, but we use :default-deps to be able to specify versions in one place, and our modules have nil instead of a version map. That works well in most cases. But I also tried :override-deps in projectA, having it specify Clojure 1.11.1, and still got 1.10.3 in the uberjar.

seancorfield17:05:47

@tcrawley What version of tools.build are you using?

tcrawley20:05:29

@seancorfield My test project is using the latest tool.build (`{:git/tag "v0.8.1" :git/sha "7d40500"}`) with clojure CLI 1.11.1.1113. I believe though that the issue is really with how clojure finds the version of clojure to use. It could also be that my mental model on how tools.deps works is flawed. For example, if I have these two deps.edns:

{:deps {other/lib {:local/root "other-lib"
                   :deps/manifest :deps}}
:aliases
 {:overrides {:default-deps {org.clojure/clojure {:mvn/version "1.11.0"}}}}}

;; other-lib's deps.edn

{:deps {org.clojure/clojure nil}}
I get 1.11.1:
$ clojure -A:overrides -Stree | grep clojure/clojure; clojure -A:overrides
org.clojure/clojure 1.11.1
Clojure 1.11.1
user=> 
where I would expect to get 1.11.0, since 1.11.1 doesn't exist in my dependency tree - it comes from the tool itself. If I change the :deps in the first deps.edn to include org.clojure/clojure nil, I get 1.11.0:
$ clojure -A:overrides -Stree | grep clojure/clojure; clojure -A:overrides
org.clojure/clojure 1.11.0
Clojure 1.11.0
user=>
which implies that 1.11.1 isn't part of th dependency graph in consideration. However, if we change the first deps.edn to rely on tools.build, and have a build.clj :
{:deps {other/lib {:local/root "other-lib"
                   :deps/manifest :deps}}
:aliases
 {:overrides {:default-deps {org.clojure/clojure {:mvn/version "1.11.0"}}}}

 :paths ["src"]

 :build
 {:deps {io.github.clojure/tools.build {:git/tag "v0.8.1" :git/sha "7d40500"}}
  :ns-default build}}

;; build.clj

(ns build
  (:require [clojure.tools.build.api :as b]))

(def lib 'my/lib1)
(def version "1.2.3")
(def class-dir "target/classes")
(def basis (b/create-basis {:project "deps.edn" :aliases [:overrides]}))
(def uber-file (format "target/%s-%s-standalone.jar" (name lib) version))

(defn clean [_]
  (b/delete {:path "target"}))

(defn uber [_]
  (clean nil)
  (b/copy-dir {:src-dirs ["src" "resources"]
               :target-dir class-dir})
  (b/compile-clj {:basis basis
                  :src-dirs ["src"]
                  :class-dir class-dir})
  (b/uber {:class-dir class-dir
           :uber-file uber-file
           :basis basis
           :main 'my.lib.main}))
And build an uberjar, I get 1.11.1 in the tree, but 1.10.3 in the jar:
$ clojure -A:overrides -Stree | grep clojure/clojure; clojure -T:build uber; java -cp target/lib1-1.2.3-standalone.jar clojure.main
org.clojure/clojure 1.11.1
Clojure 1.10.3
user=> 
Which is surprising. 1.10.3 seems to come in from the tools.build dependency. Finally, if I add org.clojure/clojure nil as a dependency to the primary deps.edn, I get what I expect in the jar:
$ clojure -A:overrides -Stree | grep clojure/clojure; clojure -T:build uber; java -cp target/lib1-1.2.3-standalone.jar clojure.main
org.clojure/clojure 1.11.0
Clojure 1.11.0
user=> 
I think the clojure version resolution is surprising, and possibly a bug. But I think tools.build not honoring the resolved version in the basis is a bug. Again, unless I don't understand how the tooling is supposed to work.

hiredman20:05:04

do you have a user level deps.edn? ~/.clojure/deps.edn or whatever, I think -Sdescribe will tell you what the path to it is

tcrawley20:05:16

Good call out - I do not have one.

tcrawley20:05:13

We are working around this now by adding org.clojure/clojure nil as a dependency to each of our "service modules" (they don't contain any Clojure code, but assemble other :local/root modules into a service package). So we're fine, but I wanted to point this out in case others run in to it/the tools.build/`tools.deps` maintainers consider it a bug.

tcrawley20:05:03

@seancorfield I saw a notification from Slack asking about :override-deps, but don't see that message here (you may have deleted it, and that's ok!). With :override-deps instead of :default-deps, I get 1.11.0 in the above examples.

tcrawley20:05:43

However, our actual production case is more complicated than the above - we are still using depstar, where we get very similar results, but :override-deps doesn't work, but my workaround does. I don't want to dig in to that too much though, since depstar is deprecated and we plan to move to build-clj

seancorfield20:05:44

That's what I would expect.

seancorfield20:05:41

(that :override-deps would work with build.clj but I'd have to see exactly how you're using depstar -- and which version -- to comment further on things not working with depstar)

seancorfield20:05:51

It sounds like, if you move to build.clj and use :override-deps, it will all work. If you want me to help you fix your current setup with depstar, feel free to DM me with details of how you're invoking it.

seancorfield20:05:19

(but I would not expect :default-deps to do what you need in this situation)

favila20:05:17

Why not? :default-deps does indeed seem to do what’s expected for other transitive but declared deps with a nil version?

favila20:05:25

For everything other than clojure that is.

seancorfield20:05:15

Because org.clojure/clojure has a default version specified by the CLI itself.

favila20:05:43

ok, but 1) we’re not getting that version 2) why would that matter for basis calculation?

seancorfield20:05:12

Because the basis is root (system) + user + project unless you explicit exclude any of them. [edited to cross out the user deps.edn reference because tools.build's create-basis excludes that by default]

seancorfield21:05:41

@U09R86PA4 see my longer thread with @tcrawley for the full explanation of what is going on (and why you need :override-deps and not :default-deps)

tcrawley20:05:07

Thanks! I think we're moving to tools.build relatively soon, and those service modules are going to likely get some clojure code, so should have a Clojure dependency. So I'm not worried about getting things sorted out further with our depstar build.

tcrawley20:05:40

The plot thickens! :default-deps doesn't matter at all here. If my :local/root depends on org.clojure/clojure 1.10.1, I stll get 1.11.1 in my tree, and at the repl. And I still get 1.10.3 in the uberjar.

seancorfield20:05:20

What version of the CLI? I think you said 1.11.1.1113 -- so that has a system default Clojure version of 1.11.1

tcrawley21:05:57

Right, but I would expect at least 1.11.1 in my jar then, not 1.10.3

seancorfield21:05:50

I repro'd and printed out the basis in build.clj and it shows 1.10.3 which I suspect is coming in via the :build alias as the default Clojure version something in there was built from... checking...

seancorfield21:05:31

Hmm, nope, that's not it.

seancorfield21:05:10

There's a default of 1.10.3 coming from somewhere in tools.build or tools.deps.alpha it seems... even with :override-deps in place, printed the basis still reports 1.10.3 in the :deps for project A, although it builds the uberjar correctly with :override-deps

seancorfield21:05:18

So, when you run -Stree or -X:deps tree you're getting whatever the CLI has as a default:

(! 568)-> /usr/local/Cellar/clojure\@1.10.3.1087/1.10.3.1087/bin/clojure -X:deps list :aliases '[:overrides]'
org.clojure/clojure 1.10.3  (EPL-1.0)
org.clojure/core.specs.alpha 0.2.56  (EPL-1.0)
org.clojure/spec.alpha 0.2.194  (EPL-1.0)
other/lib /Users/sean/clojure/fresh/toby/b 

Mon May 02 14:37:13
(sean)-(jobs:0)-(~/clojure/fresh/toby/a)
(! 569)-> clojure -X:deps list :aliases '[:overrides]'
org.clojure/clojure 1.11.1  (EPL-1.0)
org.clojure/core.specs.alpha 0.2.62  (EPL-1.0)
org.clojure/spec.alpha 0.3.218  (EPL-1.0)
other/lib /Users/sean/clojure/fresh/toby/b 
That's with :overrides using :default-deps

seancorfield21:05:48

Because your project A has no specific version of Clojure specified, it will pick up whatever the tooling environment provides, and because of :replace-deps for :build you'll get what tools.build itself specifies -- but you can't see that with -Stree or -X:deps tree because those code paths pick up the CLI's default (and your CLI default is 1.11.1).

seancorfield21:05:58

:default-deps provides a default to use if nothing else is specifying a version but for Clojure itself the tooling is going to provide a version (and you'd see this same issue for any libraries that tools.build itself depends on if you tried to use :default-deps on those for your own code and a nil version).

favila21:05:23

I guess it’s weird to me that basis computation, which is only ever going to run your own code, would specify a clojure version. (I understand why the clojure cli commands would specify a clojure.) This means you can’t ever not have a clojure dependency in a tools build basis?

seancorfield21:05:21

To run build.clj, you have to have some version of Clojure in place so there has to be a default for it somewhere.

favila21:05:46

yeah, to run build.clj, but not to compute the classpath of what it builds, right?

favila21:05:27

if I have a deps.edn {:deps {}} and compute a classpath with -Srepro shouldn’t I get an empty classpath? It sounds like I will at least get clojure inherited from tools.build (or somewhere).

seancorfield21:05:32

But your basis calculation is more than just the project deps.edn: it includes root and user. Add :root nil :user nil to your basis calculation to exclude those.

seancorfield21:05:57

Then you'll get your :default-deps for Clojure.

seancorfield22:05:09

(! 575)-> cat deps.edn 
{:deps {other/lib {:local/root "../b"
                   :deps/manifest :deps}}
 :aliases
 {:overrides {:default-deps {org.clojure/clojure {:mvn/version "1.11.0"}}}

 :paths ["src"]

 :build
 {:deps {io.github.clojure/tools.build {:git/tag "v0.8.1" :git/sha "7d40500"}}
  :ns-default build}}}

(! 576)-> cat build.clj 
(ns build
  (:require [clojure.tools.build.api :as b]))

(def lib 'my/lib1)
(def version "1.2.3")
(def class-dir "target/classes")
(def basis (b/create-basis {:root nil :user nil :project "deps.edn" :aliases [:overrides]}))
((requiring-resolve 'clojure.pprint/pprint) basis)
(def uber-file (format "target/%s-%s-standalone.jar" (name lib) version))

(defn clean [_]
  (b/delete {:path "target"}))

(defn uber [_]
  (clean nil)
  (b/copy-dir {:src-dirs ["src" "resources"]
               :target-dir class-dir})
  (b/compile-clj {:basis basis
                  :src-dirs ["src"]
                  :class-dir class-dir})
  (b/uber {:class-dir class-dir
           :uber-file uber-file
           :basis basis
           :main 'my.lib.main}))

(! 577)-> clojure -T:build uber

(! 578)-> java -cp target/lib1-1.2.3-standalone.jar clojure.main
Clojure 1.11.0
user=> 

favila22:05:29

ok, so I guess -Srepro doesn’t exclude the root alias, and that’s why depstar was doing this.

seancorfield22:05:10

-Srepro only excludes the user deps.edn

seancorfield22:05:04

It's definitely an edge case...

favila22:05:08

alright, well, I guess I’ll just keep that in mind if we switch to build.clj one day.

favila22:05:39

It makes me wonder how many build tools are being careful about :root nil

seancorfield22:05:10

Most folks specify an explicit version of Clojure in deps.edn.

seancorfield22:05:43

And, like I say, using :override-deps would solve this.

favila22:05:51

fair. If this weren’t a wrapper application project we would have done the same. But we literally had no src directory

favila22:05:48

Semantically we’re trying to keep override-deps away from dependencies we declare explicitly (even transitively via other modules). We use it just to alter dependencies that we don’t use directly.

favila22:05:55

to address CVEs etc

seancorfield22:05:19

Just checked the tools.build docs and :user nil is not needed, as that's the default (basically -Srepro by default).

👍 1
seancorfield22:05:53

My tools.build wrapper, build-clj, assumes it is safe to include the root deps.edn so you would trip over this with my wrapper, unless you explicitly created your own basis with :root nil and passed that into my wrapper functions. Just a heads-up in case you switch to tools.build and think about using my wrapper to reduce boilerplate in your build.clj file!