Fork me on GitHub
#tools-build
<
2021-10-31
>
weavejester12:10:35

I’ve been looking into replacing my Leiningen build chain with tools.deps and tools.build, but I notice that write-pom doesn’t currently support description or license elements. Would there be interest in me putting together a patch?

deactivateduser18:11:13

That’s a use case this library was developed for: https://github.com/pmonks/tools-pom

borkdude12:10:28

Usually the flow is http://ask.clojure.org -> JIRA -> patch

weavejester12:10:51

I was wondering about http://ask.clojure.org, as I noticed it on previous patches, but it seemed more a place for community answers to questions, rather than requests for features.

weavejester12:10:44

I can certainly post on there if that’s the common flow, though.

borkdude12:10:46

Let's wait for Alex's answer, I was just writing how I usually see things being handled here.

Alex Miller (Clojure team)13:10:16

We don't support lots of elements as we can sync from a source pom

Alex Miller (Clojure team)13:10:44

So write the pom template with whatever you need and sync that

Alex Miller (Clojure team)13:10:05

And ask Clojure is a good place to ask for features too (use the request tag)

weavejester13:10:31

That’s fair. In which case I might fork then. I’d prefer to keep the pom out of source control.

2
♾️ 1
Alex Miller (Clojure team)14:10:26

what if you made a pre-task that made the static source pom?

lread14:10:34

What's your thinking @weavejester, if you don't mind me poking your big brain? For projects where I have adopted tools-build my pom.xml has become less information-rich but still present. Some of the data it held now lives in build.clj. So I have build info in declarative deps.edn pom.xml and programmatic build.clj. Are you shooting for a declarative source of build truth or something different?

dcj18:10:57

I want/feel that pom.xml is a generated "artifact" and I'd prefer not to have a template at all. I'd prefer if everything needed for most pom.xml files be generated from other sources of truth. deps.edn is one potential source of truth, and earlier this year I worked with @U04V70XH6 (`depstar`) and created a PR for deps-deploy to support centralizing and abstracting various aspects, enabling a deps.edn like this (for example):

:mvn/repos {"acn-snapshots" {:url ""}
             "acn-releases"  {:url ""}}

 :aliases
 {:mvn/artifact-id database
  :mvn/group-id    org.aircraft-noise
  :mvn/version     "0.3.4-SNAPSHOT"
  :jar/file-name   "database.jar"

  :jar {:replace-deps
        {com.github.seancorfield/depstar {:mvn/version "2.0.216"}}
        :exec-fn hf.depstar/jar
        :exec-args {:jar         :jar/file-name
                    :artifact-id :mvn/artifact-id
                    :group-id    :mvn/group-id
                    :version     :mvn/version
                    :sync-pom    true}}

  :deploy {:extra-deps {com.dcj/deps-deploy
                        {:git/url ""
                         :sha "cf0f5e612eab04789c757a98f544eb8dea61a45b"}}
           :exec-fn deps-deploy.deps-deploy/deploy
           :exec-args {:installer :remote
                       :sign-releases? false
                       :artifact :jar/file-name
                       :repository "acn-snapshots"}}
  }
(with the credentials for my private Maven repo getting pulled out of ~/.m2/settings.xml) This enabled me to provide most of what I needed for pom.xml in a DRY manner, with no need for a template/basic pom.xml, but obviously not every desirable pom directive was supported (e.g. VCS, license, etc) N.B that this thinking/work PRECEEDED tools.build... I can appreciate that most of this info is not about dependencies. so putting it into deps.edn is not ideal. Revisiting this in the light of tools.build is on my list of things to do (eventually) The idea (later in this thread) of project.edn has crossed my mind, and could have merit. The above got me to about 80% of what I wanted a completely generated pom.xml to contain. Seems like with latest versions of tools we could support even more POM stanzas.....

☝️ 1
rickmoynihan09:11:03

Yeah - I made a similar PR last year: https://github.com/slipset/deps-deploy/pull/21 but it predates tools.build, so would want to be rethought now

seancorfield18:11:25

Given that the source template for the pom.xml file doesn't need to be called pom.xml, and can be very minimal -- just containing the specific non-default bits of data you want -- I think it's reasonable to have that under version control, just like you have any other "config/control" files under version control (`deps.edn`, your build.clj file etc). It seems weird to me to talk about "forking tools.build" because of that, unless I'm misunderstanding your comment @weavejester?

deactivateduser18:11:56

@U04V70XH6 the problem with that is that those minimal / incomplete pom.xml files confuse other Maven-based tooling.

deactivateduser18:11:24

(including GitHub itself, for example, when it tries to do its dependabot security scanning)

deactivateduser18:11:01

@U07TTE6RH your framing absolutely resonates with me, and I threw this together to scratch my own itch on this problem: https://github.com/pmonks/tools-pom I’d greatly appreciate your thoughts (especially on the general approach - the code itself is a bit of a rush job though…).

seancorfield18:11:41

@U0MDMDYR3 That's why I made a point of saying they do not need to be called pom.xml!

deactivateduser18:11:04

Sure, but defaults have inertia.

p-himik18:11:19

That is exactly a concern someone has raised and has been discussing on Discljord Discord server. IIUC, a library author might not care about the default and be completely fine with a template pom.xml lingering at the rood. But a library consumer will have potential problems if their tooling decides to look at that file.

seancorfield18:11:36

No need to throw the baby out with the bathwater. build.clj can just specify :src-pom and you can call the file pom_template.xml or whatever you want. Heck, you could even have build.clj copy the generated file back to the root of the project instead so that the complete, correct pom.xml is at the root, can be checked into git, and can be relied on by other tooling. The choice is entirely up to you.

seancorfield18:11:54

Complaining about defaults and inertia when you have full control over this is silly.

p-himik18:11:21

You as a library author have full control over where pom.xml goes - yes. You as a library consumer have full control iff you fork the library. If the author of the library has used the default pom.xml at the root of the project as a template, then all your tooling that knows about pom.xml files will be screwed. Or am I misunderstanding something?

seancorfield19:11:58

If the project has a pom.xml and it is out of date and negatively affects tooling, that's a bug. Create an issue and get the library owner to fix it.

seancorfield19:11:14

That's what you'd do if tools.build wasn't even in the mix, right?

seancorfield19:11:52

Consumers of the library are generally going to be depending on the pom.xml inside the JAR tho', right?

seancorfield19:11:43

The pom.xml that is uploaded to Clojars, for example, should be the complete, generated one. That's what should be passed to deps-deploy for example.

p-himik19:11:48

Generally - yes. But not in all cases. It's about applying my tools to someone else's code. Not a JAR. Not a Git dep. The code.

seancorfield19:11:17

Then raise an issue and get the library author to fix it. It's a bug, plain and simple.

seancorfield19:11:29

Either don't have pom.xml or have a correct one.

p-himik19:11:37

Indeed! That's exactly the point - a template pom.xml a priori is not correct by itself. But the default template path is exactly pom.xml. So it's a bug affordance.

seancorfield19:11:23

Then complain on http://ask.clojure.org and see if you can persuade the core team to change that?

seancorfield19:11:58

I think the burden is on the library author, personally, but you don't agree.

p-himik19:11:34

Then the tool imposes the burden on library authors. :) My main intention here was to figure out if I understand the situation correctly, and seems like it's the case now. Someone that actually uses template POM files and happens to have an alternative approach in mind would be in a much better position to create an question.

seancorfield19:11:32

Oh, the irony! So I decided to follow that advice and rename pom.xml to pom_template.xml and now the GitHub Actions setup_java fails because it requires that your repo includes a pom.xml 😞

seancorfield19:11:26

Updated that section to suggest template/pom.xml -- which satisfies GitHub Actions' setup_java and still avoids having an (incomplete and/or outdated) pom.xml in the project root.

deactivateduser19:11:17

@U04V70XH6 that may be something else. I have a project (several in fact) that use the setup-java GitHub Action, but do not include a pom.xml anywhere in the checked in code, and that seems to be working fine. Here’s an amusingly meta example of that: • Project: https://github.com/pmonks/tools-pom • Use of setup-java: https://github.com/pmonks/tools-pom/blob/main/.github/workflows/deploy.yml#L18 • Confirmation that this repo does not include a pom.xml anywhere: https://github.com/pmonks/tools-pom/search?q=pom.xml (observe that none of the search results are for a file called pom.xml).

seancorfield20:11:11

I was surprised by it. Are you using v1 or v2? I see you're using v2, same as me.

seancorfield20:11:37

Oh, you're not enabling the maven cache so that's probably it.

deactivateduser21:11:28

Ah yes well spotted. I’d been wondering about enabling that on the off chance it might help with tools.build builds, but this finding suggests that that might not work. 😉

seancorfield21:11:26

Well, it does help since it caches the ~/.m2/repo stuff between builds so you don't have to "download the world" for each build, so your tests/builds run faster. But you need a pom.xml somewhere -- in my repos, it's template/pom.xml which satisfies the GH setup and provides cache from run-to-run -- and still lets me have a :src-pom for write-pom without needing a misleading pom.xml in the project root.

seancorfield21:11:54

Now, I suspect the cache is only recalculated if that template/pom.xml changes so if I update deps.edn, it will have to download any new deps but that's still better than no cache (since the cache will also include all the tools.deps stuff as well which doesn't change very often).

deactivateduser23:11:42

Yeah I was leaning more towards using the https://github.com/marketplace/actions/cache directly, since that lets one specify file paths (which could include both ~/.m2/repository and ~/.gitlibs/libs ), and doesn’t require a pom.xml or whatever (it’s also what the Maven caching mechanism uses under the hood). Obviously what would be even better would be a comprehensive tools.build based GitHub Action that just rolled all this stuff up, but I’m currently 2 side projects away from my primary development priority, so probably can’t justify biting that off just yet. 😉

1
seancorfield00:11:21

Yeah, that cache action is more like what I am using in BitBucket Pipelines where I cache those two and also ~/.clojure and ~/.cpcache which also avoids classpath recomputation in the CLI etc.

👌 1
borkdude14:10:05

Having a pom.xml in source control also gives GitHub some hints about your project, although probably not very important

weavejester14:10:36

If the pom contains both generated elements and non-generated elements, effective source control becomes a little tricky. Ideally I want to separate the two, so I can keep the generated parts of my project out of source control, and the parts that aren’t generated in source control.

weavejester14:10:12

I could have a task that generates a pom, and then use write-pom to update it, but I think it would be easier to grab gen-pom from the tools.build source and adapt it.

lread14:10:29

Thanks. I don't have that situation (yet) where I need to commit the pom.xml enriched by build.clj.

weavejester14:10:50

I’m mostly doing it because I don’t want to feel like I’m going backward moving from Leiningen to tools.build. I have the possibly unrealistic goal of being able to do everything in my old workflow in my new one.

weavejester14:10:45

I’m experimenting with a combination of tools.deps, tools.build and babashka task runner.

lread14:10:21

I'm personally finding that combo works nicely. Babashka allows me to succinctly describe command-line friendly build tasks. And the cross-platform scripting language is pretty sweet too!

weavejester14:10:33

Babashka’s task runner is certainly a hidden gem.

lread14:10:04

Are you experimenting with this for your open-source projects, or is this a closed-source/work-thing endeavor?

lread14:10:59

If it is open-source, I'm looking forward to seeing what you come up with. As you know, I do admire your knack for combining clarity with succinctness!

borkdude14:10:34

We have some examples of tools.build + bb tasks in eg clj-easy/graal-build-time. I also have one experimental branch in that project where I launch a JVM prepl once lazily and then send build tasks as needed :-)

weavejester14:10:00

@lee open source projects. Every so often I’ll check out the tools.deps toolchain and see how viable it is to switch. I like the design of tools.deps, but Lein is very convenient for having “batteries included”.

Alex Miller (Clojure team)14:10:52

if you have an idea for extending the pom stuff in a generic way, I'm interested in that for tools.build. that is, I don't want to add 100 specific keys and code to handle them, but if you worked on a generic "turn clojure data into pom xml", would be happy to integrate that.

lread14:10:10

project.edn? simple_smile

weavejester14:10:19

It did cross my mind…

lread14:10:26

I was slightly surprised, without understanding all the hard work that went into making the decision, that while tools deps source of truth is declarative, tools build source of truth is programmatic.

weavejester14:10:54

I understand the reasoning behind it. I think a project.edn would only make sense if you could pick and choose the “build library” that goes along with it.

lread14:10:57

And I guess there's really nothing stopping anybody from rolling their own build truth stored in a project.edn while using tools build.

weavejester14:10:22

Sure, @alexmiller. I think I’ll aim for something a little more specific at first, simply because it seems easier, and I don’t have to worry about updates. I understand why you don’t want to include additional keys without a generic solution.

weavejester14:10:56

That said, the description element is exposed as part of clojars interface, so I’d advocate that particular key at least, even if none of the others are included.

weavejester14:10:25

But I also understand the need to say “no” on a project — I’ve said it often enough myself 🙂

lread14:10:56

(yes, yes, you have! simple_smile)

borkdude14:10:58

What if you can have a hook in write pom to operate on the s-expr before it’s turned into XML for instance. There’s lots of other considerations and places where one could intercept. It feels like it’s almost worth a spreadsheet ;-)

Alex Miller (Clojure team)16:10:19

Also seems like a viable option

lread14:10:22

That sounds pretty flexible. Users could then simply add in what they like.

weavejester14:10:05

On another note, could build.clj potentially be part of a dependency? I assume so, but I’m not completely certain on how clj -T: works.

lread15:10:27

If I understand correctly, build.clj is merely a convention. The https://clojure.org/guides/tools_build.

borkdude15:10:52

@weavejester yes, there is nothing specifically built into deps regarding tools.build, you can just use it as a dependency

Alex Miller (Clojure team)15:10:12

for sure, the expectation is that you can pull in other libs as part of your build

lread15:10:08

FWIW, @weavejester the things I clearly prefer in tools deps/build over leiningen are: • less mysterious (for me) to extend • deps resolution makes more sense to me

weavejester16:10:53

Yeah, for sure. I’d certainly like to use it — I’ve just been waiting for it to mature enough that I can use it to deploy to Clojars without losing any of the functionality Leiningen gives me out of the box. I think the ecosystem is about 95% there now.

☝️ 1
tugh21:10:08

Did anyone have success building an application(both clj and cljc sources) as uberjar? I'm using the same config and build.clj from https://clojure.org/guides/tools_build with small edits to comply with my project. This is a representation of my projects directory structure:

src
├── clj
├── cljs
└── cljc
    └── my-project/specs.cljc (contains 1 reader conditional in ns form)
The reader conditional is:
(:require [#?(:clj  clojure.spec.alpha
              :cljs cljs.spec.alpha) :as spec])
--- The clj -T:build uber command stucks at compile-clj step without any warnings or errors but it creates an uberjar if I port specs.cljc to specs.clj under src/clj/ directory. Everything works great when I'm developing with a REPL with the same deps.edn file. What could be wrong with this?

borkdude21:10:57

@tugh my-project/specs.cljc should probably be my_project/specs.cljc but not sure if this has anything to do with your error

tugh21:10:03

It's actually a single word. there is no dash or underscore in the project/folder names.

borkdude21:10:36

ok that's definitely not it then. perhaps you can make a public repository with a repro

tugh10:11:09

Thanks. Will check it out...

dpsutton21:10:44

Been a bit but don’t you not even need a reader conditional for that? Can’t you just use clojure and the cljs analyzer or some step does the write thing?

dpsutton22:10:48

% clojure -A:cljs -M -m cljs.main -re node -r
ClojureScript 1.10.773
cljs.user=> (require '[clojure.spec.alpha :as spec])
nil
cljs.user=> (dir spec)
&
*
...

dpsutton22:10:29

I doubt this will solve your issue but you call it out as if you suspect it might be related and this can eliminate it

👍 1
tugh10:11:49

Thanks but I'm using shadow-cljs for cljs builds. This only applies if I'm using tools.build for the cljs builds too, right?

dpsutton13:11:30

No this is just the clojurescript compiler. Which shadow uses as well. I’m sure it will work there as well

👍 1