Fork me on GitHub
#tools-deps
<
2021-07-20
>
idas03:07:57

my apologies, i asked this question a while back in this channel, but i've forgotten the response many kind folks gave me earlier. i'd like to a make mixed JVM language project using both clojure and java, where clojure code is calling java code. what is the best way to structure this kind of a project using clj-tools/tools-deps? is there an existing example project to look at?

Alex Miller (Clojure team)03:07:09

you can use the brand new tools.build to help you with such a thing

πŸŽ‰ 6
Alex Miller (Clojure team)03:07:01

I don't have a good example for you (yet) but that would be a great one to add to the guide at https://clojure.org/guides/tools_build

Alex Miller (Clojure team)03:07:48

but really it's making a build script that uses tools.build javac task to compile java into a classes dir, which is on the :paths

idas04:07:36

yeah, this kind of project scenario definitely looks like a great use case for tools.build. a while back when i asked this question i was referred to github projects that used deps.edn -X aliases

didibus04:07:05

I'm not sure how much Java you intend to have, if its majority Clojure, tools.build should work great with its javac task, and you can also leverage java-command and process for slightly more complicated tasks. If you intend to have a good chunk of Java like 50% java, or you have an existing Java code base and want to start adding some Clojure to it, you might be better served by a Java build tool like Gradle that uses a Clojure extension such as Clojurephant or nebula-clojure-plugin. These will make it easier to use Eclipse or IntelliJ for the Java development, run Java code coverage, unit tests, checkstyle, Lombok, java doc, and all that, if that's needed for your use case.

idas14:07:04

mostly clojure. just a few java classes for some more performant sections. mainly because my clojure understanding isn't that great yet

didibus18:07:09

Ya, then tools.build will be good enough, and better for the Clojure

seancorfield04:07:38

Just remember that we're a Clojure-focused community here wanting to use some Java with Clojure πŸ™‚

πŸ’― 2
πŸ‘ 2
seancorfield05:07:58

At work, we've already adopted tools.build and we're leveraging java-command and process to run various things that are outside the scope of the built-in tasks, such as database migrations and other environment setup. We're also running our tests via those two functions -- so we can run clojure -T:build run-tests and that creates a basis for running our test suite and then creates a java-command to invoke Cognitect's test-runner with the appropriate arguments, and then process to actually run it. So far it's all working pretty nicely (but see TBUILD-6 for -X style invocation which was something we needed to implement ourselves, so far).

FiVo09:07:24

@idas I made a simple example project testing exactly that the other day: https://github.com/FiV0/tools.build-testing

idas14:07:19

Nice! Thanks for sharing this!

Ed11:07:12

Hi. If I was to be trying to generate an uberjar with tools.build, is there a way to generate a manifest file so we can java -jar ... to start it?

Ed11:07:12

nevermind ... I'm just being dumb ... found it πŸ˜‰

dominicm15:07:05

That's giving me a certificate error 😧

dominicm15:07:07

> Web sites prove their identity via certificates. Firefox does not trust this site because it uses a certificate that is not valid for http://blog.fogus.me. The certificate is only valid for the following names: http://s444.sureserver.com, http://www.s444.sureserver.com

fogus (Clojure Team)15:07:10

adding an s to the protocol should fix it.

dominicm15:07:00

I am using https πŸ™‚ That's what gave me the error.

dominicm15:07:20

Who uses http:// these days? πŸ˜„

fogus (Clojure Team)15:07:55

Weird. Looks like I have to check my cert.

fogus (Clojure Team)15:07:52

OK, should be fixed now..... 🀷

seancorfield15:07:10

@fogus You say "One interesting aspect of the jar target is that it uses a source pom as the base for the new pom." -- it might make it clearer to new tools.build users if you additionally clarify that "the new pom" is written to the :class-dir folder. Folks who've used clojure -Spom will be used to the CLI updating the pom.xml file in place.

seancorfield15:07:38

(and :class-dir is not as intuitive as :target-dir, I suspect, for folks used to Leiningen terminology since Leiningen writes pom.xml directly into target as I recall?)

seancorfield15:07:14

I have to keep reminding myself that :class-dir is "inside the target folder"! πŸ™‚

seancorfield15:07:43

Hopefully, I'll get around to writing up our tools.build usage at work soon. Need to figure out how one piece is going to be tackled, which I've discussed with @alexmiller (TBUILD-6). I'm kinda holding off until I have a better sense of whether that's going to get addressed sooner, or later πŸ™‚

fogus (Clojure Team)15:07:45

I'm not keen on class-dir or target-dir --- I wonder if there's an even better name for that place?

seancorfield15:07:08

Naming is hard πŸ˜•

seancorfield15:07:34

I mean, in tools.build it is more a "classes" folder than just a "target" folder. In depstar, I used :target-dir but that's for the top-level folder -- it always puts classes in that folder if you compile .clj files, but it puts the JAR and the pom.xml directly into that top-level folder, like Leiningen does (and it doesn't copy intermediate files there -- to improvement performance: they are copied directly from their original locations into the JAR file -- but I can see the additional flexibility that copying files brings).

dominicm15:07:52

cljs already uses out-dir

dominicm15:07:56

Might be confusing

borkdude15:07:34

or it might be consistent with other clojure tooling ;)

dominicm15:07:17

Except you'd likely have the cljs out-dir pointing inside the clj out-dir

fogus (Clojure Team)15:07:10

directories are hard too!

didibus19:07:07

Interesting, so tools.build intends to limit itself only to packaging tasks? And it would still be expected that other development and release tasks would use other tools? And be orchestrated through aliases again? Such as running tests, linting, doc generation, formatting, starting repl, etc. ?

seancorfield19:07:47

@U0K064KQV I'm not sure where you get that impression from the above thread?

Alex Miller (Clojure team)19:07:33

Many of those already have good answers available now, so there is no need to recreate them in tools.build

seancorfield19:07:37

At work, we're using tools.build to run database migrations, run tests, build JARs, push deployments...

Alex Miller (Clojure team)19:07:56

But if you want a build that does other stuff, it’s your program, so go for it!

didibus19:07:09

Well it talks about an amalgamation of orthogonal tools, and explains that tests for example are not meant to be a part of tools.build, so some other tool must be used and configured as an alias

seancorfield19:07:29

Your build script can invoke a test runner.

seancorfield19:07:26

Our CI at work is basically just clojure -T:build run-all-tests and clojure -T:build build-and-deploy-jars.

seancorfield19:07:16

(and run-all-tests is an exec function that runs our database setup, migrations, search engine setup, content publishing, and then our complete test suite πŸ™‚ )

didibus19:07:04

I know but that's not how Fogus set it up in his article. That's why I was curious. My thoughts was that you'd orchestrate all tasks with tools.build, and just that maybe it doesn't "yet" have a built-in task for everything, but could eventually or those could be added by other libs which follow the same pattern and can be used inside build.clj So that later the CLI would have a way to chain tasks and compose them, and I was expecting the current arg on the build steps would probably eventually be a way to pass context around between steps. But in the article I understood that testing maybe isn't meant to be a part of that, and so other aliases are expected to be used for those. Maybe I understood wrong, but that's two very different philosophies though.

seancorfield19:07:07

"builds are programs" πŸ™‚

seancorfield19:07:23

Given that your tests generally need to be run in the context of your test runner but not in the context of your build script (something that has caused us a bit of grief until we took the subprocess approach), you could just use

clojure -X:test && clojure -T:build build-and-deploy
and that's nice and simple and composable.

seancorfield19:07:15

(our monorepo setup is sufficiently complex that clojure -X:test is not enough to run our tests -- because the test runner needs to be told about all the separate subprojects that contain tests -- which is why we're using exec functions in our build.clj script to figure that out and create the right context for executing the test runner!)

seancorfield19:07:30

I will be blogging about this in due course...

didibus20:07:43

I was mostly interested in understanding the vision as things evolve. At first I thought that the vision is that all tasks would be managed by tools.build. And so in the future, you wouldn't create a test runner that is a CLI, or a lein-plugin, etc. It would just be a Clojure lib which you'd call inside build.clj Thus all tooling would just become Clojure libs, and build orchestration would become a Clojure program in the form of build.clj And the CLI would play nicely with build.clj, such as being able to call each of the "tasks" defined in it individually, or in the future compose an order of them together. As for the "context" each build tool must run under, like what classpath, what root folder, what resources, what user, JVM args, properties, etc. Well I thought that was maybe still an open question. But that either you'd fork a new process, or possibly you could have sub-builds running with a different context. Something like maybe I can have a test-build.clj where tasks in there run under the test alias.

didibus20:07:40

But now having read the article, I see maybe the vision is different, and actually build.clj is really meant for things that can happen under the build context, like packaging a jar. But everything else is still meant to be used as aliases, so other build tools would be -X or -M invokable, and shell would be used to compose and orchestrate them.

Alex Miller (Clojure team)20:07:46

I would go more with the latter vision :)

Alex Miller (Clojure team)20:07:08

tools.build is a TOOL to BUILD artifacts (it's right there in the name!) :)

Alex Miller (Clojure team)20:07:37

that said, your build program (build.clj or whatever) is your program - it can use stuff from tools.build to make artifacts, or do other things using functionality from other libraries

Alex Miller (Clojure team)20:07:12

and then deps.edn aliases are a way to set up other tools a level above that - at that level you're choosing a classpath, and building artifacts and running tests and generating docs may all need different classpaths

Alex Miller (Clojure team)20:07:20

we're not actively running at trying to provide functionality to run multiple programs - you have many OS-specific tools designed to do that

seancorfield20:07:12

For "most" projects -- where a simple clojure -X:test can run all the project's tests -- I would expect folks to use tools.build for "just" the build-related stuff (packaging, versioning, etc) since they already have a solid command for running tests and, with something like deps-deploy, they have a solid command for installing to the local ~/.m2 or up to Clojars. For more complicated projects, tools.build provides a bunch of useful stuff that helps orchestrate other tasks -- the create-basis/`java-command`/`process` stuff -- and in those cases it is a nice "leg up" to writing more complex "build" scripts. So I'm pleased with what it provides.

Alex Miller (Clojure team)20:07:19

you can use clj -X:deps mvn-install for installing locally. we may end up making something for deployment - I think there is more that can be done there.

seancorfield20:07:32

For "just" building JAR/uberjar files, I'll probably stick to depstar, even for small projects -- especially with nasty stuff like the log4j2 plugins cache file merging: that alone prevents us using tools.build.api/uber.

seancorfield21:07:56

I will probably add a tools.build-based JAR building script to library projects created by clj-new though (the log42j stuff does not affect libraries, and I want to encourage folks to use tools.build).

seancorfield21:07:55

(once the new CLI changes get to a stable release so that I don't introduce a dependency on a prerelease CLI if you create a new project with clj-new!)

seancorfield21:07:45

For the app template in clj-new, I could also add a tools.build-based script but have it use hf.depstar.api/uber for that step πŸ™‚

didibus01:07:19

I see, but I guess there is tools.build the library, and there is also changes to the CLI related to tools. So tools.build is just a lib with whatever useful convenient functions that may help you with writing a build program and doing common things. I guess more interesting is the convention of having a file called build.cljwhich it establishes, and to have a -T invokable alias called build in your deps.edn. While just a convention, this is starting to look like a build framework, as it will become the expected frame for how a project sets up its build and how someone would expect to be able to run the project build tasks by using clojure -T:build <task> Similarly, I'm guessing from this convention, if you were to poke around some projects and wanted to understand what are all the build steps available for it, one would think to look at the build.clj It seems the -T option of the CLI and build.clj being established as the ns-default are really the important bits here. As well as what's coming to chain -T commands from the CLI > We expect to be able to do these both together on the command line but that is a work in progress > As well as the convention to parameterize using a key/value map, and have the "tasks" functions return that map for whenever support for command chaining is added. Personally when I see all this, I would find it weird to expect other build related tools, like tests and linting for example, to be setup as.`-X`. If so, now you have to learn a whole other tool and it's commands and all that, and won't be able to use whatever the command chaining when it lands for -T. So I feel it's really asking for people to release their additional tools in the same convention and style as tools.build , and for projects to adopt the build.clj as the place where tasks are defined. For example, I'd expect someone could say release a nRepl task where you'd do:

(ns build
  (:require [clojure.tools.build :as b]
    [nrepl.nrepl.build :as nb]))

(def lib 'my/lib1)
(def version (format "1.2.%s" (b/git-count-revs nil)))
(def class-dir "target/classes")
(def basis (b/create-basis {:project "deps.edn"}))

(defn nrepl [params]
  (nb/start-nrepl
    {:basis basis
     :src-dirs ["src"]})
  params)
And similar for testing, linting, and all others. At least I feel this is the convention that's being established. If that happened, the use of -X or -M would eventually get smaller and smaller.

Alex Miller (Clojure team)01:07:21

you often don't need that level of customization and control though. for example I have a couple of aliases in the tools.build deps.edn for testing and doc gen https://github.com/clojure/tools.build/blob/master/deps.edn

Alex Miller (Clojure team)01:07:01

tools.build isn't (currently) released as an artifact and doesn't need a build and doesn't have a build.clj

seancorfield01:07:08

@U0K064KQV I would hope that with the improved support for source libs in the latest CLI/`tools.deps.alpha` we will see more libraries switch to that mode and perhaps stop publishing JARs at all.

Alex Miller (Clojure team)01:07:54

if I could loosely cut up the world (ignoring client-side cljs stuff atm): β€’ tools - usually released as git deps, usually don't need a build β€’ libs - usually deployed as jars to clojars or released as source deps to git. may have a "simple" build.clj that makes a source jar β€’ apps - not deployed, usually have a build.clj that makes an uberjar

seancorfield01:07:54

Even though both clj-new and depstar are currently published to Clojars, there's really no reason for me to do so -- both libraries could be installed as local "tools" (`clojure -Ttools install`) and clj-new arguably should be; and depstar is likely to have a career as a library to supplement tools.build, at least for a while, and that can be added as a source dependency.

seancorfield01:07:28

"other build related tools, like tests and linting" -- those aren't "build related" though.

Alex Miller (Clojure team)01:07:43

note that the new usage stuff for tools is taken from the deps.edn from source-based deps so you can only use that feature with source-based git deps right now (I've thought of some options for artifacts but nothing implemented atm)

seancorfield01:07:42

Leiningen blurred the lines because it runs tests, runs a REPL, builds JARs, writes poms, creates new projects, etc, etc, etc -- and with plugins it runs linters and all sorts of other tools. It's "easy" but not "simple" in the Clojure sense. I don't think we should expect to have "One Tool to rule them all".

seancorfield01:07:06

Most of my projects today have these three commands:

clojure -X:test
clojure -T:jar
clojure -T:deploy
and I can use the former to run tests using any version of Clojure I want to check (either through aliases in the project or my user deps.edn file). A lot of projects don't need any more than that -- and if we move more to source libs we'll see less need for the latter two commands.

Alex Miller (Clojure team)01:07:07

leiningen was just drafting maven in that respect

didibus01:07:40

I think all these names are kind of irrelevant. People have tasks they need to run, those are needed during development, debugging, CI, deployments, release and all that. Generally languages have a task framework so all these tasks play nice with each other and can also become part of the ubiquitous language between developers. Because these tasks often also required specific dependencies to run, such task framework is often coupled with a dependency manager. In my opinion this task framework is a good thing. Now a specific framework like lein maybe is too hard to extend or branch out when you need to do complicated things. But the idea of having at least a common convention you can be familiar with is pretty good.

didibus01:07:58

I'm okay with -X, but it makes tools.build the odd one out. Like why isn't it also just an alias to be called with -X then?

Alex Miller (Clojure team)01:07:30

but the main difference is that -X uses your project classpath + mods. -T does not start from your project classpath.

Alex Miller (Clojure team)01:07:08

testing is typically your project classpath (-X) + testing framework

Alex Miller (Clojure team)01:07:26

building is typically NOT your project classpath (-T) + build framework

Alex Miller (Clojure team)01:07:34

the other subtle difference is that -T uses :paths ["."] by default so you can reach things like a build.clj at the root without needing to add that to your alias

seancorfield01:07:16

I just edited my "three commands" above since :jar and :deploy could both be invoked with -T now -- that just wasn't the case before.

seancorfield01:07:36

Now running tests is the odd one out @U0K064KQV πŸ™‚ πŸ™‚

Alex Miller (Clojure team)01:07:39

I am still working out the best way to talk about this stuff. there is a lot of intention in it but I have been living in it too long to have the beginner mind. so I appreciate these questions didibus!

didibus01:07:05

So the "framework" here is really with -X and -T. Like if I want to create a task that others would use I should have it support proper invokation on either -T or -X. But if I need as a user to do anything more complex, which is where lein failed, I would need to write my own program, in build.clj Now in my own program, I would need to probably leverage other things to make it easier for me, like tools.build. So I guess anything setup for -T or -X can be used as a library as well, since it's just a clojure function that takes a map. But -X and -T currently don't compose at the command line. And -X has the issue when used as a lib that it might require a very different classpath and root folder and all that

seancorfield01:07:03

@U0K064KQV It's early days with this stuff. I expect we'll get multi-function invocation back for both -X and -T (right @alexmiller?) so it really will come down to knowing whether a "tool" (in the broader sense) needs to run in your project classpath (`-X`) or not (`-T`) and some tools won't care (`:jar` and :deploy examples above could be either).

didibus02:07:14

So personally I find the -T and -X confusing. I feel it would have been easier if the alias could say it's a tool or not. So as a user you just copy/paste and -X like always. It seems a burden to need to remember which invokation is the right one for the alias. I don't know maybe there's a reason for it needing to be user provided every time?

seancorfield02:07:54

What would happen if you used -X:foo:bar and :foo declared it wanted the project classpath and :bar declared it didn't? You still can't compose those two types of tools.

didibus02:07:39

But my biggest maybe uncertainty I have right now, and hopefully no hard feelings haha I am just trying to provide some feedback I can be absolutely wrong too. It's around being able to compose and reuse tasks. It's great whenever the -X and -T command composition is going to come, but there also needs a way to compose those in a Clojure program no? Specifically the build.clj ? Is that the issue your referring to Sean?

seancorfield02:07:10

If you don't mind using :replace-paths ["."] :replace-deps {...} then you can use -X for tools that don't need your project classpath.

didibus02:07:28

Well, that problem you described is the same-ish as being able to compose -X in a program no? That they might need different classpath and all?

Alex Miller (Clojure team)02:07:44

that's out of scope imo

3
Alex Miller (Clojure team)02:07:21

maybe future in-scope, but not right now. at the CLI you are always running one program in one classpath.

didibus02:07:17

You mean one function? Or like you'll be able to run multiple functions just not in different classpaths?

Alex Miller (Clojure team)02:07:36

like multiple functions in a build program

seancorfield02:07:46

Multiple functions in a single program in a single classpath.

Alex Miller (Clojure team)02:07:08

but I think the use cases for that are limited - that's more the exception than the norm

Alex Miller (Clojure team)02:07:18

> I feel it would have been easier if the alias could say it's a tool or not. is a good question

didibus02:07:05

Ok ya. Maybe this is where I latch on to tools.build Because it lets you use deps.edn to form a basis to run another task in a sub-process. If that other task is a -X test.runner for example, well then build.clj looks like a great way to compose those things. And then I could do clojure -T:build copy-config compile test

didibus02:07:19

Even though each of those might need different classpaths

seancorfield02:07:34

Well, you can't easily run a -X function in a subprocess right now.

Alex Miller (Clojure team)02:07:38

I don't think you're going to be happy forking processes to do those things

seancorfield02:07:43

That's what TBUILD-6 is about πŸ™‚

πŸ‘€ 3
didibus02:07:17

I mean, my only alternative right now is shell and that just forks processes. If I do: clojure -X:copy-config && clojure -T:build compile && clojure -X:test

seancorfield02:07:21

tools.build makes it possible to run "arbitrary" java commands as subprocesses but it's more work that just clojure.java.shell/sh or even ProcessBuilder interop -- but in exchange you get conveniences for creating the classpath (via basis creation) etc.

didibus02:07:23

And as a user, eventually I'd expect that I wouldn't need to fork the process myself, but people would have their -X function takes an optional basis that then forks it and executes it within the context of the basis for me.

seancorfield02:07:19

@U0K064KQV As I said yesterday, maybe instead of contemplating this in the abstract, you should use tools.build and build some of this stuff yourself and see how well it does or doesn't work for you.

didibus02:07:26

If they are an -X that needs to run in specific deps and all that

seancorfield02:07:00

For what it's worth depstar already forks a process to run AOT compilation (so that it can guarantee it happens in the correct classpath/environment). But I don't think it's realistic to expect test runners to do that when it's just extra complexity and overhead: they could simply run in your project's classpath without any of that.

seancorfield02:07:04

Leiningen and Boot essentially did force everything to work "the same way" so you have a barrier to entry immediately for folks looking to write simple tools that can be used from the CLI.

didibus02:07:21

Test runners need to run in the project test classpath though, which will often add mocking libs and other test only deps

seancorfield02:07:11

Well, project classpath + test runner + test deps. That's still different to "no project classpath and having to create the basis and spawn a new process" etc.

didibus02:07:18

Ya, but you can say tell users: Add the test runner to the build alias deps Add a :test alias to deps.edn and add your test deps in there. Copy this function inside your build.clj Add this require to build.clj Run: clojure -T:build test Done.

seancorfield02:07:22

Or: Add a :test alias to deps.edn and add your test deps and the test runner in there. Run: clojure -X:test Done.

seancorfield02:07:53

What if they don't have build.clj? A lot of projects won't need it, as Alex said above.

seancorfield02:07:17

If you have a pure Clojure source library and you expect folks to use it via git then you really only need :test alias and run it via -X.

didibus02:07:08

Yes that's an alternative, but then I need to know if it's -X:test or -T:test or -Ttest or -T:build test cause now I'm dealing with different tasks using all four. And if I'm a power user, and I need to do more things before my tests can run, say generate some client, grab some creds, and I want to put those step in my build.clj and then start the test runner all with -T:build test ?

seancorfield02:07:03

This is something I have to consider for clj-new -- how simple should the generated projects be? Currently, they have a deps.edn with aliases for :test, :jar, :deploy and can be run via -X. But should those (simple) projects all have build.clj by default? Probably not, so maybe I need new template names, or I handle this via additional command-line options or something else...

seancorfield02:07:02

You're boiling the ocean: you're trying to solve every use case and trying to make them all the same. I don't think that's a very helpful or productive approach.

seancorfield02:07:24

-Ttest would be a locally-installed tool (in ~/.clojure/tools) and is different from -T:test -- and you would know because you'd installed that tool yourself (`clojure -Ttools install some/coord '{...}' :as test`)

didibus02:07:01

That one I'm not super concerned, though I think a convenience if like -T:test if no such alias is in the project or global deps.edn to check if a global tool of that name exists would be nice too. But ya, I'm more thinking in terms of the convention

didibus02:07:46

Like it seems it be more straightforward to me to just establish build.clj as the build script with all entry points into all tasks. And for the Clojure CLI to become the task runner for them.

Alex Miller (Clojure team)02:07:49

that is the not the model

didibus02:07:57

Otherwise people will still have to lean on shell scripts, makefile, lein, npm scripts, Babashka tasks runner, etc.

Alex Miller (Clojure team)02:07:06

that's what those things are good at

Alex Miller (Clojure team)02:07:32

the original and still intent of the CLI is to run programs in a classpath

Alex Miller (Clojure team)02:07:48

a build that makes an artifact is one example of a program

didibus02:07:54

Hum... Okay maybe I just don't see what problem tools.build is solving then?

Alex Miller (Clojure team)02:07:09

it's helpers to write programs that make artifacts

didibus02:07:13

Ok, so it's just another depstar and company? But tasks are still meant to be standard deps.edn and be clojure programs either invokable as -X, -T or -M and orchestrated with some other tool like Makefile or a shell ?

didibus02:07:08

Ah! 🀯, ok, well maybe it was just me being very confused, but I'd suspect maybe some others are similarly confused.

Alex Miller (Clojure team)02:07:13

I started working on tools.build in Feb 2020 so depstar was not quite the same level of popularity then

didibus02:07:41

Fair fair. This really likes clarifies things. I totally projected my problem and latched on tools.build thinking it solved it, which was that -X and -M alias are hard to make work together in a coherent way and orchestrate

Alex Miller (Clojure team)02:07:23

@U04V70XH6 on the clj-new question above, I think the intent of what you're building matters - tool/lib/app

Alex Miller (Clojure team)02:07:23

for a tool, I'd say no build.clj. for a lib, I think I'd lean towards no but maybe having a basic build.clj that made a source jar would be useful. for an app, definitely.

seancorfield02:07:18

Agreed. But tools and libs don't "need" JAR/deploy -- unless they do πŸ™‚ and an app can be a simple think or might be complex enough to warrant a build.clj file. Right now, clj-new's lib (and template) templates have a :jar alias that uses depstar to build a source JAR.

Alex Miller (Clojure team)02:07:54

well, I'd stick with having a build for that then

seancorfield02:07:25

Once the CLI gets to a stable release, I'll def. revisit the whole clj-new template thing.

Alex Miller (Clojure team)02:07:27

I mean, I think that's the most common case right now - building a lib with a source artifact deployed to clojars

πŸ‘ 3
Alex Miller (Clojure team)02:07:08

I'm in another window trying to debug the Windows updates so that we can move towards a stable release :)

didibus02:07:43

As an aside, and maybe a selfish request, but I would like it if all libs still defaulted to being able to publish a Jar to either Clojars or Maven Central. It makes companies like mine which have policies that deps must only come from internal repository mirrors and where GitHub and git is not something they support being mirrored from, mean that I'd have to manually import all source only libs and do a jar myself to put in our mirrored Maven repo. (I had to do that with the Cognitect test runner)

seancorfield02:07:52

Just so everyone reading this (long!) thread is on the same page regarding clj-new and lib projects right now:

(! 588)-> clojure -Tnew create :template lib :name alexmiller/example
Generating a project called example based on the 'lib' template.
The lib template is intended for library projects, not applications.
(! 589)-> cat example/deps.edn 
{:paths ["src" "resources"]
 :deps {org.clojure/clojure {:mvn/version "1.10.3"}}
 :aliases
 {:test
  {:extra-paths ["test"]
   :extra-deps {org.clojure/test.check {:mvn/version "1.1.0"}
                io.github.cognitect-labs/test-runner
                {:git/url ""
                 :sha "62ef1de18e076903374306060ac0e8a752e57c86"}}
   :exec-fn cognitect.test-runner.api/test}
  :jar {:replace-deps {com.github.seancorfield/depstar {:mvn/version "2.1.267"}}
        :exec-fn hf.depstar/jar
        :exec-args {:jar "example.jar" :sync-pom true}}
  :install {:replace-deps {slipset/deps-deploy {:mvn/version "0.1.5"}}
            :exec-fn deps-deploy.deps-deploy/deploy
            :exec-args {:installer :local :artifact "example.jar"}}
  :deploy {:replace-deps {slipset/deps-deploy {:mvn/version "0.1.5"}}
           :exec-fn deps-deploy.deps-deploy/deploy
           :exec-args {:installer :remote :artifact "example.jar"}}}}
(! 590)-> ls example/
CHANGELOG.md	LICENSE		README.md	deps.edn	doc		pom.xml		resources	src		test
(edited to show a library project!)

didibus02:07:28

I'm not sure how many companies are in this boat, but all the ones that use services like artifactory or AWS code artifact might have it

Alex Miller (Clojure team)02:07:57

the cool thing about templates is you can have more than one :)

seancorfield03:07:36

I realized I missed a bit of the back and forth up above around depstar and just wanted to provide a bit of historical context: I took over depstar in October 2018 and by February 2020 it was already doing AOT, Multi-Release stuff, etc but it was all still trying to do it in the project classpath (which was getting painful because it meant depstar could not have any dependencies except Clojure itself). It didn't become a "tool" -- running separately from the project classpath -- until December 2020, at which time it was leveraging t.d.a etc. So I think that really highlights how 2020 has been a critical year for a lot of the work around deps.edn and CLI in terms of massive changes/evolution in tooling.

dharrigan15:07:58

I have a little question

dharrigan15:07:54

It's too early, given the new tools build feature, and I think a consensus has to arise (maybe?), but would the thoughts be around including the build clj programs as part of your project, or perhaps bundling them up in a jar and depending up that jar (i.e., if one has say 10 or so Clojure projects, it would feel perhaps a bit burdensome to include the build scripts for every single project, but perhaps to depend upon a jar that includes those scripts might be the way?)

dharrigan15:07:01

What's the thoughts around this?

dominicm15:07:11

@dharrigan I think a common build script for common uses cases makes loads of sense

dominicm15:07:18

notleiningen.build.clj πŸ˜›

seancorfield15:07:29

Since your build script can be an arbitrary namespace full of functions, it could also be a library.

βž• 2
seancorfield15:07:37

But the "variables" between projects are going to be the group/artifact/version/etc which would need to be per-project -- so you could have most of your programmatic build in a library and just the variables part in your local build.clj which the library then requires to get at those...

seancorfield15:07:28

And you could treat your "build" library as a "tool" and clojure -Ttools install it so it's available "everywhere".

dpsutton15:07:32

interesting, but they probably need to be kept from any jars. Since they are all single namespace any other build.clj files in a jar could interfere with finding your own build.clj right? And this would rely on your source directories position in the path to always resolve your own

seancorfield15:07:01

@dpsutton build.clj isn't in src so it doesn't end up in the JAR.

dpsutton15:07:26

> but would the thoughts be around including the build clj programs as part of your project, or perhaps bundling them up in a jar and depending up that jar was responding to a suggestion for them to end up in jars

seancorfield15:07:54

I took that to mean "build script as library" rather than the "project JAR"

Ed15:07:42

yeah ... I think he's talking about sharing common build tasks between multiple repos

dharrigan15:07:26

my intention (and probably I'm misunderstanding), is that my project deps.edn would have a com.foo/build dep and in that build.jar would be the scripts that are common across all projects.

dharrigan15:07:37

does it work that way?

Ed15:07:01

I think you add a :deps keyword to your tooling alias

Ed15:07:47

and that defines the dependencies that the tool uses when it runs ... independent of the dependencies of your project itself

seancorfield15:07:54

Yes, the build is "just" a dependency.

dharrigan15:07:32

Right, so something like this?

dharrigan15:07:34

{:paths ["src"]
 :aliases
 {:build {:deps {com.foo/build {:mvn/version "0.1.0"}}
          :ns-default build}}}

seancorfield15:07:35

When you run clojure -T:some-alias it does :replace-paths ["."] I think which makes it easy to pick up a script in the root of the project, but our :build alias at work also includes a dependency on depstar as a library and on a small artifact uploader "library" locally.

seancorfield15:07:46

Our :build alias:

:build
  {:deps {org.clojure/clojure {:mvn/version "1.11.0-alpha1"}

          io.github.clojure/tools.build {:git/tag "v0.1.3" :git/sha "688245e"
                                         :exclusions [org.slf4j/slf4j-nop]}
          ;; force an updated version of t.d.a:
          org.clojure/tools.deps.alpha {:mvn/version "0.12.1003"
                                        :exclusions [org.slf4j/slf4j-nop]}
          ;; add depstar for building uberjars:
          com.github.seancorfield/depstar {:git/tag "v2.1.267" :git/sha "1a45f79"
                                           :exclusions [org.slf4j/slf4j-nop]}
          ;; and local build tools:
          worldsingles/build {:local/root "build"}
          poly/base-artifact-uploader-cli {:local/root "bases/artifact-uploader-cli"}
          poly/artifact-uploader {:local/root "components/artifact-uploader"}}
   :ns-default build}

seancorfield15:07:18

(we have some code in our build.clj and its deps that rely on the 1.11 hash map arg processing, hence needing to declare Clojure in that alias)

dharrigan15:07:53

is that on your parent directory, so the various sub projects would see the :local/root "build" directory?

seancorfield15:07:59

We execute all CLI commands from the project root with its deps.edn file. Each subproject has its own deps.edn file, including the build subproject.

Ed15:07:53

do you start a different repl for developing build scripts from developing the application?

Ed15:07:09

or do you start a jvm with them all on the classpath?

seancorfield15:07:12

@l0st3d Our dev REPL gets started with -M:everything:dev:test:runner plus whatever local tooling a developer wants. Our build REPL can be started with -M:build -i build.clj -r

seancorfield15:07:53

(`:dev` and :test are our Polylith aliases for the portion of our monorepo that we've migrated, :everything and :runner are the "equivalent" aliases for the portion that has not yet been migrated)

seancorfield16:07:56

See various posts on http://corfield.org for our monorepo setup with CLI/`deps.edn` and our migration to Polylith.

Ed16:07:25

cool ... so if you need to debug the build script, your current workflow is to start a new repl from your "all the modules in the system" repl?

seancorfield16:07:29

My personal REPL for work is started like this:

SOCKET_REPL_PORT=5000 clojure -Sforce -M:rebel:reflect:jedi-time:reveal:j14:classes:everything:dev:test:runner:dev/repl
πŸ‘€

Ed16:07:51

ah ... so no ... you've got them all on there

seancorfield16:07:12

On a branch we have the :build alias above and a fully-migrated setup, where we run tests in the subprocess via java-command and process in our build script. On our main branch we still have :build with :extra-deps etc... and that's where I still need :build in my REPL startup command.

seancorfield16:07:57

(hopefully that edit makes it less confusing)

Ed16:07:20

πŸ˜‰ ... I'm easily confused

Ed16:07:33

thanks for sharing ....

Ed16:07:50

I'm just interested in the potential workflows with these tools

seancorfield16:07:00

Our dev setup is pretty fluid -- we've been changing our build/CI pipeline a lot in the last month or so, trying to align with tools.build.

seancorfield16:07:09

In CI we have (on that fully-migrated build branch):

# ensure we have all the deps for our cache
          - ../build/clojure/bin/clojure -P -A:build:everything:dev:test:runner
          - FORCE_POLLING_ONLY=true
              ../build/clojure/bin/clojure -T:build all-tests-ci
          - ../build/clojure/bin/clojure -T:build tag-build-and-upload-all
(we have the Clojure CLI vendored into our repo so we can rely on a known version on all servers)

Ed16:07:55

where I'm currently working have some python based tools that are used to build docker images and wrap terraform / aws calls, etc ... and there's talk of migrating away from python to something more clojurey since it's mostly building clojure apps ... babashka tasks is on the list ... and I'm just looking around at this to see what is offered πŸ˜‰

seancorfield16:07:15

We previously had a mix of bash scripts and some Clojure stuff. Now it's all Clojure (although we still have a legacy app that relies on ant for build/deploy πŸ™€).

Ed16:07:31

oh the horror

borkdude16:07:22

@l0st3d doing a talk on babashka tasks in one hour and thirteen minutes at London Clojurians

Ed16:07:39

yes .. I know ... I'm afraid I can't make it 😞

borkdude16:07:00

It will be recorded

Ed16:07:15

yeah ... I figured it would ... I've been stalking you on the youtubes watching your videos ... it's all much appreciated πŸ˜‰

seancorfield17:07:53

@alexmiller The 1.10.3.920 prerelease seems to be broken for -T?

(! 1203)-> /usr/local/Cellar/clojure\@1.10.3.912/1.10.3.912/bin/clojure -Sforce -T:build uberjars :projects '[api auth]'

Running: git describe --tags

Running: depstar on api
2021-07-20 10:32:33,489 INFO  [hf.depstar.aot] - {} - Compiling api.main ...
2021-07-20 10:32:56,259 INFO  [hf.depstar.uberjar] - {} - Building uber jar: projects/api/../../../build/uberjars/api-1.0.0.jar

Running: depstar on auth
2021-07-20 10:33:06,394 INFO  [hf.depstar.aot] - {} - Compiling auth.main ...
2021-07-20 10:33:25,174 INFO  [hf.depstar.uberjar] - {} - Building uber jar: projects/auth/../../../build/uberjars/auth-1.0.0.jar
(! 1204)-> !!:gs/912/920/
/usr/local/Cellar/clojure\@1.10.3.920/1.10.3.920/bin/clojure -Sforce -T:build uberjars :projects '[api auth]'
Unqualified function can't be resolved: uberjars

Alex Miller (Clojure team)17:07:28

I don’t see that in local things I’ve done? Can you do something I can repro?

seancorfield17:07:09

It looks like the fix for the -X regression removed "${exec_args[@]}" and the --aliases ... piece is missing now? Based on https://github.com/clojure/brew-install/commit/8b556085a5e23cbce0a5cbc55766f4f1018d7b10

Alex Miller (Clojure team)17:07:58

Lots of changes in the handoff of this data

Alex Miller (Clojure team)17:07:11

You may need to -Sforce possibly to remake basis

seancorfield17:07:18

Oh... this is because I stashed a version of clojure.run.exec to implement our exec-command stuff!

Alex Miller (Clojure team)17:07:23

I can't tell what you're doing here, but if your exec copy is involved, that is broken

Alex Miller (Clojure team)17:07:37

yeah, I mentioend that to you yesterday :)

seancorfield17:07:33

Right. I didn't make the connection between updating the CLI and updating exec.clj as well πŸ˜•

seancorfield17:07:51

All fixed. Although now I can't easily tell the difference between a -X/`-T` invocation of build.clj and a -M invocation and my exec-command code doesn't work any more...

Alex Miller (Clojure team)17:07:56

for what reason do you need to tell the difference?

Alex Miller (Clojure team)17:07:39

-T can be seen as a pre-stage for -X that gathers the tool's lib/coord and its usage info from the deps.edn and then executes as a -X program

Alex Miller (Clojure team)17:07:10

there is new stuff in the basis (:execute-args) that may be able to tell you something useful, but not sure what you need

seancorfield17:07:41

Our build script had a mode where you could invoke it with -M and "regular" arguments and it would run multiple functions with interspersed arguments as part of a transition from our old shell script.

Alex Miller (Clojure team)17:07:22

not sure if you're writing more but I don't understand what the problem is from that

seancorfield17:07:49

As a shortcut, the script didn't need a -main, it just read *command-line-args* at the end of the script πŸ™‚ Now it needs a -`main` for that.

seancorfield17:07:45

I'm not sure what to do about invoking the subprocess via -X now since the updated exec.clj reads the basis from the system property location and that's where it tries to get :execute-args from -- so my trick of passing that system property through and being able to specify the additional aliases for exec-args no longer works.

seancorfield18:07:06

(this is all on a branch so it's not a blocker -- but it does mean that we can't use that branch unless we roll back to CLI 1.10.3.912 and the previous exec.clj code)

seancorfield18:07:38

I hope TBUILD-6 is going to be implemented "soon" so we can move forward with running -X style subprocesses from the build script... πŸ™‚

Alex Miller (Clojure team)18:07:11

well I think where things are now is substantially better in a lot of ways so not planning on going back (change let me delete a lot of fiddly code in multiple places)

Alex Miller (Clojure team)18:07:39

I'm not likely to work on TBUILD-6 soon

Alex Miller (Clojure team)18:07:04

(partly because I have 3 vacations in the next 2.5 weeks) and partly b/c I think other stuff is higher priority

πŸŽ‰ 2
Alex Miller (Clojure team)18:07:03

one possible way to go with it is to move exec into something consumable and the recent api changes make me feel better about that option, but that's a bunch of stuff to do

seancorfield18:07:09

Yeah, definitely an improvement for the system as it stands. I may take the old exec.clj and put it under a different name and use it "as-is" which will allow us to continue working on exec-command without breaking what the CLI is doing with updates to exec.clj.

seancorfield18:07:06

OK, I copied the old exec.clj to ws.clojure.run.exec for now so we can continue with our exec-command experiment, and now the following three types of invocation all work:

# legacy command-line invocation:
clojure -M:build -m build test lowlevel test worldsingles-web
# interactive "build REPL":
clj -M:build -i build.clj -r
# new "tools" invocation:
clojure -T:build run-tests :projects '[lowlevel worldsingles-web]'
with the latter running tests in a subprocess via our exec-command stuff and that older clojure.run.exec code.

seancorfield18:07:08

(to make the legacy invocation work, I had to add :paths ["."] to the :build alias because -T does that but -M does not)

seancorfield18:07:06

And just in case it sounds like I'm grumbling, I'll just reiterate that I really appreciate all the work you're doing on the CLI and the build tool chain -- thank you, @alexmiller -- and enjoy your vacations and I'll try to stop pestering you for a few weeks πŸ™‚

Alex Miller (Clojure team)18:07:05

io.github.clojure/tools.build {:git/tag "v0.1.4" :git/sha "169fef9"} is now available β€’ jar - add support for custom :manifest attributes β€’ uber - add support for custom :manifest attributes β€’ create-basis - make more tolerant of missing deps.edn file β€’ update tools.deps.alpha dependency to latest

πŸŽ‰ 15
seancorfield23:07:43

Related to the new ability to reference a project via :git/tag and :git/sha -- since t.d.a checks that the tag and the short SHA match, that means that any docs for the project that give examples of installation have to be updated after the "release" (after the tag has been applied)... which means that any service, such as cljdoc, which fetches doc source from GH based on a tag will pull back the outdated docs with an old SHA. Any thoughts on how this could be addressed?

seancorfield23:07:59

(currently, cljdoc requires a JAR release to clojars but some of us have been discussing the possibility of cljdoc importing direct from GH so this scenario becomes more likely)

Alex Miller (Clojure team)23:07:45

for tools.build and test-runner, I tag, then update the changelog and readme with tag+sha

Alex Miller (Clojure team)23:07:33

I don't think there's any way around that with the sha (although you could update the docs with the tag before you tag

Alex Miller (Clojure team)23:07:14

(also note that in the special case of clj -Ttools install - the sha is not required and will be inferred from the tag)

seancorfield23:07:56

Yeah, it only affects projects that expect to be used as source libs that have the short SHA in their docs -- and their docs are generated by an external system based on a tag.

Alex Miller (Clojure team)23:07:59

one helpful tool might be to look up the sha (although then maybe you're just opening up to all the problems we're trying to avoid)

Alex Miller (Clojure team)23:07:04

git ls-remote --short v0.1.4 for example

seancorfield23:07:24

I'll see what the cljdoc folks suggest. This came up for depstar since I document how to use it with tools.build and I wanted to use a source dep but then realized it would always have to be out of date...

Alex Miller (Clojure team)23:07:48

it is definitely annoying

seancorfield23:07:19

Hah, and I just shot myself in the foot by updating depstar's deps.edn file and referring to test-runner via the new coords and my CI failed because it's not on a recent enough CLI version 😐 Easy to get used to these new affordances!

Alex Miller (Clojure team)23:07:45

I have done that over and over in the last few months