Fork me on GitHub
#tools-deps
<
2021-07-16
>
seancorfield16:07:29

We are officially using tools.build at work for part of our build.clj script (although not for anything very substantial: just process, instead of some ProcessBuilder/interop stuff, and some file copying/deletion).

seancorfield16:07:36

Part of me really wants to have build tasks that are like "run the tests and if successful build the JAR" but you can't "run the tests" if your build script is invoked via :replace-deps (or just :deps). Am I barking up the wrong tree with that or are folks resigned to having "run the tests" always be a separate clojure invocation to "build the JAR"?

Alex Miller (Clojure team)16:07:59

why can't you run the tests in your build script? something to do with bring in the test runner dep?

seancorfield16:07:30

If your build script is invoked with just "." for paths and just c.t.b for :deps, how do you get the test runner on the classpath to invoke it and how does the test runner find all your source/test code to actually run it?

seancorfield16:07:42

(currently, we're invoking the build script with a bunch of extra paths/deps and just running all the tests with c.t.b and t.d.a on the classpath as well which is what caused the S3 code loading race conditions that @hiredman was complaining about the other week)

seancorfield16:07:24

unless you use c.t.b process to shell out and invoke clojure -X:test... I guess which is... ugh!

Alex Miller (Clojure team)16:07:22

that's one option (personally I prefer to have it as a separate thing)

Alex Miller (Clojure team)16:07:59

if you have it as a separate thing, you can use a combo of create-basis, java-command, and process to execute it

seancorfield23:07:26

FWIW, I just started to look into this and whilst it might work fine for a -main with regular string command-line arguments, it doesn't look so easy to replicate a -X style invocation, which is what we're using. We have a custom wrapper for test-runner's API that does some per-run setup and then invokes the runner, both of which happen via the -X entry points right now.

seancorfield23:07:59

If I'm following the CLI logic correctly, it looks like that routes through clojure.run.exec which I believe is only in the exec.jar and can't be added to a basis easily?

seancorfield23:07:50

I thought maybe I could depend on brew-install as a git dep but the deps.edn there isn't a "real" one (`:paths ["x"]` etc) so that's not possible.

seancorfield00:07:28

OK, I've worked around that by temporarily copying clojure.run.exec into our repo so it can be on the classpath and I have a working version of running tests in a subprocess this way. I noticed that clojure.run.exec assumes the clojure.basis system property exists, so I had to propagate that into :java-opts so it could read the basis...

seancorfield00:07:05

FYI

(let [test-dirs (map #(pr-str (str % "/test")) projects)
        aliases   [:everything :dev :test :runner]
        basis     (b/create-basis {:aliases aliases})
        cmd-args
        (b/java-command {:basis basis :main 'clojure.main
                         :java-opts
                         ;; propagate the basis from the CLI:
                         [(str "-Dclojure.basis=" (System/getProperty "clojure.basis"))
                          ;; ideally we should pull all the :jvm-opts from the :runner
                          ;; alias into this setup...
                          "-Dclojure.tools.logging.factory=clojure.tools.logging.impl/log4j2-factory"
                          "-Dlog4j2.configurationFile=log4j2-silent.properties"]
                         :main-args
                         ["-m" "clojure.run.exec"
                          ;; need :exec-args from this alias:
                          "--aliases" ":runner"
                          "ws.build.runner/test"
                          ":dirs" (str "[" (str/join " " test-dirs) "]")]})
        {:keys [exit]}
        (b/process cmd-args)]
    (when-not (zero? exit)
      (System/exit 1)))

Alex Miller (Clojure team)01:07:40

Yeah, you should rely on nothing about clojure.exec.run’s api - it changes frequently in breaking ways

Alex Miller (Clojure team)01:07:54

But I think having support to invoke a -X entry point as part of tools.build is really interesting

seancorfield05:07:36

Some sort of tool.build.api/exec perhaps? I'll try converting a few more of our build tasks over and see what abstraction I can pull out of it...

seancorfield05:07:28

I think with a decent wrapper, this can be quite clean -- but it needs some sort of standard -X style adapter for java-command?

seancorfield17:07:17

Thanks for that @alexmiller -- do you want me to share with you the code I created for that?

Joshua Suskalo16:07:05

This is the sort of thing that I feel like just, the task runner, is good for.

seancorfield16:07:22

@alexmiller That sounds very low-level and a lot of work when the Cognitect test-runner has a nice -X function so it could be run in-process, if only there was a nice way to have basis-isolated contexts in c.t.b 🙂

Alex Miller (Clojure team)17:07:27

well, not planning to make that

Alex Miller (Clojure team)17:07:04

basically re-making boot pods at that point

seancorfield17:07:25

Yeah, they were great but they were also problematic 😞

Alex Miller (Clojure team)17:07:50

which is probably cool, but farther down the list

seancorfield17:07:19

I'd be interested in seeing an example in the tools.build docs of using "a combo of create-basis, java-command, and process to execute it", preferably for test-runner...

Alex Miller (Clojure team)17:07:45

but if someone else wanted to make it, I think the provided basis stuff probably does all the hard work, just need to combine it with classloader isolation stuff

Alex Miller (Clojure team)17:07:59

@seancorfield can you file a ticket for that at clojure-site ?

seancorfield17:07:41

Yeah, which @hiredman has done at work but it's kinda gnarly (classloader isolation). And it's something that Polylith has also had to do. I think everyone reinvents that particular wheel right now.

seancorfield17:07:59

Sure, I'll go open an issue on the GH repo for that...

Cam Saul19:07:52

Is there any way to customize the JAR manifest when using tools.build uber? https://docs.liquibase.com/tools-integrations/extensions/extension-upgrade-guides/lb-2.0-upgrade-guide.html. Also having it not include Multi-Release: true automatically (even if some of our libs are multi-release) would be nice -- log4j is several orders of magnitude slower in a Multi-Release JAR. Maybe a way to override the default mainfest that tools.build generates with my own custom one?

Alex Miller (Clojure team)19:07:57

currently, no - could you ask it at https://ask.clojure.org and I'll turn it into a ticket from there

Cam Saul19:07:06

sure, np. Thanks!

seancorfield20:07:53

@camsaul depstar lets you customize the manifest and has a drop-in replacement for tools.build.api/uber in hf.depstar.api

Cam Saul23:07:14

@seancorfield thanks, I'll take a look at that

seancorfield23:07:48

(I added some notes to your ask post about it)

👍 2
seancorfield20:07:06

I'm surprised about log4j and the MR stuff but I will point out that if you're combining libs that use log4j2 (not 1.x) you'll also run into the problem that tools.build's uber task does not merge plugin cache files properly (which depstar does).

borkdude20:07:50

> This is the sort of thing that I feel like `just`, the task runner, is good for.  FWIW, this is also one of the main use cases of the bb task runner. https://book.babashka.org/#tasks

👍 2
borkdude21:07:09

Or find a better solution to pass literal strings to -X. I think that would make things between shells and clojure smoother in general (https://clojurians.slack.com/archives/C6QH853H8/p1626295192392500?thread_ts=1626294554.390300&amp;cid=C6QH853H8)

borkdude21:07:49

$ echo '(ns foo) (defn -main [& {:strs [-a -b --long-opt]}] (prn {:a -a :b -b :long-opt --long-opt}))' > src/foo.clj
$ clojure -M -m foo -a foo -b bar --long-opt baz
{:a "foo", :b "bar", :long-opt "baz"}
;)