This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2021-07-16
Channels
- # announcements (3)
- # babashka (25)
- # beginners (71)
- # calva (18)
- # clj-kondo (52)
- # cljs-dev (94)
- # cljsrn (12)
- # clojure (33)
- # clojure-europe (52)
- # clojure-nl (2)
- # clojure-uk (27)
- # clojurescript (18)
- # clojureverse-ops (4)
- # datomic (64)
- # deps-new (27)
- # depstar (5)
- # events (5)
- # fulcro (5)
- # graalvm (12)
- # graalvm-mobile (82)
- # helix (2)
- # introduce-yourself (1)
- # juxt (5)
- # lsp (10)
- # malli (7)
- # missionary (1)
- # off-topic (41)
- # pathom (69)
- # pedestal (6)
- # re-frame (4)
- # reagent (8)
- # releases (9)
- # remote-jobs (8)
- # shadow-cljs (3)
- # sql (46)
- # tools-deps (44)
- # uncomplicate (1)
- # vim (83)
We are officially using tools.build
at work for part of our build.clj
script (although not for anything very substantial: just process
, instead of some ProcessBuilder
/interop stuff, and some file copying/deletion).
Part of me really wants to have build tasks that are like "run the tests and if successful build the JAR" but you can't "run the tests" if your build script is invoked via :replace-deps
(or just :deps
). Am I barking up the wrong tree with that or are folks resigned to having "run the tests" always be a separate clojure
invocation to "build the JAR"?
why can't you run the tests in your build script? something to do with bring in the test runner dep?
If your build script is invoked with just "."
for paths and just c.t.b for :deps
, how do you get the test runner on the classpath to invoke it and how does the test runner find all your source/test code to actually run it?
(currently, we're invoking the build script with a bunch of extra paths/deps and just running all the tests with c.t.b and t.d.a on the classpath as well which is what caused the S3 code loading race conditions that @hiredman was complaining about the other week)
unless you use c.t.b process
to shell out and invoke clojure -X:test...
I guess which is... ugh!
that's one option (personally I prefer to have it as a separate thing)
if you have it as a separate thing, you can use a combo of create-basis, java-command, and process to execute it
FWIW, I just started to look into this and whilst it might work fine for a -main
with regular string command-line arguments, it doesn't look so easy to replicate a -X
style invocation, which is what we're using. We have a custom wrapper for test-runner
's API that does some per-run setup and then invokes the runner, both of which happen via the -X
entry points right now.
If I'm following the CLI logic correctly, it looks like that routes through clojure.run.exec
which I believe is only in the exec.jar
and can't be added to a basis easily?
I thought maybe I could depend on brew-install
as a git dep but the deps.edn
there isn't a "real" one (`:paths ["x"]` etc) so that's not possible.
OK, I've worked around that by temporarily copying clojure.run.exec
into our repo so it can be on the classpath and I have a working version of running tests in a subprocess this way. I noticed that clojure.run.exec
assumes the clojure.basis
system property exists, so I had to propagate that into :java-opts
so it could read the basis...
FYI
(let [test-dirs (map #(pr-str (str % "/test")) projects)
aliases [:everything :dev :test :runner]
basis (b/create-basis {:aliases aliases})
cmd-args
(b/java-command {:basis basis :main 'clojure.main
:java-opts
;; propagate the basis from the CLI:
[(str "-Dclojure.basis=" (System/getProperty "clojure.basis"))
;; ideally we should pull all the :jvm-opts from the :runner
;; alias into this setup...
"-Dclojure.tools.logging.factory=clojure.tools.logging.impl/log4j2-factory"
"-Dlog4j2.configurationFile=log4j2-silent.properties"]
:main-args
["-m" "clojure.run.exec"
;; need :exec-args from this alias:
"--aliases" ":runner"
"ws.build.runner/test"
":dirs" (str "[" (str/join " " test-dirs) "]")]})
{:keys [exit]}
(b/process cmd-args)]
(when-not (zero? exit)
(System/exit 1)))
Yeah, you should rely on nothing about clojure.exec.run’s api - it changes frequently in breaking ways
But I think having support to invoke a -X entry point as part of tools.build is really interesting
Some sort of tool.build.api/exec
perhaps? I'll try converting a few more of our build tasks over and see what abstraction I can pull out of it...
I think with a decent wrapper, this can be quite clean -- but it needs some sort of standard -X
style adapter for java-command
?
added https://clojure.atlassian.net/browse/TBUILD-6 to track this
Thanks for that @alexmiller -- do you want me to share with you the code I created for that?
if you like
This is the sort of thing that I feel like just
, the task runner, is good for.
@alexmiller That sounds very low-level and a lot of work when the Cognitect test-runner
has a nice -X
function so it could be run in-process, if only there was a nice way to have basis-isolated contexts in c.t.b 🙂
well, not planning to make that
or at least not soon
basically re-making boot pods at that point
Yeah, they were great but they were also problematic 😞
which is probably cool, but farther down the list
I'd be interested in seeing an example in the tools.build
docs of using "a combo of create-basis, java-command, and process to execute it", preferably for test-runner
...
but if someone else wanted to make it, I think the provided basis stuff probably does all the hard work, just need to combine it with classloader isolation stuff
@seancorfield can you file a ticket for that at clojure-site ?
Yeah, which @hiredman has done at work but it's kinda gnarly (classloader isolation). And it's something that Polylith has also had to do. I think everyone reinvents that particular wheel right now.
Sure, I'll go open an issue on the GH repo for that...
Is there any way to customize the JAR manifest when using tools.build
uber
? https://docs.liquibase.com/tools-integrations/extensions/extension-upgrade-guides/lb-2.0-upgrade-guide.html. Also having it not include Multi-Release: true
automatically (even if some of our libs are multi-release) would be nice -- log4j is several orders of magnitude slower in a Multi-Release
JAR. Maybe a way to override the default mainfest that tools.build generates with my own custom one?
currently, no - could you ask it at https://ask.clojure.org and I'll turn it into a ticket from there
@camsaul depstar
lets you customize the manifest and has a drop-in replacement for tools.build.api/uber
in hf.depstar.api
@seancorfield thanks, I'll take a look at that
I'm surprised about log4j and the MR stuff but I will point out that if you're combining libs that use log4j2 (not 1.x) you'll also run into the problem that tools.build
's uber
task does not merge plugin cache files properly (which depstar
does).
> This is the sort of thing that I feel like `just`, the task runner, is good for. FWIW, this is also one of the main use cases of the bb task runner. https://book.babashka.org/#tasks
Or find a better solution to pass literal strings to -X. I think that would make things between shells and clojure smoother in general (https://clojurians.slack.com/archives/C6QH853H8/p1626295192392500?thread_ts=1626294554.390300&cid=C6QH853H8)