This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2022-01-07
Channels
- # adventofcode (3)
- # announcements (6)
- # babashka (20)
- # beginners (53)
- # calva (11)
- # clj-kondo (11)
- # clojure (50)
- # clojure-argentina (4)
- # clojure-dev (1)
- # clojure-europe (14)
- # clojure-houston (1)
- # clojure-italy (2)
- # clojure-nl (4)
- # clojure-norway (3)
- # clojure-seattle (3)
- # clojure-uk (13)
- # clojurescript (2)
- # cloverage (1)
- # code-reviews (4)
- # conjure (2)
- # cursive (5)
- # datalevin (4)
- # datascript (33)
- # datomic (16)
- # events (1)
- # graphql (10)
- # gratitude (1)
- # honeysql (6)
- # introduce-yourself (2)
- # jobs (1)
- # lsp (88)
- # malli (8)
- # off-topic (3)
- # other-languages (4)
- # polylith (3)
- # re-frame (16)
- # reagent (17)
- # reitit (3)
- # releases (2)
- # remote-jobs (1)
- # rewrite-clj (3)
- # shadow-cljs (3)
- # slack-help (2)
- # sql (36)
- # testing (31)
- # tools-deps (41)
- # xtdb (23)
@fabrao You don't use it with test-runner
. Cloverage is a test runner itself.
We have a wrapper for it, called from our build.clj
script:
(defn coverage
"Invoked via exec-fn (-X). Accepts all the same options that Cloverage's
`run-project` function accepts (and passes them through).
Looks for all `src` and `test` paths on the classpath. For any `/test`
path, assume there's an equivalent `/src` path because we tend to bring
in source dependencies via `:local/root` as a project but we bring in
test dependencies via `:extra-paths`. Pass the final set of source and
test paths into Cloverage and let it run all the tests it can find."
[{:keys [src-ns-path test-ns-path] :as options}]
(let [paths (when-not (and src-ns-path test-ns-path)
(into #{}
(comp (remove #(str/starts-with? % "/"))
(mapcat #(vector % (str/replace % #"/test$" "/src"))))
(str/split (System/getProperty "java.class.path") #":")))
sources (or src-ns-path
(filter #(str/ends-with? % "/src") paths))
tests (or test-ns-path
(filter #(str/ends-with? % "/test") paths))]
((requiring-resolve 'cloverage.coverage/run-project)
(assoc options :src-ns-path sources :test-ns-path tests))))
I run this
clojure -M:test:runner --exclude :integration
...
with
:runner
{:extra-deps {com.cognitect/test-runner
{:git/url ""
:sha "b6b3193fcc42659d7e46ecd1884a228993441182"}}
:main-opts ["-m" "cognitect.test-runner"
"-d" "test"]}
As I recall, the way Cloverage instruments your code causes it to lose type hints which can definitely cause some code to fail.
Yeah, if you're excluding some tests with test-runner and it passes, I expect cloverage is running those tests...
How are you running Cloverage? Check its docs and see if it has options to exclude tests (although that kind of flies in the face of calculating coverage since you'll have gaps in your tests that way).
I´m using this
:cloverage
{:extra-paths ["test"]
:extra-deps {cloverage/cloverage {:mvn/version "1.2.2"}
babashka/process {:mvn/version "0.0.2"}
spec-provider/spec-provider {:mvn/version "0.4.14"}}
:main-opts ["-m" "cloverage.coverage"
"--no-html"
"-p" "src"
"-s" "test"
"-t" "^(?!.*?(?:integration)).*$"
"--fail-threshold" "90"]}
I think the -t
parameter is not the way to do thisNo idea. My experience of Cloverage is that it's pretty flaky if you have complex code.
It's been a while since I ran it on our code but I know some of our tests fail with Cloverage but pass with test-runner
...
I'm running it now to see if I can figure out a pattern in what fails. But because of the instrumentation, we can't run it on our entire codebase in one go so I have to run it on chunks of subprojects which takes a while and it's a bit manual.
Ah, I can't run it on some subprojects now because we're using :as-alias
from Clojure 1.11 and clojure.tools.namespace
(which Cloverage uses) doesn't support that properly yet:
clojure.lang.ExceptionInfo: Circular dependency between ws.profile.field and ws.domain.member.validation {:reason :clojure.tools.namespace.dependency/circular-dependency, :node ws.profile.field, :dependency ws.domain.member.validation}
(there is no circular dependency -- c.t.n.dependency is "broken" regarding :as-alias
)in my case, will be difficult because all the regex stuffs are only for namespaces, maybe using your wrapper for build will be possible if we can filter functions with metadata.
@fabrao If it helps, I've just found one of our subprojects that breaks horribly under Cloverage and it seems to be related to multi-methods...
#error {
:cause "No method in multimethod 'clear-redis' for dispatch value: class ws.redis.jedis_pool$pooled_jedis_object$reify__34674"
:via
[{:type java.lang.IllegalArgumentException
:message "No method in multimethod 'clear-redis' for dispatch value: class ws.redis.jedis_pool$pooled_jedis_object$reify__34674"
:at [clojure.lang.MultiFn getFn "MultiFn.java" 156]}]
:trace
[[clojure.lang.MultiFn getFn "MultiFn.java" 156]
[clojure.lang.MultiFn invoke "MultiFn.java" 229]
[ws.redis_expectations$new_test_redis invokeStatic "redis_expectations.clj" 43]
[ws.redis_expectations$new_test_redis invoke "redis_expectations.clj" 42]
So the instrumentation is breaking that code.(but you should share details of what fails under Cloverage with your tests so folks can offer more specific feedback)
@fabrao cloverage has a variety of open issues unfortunately, so in practice it's apt for some projects, and for others it isn't The larger your project is and the more techniques it uses, the more likely it is that it hits some funny edge case. My recommendation (having used cloverage the last few years) would be to consider it an 'informative' step in the CI pipeline; it's no replacement for a more standard test runner.
Yup, definitely that ☝️:skin-tone-2:
(and I consider coverage information to be "interesting" rather than "useful" -- and any goals of percentage coverage for "all" code to be... misguided)
In all honesty, last time I gave real use to coverage metrics was in my Ruby days. Especially in 'rescue' projects with absent test suites. These clojurey days any email from Coveralls or such will be graciously deleted by a gmail filter :) OTOH I tend to be pretty strict self-reviewing my code, before each commit and after creating every PR. This includes a coverage assessment. So I do see value in ensuring team-wide standards, but at a personal productivity level coverage tooling is irrelevant.
our code fails only in this kind of test (deftest ^:integration...)
. I´m thinking use like sean said, in build-clj
process, adding filtering tests without ^:integration
.
@fabrao What sort of failures? Just test failures, or exceptions?