Clojurians Log v2

Hello, any one used cloverage with cognitec test-runner ?

@fabrao You don't use it with test-runner. Cloverage is a test runner itself.

why some tests fail in cloverage but in test-runner don´t?

We have a wrapper for it, called from our build.clj script:

(defn coverage
  "Invoked via exec-fn (-X). Accepts all the same options that Cloverage's
  `run-project` function accepts (and passes them through).

  Looks for all `src` and `test` paths on the classpath. For any `/test`
  path, assume there's an equivalent `/src` path because we tend to bring
  in source dependencies via `:local/root` as a project but we bring in
  test dependencies via `:extra-paths`. Pass the final set of source and
  test paths into Cloverage and let it run all the tests it can find."
  [{:keys [src-ns-path test-ns-path] :as options}]
  (let [paths   (when-not (and src-ns-path test-ns-path)
                  (into #{}
                        (comp (remove #(str/starts-with? % "/"))
                              (mapcat #(vector % (str/replace % #"/test$" "/src"))))
                        (str/split (System/getProperty "java.class.path") #":")))
        sources (or src-ns-path
                    (filter #(str/ends-with? % "/src")  paths))
        tests   (or test-ns-path
                    (filter #(str/ends-with? % "/test") paths))]
    ((requiring-resolve 'cloverage.coverage/run-project)
     (assoc options :src-ns-path sources :test-ns-path tests))))

I run this

clojure -M:test:runner --exclude :integration

... 
with
:runner
  {:extra-deps {com.cognitect/test-runner
                {:git/url ""
                 :sha     "b6b3193fcc42659d7e46ecd1884a228993441182"}}
   :main-opts  ["-m" "cognitect.test-runner"
                "-d" "test"]}

As I recall, the way Cloverage instruments your code causes it to lose type hints which can definitely cause some code to fail.

may be something in integration ?

is there any ideas for considering this hints?

Yeah, if you're excluding some tests with test-runner and it passes, I expect cloverage is running those tests...

so, do I have to manually excludes this (deftest ^:integration...)?

How are you running Cloverage? Check its docs and see if it has options to exclude tests (although that kind of flies in the face of calculating coverage since you'll have gaps in your tests that way).

I´m using this

:cloverage
  {:extra-paths ["test"]
   :extra-deps {cloverage/cloverage {:mvn/version "1.2.2"}
                babashka/process {:mvn/version "0.0.2"}
                spec-provider/spec-provider {:mvn/version "0.4.14"}}
   :main-opts ["-m" "cloverage.coverage"
               "--no-html"
               "-p" "src"
               "-s" "test"
               "-t" "^(?!.*?(?:integration)).*$"
               "--fail-threshold" "90"]}

No idea. My experience of Cloverage is that it's pretty flaky if you have complex code.

It's been a while since I ran it on our code but I know some of our tests fail with Cloverage but pass with test-runner...

thank you

I think I´m going try to exclude the tests manually

I'm running it now to see if I can figure out a pattern in what fails. But because of the instrumentation, we can't run it on our entire codebase in one go so I have to run it on chunks of subprojects which takes a while and it's a bit manual.

Ah, I can't run it on some subprojects now because we're using :as-alias from Clojure 1.11 and clojure.tools.namespace (which Cloverage uses) doesn't support that properly yet:

clojure.lang.ExceptionInfo: Circular dependency between ws.profile.field and ws.domain.member.validation {:reason :clojure.tools.namespace.dependency/circular-dependency, :node ws.profile.field, :dependency ws.domain.member.validation}

in my case, will be difficult because all the regex stuffs are only for namespaces, maybe using your wrapper for build will be possible if we can filter functions with metadata.

@fabrao If it helps, I've just found one of our subprojects that breaks horribly under Cloverage and it seems to be related to multi-methods...

#error {
 :cause "No method in multimethod 'clear-redis' for dispatch value: class ws.redis.jedis_pool$pooled_jedis_object$reify__34674"
 :via
 [{:type java.lang.IllegalArgumentException
   :message "No method in multimethod 'clear-redis' for dispatch value: class ws.redis.jedis_pool$pooled_jedis_object$reify__34674"
   :at [clojure.lang.MultiFn getFn "MultiFn.java" 156]}]
 :trace
 [[clojure.lang.MultiFn getFn "MultiFn.java" 156]
  [clojure.lang.MultiFn invoke "MultiFn.java" 229]
  [ws.redis_expectations$new_test_redis invokeStatic "redis_expectations.clj" 43]
  [ws.redis_expectations$new_test_redis invoke "redis_expectations.clj" 42]

(but you should share details of what fails under Cloverage with your tests so folks can offer more specific feedback)

@fabrao cloverage has a variety of open issues unfortunately, so in practice it's apt for some projects, and for others it isn't The larger your project is and the more techniques it uses, the more likely it is that it hits some funny edge case. My recommendation (having used cloverage the last few years) would be to consider it an 'informative' step in the CI pipeline; it's no replacement for a more standard test runner.

Yup, definitely that point_up::skin-tone-2

(and I consider coverage information to be "interesting" rather than "useful" -- and any goals of percentage coverage for "all" code to be... misguided)

In all honesty, last time I gave real use to coverage metrics was in my Ruby days. Especially in 'rescue' projects with absent test suites. These clojurey days any email from Coveralls or such will be graciously deleted by a gmail filter :) OTOH I tend to be pretty strict self-reviewing my code, before each commit and after creating every PR. This includes a coverage assessment. So I do see value in ensuring team-wide standards, but at a personal productivity level coverage tooling is irrelevant.

our code fails only in this kind of test (deftest ^:integration...) . I´m thinking use like sean said, in build-clj process, adding filtering tests without ^:integration.

maybe it can be an issue for cloverage later

@fabrao What sort of failures? Just test failures, or exceptions?

testing failures

integration will check some testing that will always fail in test enviroment

integration tests will be in another task