testing 2021-05-29 | Slack Archive

valsen11:05:22

Hi, running my tests always returns with exit code 0 even when some tests fail. This makes running the test suite in CI useless. Am I doing something wrong or how do others run their tests in CI?

nbardiuk11:05:38

lein test returns non zero error code when test fails. What do you use to run tests?

valsen11:05:25

Ah I don’t use lein, my poject uses deps

valsen11:05:58

right now my work-around looks something like this:

(defn -main []
  (let [{:keys [fail error]} (test-all)]
    (System/exit (if (zero? (+ fail error)) 0 1))))

And test-all is just a function that finds all my namespaces and runs (apply run-tests namespaces)

valsen12:05:09

And I execute the program with clj -M:test . So, it kind of works thanks to the System/exit call but that feels like a hack

nbardiuk12:05:22

I don't use deps, but last time I've looked into it people used test runners like https://github.com/cognitect-labs/test-runner or https://github.com/lambdaisland/kaocha. With those you could remove logic that discovers tests and produces exit code

valsen12:05:55

thanks I’ll check those out!:)

seancorfield16:05:42

@U01E2D1JLMC We use Cognitect’s test-runner in our BitBucket Pipelines CI at work and like it a lot (especially now it has the -X / exec args invocation so we can merge test arguments across multiple aliases.

😃 3

seancorfield16:05:02

(you can’t merge :main-opts across aliases — but you can merge :exec-args!)

plexus07:05:23

You've basically created your own minimal test runner @U01E2D1JLMC :)

2021-05-29

Channels