Fork me on GitHub
#kaocha
<
2021-03-03
>
danielneal16:03:55

Is there a way of getting kaocha to log each test it is running? I've hit an issue with some tests that have a subtle dependency between them, and I'm trying to debug it, but I need to know what order they are being run in.

dharrigan16:03:55

I have this in my project root

dharrigan16:03:01

❯ cat tests.edn 
#kaocha/v1
{:plugins [:kaocha.plugin/junit-xml
           :kaocha.plugin/cloverage
           :kaocha.plugin.alpha/spec-test-check]
 :reporter [kaocha.report/documentation]
 :fail-fast? true
:kaocha.plugin.junit-xml/target-file "target/junit.xml"}

dharrigan16:03:18

running the tests will produce a. an xml file and b. a html file which you can open up in a browser

dharrigan16:03:21

maybe that might help?

danielneal16:03:34

oh yes, I think I have that too

danielneal16:03:43

in fact I think it was your config I used 🙂

dharrigan16:03:56

I do think that the ordering of the tests is non-deterministic

danielneal16:03:20

yeah, it's randomized with a seed

danielneal16:03:25

the error has gone away now 😄

danielneal16:03:29

need to be ready next time

dharrigan16:03:33

can you provide your own seed?

danielneal16:03:38

I think you can

dharrigan16:03:39

so that it's run the same way each time?

royalaid16:03:20

You can also disable randomization

royalaid16:03:00

add

:kaocha.plugin.randomize/randomize? false

royalaid16:03:07

to the top level map in your tests.edn

lread17:03:14

If I remember correctly, even with randomize false, your tests are still first sorted by name before they are run. Clojure.test offers a way to specify test order via test-ns-hook but kaocha currently https://github.com/lambdaisland/kaocha/issues/29. Running tests in a specific order is of interest for a project I am working on that converts code blocks in docs to tests. These code blocks sometimes rely on each other, so running in order is important. I worked around the issue by using randomize false and generating test names that would sort appropriately.

lilactown17:03:27

has anyone experienced this error when running kaocha on node.js in GitHub's CI?

Exception: clojure.lang.ExceptionInfo: Failed initializing ClojureScript runtime
full stack trace: https://github.com/lilactown/autonormal/runs/2006021070?check_suite_focus=true

lilactown17:03:18

it's intermittent; re-running the tests typically fixes it. anecdotally I would say it fails every other time in CI. I haven't repro'd it locally yet

Alys Brooks17:03:56

I haven't tried kaocha on GitHub CI myself. One thing you could try is experimenting with is the :cljs/timeout option to a value higher than 10000 (=10 s). It's possible node.js isn't consistently ready by the time those tests run.