I’m trying to write a reporter for kaocha which will eventually spit a JSON summary to a file.

I have this snippet so far and it is wired properly since it does run, but eventually after running, it fails with

Error encountered performing task 'run' with profile(s): 'base,system,user,provided,dev,kaocha'
Suppressed exit

@srijayanth you want to use a plugin for this, not a reporter. Have a look at the junit.xml plugin. In a post-run hook you get a complete data structure with all the information you need, just traverse it and convert it to the JSON you need. https://github.com/lambdaisland/kaocha-junit-xml/blob/master/src/kaocha/plugin/junit_xml.clj#L153-L156

See https://cljdoc.org/d/lambdaisland/kaocha/0.0-573/doc/9-extending for information on writing plugins. Some tips: - avoid the kaocha. or kaocha.plugin. namespaces, instead use a prefix of your own, e.g. based on your github username - a test-plan or test result is nested, typically three levels deep but that's arbitrary. You can use kaocha.testable/test-seq to instead get a flat list. - to only get the "leaf" tests, and skip any grouping levels (i.e. test vars, not test suites or test namespaces), filter by kaocha.hierarchy/leaf?

as for your original question, not sure but I'm guessing you need at least a (defmethod report :default [_])

Note that testables that represent a grouping of tests, like namespace testables, can still fail, e.g. when a namespace fails to load then that's reported as a test failure on the :kaocha.type/ns testable. Hence why kaocha-junit-xml emits entries for all leaf testables and any other testable that has a failure https://github.com/lambdaisland/kaocha-junit-xml/blob/master/src/kaocha/plugin/junit_xml.clj#L31-L34

Thanks a lot @plexus

Had a good pairing session with @slawek098, which resulted in https://github.com/lambdaisland/kaocha/pull/132

note that this changes the behaviour of focus/skip. Before we would merge any focus/skip from tests.edn with those specified at the command line, and process them all together. With this change we process in two steps: we first mark tests to be skipped based on what's in tests.edn, then do a second pass and mark tests to be skipped based on CLI options. This means that CLI options can only narrow what's being run, which seems more intuitive. Feeback welcome!

we also noticed some cases where focus/focus-meta interact in a way that can be quite surprising, but we felt this needs some more hammock time to figure out how it should behave exactly, so we didn't change anything there yet