yeah sorry; I meant outside of watch… e.g. in CI. Even printing a copy-pastable snippet of --focus args to rerun the last run (when run without watch) might be useful.

you mean like --plugin kaocha.plugin/print-invocations ? 🙂

oooo 👀

Also in CI I’d quite like: 1. kaocha to be able to save the info you’re suggesting, cache it for the next run; and then execute the previously failing tests first. (So it will fail faster next time if that is still a failing test) 2. To be able to annotate selected tests with ^:flaky-test metadata, and get kaocha to retry those tests a set number of times if they fail. Ideally we wouldn’t need this one of course; but from time to time when a timing issue pops up in an integration test; it’d be useful to temporarily patch it up in this manner; rather than retry the whole build 😞 or be forced to debug a flakiness issue at a time when there other more pressing fires to put out.

Oh one other thing whilst I’ve been meaning to mention whilst I’m here. It’s great that you can use aero in config… however one thing I’ve found (both in using aero with kaocha and in numerous projects using aero elsewhere). Is that when trying to build profile cascades with aero; where you have a base config and a local.edn file of develper specific customisations, aero gets really messy and hard to DRY up. e.g. I have this base test.edn config:

#kaocha/v1
#merge [{:tests [{:id          :static-data
                  :test-paths  ["test"]
                  :source-paths ["src"]
                  :ns-patterns ["-test$"]
                  :skip-meta [:cleans-database]}
                 {:id         :mutating-data
                  :test-paths  ["test"]
                  :source-paths ["src"]
                  :ns-patterns ["-test$"]
                  :focus-meta [:cleans-database]}]
         :reporter kaocha.report/documentation
         :plugins [:kaocha.plugin/instrument-specs
                   :kaocha.plugin.some.required.project.specific-plugin
                   :kaocha.plugin/profiling
                   :kaocha.plugin/instrument-timings
                   :kaocha.plugin/print-invocations]}
        #include "tests.user.edn"]

(defmethod aero/reader 'meta-merge [opts _tag value]
  (apply meta-merge value))

Those all sounds like great ideas to explore in plugins. I'm happy to make time to pair with you on that, or to answer questions and provide feedback through code review.

@plexus: thanks… I’ve been meaning to mention them, here first; incase there was something in the works or that did it already — and if not write an issue up for them, and then either write the plugin myself, or wait for someone else to do it (as happened with the spec check stuff 🙂 — which was a nice surprise; though I’ve yet to try it out) .

#meta-merge would be useful and welcome in core

That would be awesome

and is a PR I might even have time to implement 🙂

here's the code I mentioned, it creates a .kaocha-stats directory with timestamped files for each run, containing the test id, duration and result https://gist.github.com/70ccf307581d51134f853b4eaf7d886e

this is two thirds of the way of implementing a --re-run-failed option, you'd need a cli-options and config hook to support the CLI flag, and a post-load hook to skip everything that passed in (latest-stats)

Thanks that’s interesting

https://github.com/lambdaisland/kaocha/pull/128

thanks! not so sure about the k/meta-merge. Either make it kaocha/meta-merge, but I think just meta-merge is fine in this case.

also for next time please add an entry to the CHANGELOG. It's fine for now though, I'm happy to merge this and make these changes myself.