kaocha

2024-03-22T17:47:41.429539Z

We've run into a medium/big issue with :once fixtures: we have a feature-flag fixture that runs all tests in a namespace twice, once with the feature flag off and once with it on. (This way we can catch regressions.) It's implemented like this:

(defn create-on-off-ff-fixture
  [& flags]
  {:pre [(seq flags) (every? keyword flags)]}
  (let [off (redef-feature-flag-fixture false flags)
        on (redef-feature-flag-fixture true flags)]
    (fn [f] (off f) (on f))))
and then used like this: (use-fixtures :once (fix/create-on-off-ff-fixture ::core.ff/temp-example)) If a test fails in the first invocation and the passes in the second invocation, the failure is printed but it's not reported in the output, which means that the total failures will be 0 and the script will return with code 0. I have implemented a fix, along with a test, in https://github.com/lambdaisland/kaocha/pull/432.

plexus 2024-03-25T08:10:47.723129Z

Interesting use of fixtures, I hadn't seen that pattern of calling the tests multiple times, but I suppose it's a valid thing to do.