We have a global env variable that the feature flags sit in, and can overwrite those with redefs, but that only works on a local test scale, I can't easily test both switches on/off, nor combinations when we get more options. (I'm fine with the test suite taking a while to run, it's not going to be run constantly)

is your env defined as ^:dynamic? because that way u can do something like:

(ns feature-flags-test
  (:require [clojure.test :refer :all]
            [clojure.math.combinatorics :as combinatorics]))

(defn ? [x] (prn x) x)

(def ^:dynamic env #{})

(defn some-logic [x]
  (+ x (if (contains? env :feature/x)
         1
         2)))

(deftest some-logic-test
  (doseq [feature-set (->> #{:feature/x :feature/y}
                           vec
                           combinatorics/subsets
                           (remove empty?)
                           (map set))]
   (binding [env feature-set]
     (testing (format "with features %s" (pr-str (set feature-set)))
       (is (< 1 (some-logic 10)))))))

(kaocha.repl/run #'some-logic-test)

and the output would be:

--- unit (clojure.test) ---------------------------
feature-flags-test
  some-logic-test
    with features #{:feature/y}
    with features #{:feature/x}
    with features #{:feature/y :feature/x}

1 tests, 3 assertions, 0 failures.
=> #:kaocha.result{:count 1, :pass 3, :error 0, :fail 0, :pending 0}

It's using with-redefs since testing is single threaded, but yes that's generally how I'm doing it

that way u could test some fixed points in the behaviour of your function (aka system under test)

This works well for single cases, I'm looking at running the entire test suite with different combinations is feature flags

i see. well, in that case, if u notice the last line, u can run your test suite programmatically, so u can put the looping over the feature-sets around that kaocha.repl/run call.

Ah I see. Appreciated, I'll play with it Monday

(deftest some-logic-test
  (testing (format "with features %s" (pr-str env))
    (is (< 1 (some-logic 10)))))

(doseq [feature-set (->> #{:feature/x :feature/y}
                         vec
                         combinatorics/subsets
                         (remove empty?)
                         (map set))]
  (binding [env feature-set]
    (kaocha.repl/run :unit)))

and u can also define this doseq logic as a deftest and exclude it from your default set of tests.

either way, i don't think there is anything in kaocha to support this kind of testing, and probably it doesn't even make sense to have any kind of explicit support, because it's quite application specific what u want to do with your feature flags. on the other hand, it also doesn't get in the way of implementing your own logic on top of kaocha to do what u want. 🙂