This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2023-09-12
Channels
- # ai (1)
- # announcements (7)
- # babashka (32)
- # beginners (23)
- # biff (9)
- # calva (1)
- # cljs-dev (13)
- # clojure (32)
- # clojure-belgium (1)
- # clojure-chicago (15)
- # clojure-europe (24)
- # clojure-india (3)
- # clojure-nl (3)
- # clojure-norway (55)
- # clojure-uk (4)
- # clojurebridge (1)
- # clojurescript (5)
- # core-async (17)
- # data-science (9)
- # datomic (29)
- # events (3)
- # fulcro (16)
- # graalvm-mobile (4)
- # helix (15)
- # hyperfiddle (74)
- # introduce-yourself (1)
- # jobs (4)
- # kaocha (12)
- # leiningen (27)
- # lsp (16)
- # shadow-cljs (6)
- # spacemacs (20)
- # sql (27)
- # squint (7)
- # tools-deps (29)
- # vim (2)
- # xtdb (10)
We've started introducing a few feature flags in our code, and I want to run the test base across the range of possibilities. Right now I'm working with a wrap-run hook that does a few for loops, then runs the tests individually. I'm unsure of how to properly report these back, or how I should merge these results back to report on which combination it went wrong with. Any pointers to docs would be welcome, I can't find how to do it at the moment
We have a global env
variable that the feature flags sit in, and can overwrite those with redefs, but that only works on a local test scale, I can't easily test both switches on/off, nor combinations when we get more options.
(I'm fine with the test suite taking a while to run, it's not going to be run constantly)
is your env
defined as ^:dynamic
?
because that way u can do something like:
(ns feature-flags-test
(:require [clojure.test :refer :all]
[clojure.math.combinatorics :as combinatorics]))
(defn ? [x] (prn x) x)
(def ^:dynamic env #{})
(defn some-logic [x]
(+ x (if (contains? env :feature/x)
1
2)))
(deftest some-logic-test
(doseq [feature-set (->> #{:feature/x :feature/y}
vec
combinatorics/subsets
(remove empty?)
(map set))]
(binding [env feature-set]
(testing (format "with features %s" (pr-str (set feature-set)))
(is (< 1 (some-logic 10)))))))
(kaocha.repl/run #'some-logic-test)
and the output would be:
--- unit (clojure.test) ---------------------------
feature-flags-test
some-logic-test
with features #{:feature/y}
with features #{:feature/x}
with features #{:feature/y :feature/x}
1 tests, 3 assertions, 0 failures.
=> #:kaocha.result{:count 1, :pass 3, :error 0, :fail 0, :pending 0}
It's using with-redefs since testing is single threaded, but yes that's generally how I'm doing it
that way u could test some fixed points in the behaviour of your function (aka system under test)
This works well for single cases, I'm looking at running the entire test suite with different combinations is feature flags
i see. well, in that case, if u notice the last line, u can run your test suite programmatically, so u can put the looping over the feature-sets around that kaocha.repl/run
call.
(deftest some-logic-test
(testing (format "with features %s" (pr-str env))
(is (< 1 (some-logic 10)))))
(doseq [feature-set (->> #{:feature/x :feature/y}
vec
combinatorics/subsets
(remove empty?)
(map set))]
(binding [env feature-set]
(kaocha.repl/run :unit)))
and u can also define this doseq
logic as a deftest
and exclude it from your default set of tests.
either way, i don't think there is anything in kaocha to support this kind of testing, and probably it doesn't even make sense to have any kind of explicit support, because it's quite application specific what u want to do with your feature flags. on the other hand, it also doesn't get in the way of implementing your own logic on top of kaocha to do what u want. 🙂