Fork me on GitHub
#testing
<
2023-02-07
>
Phillip Mates14:02:35

hi there, today I came across a limitation to clojure.test's fixture implementation: there is no way to know which test is being run inside the fixture implementation There is an old issue/patch for this that hasn't had much attention recently: https://ask.clojure.org/index.php/3835 I'm curious, have others hit this limitation? And if so, I'd appreciate it if you upvote that issue / provide your use case on ask.clojure

Noah Bogart14:02:07

If you don't mind, how would you use it?

Phillip Mates14:02:07

There was also mention of it in this thread: https://clojurians.slack.com/archives/C03S1L9DN/p1666477043602029 I tried the suggested approach but it didn't work in the context of a fixture, just inside the actual call to the test

seancorfield15:02:54

I don't understand your comment there - and the linked code doesn't clarify it either. My feeling is: if your fixtures need to know which test they are running then you should split your tests up into more namespaces and have more specific fixtures.

Phillip Mates15:02:55

I'll elaborate a bit more here: I write a lot of tests that require setup of state (starting a fresh instance of the service on every deftest invocation). You can do these custom setup invocations via fixtures, via a manual call inside the deftest , or via a custom macro that expands to a variation of what deftest expands to. Fixtures is the easiest if you want to your tests to be concise + work with existing tooling (hence my interest in this clojure ticket) In my case, each test may require different setup state for the service. For example, some tests exercise code that that uses special db schemas we haven't yet shipped to prod, thus requiring us to inject different seed schemas into the service startup state. I don't think this warrants moving tests around, because they otherwise logically operate over the same feature under test.

Phillip Mates15:02:09

my use case in-and-of-itself doesn't warrant implementing the feature (there are several work-arounds by way of macro etc). But if there are other use cases as well it might become more compelling

Phillip Mates15:02:40

assigning fixtures via metadata on single deftests ( https://clojure.atlassian.net/browse/CLJ-2719 / https://ask.clojure.org/index.php/12112) is another way forward for my example problem

seancorfield15:02:44

If the behavior of a "fixture" is specific to the test that is running it, it's not a "fixture" -- it's part of the test, IMO. I think this is something that is better solved by existing options: I don't think encouraging test-specific behavior to be written in (generic) fixtures is a good way to go.

seancorfield16:02:49

(I've spent a fair bit of time working with tests and testing libraries -- including two versions of Expectations, one of which is compatible with clojure.test -- and I think it's definitely an area we don't talk about enough as a community, and I used to think clojure.test was far too basic and was deficient in several ways but over the years I've come to believe it is mostly "just right"... I think the actual/expected diff stuff needs a better extension point but that's really the only thing I'd want to "fix"... I would like to see more "test matches" built-in, or an easier way to add new ones, but overall it is small, focused, and "does the job")

Phillip Mates16:02:18

I'm fine with not calling it a fixture and trying to avoid abusing the fixture concept in that way. I do still desire a mechanism to do customized setup, via vanilla deftest, without a lot of repetition one can do the following but it gets noisy if you have many tests with this binding repetition:

(deftest my-experimental-test
  (binding [*system* (start-system {:config-override :experimental})]
    (is (= :baz :qux))
    ...))

(deftest my-other-stable-test
  (binding [*system* (start-system {:config-override :stable})]
    (is (= :foo :bar))
    ...))
so what I would love to be able to write is
(use-fixture :each (fn [t] (binding [*system* (start-system (-> t meta :config-override))] (t))))

(deftest ^{:config-override :experimental} my-experimental-test
  (is (= :baz :qux))
  ...)

(deftest ^{:config-override :stable} my-other-stable-test
  (is (= :foo :bar))
  ...)

seancorfield16:02:01

I'd reach for a with-system macro there to reduce boilerplate but still be explicit.

Phillip Mates16:02:39

Having spent years maintaining Midje and also rewriting the best parts of it as standalone test libs, I feel like I went through a similar trajectory: I used to find clojure.test super minimal (in a bad way) and "without batteries" and over time came to appreciate how small and extensible it is 🙂

seancorfield16:02:42

(deftest my-experimental-test
  (with-system :experimental
    ... tests ... ))

seancorfield16:02:58

Midje is... a monstrosity 😞

Phillip Mates16:02:08

never said it wasn't 😛

Phillip Mates16:02:27

yeah, with-system sounds good, thanks for the consideration and suggestion with this

Phillip Mates16:02:23

but to Midje's credit, it inspired matcher-combinators and mockfn, which I really like to use (though I'm super biased)

seancorfield16:02:24

I can understand how and why it came to be, back when it happened -- DSLs were very cool because of macros -- but it's all sorts of non-idiomatic and none of the tooling is compatible with anything. I went through that tooling issue with Expectations -- so much custom tooling! -- and that's why I wrote the clojure.test-compatible version of Expectations.

👍 2
mauricio.szabo14:02:44

We went to the same path, @U04V70XH6 - I did an adaptation of Expectations and Midje that worked with clojure.test. Both reside on my check library, which also supports a clojure.test-inspired matcher situation. And I also made the mistake of doing some implicit things, now I'll probably need to break backwards compatibility to make things more explicit (like importing the matchers instead of relying on "side-effects requires"). These things start to matter when the code starts to get big Answering the question, I also advise for a with-system macro. In fact, I don't remember when was the last time I didn't do something like this....

☝️ 2
2
Noah Bogart15:02:43

At my company, we have such a large usage of a "with-system" approach that we built a macro that expands to two use-fixtures calls, one for :once and one for :each, so we can just say (xbt-test {:dataset :standard}) or (xbt-test {:each-fixtures [#'local-fixture] :dataset :special-case}) at the top of a test file and it "does the right thing"

Noah Bogart15:02:02

that approach doesn't work if you have a lot of tests in a given namespace that rely on different systems, of course

fuad01:03:50

I just wrote myself a with-system macro to more or less solve the same problem. I did favour the explicit, named binding compared to an implicit, dynamic one because I think it's easier to reason about it that way. https://clojurians.slack.com/archives/C030C4Z2W0Y/p1677632141804569?thread_ts=1676838664.145969&amp;cid=C030C4Z2W0Y