I have a macro generating tests "on the fly", is there a way I can have Kaocha run only the tests by that macro? Adding a metadata key to the generated deftest does not work, I assume Kaocha is statically parsing and looking at the test files.

Hmm, I believe that adding the metadata should work. Are you sure your macro adds the metadata correctly, though? It's a bit tricky, If you do it like make-test-1 below, it won't work, but I think make-test-2 would work

(defmacro make-test-1 [n] `(deftest ^:my-test ~n))
(defmacro make-test-2 [n] `(deftest ~(with-meta n {:my-test true})))

You can check it in REPL by requiring the namespace with macro-generated tests and running (meta #'your-namespace/name-of-the-test) . If the metadata is in the result, then Kaocha should see it.

@miikka is correct, your test files get loaded first, then we look for metadata on the vars. You can also check --print-test-plan to see the metadata Kaocha has found for each var

Oh, it needs ~(with-meta), yea I just used ^:my-test.

Thanks! And I'll try --print-test-plan

Where would the test name go though, before or after the with-meta block? Because neither seems to work for me

inside the with-meta, you add the metadata to the symbol that becomes the name of the var.