Fork me on GitHub
#kaocha
<
2023-04-17
>
amithgeorge17:04:45

In a codebase at work, I often struggle with the issue where I don't know which tests actually cover a certain function I am working on. For various reasons, it is not feasible for me to run the entire suite locally. Currently I manually find references and then keep going up the reference chain till I find tests that will indirectly/directly invoke the function I care about. I add focus metadata to their namespaces and run the tests. I was wondering given code coverage tools know which lines of code are covered by tests, can they (or other tools) also list which tests indirectly execute a certain function?

Alys Brooks19:04:05

Some tools, including Conjure and CIDER can run relevant tests automatically, although I think they all just go off of the namespace name, e.g., if you activate the feature for namespace foo, it tries to run tests in foo-test . I think you're right that running tests for a function should be possible using the same techniques as code coverage tools (or perhaps the coverage data directly), but I'm not aware of a current way to do it. A coverage tool could theoretically let you go to the granularity of a line or form, so you could, e.g., just run tests that exercise a particular branch. Clojure-LSP can find all references to a symbol, so while that doesn't directly run them, that might save you a bit of time by finding tests that reference the relevant function. However, that would miss tests that call a function indirectly. That might be a strength in some cases: A core function might have both direct unit tests and also used indirectly to set up a large number of other tests—the "find symbol" approach would only find the tests that specifically validate its behavior. I think this function could be implemented as a plugin that invokes an LSP or code coverage tool and identifies all tests that cover the specified funciton, skipping all other tests.

amithgeorge19:04:30

Thank you for articulating my approach better and in more detail, and listing the pro/cons. You described my issue perfectly 👍 Yeah, I am doing currently using LSP find references multiple times to manually create a call tree and eventually find interesting tests that indirectly call the function I care about. It gets tiresome very quickly. > as a plugin that invokes an LSP or code coverage tool and identifies all tests that cover the specified funciton, I think clojure-lsp uses clj-kondo under the hood to do the analysis and maintain references. I wonder whether I should look into using clj-kondo directly...

frank15:04:16

It would be super cool if, as a part of marking a line of code covered, cloverage also noted the test(s) that covered it. However, I'm also unaware of a tool that currently does this.

Alys Brooks02:04:40

Glad to help, @U0A5B1LJU. I haven't looked into using clj-kondo as a library (as opposed to running it on my code), so I'd be curious how easy it is to use in that context. @U02E7A2NGKD, that is an interesting idea. I did find the coverage-testing function (https://github.com/cloverage/cloverage/blob/87fd10f82ea7c0f47c03354105e513b160d1e047/cloverage/src/cloverage/coverage.clj#L100), and it seems to just add each form to a list of covered forms. I suppose you could use a dynamic variable with the current test, and then record that test symbol as having invoked that form?

plexus17:04:22

Clerk also has some pretty good tracking code that we could possibly steal. See here for an earlier write-up https://nextjournal.com/lambdaisland/kaocha-autotest