kaocha

amithgeorge 2023-04-17T17:31:45.428609Z

In a codebase at work, I often struggle with the issue where I don't know which tests actually cover a certain function I am working on. For various reasons, it is not feasible for me to run the entire suite locally. Currently I manually find references and then keep going up the reference chain till I find tests that will indirectly/directly invoke the function I care about. I add focus metadata to their namespaces and run the tests. I was wondering given code coverage tools know which lines of code are covered by tests, can they (or other tools) also list which tests indirectly execute a certain function?

Alys Brooks 2023-04-20T02:08:40.268619Z

Glad to help, @amithgeorge. I haven't looked into using clj-kondo as a library (as opposed to running it on my code), so I'd be curious how easy it is to use in that context. @znwindy, that is an interesting idea. I did find the coverage-testing function (https://github.com/cloverage/cloverage/blob/87fd10f82ea7c0f47c03354105e513b160d1e047/cloverage/src/cloverage/coverage.clj#L100), and it seems to just add each form to a list of covered forms. I suppose you could use a dynamic variable with the current test, and then record that test symbol as having invoked that form?

Alys Brooks 2023-04-17T19:18:05.724889Z

Some tools, including Conjure and CIDER can run relevant tests automatically, although I think they all just go off of the namespace name, e.g., if you activate the feature for namespace foo, it tries to run tests in foo-test . I think you're right that running tests for a function should be possible using the same techniques as code coverage tools (or perhaps the coverage data directly), but I'm not aware of a current way to do it. A coverage tool could theoretically let you go to the granularity of a line or form, so you could, e.g., just run tests that exercise a particular branch. Clojure-LSP can find all references to a symbol, so while that doesn't directly run them, that might save you a bit of time by finding tests that reference the relevant function. However, that would miss tests that call a function indirectly. That might be a strength in some cases: A core function might have both direct unit tests and also used indirectly to set up a large number of other tests—the "find symbol" approach would only find the tests that specifically validate its behavior. I think this function could be implemented as a plugin that invokes an LSP or code coverage tool and identifies all tests that cover the specified funciton, skipping all other tests.

amithgeorge 2023-04-17T19:41:30.782719Z

Thank you for articulating my approach better and in more detail, and listing the pro/cons. You described my issue perfectly 👍 Yeah, I am doing currently using LSP find references multiple times to manually create a call tree and eventually find interesting tests that indirectly call the function I care about. It gets tiresome very quickly. > as a plugin that invokes an LSP or code coverage tool and identifies all tests that cover the specified funciton, I think clojure-lsp uses clj-kondo under the hood to do the analysis and maintain references. I wonder whether I should look into using clj-kondo directly...

plexus 2023-04-20T17:34:22.416179Z

Clerk also has some pretty good tracking code that we could possibly steal. See here for an earlier write-up https://nextjournal.com/lambdaisland/kaocha-autotest

frank 2023-04-18T15:47:16.180649Z

It would be super cool if, as a part of marking a line of code covered, cloverage also noted the test(s) that covered it. However, I'm also unaware of a tool that currently does this.