Clojurians Log v2
announcements 2025-10-09
Announcing test-filter (not feeling creating on naming š ) Hereās the idea: I work on very large projects. Our test suite runs 7 different groups of tests and still takes 12 minutes. This wastes a lot of time and CI money. But, what if you had a tool that could analyze your source code, find all the transitive dependencies, look at your recent work (compare the SHA of current functions with the last successful run of that function) and tell you which tests could possibly be affected? Thatās the goal of test-filter: Find the tests that matter, so you can just run those. So, far I know there are cases Iāve missed, and I think right now the best use of this tool is from the REPL during development to quickly run things you might have broken before handing it off to CI. Here are the cases I think it handles right: ā¢ Direct transitive dependency analysis. If test calls f calls g calls h, then a change to f, g, or h will run the test ā¢ Content normalization (remove docstrings) and hashing (every functionās code is hashed SHA256) so that trivial formatting edits/rearrange donāt affect tracking ā¢ Integration tests via metadata (integratoin tests probably use EVERY function in your code, so you can mark what is actually under test via metadata) ā¢ Work flow for REPL and CI Here are things that I have thought about, but wonāt work: ā¢ Changing a global var (e.g. alter-var-root). For example imagine you change the global locale of your program, and a bunch of strings change. ā¢ More careful git analysis (in various ways) In fact, I really didnāt look at the git logic much at all, and havenāt personally tested it much. Consider this an alpha phase, even though it might look a bit polished in the readme š https://github.com/fulcrologic/test-filter
Of course. Iāve been doing this for a few years, you know? ;)
Hey, I try not to assume anything š How many people do you know who profiled their tests?
;; btw all of this is not meant to knock your work, it's a good problem to solve and a good solution, I'm just biased against things being slow
Oh believe me so am I. I would have never written this many integration tests. But at a startup in the early phases you take what you can get to get things making money, and then you deal with it later.
and Iām an optimization/profiling fanatic. Look at the library code I just releasedā¦completely instrumented with tufte. Analyze on 250k LOC went from 3 minutes in the first iteration to about 1, etc. Kaocha has a nice plugin for profiling, but all of my slow tests are āspin up server, run REST call, drop serverāā¦The actual ātestā guts is fast. Itās all that up/down/reseed db thatās the mess
That's what I meant btw in another comment, can't you keep the server and DBs live between tests?
Combinatorial testing via integration testsā¦sooo bad š
The db is h2 in memory for tests or datomic local. Those two are actually really fast. But the stack also spins up lots of other things (e.g. for job processing etc). Itās all in-memory stuff.
Iām sure there are ways I could make it faster, but not as fast as not having to run the tests to begin with. Most of these endpoints rarely change. Running them every time is the real waste
Absolutely, the fastest code is the one which isn't ran
And Iām actually really pumped to have this workingā¦Iāve been sitting on the idea for some time. Occurred to me Claude might push me over the edge for building it.
and in 3 days I got what would have probably take 3 weeks to get to this point. I had to direct it a lot, and do a lot of manual testing and suggesting automated tests
but I was also able to work on 2 other things (using CC) at the same time š
I'm actually suffering some buyer's remorse for having used a LLM for writing something, hold the praise for Mr Certainly for a month or so š
Can I ask why tests were taking 12 minutes to begin with?
We have a LOT of integration tests and 250k lines of code. So, loading the code in the JVM with tests takes a couple of minutes. Caching restore or deps download takes a few minutes. Roughly 5-6 minutes of the time is just overhead BEFORE the test even start running. So the tests themselves are about 5-7 minutes. There are 4379 tests total, with maybe a total of 20k or more assertions. Make even 10% of those integration tests that have to spin up a resources and it gets bad quickly
So, an MR that does a simple front-end fix (no CLJ changes) can now skip all of the tests. Someething that fixes a leaf node bug in clojure: probably skips 99% of them. I donāt have it fully integrated yet, but itās going to make things a whole lot better.
Depending on the CI environment, caching the dependencies can shave that down from minutes to a few seconds from my experience. Integration tests I also tweaked to persist external dependencies between tests in the same run and just clear their states. But judicious change detection and running only dependent tests is cool. Unison does that too.
You also have a use case for asking for a faster compiler š
Of course Iām using caching, and we just happen to have a lot of transitive deps, so it isnāt seconds (unfortunately). There are all sorts of ways I can get the tests started quicker, but the reality is Iāve got 6 minutes (avg) for each of the parallel groups if I run all tests. So, with an instant cache AND instant compile Iāve still got six minutes. With this tool I hope to turn that into seconds (on average)
Also, with the code analysis it is much more likely that a dev will be able to run the set of tests in their already-initialized REPL in seconds, but will catch these ādependent casesā before ever pushing an MR. Thatās probably the real win.
It's a bit of a long shot but have you ever tried profiling the tests to find if there are any glaring CPU hogs?
I have a number of almost immediate improvements in progress: change detection by function SHA, support for fulcro-specās specifications, better clarity around whatās cached, etc.
This looks great, thank you! Do you plan (or would you be open to a contribution) adding support for leiningen by any chance?
Contribute away! š Iām doing this for my own companyās needs for faster turnaround in CI. So, my main interest is seeing my CI times go from 12 minutes to less than 5 (just restoring the caches and loading the project in the JVM takes 3-4 minutesā¦at that point I can rework the runner infra and maybe shave off another few) I got a lot more done on it earlier today, but itās crashing on my large project, so a bit more bug fixing to go still.
Oh, and a shout-out to @borkdude. What Iāve got so far took 3 hours with Claude Code, but only because clj-kondo already has source analysis and works as a library!
exactly what we were looking for! Awesome