Clojurians Log v2

Of course. I’ve been doing this for a few years, you know? ;)

Hey, I try not to assume anything 🙃 How many people do you know who profiled their tests?

;; btw all of this is not meant to knock your work, it's a good problem to solve and a good solution, I'm just biased against things being slow

Oh believe me so am I. I would have never written this many integration tests. But at a startup in the early phases you take what you can get to get things making money, and then you deal with it later.

and I’m an optimization/profiling fanatic. Look at the library code I just released…completely instrumented with tufte. Analyze on 250k LOC went from 3 minutes in the first iteration to about 1, etc. Kaocha has a nice plugin for profiling, but all of my slow tests are “spin up server, run REST call, drop server”…The actual “test” guts is fast. It’s all that up/down/reseed db that’s the mess

That's what I meant btw in another comment, can't you keep the server and DBs live between tests?

Combinatorial testing via integration tests…sooo bad 😞

The db is h2 in memory for tests or datomic local. Those two are actually really fast. But the stack also spins up lots of other things (e.g. for job processing etc). It’s all in-memory stuff.

I’m sure there are ways I could make it faster, but not as fast as not having to run the tests to begin with. Most of these endpoints rarely change. Running them every time is the real waste

Absolutely, the fastest code is the one which isn't ran

And I’m actually really pumped to have this working…I’ve been sitting on the idea for some time. Occurred to me Claude might push me over the edge for building it.

and in 3 days I got what would have probably take 3 weeks to get to this point. I had to direct it a lot, and do a lot of manual testing and suggesting automated tests

but I was also able to work on 2 other things (using CC) at the same time 😄

I'm actually suffering some buyer's remorse for having used a LLM for writing something, hold the praise for Mr Certainly for a month or so 🙂

Can I ask why tests were taking 12 minutes to begin with?

We have a LOT of integration tests and 250k lines of code. So, loading the code in the JVM with tests takes a couple of minutes. Caching restore or deps download takes a few minutes. Roughly 5-6 minutes of the time is just overhead BEFORE the test even start running. So the tests themselves are about 5-7 minutes. There are 4379 tests total, with maybe a total of 20k or more assertions. Make even 10% of those integration tests that have to spin up a resources and it gets bad quickly

So, an MR that does a simple front-end fix (no CLJ changes) can now skip all of the tests. Someething that fixes a leaf node bug in clojure: probably skips 99% of them. I don’t have it fully integrated yet, but it’s going to make things a whole lot better.

Depending on the CI environment, caching the dependencies can shave that down from minutes to a few seconds from my experience. Integration tests I also tweaked to persist external dependencies between tests in the same run and just clear their states. But judicious change detection and running only dependent tests is cool. Unison does that too.

You also have a use case for asking for a faster compiler 😁

Of course I’m using caching, and we just happen to have a lot of transitive deps, so it isn’t seconds (unfortunately). There are all sorts of ways I can get the tests started quicker, but the reality is I’ve got 6 minutes (avg) for each of the parallel groups if I run all tests. So, with an instant cache AND instant compile I’ve still got six minutes. With this tool I hope to turn that into seconds (on average)

Also, with the code analysis it is much more likely that a dev will be able to run the set of tests in their already-initialized REPL in seconds, but will catch these “dependent cases” before ever pushing an MR. That’s probably the real win.

It's a bit of a long shot but have you ever tried profiling the tests to find if there are any glaring CPU hogs?

I have a number of almost immediate improvements in progress: change detection by function SHA, support for fulcro-spec’s specifications, better clarity around what’s cached, etc.

This looks great, thank you! Do you plan (or would you be open to a contribution) adding support for leiningen by any chance?

Contribute away! 😄 I’m doing this for my own company’s needs for faster turnaround in CI. So, my main interest is seeing my CI times go from 12 minutes to less than 5 (just restoring the caches and loading the project in the JVM takes 3-4 minutes…at that point I can rework the runner infra and maybe shave off another few) I got a lot more done on it earlier today, but it’s crashing on my large project, so a bit more bug fixing to go still.

Oh, and a shout-out to @borkdude. What I’ve got so far took 3 hours with Claude Code, but only because clj-kondo already has source analysis and works as a library!

exactly what we were looking for! Awesome