Clojurians Log v2

There isn't, but that's a good idea, I'll see about being able to mark the source folder itself.

I would absolutely adore to have LazyTest integration with Cursive. I’m converting our whole codebase over, and it’s fantastic, except Cursive doesn’t support it. I run my tests using a couple of REPL commands now, but obviously there’s no in-editor reporting or anything like that. If there’s anything I, as a regular user, could do to help, I’m all in.

I am not sure about the internals (obviously) but that’s how it looks from end-user perspective (config and output):

And here’s the file that cursive “compiles” to run the tests…

Assembles a classpath, and runs the file above.

Ah, makes sense that it basically wraps clojure.test directly. I suspected that might be the case. And it assumes clojure.test reporting machinery as well.

It’s nice, but in reality it’s just a glorified terminal output with a list of tests… Jumping to source of a failed test is nice, but not sure if that’s ground breaking. (as in, you can probably just run any test framework in the terminal and get 90% of the benefit). Maybe there are features that I am not using/realizing 😅 I think if something fails, it might show a diff of “expected” vs “current” data… (don’t remember)

Splitting the stdout/err output by test is nice

So it’s not just one huge blob of text.

Diff of expected/actual (useful)

(not a great example, but if there’s a subtle departure of expected/actual data, it can help a ton)

There are two integrations that Cursive has - what Rangel's showing there is the test runner, but you can also run tests interactively in the REPL, like this: https://cursive-ide.com/userguide/testing.html, which I think is what Martin is talking about. But they're definitely both tied to clojure.test very heavily right now. I would love an excuse to change that though! In particular, a solution which permits CLJS testing would be great. Is there any doc about how lazytest works? I assume it's not built on clojure.test, having read about it fairly briefly?

hi, i made lazytest. https://github.com/NoahTheDuke/lazytest#intellij is what i figured out for making cursive commands to run lazytest tests.

It's not built on clojure.test. what kind of information do you need about how it works? how can I help?

Ah, didn’t realize it can run tests interactively in the REPL… I was just eval-ing part of the test code to do that, should try the built-in option 🙂

> Ah, didn’t realize it can run tests interactively in the REPL… This is the sort of moment that makes me realise I really need to get on to making short videos of Cursive features 🙂

@nbtheduke Those commands look good for a semi-interactive integration, and are probably the best you can do in user space right now. There are two parts to the integration: one is running the tests, which is easy enough - I have commands for "run all tests in this ns" and "run this test under the caret" - basically what you have in your commands. The trickier bit is the reporting - there isn't a good way to get bidirectional communication with the REPL out of the box, so at the moment the integration binds the report var and just writes specially formatted messages to stdout, and the REPL integration parses those and annotates the results. What does lazytest provide in terms of test result events?

> This is the sort of moment that makes me realise I really need to get on to making short videos of Cursive features Oh yes…

gif “video” works great for stuff like that, I think. Example: https://github.com/apple/container

@cfleming I'd point you to the documentation but i still haven't written it yet. 😅 lazytest supports arbitrary custom reporters, which can be combined to run in the order provided. Each reporter is a function that is called with each suite before and after, and with each test case before and after, as well as each test case result. Typically, the reporter is a multi method that dispatches on :type but lazytest does not make assumptions. I can go into details about the various types if desired, and I will make some time to write proper documentation about this in general later today.

@nbtheduke Ok, that sounds like it should be relatively easy to integrate with. Do you provide reports for individual assertions? i.e. expect or it blocks?

Also, something else useful is that in clojure.test I get the expected and actual values when an = test fails. It's not ideal in that I have to fish it out of the macro expanded form, but it's there and essential for showing diffs.

One other good thing to cover in the doc would be how the reporting works for generative testing, if there's any relevant detail there.

Lazytest doesn't report individual assertions because it deals in "test cases". A test case runs the given body, and if it doesn't throw an exception, it's counted as a success. This means that if you want to have granular reporting on each assertion, you should split them into separate test cases. for example:

(describe "some outer context"
  (it "is a test case"
    (expect (= 1 1))
    (expect (true? false))
    (expect (delete-the-database))))

it has a similar expected vs actual tracking that clojure.test/is has, which is passed to the reporters

I don't know much about generative testing so I can't speak much to that, but test.check's defspec is just a thin wrapper over test.check's quick-check passed to clojure.test/is, so i think doing the same in Lazytest should be... easy? i hesitate to say it because that always blows up in my face lol

Yeah, the reporting in test.check is tricky IIRC.

That's a shame about the individual assertions. Is there any chance of reconsidering that? That allows things like the little pass/fail icons in Cursive's gutter area which are really nice.

Well, the pass/fail icons could go on the it (or expect-it) lines since those are the test cases?

(I mean, yeah, it would be nice to indicate the pass/fail/not-run on the expect lines too)

Sure, they could, but that's not as nice as seeing each individual... right.

It looks like this in Cursive, I (and most everyone else) like it a lot: https://cursive-ide.com/userguide/images/testing/test-results.webp

As a user of LazyTest, I really like the separation of test cases from assertions, TBH, and that a test case fails on the first assertion that fails. I've often wished clojure.test did that instead of ploughing on and producing additional failures.

Thinking about this, I don't know how I would support what Cursive does. Test running is not just a pass/fail, in the case of a fail you want to see the diff between expected/actual. I can do that with clojure.test because I get individual assertion reports - with lazytest I would not be able to do that.

If you just want pass/fail reporting, then @nbtheduke's existing REPL commands are probably good enough already.

I guess if that detail is surfaced in the exception then I could potentially show it there, and use the exception stack trace to identify the failing assertion.

That depends on that detail being surfaced in the reporter though....

LazyTest supports custom reporters. I think the way forward here is a custom reporter that provides the information Cursive needs.

lazytest uses very similar evaluation logic to clojure.test, where in the event of a failed assertion, it captures the input form, the evaluated result, and when given a "function call", the evaluated arguments as well. it doesn't capture passing assertions

if the logic for generating the test runner code is public, i can try writing a lazy test reporter for it

@nbtheduke In a failure does it also capture the expected value?

sure? what's the expected value of (expect (zero? (foo ...)))? there's no built in handling of clojure.core/=, if that's what you mean.

i'm on my phone so my apologies up front. given (expect (= 5 (foo))), the resulting ex-data map will be

{:ns user
 :expected '(= 5 (foo))
 :evaluated (list #function[clojure.core/=] 5 :cool-keyword)
 :actual false
 :message nil}

Right, so Cursive handles = specially, but I can do that in your example there by checking the form in :expected. In your example is false (from :actual) the result of (foo)?

and given (expect (or (foo) (bar)) "this better work!"), the resulting map will be

{:ns "user"
 :expected '(or (foo) (bar))
 :actual nil
 :message "this better work!"}

yeah, it's the result of the evaluated form

Actually, wait - false is actually the value of (= 5 (foo)), right? Not (foo)?

And it looks like :cool-keyword is (foo)?

yes, right

Ok. Can I customise the way the reporting is done, so that on a failure I can create an exception object and store that, then I'll use the stack trace to figure out where the assertion is in the code?

Then I can paint pretty icons on it.

yes. i didn't include that info in my example but the location data is all included

https://github.com/NoahTheDuke/lazytest/blob/77d9a01e87636fa2196d3d4db78e52a7219cbdf9/src/clojure/lazytest/reporters.clj#L149 here is existing code that pretty prints equality checks

I guess it's also potentially in metadata on :expected too, right?

yeah, but i also include that directly in the data map

Oh ok great.

Ok, I'm working on something else right now but I'll see if I can get this in this release cycle, it'll be an interesting test of whether I can extend the test functionality I have easily or not.

for better or worse, i built a custom ExceptionInfo that doesn't generate stack traces because ex-info is really really slow, which is why i include the metadata directly.

this is a lot to ask of you so let me know if i can help

I will do, thanks - it looks like lazytest is pretty well set up with the info I need, so thanks for that!