Clojurians Log v2

i'm excited to see you play with it! What's got you interested in it? I tend to think of you as sticking with the official or near-official tools so I'm a little surprised that you're going to start using it

Well, we were using classic Expectations at work a bit but the custom tooling was a pain, which is why I wrote Expectations 2.0. But you've already exposed a similar API to clojure.test to run tests, and at work we use a custom test runner plugged into Polylith so I control that aspect and can add LazyTest support. And you've done a lot of stuff that I sort of wanted to do with Expectations, but I'm held back by clojure.test at this point. I use Jest at work for frontend stuff and like the expressiveness...

And for all my talk on the LA Meetup about editor integration -- both Cursive and Calva integrate testing into the host editor's test reporting (showing pass/fail in the editor with highlighted failing tests) -- I don't actually use any of that:grin:My editor integration for test running is via custom REPL snippets and, again, that control that test running so I can make it run both types of tests.

https://github.com/NoahTheDuke/lazytest/issues/5

https://github.com/NoahTheDuke/lazytest/issues/2

Lazytest has access to the entire test run at the end, filters to just the test cases, and then uses those to determine the summary entries: :total is the total number of test cases (`it` calls, and equivalent), :pass is the number of test cases that didn't throw an exception, and :fail is the number of test cases that did throw an exception

the clojure-test reporter does mimic clojure.test's presentation by filtering for test vars and saying "Ran N tests containing Z test cases", so maybe something like :vars could be exposed through the summary

however, given that test suites are composable, that tests can be attached to the namespace (which skips looking for test vars entirely), and test cases can be generated programmatically, I opted for the default to merely look at test cases

when i was transitioning #splint’s test suite from clojure.test/expectations to lazytest, i had a custom test runner api that papered over my issues with cognitect/test-runner: https://github.com/NoahTheDuke/splint/blob/f851ef7933a74f54187d466a29482a1e4b4b577b/test/noahtheduke/splint/utils/test_runner.clj

i'd merge the results of ctr/test and lazytest/run, and then print them, treating :test and :total as the same:

(defn- print-summary
  "Adapted from clojure.test/report :summary"
  [{:keys [result elapsed]}]
  (printf
    "Ran %s tests containing %s assertions in %s msecs.\n%s failures, %s errors.\n"
    (+ (:test result 0) (:total result 0))
    (+ (:pass result 0) (:fail result 0) (:error result 0))
    elapsed
    (:fail result 0)
    (:error result 0))
  (flush))

not ideal, but it worked in the interim

Yeah, what I've done in my Calva setup is change my hot keys to run both clojure.test/run-tests and lazytest.repl/run-tests and merge-with + on the results, which is working pretty well.

I decided to not differentiate between "errors" and "failures" because i'm relying on exceptions. will all non-ExpectationFailed exceptions be errors? If so, that means you can't use your own exception or even write (assert (= 1 2)) which is something i was hoping to support

Yeah, it's much cleaner, TBH. I've always hated that aspect of clojure.test.

It's slightly more complicated than that, as seen here: https://github.com/NoahTheDuke/lazytest/blob/068b529350050213d8a7cbb7b13b03a8f783807f/src/clojure/lazytest/find.clj#L10-L34

it looks for, in order: • :type :lazytest/var metadata • suite value (def foo (suite ...)) • :test metadata that's a suite, test case, or function

As discussed in the other thread, the last one is problematic

Yeah, having LazyTest's runner just happen to run all clojure.test/deftest tests but have them pass because they don't throw exceptions is... weird, at best 🙂

Usurping :test metadata -- when that is already used by clojure.test -- and other tooling(?), is definitely problematic. Maybe use :lazytest or :lazytest/test instead?

I think that the lazytest runs the clojure.test tests as well because the lazytest runner will run vars that have the :test metadata on them.

Ah that's a great point, something I hadn't thought of.

Lazytest searches for vars that have :test metadata that's a function, and wraps it in a lazytest test case, calling it in the normal course of the test run. I forgot that clojure.test also puts the tests under :test metadata

I'm not sure of the best course here, truth be told. i suspect switching to :lazytest/test or similar is correct, as clojure.test must be treated as the only user of :test metadata

https://github.com/NoahTheDuke/lazytest/issues/4

Hah, just made that suggestion in the other thread. Yeah, if you switch to :lazytest/test then folks can fairly easily migrate from clojure.test (or Expectations) to LazyTest incrementally. I've already got a partially working version of the Polylith external test runner that looks for and runs both sets of tests 🙂

that's killer, dude, very kind of you to do all that

now i gotta get serious about no breaking changes hah

If you could expose lazytest.find/test-var? so that tooling could use it for scanning nses instead of replicating all that logic, that would be great. Scanning for clojure.test stuff is easy -- it's just (-> v meta :test) -- but LazyTest has three or four ways to define tests...

https://github.com/seancorfield/polylith-external-test-runner/blob/main/bases/external-test-runner-cli/src/org/corfield/external_test_runner_cli/main.clj hasn't been fully updated to handle LazyTest -- but that's where my work is going to be 🙂

(`contains-tests?` needs that test-var? query; I'm not so worried about trying to figure out filter/restore -- but could do it for :lazytest/test meta data only perhaps)

lazytest does its own filtering too. is that something you'd want to try to support? i see you've copied in a bunch of cognitect/test-runner, including the fitlering logic

Further down the road maybe. But right now, every test runner implements some variant of this (b/c clojure.test doesn't support it).

i guess i should be more clear. if you call lazytest.repl/run-tests, it will perform the lazytest specific finding and filtering: gathering lazytest-specific vars, focusing :focus suites, skipping :skip suites, and building the test run suite. then it'll execute that suite.

looking at your code, you're checking whether to even run ctr/test on a given namespace before executing?

Yes, b/c there's no filtering at all in clojure.test so the runners all do it.

(and I mostly don't use that stuff anyway)

I took care of the :test metadata overlap. That functionality now relies on :lazytest/test, which matches the other namespaced metadata keywords

For contains-test? , you can use lazytest.find/find-var-test-value: (some #(or (:test (meta %)) (lazytest.find/find-var-test-value %)) (-> ns ns-publics vals))

that function will return a truthy value if the var points to a lazytest "testable object", so to speak

it's a little more computationally expensive than (comp :test meta) lol but some is lazy so I don't think I should worry about it

Doesn't that have a set-var call in it?

poorly named, lol: (assoc value :type :lazytest/var :var this-var)

it changes the suite's type to :lazytest/var and attaches the var to the suite

no side effects

Haha... Okay. Then I'll make Polylith use that. Well, my runner. Thanks.

I'll make the runner smarter about checking for test types separately and only running the run-tests needed for the type of tests in a given ns.

I don't see this behavior:

noah@Noahs-MacBook-Pro ~/personal/lazytest  [main ≡ +1 ~1 -0 !]
$ cat example.clj
(ns example
  (:require [lazytest.core :refer [defdescribe expect-it]]))

(defdescribe hello-sean
  (expect-it "works" (= 1 1)))
noah@Noahs-MacBook-Pro ~/personal/lazytest  [main ≡ +1 ~1 -0 !]
$ clj-kondo --lint example.clj
linting took 12ms, errors: 0, warnings: 0

If you require a docstring from all defs (with :missing-docstring), then that config won't work

i thought that using deftest as the target would trigger :unused-value when given a docstring, but it looks like it doesn't, which i suspect is a bug in clj-kondo

https://github.com/NoahTheDuke/lazytest/issues/3

We have :missing-docstring enabled as :warn at work, which is how I noticed it.

Hmm tricky. Maybe I need to write a custom hook. I don't want to require docstrings, but docstrings are allowed

Can you configure Kondo to ignore missing docstring for defdescribe?

(I'm not sure how flexible config.edn is for that)

Hmm I'll see

Switching defdescribe and its pals over to lint as deftest also solves the "Unused public var" warning. Just FYI.