This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2020-08-14
Channels
- # atom-editor (5)
- # babashka (6)
- # beginners (29)
- # calva (16)
- # cider (1)
- # clj-kondo (20)
- # cljs-dev (44)
- # clojure (29)
- # clojure-europe (19)
- # clojure-nl (8)
- # clojure-norway (7)
- # clojure-spec (2)
- # clojure-sweden (1)
- # clojure-uk (56)
- # clojurescript (32)
- # code-reviews (30)
- # conjure (24)
- # cursive (49)
- # datomic (4)
- # fulcro (31)
- # helix (3)
- # instaparse (4)
- # kaocha (100)
- # lambdaisland (2)
- # mid-cities-meetup (1)
- # monads (1)
- # off-topic (42)
- # pathom (13)
- # pedestal (6)
- # portal (5)
- # re-frame (6)
- # reagent (9)
- # reitit (11)
- # remote-jobs (1)
- # rewrite-clj (11)
- # shadow-cljs (44)
- # sql (22)
- # tools-deps (13)
- # uncomplicate (1)
- # xtdb (15)
Hello!
If I were to write a reporter which would print some extra information when a test fails, but which otherwise works as normal, is there a guide I could follow?
that's pretty specific, there's no documented recipe for that, but it should be straightforward. A reporter is just a function. You would write one which deals with the cases you want to deal with, and otherwise delegates to an existing reporter.
This is what I thought I’d do. Maybe I’m not reading the docs closely enough, but I’m in doubt about how to delegate. (I was thinking that it was just a matter of calling (existing-reporter m)
, but I’m in doubt about which existing report to refer to, and whether that would fit into the flow)
(defn my-reporter [event]
(if (kaocha.hierarchy/fail-type? event)
(println "oh no that's not good")
(kaocha.report/dots* event)))
@plexus I don’t understand this example. I tried it verbatim, but it seems like even though I have a test failure (as reported by the summary: #:kaocha.result{:count 1, :pass 0, :error 0, :fail 1, :pending 0}
), (kaocha.hierarchy/fail-type?
event)` does not return true during my run.
Ah wait, I am mistaken.
The printing confused me.
(run 'query.opgavefabrikken-test.config-test
{:reporter 'util.reporters/show-owner})
[()]
Randomized with --seed 1576729021(:pending :file :type :fail :line :error :kaocha/testable :pass :kaocha/test-plan :test)
=> #:kaocha.result{:count 1, :pass 0, :error 0, :fail 1, :pending 0}
No wait, I confused myself! The code should be like this:
(ns util.reporters
(:require [kaocha.hierarchy :as hierarchy]
[kaocha.report :as report]))
(defn show-owner [{:keys [] :as m}]
(if (hierarchy/fail-type? m)
(println "oh oh!")
(report/dots* m)))
And the output is then:
(run 'query.opgavefabrikken-test.config-test
{:reporter 'util.reporters/show-owner})
[()]
Randomized with --seed 1706915743
=> #:kaocha.result{:count 1, :pass 0, :error 0, :fail 1, :pending 0}
also add :kaocha.plugin.capture-output/capture-output? false
to your config when calling run
, that's generally a good idea when working on stuff like this to make sure things don't get hidden that you really would want to see
Both things work separately. I mean, for some definition of “work” 😄 The output is a bit funny looking, but at least it includes my prn now.
(run 'query.opgavefabrikken-test.config-test
{:reporter 'util.reporters/show-owner
:kaocha.plugin.capture-output/capture-output? false})
[(oh oh!
)]
Randomized with --seed 278474731
=> #:kaocha.result{:count 1, :pass 0, :error 0, :fail 1, :pending 0}
Will it always output whatever I prn as a part of that dots-like vector?
if you are using the dots reporter, yes. keep in mind that these reporters aren't very smart, they just get handed event after event and write some output to stdout as a result. The dot reporter prints "["
when a suite starts, "("
when a group starts, etc.
Specifically, I think it’d be neat to use something (perhaps a reporter) to react to failing tests by printing the names of the people who last touched the file that the ns belongs to, possibly via some git command that uses the file name.
It seems like all the pieces of the puzzle are available, provided that git
is available in the environment (and it can be)
I’ll give that a go 🙂
Iterating with the repl is, as always, great
I’ve hooked into the :kaocha.hooks/post-run
, but the m
doesnt’ seem to have the info I want
Nor for :kaocha.hooks/pre-report
something to be aware of here is that in kaocha you can specifiy multiple reporters, they will get merged into a single function, this is what happens in case of dots
and documentation
(def dots
"Reporter that prints progress as a sequence of dots and letters."
[dots* result])
(def documentation
"Reporter that prints an overview of all tests bein run using indentation."
[doc result])
they share a result
reporter which prints out the test summary in the end. So either you delegate to that too, or you add kaocha.report/result
as an extra reporter in tests.edn
That makes great sense, thanks @plexus!Now to iterate on it and see if I can do what I want 🙂
good look! the kaocha.report/debug
reporter can be helpful to get a sense of what data you're getting
and as demonstrated in my example it's recommended to use kaocha.hierarchy
instead of checking for concrete :type
values, so as to be maximally compatible with different test types and assertions
A "writing reporters" section in the docs would certainly be welcome. If you have the appetite for it it would be great if you could gather up these tips and stick them in a markdown file 🙂
That’s good to know. I was expecting to match directly with =
.
If this goes well, I might do a writeup 🙂
reporters are simple in theory but complex in practice, they're a concept we took directly from clojure.test
, but then kind of ran with it. We stick a lot more information in the event map than clojure.test does, and well behaved kaocha reporters use that when available. This is also because we need to support many test types (cucumber, cljs), with sometimes quite specific requirements
some examples
(defmethod doc :kaocha/begin-test [m]
(t/with-test-out
(let [desc (or (some-> m :kaocha/testable :kaocha.testable/desc)
(some-> m :var meta :name))]
(print (str "\n " desc))
(flush))))
:var
is a clojure.test specific thing, the more general thing is to use a testable's description
(defmethod fail-summary :kaocha/fail-type [{:keys [testing-contexts testing-vars] :as m}]
(println (str "\n" (output/colored :red "FAIL") " in") (testing-vars-str m))
(when (seq testing-contexts)
(println (str/join " " (reverse testing-contexts))))
(when-let [message (:message m)]
(println message))
(if-let [expr (::printed-expression m)]
(print expr)
(print-expr m))
(print-output m))
More info is great, and it’s probably what will enable me to do what I want. I just learned that I can’t go ahead and do (pprint/pprint m)
, that blows the heap space 😄 Too much info in one portion!
for clojurescript tests we do the formatting on the clojurescript side, because we can't always properly serialize/deserialize clojurescript assertions into clojurescript. So if :kaocha.report/printed-expression
is present then the reporter simply prints that instead of trying to parse and format/print the assertion
yeah you have the full testable in there which can be huge. That's why the debug
reporter limits the keys that it prints
(defn debug [m]
(t/with-test-out
(prn (cond-> (select-keys m [:type
:file
:line
:var
:ns
:expected
:actual
:message
:kaocha/testable
:debug
::printed-expression])
(:kaocha/testable m)
(update :kaocha/testable select-keys [:kaocha.testable/id :kaocha.testable/type])))))
also notice the with-test-out
, that's another clojure.test
thing that allows you to rebind where test output is printed. It defaults to just *out*
. We don't do much with it but users might. We really only use it to capture output in kaocha's own tests.
I was just messing around with printing parts of m
in a custom reporter, but I’ll try the debug
reporter now, as iterating on it is a bit slow. (I wonder how I could get lein test
to start up faster…)
Even better 😄
Seems like you’ve thought of everything 🙂
Do you happen to know how to get the path for a source file based on the ns? I was hoping that the path was available in the Kaocha map, but :file
only seems to refer to the filename itself.
Maybe I can also get it via the filename.
(let [ns 'foo.bar-test]
(io/resource (str (.. (name ns)
(replace \- \_)
(replace \. \/))
".clj")))
not sure if that's really the best way but that's roughly the idea. Might be some other option through tools.namespace
🙏 thank you very much!
(ns my.project.kaocha-hooks
(:require [kaocha.testable :as testable]
[kaocha.hierarchy :as hierarchy]
[kaocha.result :as result]))
;; use as a post-summary hook
(defn blame-report [result]
(let [clojure-test-suites (filter (comp #{:kaocha.type/clojure.test} :kaocha.testable/type)
(:kaocha.result/tests result))]
(doseq [suite clojure-test-suites
ns-testable (:kaocha.result/tests suite)
:when (result/failed? ns-testable)]
(prn (:kaocha.ns/name ns-testable)))
)
)
(comment
(require '[kaocha.repl :as repl]
'[kaocha.api :as api])
(def test-result (api/run (repl/config {})))
(blame-report test-result)
)
@reefersleepthis specifically assumes clojure.test, since you can't really generalize this, unless we make test types add a :kaocha.testable/file
to their testables, which might be a good idea
I’ll let you be the judge of that. You’re deep into datastructures that I only have surface knowledge of 🙂
Just trying this now
(defn blame-report [result]
(let [clojure-test-suites (filter (comp #{:kaocha.type/clojure.test} :kaocha.testable/type)
(:kaocha.result/tests result))]
(doseq [suite clojure-test-suites
ns-testable (:kaocha.result/tests suite)
:when (result/failed? ns-testable)
:let [ns-name (:kaocha.ns/name ns-testable)]]
(println
ns-name "last touched by"
(re-find #"Author:.*"
(:out
(sh/sh "git" "log" "-1" (str/replace (str (or (io/resource (str (.. (name ns-name)
(replace \- \_)
(replace \. \/))
".clj"))
(io/resource (str (.. (name ns-name)
(replace \- \_)
(replace \. \/))
".cljc"))))
"file:" ""))))))))
I’m trying to hook it in as a post-summary hook. I think I’m either doing it wrong, or on the wrong version of kaocha.
(run 'query.opgavefabrikken-test.config-test
{:plugins [:kaocha.plugin/hooks]
:kaocha.hooks/post-summary [util.hooks/blame-report]
:kaocha.plugin.capture-output/capture-output? false})
no sign of the output of blame-report
in the output
ah right, post-summary doesn't get run when running via kaocha.repl
, a post-run
should also work
I’m on "0.0-541"
😮
Time flies in this project. I feel like Kaocha was introduced just yesterday.
Brilliant! Now I get proper output
commit 723ff0e2ed7c63d8daf36ee39b6da57186d8686a
Author: Arne Brasseur <[email protected]>
Date: Thu Apr 12 09:16:35 2018 +0200
Initial skeleton
that is darn cool
I’m iterating towards your code, seeing that it works along the way
Yes! This is fantastic. Thanks Arne!
I’ll refine this and add it to our repo, it should help out whenever our CICD fails.
Is the missing run from post-summary
when running via kaocha.repl
intentional, or just something that hasn’t been fixed yet?
I’d like to write in our docs about how to test this stuff, and if post-summary
is more appropriate, it’d be cool to be able to test that in the REPL as well.
it's kind of intentional, post-summary is handled in kaocha.runner
, not in kaocha.api
, to handle cases like --print-test-result
which does a test run via kaocha.api
but then does completely different output handling, and we don't want plugins that print stuff at the end of the test run to clobber that
Call what and when? kaocha.runner
when testing in the REPL, or?
basically kaocha.repl and kaocha.runner both are ways to invoke kaocha.api. We don't want post-summary
to be called from api
, because there are use cases where api is used directly where it's not appropriate to call post-summary (this is the main difference between post-summary and post-run, the former is called outside kaocha.api, the latter inside)
Alrighty 🙂 So that’s a feature request for kaocha, I guess.
so yes, post-summary is the right place to print this kind of extra info at the end of a test run
@plexus how do you specify the equivalent of
{:plugins [:kaocha.plugin/hooks]
:kaocha.hooks/post-run [util.hooks/blame-report]}
When running in the terminal? I see that I can specify --plugin kaocha.plugin/hooks
, but I don’t see how to specify options for that plugin.I guess by using a tests.edn
. Trying that now.
Seems to work great! Though kaocha.hooks/post-summary
does not seem to work, only kaocha.hooks/post-run
, like we tried earlier. Kaocha is not documented to have post-summary
either. Are you ahead of the published code when talking about post-summary
? 😅
Hm, no, I see it in the source.
Expected:
true
Actual:
-true +false
171 tests, 675 assertions, 1 failures.
Error encountered performing task 'run' with profile(s): 'base,system,user,provided,dev,kaocha'
Suppressed exit
Am I maybe doing something wrong that causes the code to exit before the report is printed?