Fork me on GitHub
#kaocha
<
2020-08-14
>
reefersleep11:08:32

If I were to write a reporter which would print some extra information when a test fails, but which otherwise works as normal, is there a guide I could follow?

plexus11:08:06

that's pretty specific, there's no documented recipe for that, but it should be straightforward. A reporter is just a function. You would write one which deals with the cases you want to deal with, and otherwise delegates to an existing reporter.

reefersleep11:08:21

This is what I thought I’d do. Maybe I’m not reading the docs closely enough, but I’m in doubt about how to delegate. (I was thinking that it was just a matter of calling (existing-reporter m), but I’m in doubt about which existing report to refer to, and whether that would fit into the flow)

plexus11:08:46

(defn my-reporter [event]
  (if (kaocha.hierarchy/fail-type? event)
    (println "oh no that's not good")
    (kaocha.report/dots* event)))

reefersleep12:08:25

@plexus I don’t understand this example. I tried it verbatim, but it seems like even though I have a test failure (as reported by the summary: #:kaocha.result{:count 1, :pass 0, :error 0, :fail 1, :pending 0}), (kaocha.hierarchy/fail-type? event)` does not return true during my run.

reefersleep12:08:03

Ah wait, I am mistaken.

reefersleep12:08:31

The printing confused me.

(run 'query.opgavefabrikken-test.config-test
     {:reporter 'util.reporters/show-owner})
[()]
Randomized with --seed 1576729021(:pending :file :type :fail :line :error :kaocha/testable :pass :kaocha/test-plan :test)

=> #:kaocha.result{:count 1, :pass 0, :error 0, :fail 1, :pending 0}

reefersleep12:08:56

No wait, I confused myself! The code should be like this:

reefersleep12:08:02

(ns util.reporters
  (:require [kaocha.hierarchy :as hierarchy]
            [kaocha.report :as report]))

(defn show-owner [{:keys [] :as m}]
  (if (hierarchy/fail-type? m)
    (println "oh oh!")
    (report/dots* m)))

reefersleep12:08:13

And the output is then:

(run 'query.opgavefabrikken-test.config-test
     {:reporter 'util.reporters/show-owner})
[()]
Randomized with --seed 1706915743
=> #:kaocha.result{:count 1, :pass 0, :error 0, :fail 1, :pending 0}

plexus12:08:01

might be output capturing? try adding the t/with-test-out

plexus12:08:07

also add :kaocha.plugin.capture-output/capture-output? false to your config when calling run, that's generally a good idea when working on stuff like this to make sure things don't get hidden that you really would want to see

reefersleep12:08:09

Both things work separately. I mean, for some definition of “work” 😄 The output is a bit funny looking, but at least it includes my prn now.

reefersleep12:08:17

(run 'query.opgavefabrikken-test.config-test
     {:reporter 'util.reporters/show-owner
      :kaocha.plugin.capture-output/capture-output? false})
[(oh oh!
)]
Randomized with --seed 278474731
=> #:kaocha.result{:count 1, :pass 0, :error 0, :fail 1, :pending 0}

reefersleep12:08:16

Will it always output whatever I prn as a part of that dots-like vector?

plexus12:08:36

if you are using the dots reporter, yes. keep in mind that these reporters aren't very smart, they just get handed event after event and write some output to stdout as a result. The dot reporter prints "[" when a suite starts, "(" when a group starts, etc.

plexus12:08:12

what are you trying to achieve?

reefersleep12:08:21

Specifically, I think it’d be neat to use something (perhaps a reporter) to react to failing tests by printing the names of the people who last touched the file that the ns belongs to, possibly via some git command that uses the file name.

reefersleep12:08:46

It seems like all the pieces of the puzzle are available, provided that git is available in the environment (and it can be)

plexus12:08:45

I think that will be easier with a plugin/hook

plexus13:08:08

since that way you can just print a "report" at the end of the test run

reefersleep13:08:59

I’ll give that a go 🙂

reefersleep13:08:08

Iterating with the repl is, as always, great

plexus13:08:52

I'll write up something to get you started

reefersleep13:08:19

I’ve hooked into the :kaocha.hooks/post-run, but the m doesnt’ seem to have the info I want

reefersleep13:08:35

Nor for :kaocha.hooks/pre-report

plexus13:08:48

it's all there, just give me a moment

plexus13:08:11

managed to freeze up my emacs by printing a too big a datastructure 🙂

plexus11:08:31

something to be aware of here is that in kaocha you can specifiy multiple reporters, they will get merged into a single function, this is what happens in case of dots and documentation

plexus11:08:32

(def dots
  "Reporter that prints progress as a sequence of dots and letters."
  [dots* result])

(def documentation
  "Reporter that prints an overview of all tests bein run using indentation."
  [doc result])

plexus11:08:34

they share a result reporter which prints out the test summary in the end. So either you delegate to that too, or you add kaocha.report/result as an extra reporter in tests.edn

plexus11:08:04

#kaocha/v1
{:reporter [my.project/my-reporter kaocha.report/result]}

plexus11:08:08

does that make sense?

reefersleep12:08:37

That makes great sense, thanks @plexus!Now to iterate on it and see if I can do what I want 🙂

plexus12:08:12

good look! the kaocha.report/debug reporter can be helpful to get a sense of what data you're getting

plexus12:08:55

and as demonstrated in my example it's recommended to use kaocha.hierarchy instead of checking for concrete :type values, so as to be maximally compatible with different test types and assertions

plexus12:08:48

A "writing reporters" section in the docs would certainly be welcome. If you have the appetite for it it would be great if you could gather up these tips and stick them in a markdown file 🙂

reefersleep12:08:00

That’s good to know. I was expecting to match directly with =.

reefersleep12:08:16

If this goes well, I might do a writeup 🙂

plexus12:08:58

reporters are simple in theory but complex in practice, they're a concept we took directly from clojure.test, but then kind of ran with it. We stick a lot more information in the event map than clojure.test does, and well behaved kaocha reporters use that when available. This is also because we need to support many test types (cucumber, cljs), with sometimes quite specific requirements

plexus12:08:28

some examples

(defmethod doc :kaocha/begin-test [m]
  (t/with-test-out
    (let [desc (or (some-> m :kaocha/testable :kaocha.testable/desc)
                   (some-> m :var meta :name))]
      (print (str "\n  " desc))
      (flush))))

plexus12:08:54

:var is a clojure.test specific thing, the more general thing is to use a testable's description

plexus12:08:58

(defmethod fail-summary :kaocha/fail-type [{:keys [testing-contexts testing-vars] :as m}]
  (println (str "\n" (output/colored :red "FAIL") " in") (testing-vars-str m))
  (when (seq testing-contexts)
    (println (str/join " " (reverse testing-contexts))))
  (when-let [message (:message m)]
    (println message))
  (if-let [expr (::printed-expression m)]
    (print expr)
    (print-expr m))
  (print-output m))

reefersleep12:08:32

More info is great, and it’s probably what will enable me to do what I want. I just learned that I can’t go ahead and do (pprint/pprint m), that blows the heap space 😄 Too much info in one portion!

plexus12:08:23

for clojurescript tests we do the formatting on the clojurescript side, because we can't always properly serialize/deserialize clojurescript assertions into clojurescript. So if :kaocha.report/printed-expression is present then the reporter simply prints that instead of trying to parse and format/print the assertion

plexus12:08:47

yeah you have the full testable in there which can be huge. That's why the debug reporter limits the keys that it prints

plexus12:08:02

(defn debug [m]
  (t/with-test-out
    (prn (cond-> (select-keys m [:type
                                 :file
                                 :line
                                 :var
                                 :ns
                                 :expected
                                 :actual
                                 :message
                                 :kaocha/testable
                                 :debug
                                 ::printed-expression])
           (:kaocha/testable m)
           (update :kaocha/testable select-keys [:kaocha.testable/id :kaocha.testable/type])))))

plexus12:08:26

also notice the with-test-out, that's another clojure.test thing that allows you to rebind where test output is printed. It defaults to just *out*. We don't do much with it but users might. We really only use it to capture output in kaocha's own tests.

reefersleep12:08:55

I was just messing around with printing parts of m in a custom reporter, but I’ll try the debug reporter now, as iterating on it is a bit slow. (I wonder how I could get lein test to start up faster…)

plexus12:08:34

use kaocha.repl! no need to start a new process every time

reefersleep12:08:51

Even better 😄

reefersleep12:08:04

Seems like you’ve thought of everything 🙂

reefersleep12:08:48

Do you happen to know how to get the path for a source file based on the ns? I was hoping that the path was available in the Kaocha map, but :file only seems to refer to the filename itself.

reefersleep12:08:58

Maybe I can also get it via the filename.

plexus12:08:41

(let [ns 'foo.bar-test]
  (io/resource (str (.. (name ns)
                        (replace \- \_)
                        (replace \. \/))
                    ".clj")))

plexus12:08:26

not sure if that's really the best way but that's roughly the idea. Might be some other option through tools.namespace

reefersleep12:08:39

🙏 thank you very much!

plexus13:08:48

(ns my.project.kaocha-hooks
  (:require [kaocha.testable :as testable]
            [kaocha.hierarchy :as hierarchy]
            [kaocha.result :as result]))

;; use as a post-summary hook
(defn blame-report [result]
  (let [clojure-test-suites (filter (comp #{:kaocha.type/clojure.test} :kaocha.testable/type)
                                    (:kaocha.result/tests result))]
    (doseq [suite clojure-test-suites
            ns-testable (:kaocha.result/tests suite)
            :when (result/failed? ns-testable)]
      (prn (:kaocha.ns/name ns-testable)))
    )
  )


(comment
  (require '[kaocha.repl :as repl]
           '[kaocha.api :as api])

  (def test-result (api/run (repl/config {})))

  (blame-report test-result)
  )
@reefersleep

plexus13:08:52

this specifically assumes clojure.test, since you can't really generalize this, unless we make test types add a :kaocha.testable/file to their testables, which might be a good idea

reefersleep13:08:21

I’ll let you be the judge of that. You’re deep into datastructures that I only have surface knowledge of 🙂

reefersleep13:08:25

Just trying this now

plexus13:08:35

(defn blame-report [result]
  (let [clojure-test-suites (filter (comp #{:kaocha.type/clojure.test} :kaocha.testable/type)
                                    (:kaocha.result/tests result))]
    (doseq [suite clojure-test-suites
            ns-testable (:kaocha.result/tests suite)
            :when (result/failed? ns-testable)
            :let [ns-name (:kaocha.ns/name ns-testable)]]
      (println
       ns-name "last touched by"
       (re-find #"Author:.*"
                (:out
                 (sh/sh "git" "log" "-1" (str/replace (str (or (io/resource (str (.. (name ns-name)
                                                                                     (replace \- \_)
                                                                                     (replace \. \/))
                                                                                 ".clj"))
                                                               (io/resource (str (.. (name ns-name)
                                                                                     (replace \- \_)
                                                                                     (replace \. \/))
                                                                                 ".cljc"))))
                                                      "file:" ""))))))))

reefersleep13:08:42

I’m trying to hook it in as a post-summary hook. I think I’m either doing it wrong, or on the wrong version of kaocha.

reefersleep13:08:52

(run 'query.opgavefabrikken-test.config-test
     {:plugins [:kaocha.plugin/hooks]
      :kaocha.hooks/post-summary [util.hooks/blame-report]
      :kaocha.plugin.capture-output/capture-output? false})

reefersleep13:08:00

no sign of the output of blame-report in the output

plexus13:08:40

ah right, post-summary doesn't get run when running via kaocha.repl, a post-run should also work

reefersleep13:08:25

I’m on "0.0-541" 😮

plexus13:08:39

owwww better do something about that 🙂

plexus13:08:58

yeah that's almost a year old

plexus13:08:33

and exactly 100 commits behind 1.0.641 🙂

reefersleep13:08:04

Time flies in this project. I feel like Kaocha was introduced just yesterday.

reefersleep13:08:10

Brilliant! Now I get proper output

plexus13:08:07

commit 723ff0e2ed7c63d8daf36ee39b6da57186d8686a
Author: Arne Brasseur <[email protected]>
Date:   Thu Apr 12 09:16:35 2018 +0200

    Initial skeleton

plexus13:08:37

2.3333333 years old 🙂

reefersleep13:08:31

that is darn cool

reefersleep13:08:58

I’m iterating towards your code, seeing that it works along the way

reefersleep13:08:39

Yes! This is fantastic. Thanks Arne!

reefersleep13:08:23

I’ll refine this and add it to our repo, it should help out whenever our CICD fails.

reefersleep13:08:14

Is the missing run from post-summary when running via kaocha.repl intentional, or just something that hasn’t been fixed yet?

reefersleep13:08:08

I’d like to write in our docs about how to test this stuff, and if post-summary is more appropriate, it’d be cool to be able to test that in the REPL as well.

plexus13:08:03

it's kind of intentional, post-summary is handled in kaocha.runner, not in kaocha.api, to handle cases like --print-test-result which does a test run via kaocha.api but then does completely different output handling, and we don't want plugins that print stuff at the end of the test run to clobber that

plexus13:08:24

now we could call it from kaocha.repl as well, which I think would be appropriate

reefersleep14:08:13

Call what and when? kaocha.runner when testing in the REPL, or?

plexus14:08:49

no, call the post-summary hook from kaocha.repl

plexus14:08:51

basically kaocha.repl and kaocha.runner both are ways to invoke kaocha.api. We don't want post-summary to be called from api, because there are use cases where api is used directly where it's not appropriate to call post-summary (this is the main difference between post-summary and post-run, the former is called outside kaocha.api, the latter inside)

reefersleep21:08:58

Alrighty 🙂 So that’s a feature request for kaocha, I guess.

plexus13:08:05

so yes, post-summary is the right place to print this kind of extra info at the end of a test run

plexus13:08:42

that's also for instance what the profiling plugin uses

reefersleep21:08:04

@plexus how do you specify the equivalent of

{:plugins [:kaocha.plugin/hooks]
        :kaocha.hooks/post-run [util.hooks/blame-report]}
When running in the terminal? I see that I can specify --plugin kaocha.plugin/hooks, but I don’t see how to specify options for that plugin.

reefersleep21:08:33

I guess by using a tests.edn. Trying that now.

reefersleep21:08:43

Seems to work great! Though kaocha.hooks/post-summary does not seem to work, only kaocha.hooks/post-run, like we tried earlier. Kaocha is not documented to have post-summary either. Are you ahead of the published code when talking about post-summary? 😅

reefersleep22:08:16

Hm, no, I see it in the source.

reefersleep22:08:21

Expected:
  true
Actual:
  -true +false
171 tests, 675 assertions, 1 failures.
Error encountered performing task 'run' with profile(s): 'base,system,user,provided,dev,kaocha'
Suppressed exit
Am I maybe doing something wrong that causes the code to exit before the report is printed?

plexus12:08:03

Oh you're actually right, post-summary has been around for a while for use in plugins, but it wasn't supported by the hooks plugin as a standalone hook. That's fixed in master. I'll push a release. Are you using deps.edn? In that case you can just use the version off github

plexus12:08:18

Ah I guess you're using leiningen. I'll cut a release, might not make it today but should be out by tomorrow.