This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2019-01-18
Channels
- # aleph (1)
- # announcements (2)
- # aws (4)
- # beginners (73)
- # boot (2)
- # boot-dev (3)
- # cider (6)
- # cljs-dev (40)
- # clojure (64)
- # clojure-austin (2)
- # clojure-belgium (1)
- # clojure-dev (25)
- # clojure-estonia (1)
- # clojure-europe (16)
- # clojure-italy (11)
- # clojure-nl (4)
- # clojure-spec (90)
- # clojure-sweden (2)
- # clojure-uk (105)
- # clojurescript (58)
- # core-async (10)
- # cursive (23)
- # data-science (1)
- # datascript (3)
- # datomic (14)
- # duct (11)
- # fulcro (48)
- # graphql (1)
- # hyperfiddle (3)
- # kaocha (95)
- # liberator (1)
- # lumo (6)
- # nrepl (1)
- # off-topic (14)
- # onyx (2)
- # overtone (8)
- # portkey (3)
- # re-frame (31)
- # reagent (6)
- # shadow-cljs (185)
- # sql (12)
- # tools-deps (6)
- # vim (6)
- # yada (224)
I guess what you want is a kind of decorator on the reporter, so write a plugin with a config
hook that takes the resolved reporter, and wraps it in a function that rewrites certain clojure.test messages
hi y’all … I whipped up a terminal-notifier/libnotify post-run hook, in case that’s useful to anyone: https://gist.github.com/rgm/7cc2aaf2a131f637940ffd1d4f0bce05
seems to work but critique/ideas welcome, in particular this hacky bit of whatever for which I presume a prettier function already exists, but that I wasn’t able to find: https://gist.github.com/rgm/7cc2aaf2a131f637940ffd1d4f0bce05#file-kaocha_notifier-clj-L28-L37
Hi @rgm, this is great! You should be able to use kaocha.result/testable-totals
and kaocha.result/failed?
.
How would you feel about turning this into a plugin? You're almost there, just need to change your last notification-hook into a (defplugin ... (post-run ...))
For anything that's worth distributing I recommend making it a plugin, hooks are really for end users that want to locally patch/hack something in.
here's your code converted to a plugin: https://gist.github.com/plexus/e5befecc5b3f950d67944dd63401c580
;;deps.edn
{:deps {rgm/kaocha-notifier {:git/url ":e5befecc5b3f950d67944dd63401c580.git", :sha "1aa88253c2065568198478e9cd80e80e54210d63"}}}
@plexus yes, I mean to reverse the meaning of (is (= expected actual))
to (is (= actual expected))
... and I hope to do it without writing a custom test reporter. 🙂 I would think I could either hook into some place to rewrite the test definitions. In any case, it would be good to do so before the diffs are done in the reporter. Any ideas?
I think you need to look at the clojure.test/assert-expr
multimethod. This defaults to calling clojure.test/assert-predicate
. Kaocha adds an implementation that does the single-arity-= check, then delegates to assert-predicate
(defmethod t/assert-expr '= [msg form]
(if (= 2 (count form))
`(t/do-report {:type ::one-arg-eql
:message "Equality assertion expects 2 or more values to compare, but only 1 arguments given."
:expected '~(concat form '(arg2))
:actual '~form})
(t/assert-predicate msg form)))
You could overwrite this with your own version that changes form
before passing it on.
Aha, I see. I presume I would then overwrite your (excellent) feature then? Would an alternative be to add a hook with the data generated by clojure.test, just prior to the reporter getting it?
you can copy those two lines above to retain that excellent feature 🙂 The alternative would indeed be to manipulate the map that gets passed to clojure.test/report
, which you could do by wrapping the reporter as I suggested earlier. It'll take a bit of poking around though, you'll have to change the :expected
and :actual
keys in that map, but you can't just swap them.
(defmethod print-expr '= [m]
(let [printer (output/printer)]
;; :actual is of the form (not (= ...))
(if (and (not= (:type m) ::one-arg-eql)
(seq? (second (:actual m)))
(> (count (second (:actual m))) 2))
(let [[_ expected & actuals] (-> m :actual second)]
(output/print-doc
[:span
"Expected:" :line
[:nest (output/format-doc expected printer)]
:break
"Actual:" :line
(into [:nest]
(interpose :break)
(for [actual actuals]
(output/format-doc ((jit lambdaisland.deep-diff/diff) expected actual)
printer)))]))
(output/print-doc
[:span
"Expected:" :line
[:nest (output/format-doc (:expected m) printer)]
:break
"Actual:" :line
[:nest (output/format-doc (:actual m) printer)]]))))
Aha, okay, I thought the report was the one writing output to the terminal - so wrapping it would be too late.
so you'll have to take the value of :actual
, which will look like (not (= ...))
, and update it
Good luck! It wouldn't be a bad idea to add a report
hook (or pre-report
) so people can mess with those messages before they reach the reporter
we could easily do that in kaocha.monkey-patch
(see my earlier comment about clojure.test
😉 )
If it's a very easy change for you to do, then that's okay - otherwise I'm happy to do the work and submit a PR 🙂
I'd appreciate a PR with a test, in these final weeks of the funding I really hope to encourage people to start contributing, so it'll be good for people to see I'm not the only dev
@plexus running just the kaocha unit tests with --watch
enabled seems to fail. Running without watch is 👍
this is the challenging thing with the thing testing itself, sometimes hard to keep configuration and state isolated
aye, I was thinking the same thing. A couple friends of mine made an excellent testing library for JS (Buster.JS) which has much the same architecture as yours (very extensible with plugins and hooks) - and they had the misfortune of testing this huge thing without using their own testing library. 😛
seems it's not getting a file/line for the error event... This detecting of file/line is tricky. Not sure what's going on here, I'm just gonna loosen the test a bit for now.
at some point I hope to better study the prior art on this stuff and maybe rewrite that whole bit.
it seems to me that testing the monkey-patch would need a feature test. The hooks_plugin.feature only has one test, testing the hooks infrastructure via the pre-test hook. Do you have any thoughts on where I could add a test for :kaocha.hooks/pre-report
?
I guess you did pick a tricky thing to test. I'd still add a monkey_patch_test.clj
, even though it's tested indirectly.
something like this
(defplugin kaocha.monkey-patch-test/test-plugin
(pre-test [m]
(assoc m :been-here true)))
(binding [plugin/*current-chain* (plugin/load-all [:kaocha.monkey-patch-test/test-plugin])]
(with-redefs [clojure.test/report (fn [m] (is (:been-here m)))]
(clojure.test/do-report {:type :pass})))
(binding [plugin/*current-chain* [{:pre-test (fn [m] (assoc m :been-here true))}]]
(with-redefs [clojure.test/report (fn [m] (is (:been-here m)))]
(clojure.test/do-report {:type :pass})))
@plexus what are these reductions in kaocha.plugin.hooks/defplugin
? I'm unsure how to handle pre-report
here. Something like (reduce #(%2 %1) results (:kaocha.hooks/pre-report results))
(or just m
instead of results
?)
These reduce calls are just there to pass the value through every plugin that implements the given hook
When you call run-hook you give it a hook name, a starting value, and possibly extra arguments. In this case you don't need extra arguments, just m
hmm, yes, possibly 🙂 the general workings of the plugin chain and hooks are clear to me, but I'm a bit confused about where the hooks are to be found. I think I got it right, tho. I'll open a PR, and you can take a look.
yeah I'm sorry I was in a cab, I didn't realize you were talking about the hooks plugin. Hooks are configured in tests.edn
so they are on the config
(or test-plan
). Anyway I commented on the PR.
Are there any plans to add support for running clojure.spec.test.alpha/check
“tests” with reporting etc into kaocha?
it hasn't been explored yet. Feel free to write a ticket with a proposal with how you think this should behave.
Ok I’ve written up a brief proposal here: https://github.com/lambdaisland/kaocha/issues/45
@U06HHF230 FWIW, I have a small lib here that provides a little help around testing fdefs: https://cljdoc.org/d/respeced/respeced/0.0.1/doc/readme
It doesn’t do anything intelligent around report except relying on the behavior of clojure.test
yes I’m using it after you mentioned last week 🙂
I guess that is the other option, is use that with clojure.test… though I think you’re very limited with the error reporting if you’re wrapping check
with clojure.test/is
.
If there was a native runner that ran specs independently of clojure.test wrappers you’d get better reporting, and I think for less boilerplate.
hehe easily done… I did mean to mention it on the ticket… as I thought your check wrapper did a little more than it actually does.
I guess such a runner could also be part of a more focused lib like respeced or a plugin for koacha
if it’s more general than koacha people could use it in their everyday clojure.tests as well
I’d hope it would ship out of the box with kaocha — as spec is effectively “a core” library (though not literally) like clojure.test
We may also want a richer st/summarise-results
for my personal use I’m fine with the behavior of respeced right now. when there’s an issue, if I have enough information to fix it. but then again, I never had these high demands of pretty error reporting 😉
As I was saying the other day though - I do wonder if there’s a usecase for a conform
testing feature. You can use an fdef
for that… but if you are specing a datastructure and not a function, you may want to test that… via s/valid?
and explain-*
what I would probably do is use s/explain-data directly in the test and not call s/valid at all
yeah… again it becomes hard to get a decent error out though — hence why I think there’s a need for a custom runner. I may just be doing it wrong though. 🙂
well, in the end you’re comparing some seq of maps to what you expect it to be and then it’s up to the testing framework to figure out how to print the difference in case it fails
I’ve not thought about it too much — and could easily have just been doing it wrong but the problem is that explain data is quite detailed… so you want it prettified somewhat. It’s ok at a repl when you get ex-data
pretty printed in cider for you… but in CI you get a lot of noise printed. Then clojure.test
assert wants to print the errors in a particular way; so the spec errors aren’t part of the clojure.test
error. They end up just being a println
or something.
In the past I’ve resorted to hacks like (is (= "Success!\n" (s/explain-str ::spec data)))
just to get a half decent error out.
@plexus oh wow, thanks so much re: https://gist.github.com/plexus/e5befecc5b3f950d67944dd63401c580
(though now that I think about it for a sec given that a gist is a repo under the hood it seems logical)
I think I’d like to clojar it up but might not have the time for a while; even if distribution via gist feels ephemeral at least it’s a way to put stuff out for other people as an example.