This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2019-01-18
Channels
- # aleph (1)
- # announcements (2)
- # aws (4)
- # beginners (73)
- # boot (2)
- # boot-dev (3)
- # cider (6)
- # cljs-dev (40)
- # clojure (64)
- # clojure-austin (2)
- # clojure-belgium (1)
- # clojure-dev (25)
- # clojure-estonia (1)
- # clojure-europe (16)
- # clojure-italy (11)
- # clojure-nl (4)
- # clojure-spec (90)
- # clojure-sweden (2)
- # clojure-uk (105)
- # clojurescript (58)
- # core-async (10)
- # cursive (23)
- # data-science (1)
- # datascript (3)
- # datomic (14)
- # duct (11)
- # fulcro (48)
- # graphql (1)
- # hyperfiddle (3)
- # kaocha (95)
- # liberator (1)
- # lumo (6)
- # nrepl (1)
- # off-topic (14)
- # onyx (2)
- # overtone (8)
- # portkey (3)
- # re-frame (31)
- # reagent (6)
- # shadow-cljs (185)
- # sql (12)
- # tools-deps (6)
- # vim (6)
- # yada (224)
I guess what you want is a kind of decorator on the reporter, so write a plugin with a config
hook that takes the resolved reporter, and wraps it in a function that rewrites certain clojure.test messages
hi yāall ā¦ I whipped up a terminal-notifier/libnotify post-run hook, in case thatās useful to anyone: https://gist.github.com/rgm/7cc2aaf2a131f637940ffd1d4f0bce05
seems to work but critique/ideas welcome, in particular this hacky bit of whatever for which I presume a prettier function already exists, but that I wasnāt able to find: https://gist.github.com/rgm/7cc2aaf2a131f637940ffd1d4f0bce05#file-kaocha_notifier-clj-L28-L37
Hi @rgm, this is great! You should be able to use kaocha.result/testable-totals
and kaocha.result/failed?
.
How would you feel about turning this into a plugin? You're almost there, just need to change your last notification-hook into a (defplugin ... (post-run ...))
For anything that's worth distributing I recommend making it a plugin, hooks are really for end users that want to locally patch/hack something in.
here's your code converted to a plugin: https://gist.github.com/plexus/e5befecc5b3f950d67944dd63401c580
;;deps.edn
{:deps {rgm/kaocha-notifier {:git/url ":e5befecc5b3f950d67944dd63401c580.git", :sha "1aa88253c2065568198478e9cd80e80e54210d63"}}}
@plexus yes, I mean to reverse the meaning of (is (= expected actual))
to (is (= actual expected))
... and I hope to do it without writing a custom test reporter. š I would think I could either hook into some place to rewrite the test definitions. In any case, it would be good to do so before the diffs are done in the reporter. Any ideas?
I think you need to look at the clojure.test/assert-expr
multimethod. This defaults to calling clojure.test/assert-predicate
. Kaocha adds an implementation that does the single-arity-= check, then delegates to assert-predicate
(defmethod t/assert-expr '= [msg form]
(if (= 2 (count form))
`(t/do-report {:type ::one-arg-eql
:message "Equality assertion expects 2 or more values to compare, but only 1 arguments given."
:expected '~(concat form '(arg2))
:actual '~form})
(t/assert-predicate msg form)))
You could overwrite this with your own version that changes form
before passing it on.
Aha, I see. I presume I would then overwrite your (excellent) feature then? Would an alternative be to add a hook with the data generated by clojure.test, just prior to the reporter getting it?
you can copy those two lines above to retain that excellent feature š The alternative would indeed be to manipulate the map that gets passed to clojure.test/report
, which you could do by wrapping the reporter as I suggested earlier. It'll take a bit of poking around though, you'll have to change the :expected
and :actual
keys in that map, but you can't just swap them.
(defmethod print-expr '= [m]
(let [printer (output/printer)]
;; :actual is of the form (not (= ...))
(if (and (not= (:type m) ::one-arg-eql)
(seq? (second (:actual m)))
(> (count (second (:actual m))) 2))
(let [[_ expected & actuals] (-> m :actual second)]
(output/print-doc
[:span
"Expected:" :line
[:nest (output/format-doc expected printer)]
:break
"Actual:" :line
(into [:nest]
(interpose :break)
(for [actual actuals]
(output/format-doc ((jit lambdaisland.deep-diff/diff) expected actual)
printer)))]))
(output/print-doc
[:span
"Expected:" :line
[:nest (output/format-doc (:expected m) printer)]
:break
"Actual:" :line
[:nest (output/format-doc (:actual m) printer)]]))))
Aha, okay, I thought the report was the one writing output to the terminal - so wrapping it would be too late.
so you'll have to take the value of :actual
, which will look like (not (= ...))
, and update it
Good luck! It wouldn't be a bad idea to add a report
hook (or pre-report
) so people can mess with those messages before they reach the reporter
we could easily do that in kaocha.monkey-patch
(see my earlier comment about clojure.test
š )
If it's a very easy change for you to do, then that's okay - otherwise I'm happy to do the work and submit a PR š
I'd appreciate a PR with a test, in these final weeks of the funding I really hope to encourage people to start contributing, so it'll be good for people to see I'm not the only dev
@plexus running just the kaocha unit tests with --watch
enabled seems to fail. Running without watch is š
this is the challenging thing with the thing testing itself, sometimes hard to keep configuration and state isolated
aye, I was thinking the same thing. A couple friends of mine made an excellent testing library for JS (Buster.JS) which has much the same architecture as yours (very extensible with plugins and hooks) - and they had the misfortune of testing this huge thing without using their own testing library. š
and a good reminder that we should find a way to minimize the results of deep-diff
š
seems it's not getting a file/line for the error event... This detecting of file/line is tricky. Not sure what's going on here, I'm just gonna loosen the test a bit for now.
at some point I hope to better study the prior art on this stuff and maybe rewrite that whole bit.
it seems to me that testing the monkey-patch would need a feature test. The hooks_plugin.feature only has one test, testing the hooks infrastructure via the pre-test hook. Do you have any thoughts on where I could add a test for :kaocha.hooks/pre-report
?
I guess you did pick a tricky thing to test. I'd still add a monkey_patch_test.clj
, even though it's tested indirectly.
something like this
(defplugin kaocha.monkey-patch-test/test-plugin
(pre-test [m]
(assoc m :been-here true)))
(binding [plugin/*current-chain* (plugin/load-all [:kaocha.monkey-patch-test/test-plugin])]
(with-redefs [clojure.test/report (fn [m] (is (:been-here m)))]
(clojure.test/do-report {:type :pass})))
(binding [plugin/*current-chain* [{:pre-test (fn [m] (assoc m :been-here true))}]]
(with-redefs [clojure.test/report (fn [m] (is (:been-here m)))]
(clojure.test/do-report {:type :pass})))
@plexus what are these reductions in kaocha.plugin.hooks/defplugin
? I'm unsure how to handle pre-report
here. Something like (reduce #(%2 %1) results (:kaocha.hooks/pre-report results))
(or just m
instead of results
?)
These reduce calls are just there to pass the value through every plugin that implements the given hook
When you call run-hook you give it a hook name, a starting value, and possibly extra arguments. In this case you don't need extra arguments, just m
hmm, yes, possibly š the general workings of the plugin chain and hooks are clear to me, but I'm a bit confused about where the hooks are to be found. I think I got it right, tho. I'll open a PR, and you can take a look.
yeah I'm sorry I was in a cab, I didn't realize you were talking about the hooks plugin. Hooks are configured in tests.edn
so they are on the config
(or test-plan
). Anyway I commented on the PR.
Are there any plans to add support for running clojure.spec.test.alpha/check
ātestsā with reporting etc into kaocha?
it hasn't been explored yet. Feel free to write a ticket with a proposal with how you think this should behave.
Ok Iāve written up a brief proposal here: https://github.com/lambdaisland/kaocha/issues/45
@U06HHF230 FWIW, I have a small lib here that provides a little help around testing fdefs: https://cljdoc.org/d/respeced/respeced/0.0.1/doc/readme
It doesnāt do anything intelligent around report except relying on the behavior of clojure.test
yes Iām using it after you mentioned last week š
I guess that is the other option, is use that with clojure.testā¦ though I think youāre very limited with the error reporting if youāre wrapping check
with clojure.test/is
.
If there was a native runner that ran specs independently of clojure.test wrappers youād get better reporting, and I think for less boilerplate.
hehe easily doneā¦ I did mean to mention it on the ticketā¦ as I thought your check wrapper did a little more than it actually does.
I guess such a runner could also be part of a more focused lib like respeced or a plugin for koacha
if itās more general than koacha people could use it in their everyday clojure.tests as well
Iād hope it would ship out of the box with kaocha ā as spec is effectively āa coreā library (though not literally) like clojure.test
We may also want a richer st/summarise-results
for my personal use Iām fine with the behavior of respeced right now. when thereās an issue, if I have enough information to fix it. but then again, I never had these high demands of pretty error reporting š
As I was saying the other day though - I do wonder if thereās a usecase for a conform
testing feature. You can use an fdef
for thatā¦ but if you are specing a datastructure and not a function, you may want to test thatā¦ via s/valid?
and explain-*
what I would probably do is use s/explain-data directly in the test and not call s/valid at all
yeahā¦ again it becomes hard to get a decent error out though ā hence why I think thereās a need for a custom runner. I may just be doing it wrong though. š
well, in the end youāre comparing some seq of maps to what you expect it to be and then itās up to the testing framework to figure out how to print the difference in case it fails
Iāve not thought about it too much ā and could easily have just been doing it wrong but the problem is that explain data is quite detailedā¦ so you want it prettified somewhat. Itās ok at a repl when you get ex-data
pretty printed in cider for youā¦ but in CI you get a lot of noise printed. Then clojure.test
assert wants to print the errors in a particular way; so the spec errors arenāt part of the clojure.test
error. They end up just being a println
or something.
In the past Iāve resorted to hacks like (is (= "Success!\n" (s/explain-str ::spec data)))
just to get a half decent error out.
@plexus oh wow, thanks so much re: https://gist.github.com/plexus/e5befecc5b3f950d67944dd63401c580
(though now that I think about it for a sec given that a gist is a repo under the hood it seems logical)
I think Iād like to clojar it up but might not have the time for a while; even if distribution via gist feels ephemeral at least itās a way to put stuff out for other people as an example.