Fork me on GitHub

hi @caio, this is great! I didn't even know about Greenlight, or it would have been on my list. You're also the first person to contribute a new test type, so thank you and congratulations! I hope the process wasn't too painful.


A few remarks:


Please add a bin/kaocha as described in the documentation. This is a Kaocha convention, when you open a Kaocha project you should always be able to run all tests with bin/kaocha without having to figure out anything else.


I do have some questions w.r.t. your reporter, you're directly forwarding greenlight events to clojure.test, and then extending clojure.test/report. I would suggest not doing that. Kaocha forwards events it doesn't know about to the original clojure.test/report multimethod, but that's really a fallback for people using Kaocha with legacy assertions+reporters.


Instead the recommended thing to do is to emit custom event types, but to use Kaocha's keyword hierarchy to map them to things Kaocha knows about. To get custom formatting of errors/failures you can extend and


that said I see you're aiming for very specific output, maybe Kaocha doesn't yet have all the affordances you need. I'll have to play a bit with greenlight to get a better sense for that.


In ns you're catching all load errors and turning them into warnings, I know that's what the clojure.test implementation used to do as well, but I would no longer recommend it. Once this is merged errors can just bubble up, and they'll be reported as a failing test suite.


Finally I would suggest using a different namespace, since this is a contributed project rather than a core Kaocha project, so maybe name your test types something like :caioaao.kaocha/greenlight , :caioaao.kaocha.greenlight/ns, :caioaao.kaocha.greenlight/var and namespaces respectively. (it would also be a good idea to think about which maven org you would publish this under, and make your namespace names line up)

Christian Johansen09:02:32

FWIW, I'm not ecstatic about the bin/kaocha convention, and I don't use it. clojure -A:test is just as good a convention IMO, and my projects all have Makefiles anyway. In any given project I'd expect one of lein test clojure -A:test make test to work


clojure -A:test doesn't work in leiningen or boot projects

Christian Johansen09:02:17

make test would work for everyone 🙂


it also gives a project maintainer a place to add setup like setting env vars (perhaps later on). Finally we might conceivable start distributing a more intelligent wrapper script in the future that further smooths out the difference between boot and the rest.

Christian Johansen09:02:44

the latter is in my opinion an argument for not having custom binstubs. that wrapper is guaranteed to do other things than the binstubs people create now


less than half of Clojure projects use make, so my muscle memory can't rely on it

Christian Johansen09:02:24

I just don't see any value in trying to introduce a new convention that is very specific to just one tool

Christian Johansen09:02:14

and even if kaocha did ship a wrapper script that did smart things I would prefer not to rely on anything outside of my project configuration (e.g. global binaries from $PATH)

Christian Johansen09:02:37

but these are just my opinions, take them with a grain of salt 🙂


it's a convention that is strongly recommended from the project's side so that everyone is on the same page. All documentation will reference bin/kaocha rather than any specific tool, and people will be taught to expect a bin/kaocha script when they clone a kaocha-based project, so they don't have to look around how exactly kaocha is run in this project. If you don't want to add it then you're absolutely free not to do so. You can also name tests.edn and explain in the README to run clojure -A:dev:test:dev-cljs --config-file . Do what you must.

Christian Johansen09:02:11

I fully support having a common ground to use in docs etc


the clojure CLI tools aliases are a very good argument for a wrapper BTW. projects start out with a :test aliases but before you know it you need to add three or four to make things run, and if you add one that doesn't exist clojure CLI will barf, so you can't just use -A:dev:test to be "safe"

Christian Johansen09:02:51

honestly, I think the best (and most realistic) convention is to make it very visible in the Readme how to run tests, and that it can be done on a fresh checkout

Christian Johansen09:02:19

going to the Readme to figure out if this is a kaocha-project is just as much work as going to the Readme to discover you can type make test

Christian Johansen09:02:28

but bin/kaocha makes a whole lotta sense for docs etc

Christian Johansen09:02:42

I'm not going to battle this out anymore, was just an opinion, feel free to disregard

Christian Johansen09:02:16

wasn't my intention to be hostile


no worries, no offence taken. Apologies if my response was stronger than called for. I realize this may raise some objections, people don't like to be told how to organize their projects. I still plan to push for it and see if people can get used to it. We'll see if it catches on or not.

Christian Johansen09:02:18

I'm all for conventions where practical. I have no need to name my config file, tests.edn is just fine 🙂


This is of course the bottom line, if everybody would already do this I would be extremely happy.


I'm curious if you have some tips on using Kaocha in/with Cursive?


I'm not really a cursive user, but you should be able to make due with Kaocha's repl interface


Tim's manual describes the leiningen plugin lein-test-refresh that he uses in his workflow. That's what you call the 'Watch mode' in Kaocha?


> I just don’t see any value in trying to introduce a new convention that is very specific to just one tool Also without the wish to come across hostile, I kind of agree. A script doesn’t compose with including other aliases from deps.edn. People should be able to make the scripts specifically for their projects without relying on conventions.

😢 1

As for plugins, I wonder if there’s an equivalent of fixtures in kaocha. Right now it seems that every plugin has to re-invent how to act as a fixture around some level of test (suite, ns, var). It would be nice to see a more general approach to this. Having wrap-test-var, wrap-test-ns etc. hooks would maybe help?


var and ns are implementation details of :kaocha.type/clojure.test. Other test types will not have these. Most test types have three levels so far (e.g. clojure.test / ns / var or cucumber / feature / scenario), but that too is arbitrary. You can check however if you're dealing with a top-level testable (`:kaocha.testable.type/suite`), with a bottom-level one (`:kaocha.testable.type/test`), or with some intermediate grouping level (`:kaocha.testable.type/group`). So you would check for those in a wrap-run / pre-test / post-test hook. With the helpers in kaocha.hierarchy that should be straightforward, so I don't think adding extra hooks for those is necessary.


Yeah. The thing I was thinking about here was: Like in Ring-handlers or fixtures, you can have some state. If I did this thing in the pre-step, then I have to undo it in the post-step, but otherwise not.


And this is currently not possible with only pre and post-steps


Unless you hack some atoms into your plugin, but that’s kind of a workaround and it won’t scale with running tests in parallel


Not that I need this functionality now urgently, but I can see it being useful for doing stuff in a binding like construct, temporarily disabling instrumentation before a test and instrument again at the end of a test


Hey @U04V15CAJ, sorry for the very slow response, I had somehow missed this thread earlier.


> Like in Ring-handlers or fixtures, you can have some state. If I did this thing in the pre-step, then I have to undo it in the post-step, but otherwise not. You can store state in the testable, whatever you return from a hook gets passed further down, so you would do something like this

(defplugin foo
  (pre-test [testable test-plan]
    (assoc testable ::did-the-thing? true))
  (post-test [testable test-plan]
    (when (::did-the-thing? testable)


aah, that makes sense. thanks 🙂


(and I’m finally getting the hang of the name, I don’t write koacha that often anymore :-D))

❤️ 1

Another issue: when you write a kaocha plugin, how does it work in CLJS? I want to turn on speculative specs in CLJS, but does that work automatically when I write my plugin in .cljc?


There are no mechanisms in place yet for hooking into CLJS, plugins run in Clojure. Like you said, it's still early days for kaocha-cljs.

Christian Johansen13:02:00

I'm running tests with kaocha and cljs (nodejs), how can I print stuff? (println) seems to go missing, as does js/console.log, and I can't seem to find the node stdout log anywhere


hi @U9MKYDN4Q, output from JavaScript gets forwarded to Clojure stdout/stderr. Maybe output capturing is swallowing it? Try running with --no-capture-output. If it still doesn't show up then a reproducible case and a GH issue would be very appreciated!


@plexus thanks for the tips! I'll implement them asap. the process wasn't painful at all, but the docs could help a little bit more. I had to do some reverse engineering on the clojure.test implementation to figure some things out. and those naming conventions and reporting stuff could also be documented. other than that, it was waaay easier than I anticipated 🙂 I'm really impressed with the project


Point well taken, how test type work and the affordances provided to them by kaocha's reporters really need more documentation. I'll make some time for that.

🆒 1

I second the fixtures plugin thing. that would be nice notbad but I also don't like having the tests depend on the test runner, so it looks kinda hard without some sort of abstraction leak


I'll also check the existing report events and see if I can do something similar to what the authors of greenlight wanted for the reports