Fork me on GitHub
#kaocha
<
2019-02-01
>
plexus01:02:03

hi @caio, this is great! I didn't even know about Greenlight, or it would have been on my list. You're also the first person to contribute a new test type, so thank you and congratulations! I hope the process wasn't too painful.

plexus01:02:53

A few remarks:

plexus01:02:44

Please add a bin/kaocha as described in the documentation. This is a Kaocha convention, when you open a Kaocha project you should always be able to run all tests with bin/kaocha without having to figure out anything else.

plexus02:02:15

I do have some questions w.r.t. your reporter, you're directly forwarding greenlight events to clojure.test, and then extending clojure.test/report. I would suggest not doing that. Kaocha forwards events it doesn't know about to the original clojure.test/report multimethod, but that's really a fallback for people using Kaocha with legacy assertions+reporters.

plexus02:02:54

Instead the recommended thing to do is to emit custom event types, but to use Kaocha's keyword hierarchy to map them to things Kaocha knows about. To get custom formatting of errors/failures you can extend kaocha.report/fail-summary and kaocha.report/print-expr

plexus02:02:27

that said I see you're aiming for very specific output, maybe Kaocha doesn't yet have all the affordances you need. I'll have to play a bit with greenlight to get a better sense for that.

plexus02:02:11

In ns you're catching all load errors and turning them into warnings, I know that's what the clojure.test implementation used to do as well, but I would no longer recommend it. Once this is merged errors can just bubble up, and they'll be reported as a failing test suite. https://github.com/lambdaisland/kaocha/pull/68

plexus02:02:40

Finally I would suggest using a different namespace, since this is a contributed project rather than a core Kaocha project, so maybe name your test types something like :caioaao.kaocha/greenlight , :caioaao.kaocha.greenlight/ns, :caioaao.kaocha.greenlight/var and namespaces respectively. (it would also be a good idea to think about which maven org you would publish this under, and make your namespace names line up)

cjohansen09:02:32

FWIW, I'm not ecstatic about the bin/kaocha convention, and I don't use it. clojure -A:test is just as good a convention IMO, and my projects all have Makefiles anyway. In any given project I'd expect one of lein test clojure -A:test make test to work

plexus09:02:34

clojure -A:test doesn't work in leiningen or boot projects

cjohansen09:02:17

make test would work for everyone 🙂

plexus09:02:59

it also gives a project maintainer a place to add setup like setting env vars (perhaps later on). Finally we might conceivable start distributing a more intelligent wrapper script in the future that further smooths out the difference between boot and the rest.

cjohansen09:02:44

the latter is in my opinion an argument for not having custom binstubs. that wrapper is guaranteed to do other things than the binstubs people create now

plexus09:02:55

less than half of Clojure projects use make, so my muscle memory can't rely on it

cjohansen09:02:24

I just don't see any value in trying to introduce a new convention that is very specific to just one tool

cjohansen09:02:14

and even if kaocha did ship a wrapper script that did smart things I would prefer not to rely on anything outside of my project configuration (e.g. global binaries from $PATH)

cjohansen09:02:37

but these are just my opinions, take them with a grain of salt 🙂

plexus09:02:20

it's a convention that is strongly recommended from the project's side so that everyone is on the same page. All documentation will reference bin/kaocha rather than any specific tool, and people will be taught to expect a bin/kaocha script when they clone a kaocha-based project, so they don't have to look around how exactly kaocha is run in this project. If you don't want to add it then you're absolutely free not to do so. You can also name tests.edn and explain in the README to run clojure -A:dev:test:dev-cljs --config-file . Do what you must.

cjohansen09:02:11

I fully support having a common ground to use in docs etc

plexus09:02:43

the clojure CLI tools aliases are a very good argument for a wrapper BTW. projects start out with a :test aliases but before you know it you need to add three or four to make things run, and if you add one that doesn't exist clojure CLI will barf, so you can't just use -A:dev:test to be "safe"

cjohansen09:02:51

honestly, I think the best (and most realistic) convention is to make it very visible in the Readme how to run tests, and that it can be done on a fresh checkout

cjohansen09:02:19

going to the Readme to figure out if this is a kaocha-project is just as much work as going to the Readme to discover you can type make test

cjohansen09:02:28

but bin/kaocha makes a whole lotta sense for docs etc

cjohansen09:02:42

I'm not going to battle this out anymore, was just an opinion, feel free to disregard

cjohansen09:02:16

wasn't my intention to be hostile

plexus09:02:55

no worries, no offence taken. Apologies if my response was stronger than called for. I realize this may raise some objections, people don't like to be told how to organize their projects. I still plan to push for it and see if people can get used to it. We'll see if it catches on or not.

cjohansen09:02:18

I'm all for conventions where practical. I have no need to name my config file http://chewbacca.xxx, tests.edn is just fine 🙂

plexus09:02:12

This is of course the bottom line, if everybody would already do this I would be extremely happy.

worrelsik10:02:06

I'm curious if you have some tips on using Kaocha in/with Cursive?

plexus10:02:30

I'm not really a cursive user, but you should be able to make due with Kaocha's repl interface

worrelsik10:02:29

Tim's manual describes the leiningen plugin lein-test-refresh that he uses in his workflow. That's what you call the 'Watch mode' in Kaocha?

borkdude11:02:10

> I just don’t see any value in trying to introduce a new convention that is very specific to just one tool Also without the wish to come across hostile, I kind of agree. A script doesn’t compose with including other aliases from deps.edn. People should be able to make the scripts specifically for their projects without relying on conventions.

😢 5
borkdude11:02:34

As for plugins, I wonder if there’s an equivalent of fixtures in kaocha. Right now it seems that every plugin has to re-invent how to act as a fixture around some level of test (suite, ns, var). It would be nice to see a more general approach to this. Having wrap-test-var, wrap-test-ns etc. hooks would maybe help?

plexus04:02:38

var and ns are implementation details of :kaocha.type/clojure.test. Other test types will not have these. Most test types have three levels so far (e.g. clojure.test / ns / var or cucumber / feature / scenario), but that too is arbitrary. You can check however if you're dealing with a top-level testable (`:kaocha.testable.type/suite`), with a bottom-level one (`:kaocha.testable.type/test`), or with some intermediate grouping level (`:kaocha.testable.type/group`). So you would check for those in a wrap-run / pre-test / post-test hook. With the helpers in kaocha.hierarchy that should be straightforward, so I don't think adding extra hooks for those is necessary.

borkdude12:02:05

Yeah. The thing I was thinking about here was: Like in Ring-handlers or fixtures, you can have some state. If I did this thing in the pre-step, then I have to undo it in the post-step, but otherwise not.

borkdude12:02:15

And this is currently not possible with only pre and post-steps

borkdude12:02:42

Unless you hack some atoms into your plugin, but that’s kind of a workaround and it won’t scale with running tests in parallel

borkdude12:02:34

Not that I need this functionality now urgently, but I can see it being useful for doing stuff in a binding like construct, temporarily disabling instrumentation before a test and instrument again at the end of a test

plexus12:02:34

Hey @U04V15CAJ, sorry for the very slow response, I had somehow missed this thread earlier.

plexus12:02:56

> Like in Ring-handlers or fixtures, you can have some state. If I did this thing in the pre-step, then I have to undo it in the post-step, but otherwise not. You can store state in the testable, whatever you return from a hook gets passed further down, so you would do something like this

(defplugin foo
  (pre-test [testable test-plan]
    (assoc testable ::did-the-thing? true))
  
  (post-test [testable test-plan]
    (when (::did-the-thing? testable)
      (undo-the-thing))
    testable))

borkdude12:02:01

aah, that makes sense. thanks 🙂

borkdude11:02:12

(and I’m finally getting the hang of the name, I don’t write koacha that often anymore :-D))

❤️ 5
borkdude12:02:15

Another issue: when you write a kaocha plugin, how does it work in CLJS? I want to turn on speculative specs in CLJS, but does that work automatically when I write my plugin in .cljc?

plexus04:02:07

There are no mechanisms in place yet for hooking into CLJS, plugins run in Clojure. Like you said, it's still early days for kaocha-cljs.

cjohansen13:02:00

I'm running tests with kaocha and cljs (nodejs), how can I print stuff? (println) seems to go missing, as does js/console.log, and I can't seem to find the node stdout log anywhere

plexus03:02:25

hi @christian767, output from JavaScript gets forwarded to Clojure stdout/stderr. https://github.com/lambdaisland/kaocha-cljs/blob/master/src/kaocha/type/cljs.clj#L111-L120 Maybe output capturing is swallowing it? Try running with --no-capture-output. If it still doesn't show up then a reproducible case and a GH issue would be very appreciated!

caio15:02:49

@plexus thanks for the tips! I'll implement them asap. the process wasn't painful at all, but the docs could help a little bit more. I had to do some reverse engineering on the clojure.test implementation to figure some things out. and those naming conventions and reporting stuff could also be documented. other than that, it was waaay easier than I anticipated 🙂 I'm really impressed with the project

plexus04:02:50

Point well taken, how test type work and the affordances provided to them by kaocha's reporters really need more documentation. I'll make some time for that.

🆒 5
caio15:02:30

I second the fixtures plugin thing. that would be nice notbad but I also don't like having the tests depend on the test runner, so it looks kinda hard without some sort of abstraction leak

caio15:02:44

I'll also check the existing report events and see if I can do something similar to what the authors of greenlight wanted for the reports

plexus03:02:25

hi @christian767, output from JavaScript gets forwarded to Clojure stdout/stderr. https://github.com/lambdaisland/kaocha-cljs/blob/master/src/kaocha/type/cljs.clj#L111-L120 Maybe output capturing is swallowing it? Try running with --no-capture-output. If it still doesn't show up then a reproducible case and a GH issue would be very appreciated!

plexus04:02:07

There are no mechanisms in place yet for hooking into CLJS, plugins run in Clojure. Like you said, it's still early days for kaocha-cljs.

plexus12:02:56

> Like in Ring-handlers or fixtures, you can have some state. If I did this thing in the pre-step, then I have to undo it in the post-step, but otherwise not. You can store state in the testable, whatever you return from a hook gets passed further down, so you would do something like this

(defplugin foo
  (pre-test [testable test-plan]
    (assoc testable ::did-the-thing? true))
  
  (post-test [testable test-plan]
    (when (::did-the-thing? testable)
      (undo-the-thing))
    testable))