Fork me on GitHub
#kaocha
<
2020-03-10
>
defa07:03:32

I have a shadow-cljs based react-native project and I’d like to run some tests with kaocha however I’m getting this error:

$ clj -A:test --no-capture-output
[TypeError: Cannot read property 'error__GT_str' of undefined
    at Socket.<anonymous> ([stdin]:89:38)
    at Socket.emit (events.js:321:20)
    at Socket.EventEmitter.emit (domain.js:485:12)
    at addChunk (_stream_readable.js:297:12)
    at readableAddChunk (_stream_readable.js:269:11)
    at Socket.Readable.push (_stream_readable.js:214:10)
    at TCP.onStreamRead (internal/stream_base_commons.js:186:23)

E]
Any ideas what that might cause? I tested my configuration in a minimal project and it works. After the E] the process keeps running with no output. UPDATE: this error only happens when having thheller/shadow-cljs {:mvn/version "2.8.90"} as a dependency in deps.edn. SOLVED: making use of an additional tools.deps alias solved the problem by not having shadow-cljs on the test classpath. Thanks for reading and sorry for the noise πŸ˜‰

plexus08:03:26

hey @dharrigan thanks for reporting this, I also assumed that defexpect would just work, but they are probably using custom event types which the reporter needs to know about. Not a lot of work and something we can add support for without directly having to have expectations as a dependency

dominicm08:03:38

I don't think #ref will be able to work inside of a #kaocha/v1 btw. Aero is unable to figure out the expected ordering.

dharrigan08:03:57

Thanks @plexus that would be nice!

plexus08:03:33

Not saying I'm gonna do it πŸ™‚ got a lot of things competing for my time right now, but if you file an issue I'll pitch in with what needs to happen. Would be helpful if you could run with --reporter kaocha.report/debug, and look what values the :type keys have

dharrigan08:03:33

no problemo πŸ™‚ I'll get on to that too

plexus08:03:27

@flowthing never looked into it, I agree those stacktraces aren't great. Kaocha's stacktraces in general could be much improved. Would be useful to file an issue on kaocha-cljs to track this.

flowthing10:03:59

All right, good to know, thanks. I'll file that issue.

plexus08:03:44

@dominicm that makes sense. Note that #kaocha/v1 is really just a convenience, you should be able to do kaocha --print-config > tests.edn.new and go from there (cleaning a few things up, notably the current cli arguments)

plexus08:03:44

it's more verbose, many more namespaced keys, and all defaults need to be filled in explicitly, but it's not a terrible format to work with

dominicm08:03:58

Yeah. That's one solution. You might also want to make kaocha/v1 a aero macro, that would give it control over how things are resolved inside of itself. Haven't thought about how you'd write that yet though.

plexus08:03:50

it's already being handled by aero though, right? does it have a separate concept of "macros"?

dominicm08:03:50

Gonna expand for now, but might be one to ponder if this comes up again :)

dominicm08:03:13

Aero has reader "functions" (normal ones) and reader macros (like #profile and #ref). The macros can do special things like case, etc

plexus08:03:41

@defa great to hear you got that working! shadow-cljs is not terribly well supported right now. If your code is compatible with standard clojurescript then you're all good, but if you use shadow-specific compiler extensions you may be out of luck

plexus08:03:29

@dominicm I see, I guess it gets expanded at an earlier stage. Are there any downsides/risks to switching? anything that would behave differently? seems like a no-brainer to just go ahead with

dominicm08:03:46

You'd have to think a bit harder to have it work. Presumably users want their tags to expand without #kaocha/v1 having expanded. That's probably the easiest version. Essentially as if their map was the config file itself. The complex version involves trying to expand, then trying to add in the expanded keys if possible, then trying to continue.

dominicm08:03:54

Essentially, it's more code :)

plexus08:03:40

ok, in that case it's for future consideration πŸ™‚

defa08:03:18

@plexus no, there is no complier magic going on, just shadow-cljs on the classpath and a minimal test (is false). To clarify I’m not running the tests via shadow-cljs at all but separate via clj.

defa09:03:42

Any ideas why there is no output captured even if I pass --no-capture-output to the rest runner? My test fails and starts with:

(deftest foo
  (println "Hello!")
  ...)

plexus09:03:00

that's not enough context @defa... what does your tests.edn look like? can you create a minimal reproduction?

defa09:03:07

Sure…

;; tests.edn
#kaocha/v1
    {:tests [{:id           :unit-cljs
              :type         :kaocha.type/cljs
              :test-paths   ["src/test"]
              :ns-patterns  ["-test$"]
              }]
     }

defa09:03:22

;; deps.edn
{:deps  {}
 :paths ["src/main" "src/test"]
 :aliases
        {:test
         {:extra-deps {lambdaisland/kaocha      {:mvn/version "0.0-590"}
                       lambdaisland/kaocha-cljs {:mvn/version "0.0-71"}}
          :main-opts  ["-m" "kaocha.runner"]}}}

defa09:03:30

(ns foo.foo-test
  (:require [clojure.test :refer [deftest testing is are]]))

(deftest fail
  (println "hello!")
  (testing "will fail"
    (is false "failed by intention")))

defa09:03:41

@plexus I can upload a repo to gihub if that helps? Otherwise the relevant parts above.

plexus10:03:30

ok, so this is for clojurescript tests using node?. I would run with the debug options and see if you see any stdout messages coming back from the JS runtime. One possible explanation would be that the process exits before we receive the stdout. Do you see printlns outside the deftest?

defa10:03:48

Nope, println outside of deftest does not show up. Which debug options do you mean?

defa10:03:21

(js/console.log "...") does produce some output, inside and outside of deftest but not in an ASCII art β€˜box’.

defa10:03:20

… and yes, this is ClojureScript and I suppose node is involved in the background. I’m testing code that usually gets compiled by shadow-cljs with a react-native target.

plexus10:03:53

there are debug instructions in the kaocha-cljs README, which should produce some very verbose output.

plexus10:03:26

> (js/console.log "...") does produce some output, inside and outside of deftest but not in an ASCII art β€˜box’. Maybe you just need an (enable-console-print!)

defa10:03:37

@plexus okay, that helps :thumbsup:

defa10:03:08

However it does not look like in the docs:

FAIL in sample-test/stdout-fail-test (sample_test.clj:10)
Expected:
  :same
Actual:
  -:same +:not-same
╭───── Test output ───────────────────────────────────────────────────────
β”‚ Bu yi le hu?
╰─────────────────────────────────────────────────────────────────────────
2 tests, 2 assertions, 1 failures.
… but I’m okay with that.

defa10:03:43

The β€œTest output” ASCII-frame is missing.

plexus10:03:15

is that with or without output capturing? you'll only get the frame when output capturing is on

defa12:03:21

I run kaocha like this:

clj -A:test --watch --no-capture-output
… oh shit… I’m so stupid… sorry. I misunderstood the semantics of the command line option πŸ™ˆ

defa12:03:27

@plexus works just fine.

defa12:03:03

Sorry if I was wasting your time… too much work to few sleep.

defa12:03:17

… even without (enable-console-print!)

defa12:03:01

… but not always. In my real project

(enable-console-print!)
is necessary even without --no-capture-output

plexus12:03:35

> Sorry if I was wasting your time… No problem at all! I like to make sure people are able to make the best of Kaocha

defa17:03:14

@plexus thanks! I really like Kaocha, good work! Will be using it daily from now on.

dharrigan09:03:15

Hi @plexus I saw your comment. I think I can do that πŸ™‚

dharrigan11:03:23

@plexus is it reasonable to pull in expectations in the test alias (in deps.edn) for kaocha so that I can test the =? macro?

plexus11:03:54

I guess that's acceptable. How big is it?

plexus11:03:59

What does =? do?

dharrigan11:03:26

6.5K Mar 3 13:44 clojure-test-1.2.1.jar

dharrigan11:03:40

under 7KB πŸ™‚

dharrigan11:03:56

❯ unzip -l clojure-test-1.2.1.jar                        
Archive:  clojure-test-1.2.1.jar
  Length      Date    Time    Name
---------  ---------- -----   ----
        0  2019-12-09 09:41   expectations/
        0  2019-12-09 09:41   expectations/clojure/
    13888  2019-12-09 09:17   expectations/clojure/test.clj
        0  2019-12-09 09:41   META-INF/
       57  2019-12-09 09:41   META-INF/MANIFEST.MF
        0  2019-12-09 09:41   META-INF/maven/
        0  2019-12-09 09:41   META-INF/maven/expectations/
        0  2019-12-09 09:41   META-INF/maven/expectations/clojure-test/
      114  2019-12-09 09:41   META-INF/maven/expectations/clojure-test/pom.properties
     1895  2019-12-09 09:41   META-INF/maven/expectations/clojure-test/pom.xml
---------                     -------
    15954                     10 files

plexus11:03:35

yeah that's fine

dominicm11:03:56

Would it make sense for kaocha.report/fail-summary to look at t/testing-contexts if there's none in the message? They're only in the message from :summary

dominicm11:03:33

And that's because kaocha.history puts it there

plexus12:03:05

what's the use case/problem statement @dominicm ?

dominicm12:03:17

I'm writing a custom reporter, and I want to include details about the failure when it happens, as opposed to during the summary. This report has to be this format due to CI constraints.

plexus12:03:31

right, you basically want to undo what kaocha does, i.e. capture failure events and only print the details at the end.

dominicm12:03:17

They can print at the end too. But my error reporter needs to print context.

plexus12:03:36

and so you're calling fail-summary from your reporter on for instance a :fail event?

plexus12:03:58

I would suggest putting :testing-contexts in the map when you call it, rather than the other way around

dominicm12:03:36

That's what I'm doing for now, although it did feel like reaching as implementation details.

plexus12:03:18

building kaocha's reporting on top of clojure.test is one of the few things I regret. It really should be a separate adapter layer that knows about these clojure.test vars and makes the messages self-contained

plexus12:03:27

so that's the direction in general I would prefer to move to

plexus12:03:24

so think of it as kaocha's reporting API, where the messages are a generalization of clojure.test's messages

dominicm12:03:29

Heh, okay :) I guess everywhere else would be using the api wrong by not providing those keys?

dominicm12:03:40

Kaocha isn't picking up that some of my tests have meta when using focus-meta. I'd like to debug why. The printed plan is way too big. What else can I look at?

dominicm12:03:11

I've run the kaocha.load stuff that kaocha clojure test does, and I can see that kaocha has found the metadata on the tests, so I'm doubly confused :(

plexus12:03:28

Maybe poke at it from a REPL? (kaocha.repl/test-plan), then filter as needed

plexus12:03:09

do you have other focus/skip config? maybe they're interacting in unexpected ways

dominicm12:03:57

All of my test groups have a focus/skip. One of them has skip set to all of the other metadatas

plexus12:03:39

with test group do you mean test suite?

plexus12:03:09

and do you have test suites that have the same test-paths but are differentiated by skip/focus?

dominicm12:03:09

Yes, probably :)

dominicm12:03:16

Yep, that's exactly right.

plexus12:03:32

ok, that's an anti-pattern and will not work as you would expect

dominicm12:03:20

Will read before I ask my next question

plexus12:03:09

one problem with this is that the same test-id will occur multiple times in the test plan, we should really have a warning for that, as that will confuse plugins like the filtering plugin

dominicm12:03:10

That would solve my problem of wanting to make the groups explicitly, especially as I want to define an "inverse" (not :external, not :integration, not :devcards...)

dominicm12:03:28

The problem is that for me, unit tests are "the ones that aren't the others"

plexus12:03:42

yeah, wouldn't be exceptionally hard to implement either... I'm guesing a ~100 line plugin

plexus12:03:15

although the real recommendation would be to split your tests on the filesystem test/unit/ / test/integration etc

dominicm12:03:06

Not sure about getting buy in for that right now. The task would be difficult.

dominicm12:03:44

I quite like the idea of having a mega suite, and just defining selectors on it. It's probably impractical, but has a good hierarchy in my brain.

dominicm12:03:31

This does explain why you run all tests if no focus meta matches. I was getting really annoyed with that :)

dominicm12:03:27

Hmm, that would still be useful for running a group of tests across suites. Eg run all unit tests, if there's none in cljs then it doesn't matter.

plexus12:03:29

If I put together a PoC would you be able to get a little contribution going on the opencollective?

dominicm12:03:20

I did start floating the idea at the client about that. But they're still figuring out how to let people contribute bug fixes to oss.

plexus12:03:04

just to be clear I'm talking about a monetary contribution, not a code contribution

dominicm12:03:52

I know. Sorry, I was trying to point out that they all haven't solved the problem of that. They are totally unprepared for funding.

plexus12:03:04

I see... too bad

plexus12:03:08

in that case I would consider adding a plugin inside the project and doing it yourself, you can use the filter plugin for inspiration

plexus12:03:46

it's really just recursively walking the test-plan, and marking testables as :kaocha.testable/skip true or not

dominicm12:03:26

Yeah. Makes sense. I might just inflict a very long cli on people in the meantime..

plexus12:03:37

that also works

dominicm12:03:51

Just to check, is it okay for two suites (clj/cljs) to have overlapping paths?

plexus12:03:03

clj / cljs is fine because the ids are different, the cljs testables are prefixed with cljs:

plexus12:03:31

so it's fine to e.g. have a single directory with cljc files and consider that to be two test suites, one clj one cljs

plexus12:03:31

right, you basically want to undo what kaocha does, i.e. capture failure events and only print the details at the end.

dominicm12:03:48

Ah, great. That's one worry gone.

dominicm14:03:41

Can I disable the assertion exceptions?

dominicm14:03:05

I have a particular test which doesn't make assertions yet.

plexus14:03:02

we don't have a setting for that (yet), adding a dummy assertion is not an option?

plexus14:03:21

the other option is removing this association from the hierarchy (derive! :kaocha.type.var/zero-assertions :kaocha/fail-type)

plexus14:03:48

and redefining the report/fail-summary to do nothing

plexus14:03:13

(defmethod report/fail-summary :kaocha.type.var/zero-assertions [{:keys [testing-contexts testing-vars] :as m}])

dominicm14:03:57

It is, just wanted to check before I checked 1=1.

dominicm14:03:57

Unrelated, the missing assertion test seems to be happening for a case which very obviously has assertions. My best guess is that the test is clojurescript async, and that isn't being handled. But I'm not certain at all. Anything ring a bell?

dominicm14:03:50

Sorry, I have lots of questions today. Our test suite is apparently not in the state kaocha expects.

dominicm14:03:00

Test isn't async, so not that!

plexus14:03:11

ah it's a clojurescript test? although I guess same advice applies, as it piggiebacks on the event type and defmethod created by kaocha.type.var

plexus14:03:37

questions very welcome! it's good to know what is or isn't working for people, or what isn't clear or obvious

dominicm15:03:49

I hope my questions aren't too burdensome. Fortunately my employer does allow contributions, so if something was major, I would be able to contribute. I'm hoping that my questions help improve kaocha more quickly under funding. It's a bit of a weak hope, so mostly I just hope I'm not too much trouble :). Open source & time is a hard thing, and I appreciate your time building cool things, and answering my questions.

plexus14:03:48

and yeah when bringing on a big legacy test suite you're going to run into some things like this because kaocha is more strict about certain things (because it tries to prevent you from accidentally doing things you didn't intended). We really should have flags to disable these things to make porting to kaocha easier.

πŸ‘ 4
plexus14:03:27

yeah not sure what else would be triggering ::zero-assertions. Run the test with the debugging enabled and see what events you're getting (--focus on just that test). I would expect you are not seeing any assertions over the wire before the test finishes, in which case the question is why the test/runtime behave that way. If you are seeing assertions over the wire then the question is why kaocha.type.cljs still thinks that (= pass error fail pending 0)

dominicm14:03:25

Figured it out. We have a namespace which runs tests as a top level side effect. I'm guessing they were running in parallel

plexus14:03:17

oh yeah that would explain it

dharrigan15:03:42

@plexus I hope the commit and PR I did is okay. I tested it locally and I get nice output using expecations now πŸ™‚

plexus15:03:18

cool! I'll have a good luck. Thank you for contributing to Kaocha!

dharrigan15:03:39

you're welcome