Fork me on GitHub
#kaocha
<
2021-03-24
>
rickmoynihan14:03:22

Is there a way to make kaocha --print-test-plan emit edn?

rickmoynihan14:03:37

It appears to print vars e.g. :kaocha.var/var #'foo.bar/baz as far as I can tell #' isn’t a valid edn tagged literal

onetom15:03:21

what are you planning to do with the plan you read from a file? how would u like those vars to be represented?

rickmoynihan16:03:08

Well the plan is pretty noisy, so initially I was just wanting to post-process it with something like jet, just so I could understand it. However after looking at it, I realised it seems quite recursive; so it’s awkward to handle in jet anyway. Really I was hoping --print-plan or another option would let me figure out easily the test vars that are executed and their order (for a given focus etc), without actually running them. Obviously the plan contains enough to figure that out, it’s just not easy to read it as a human.

onetom17:03:13

aaand what would u use the processed information for? just looking at it, so u can learn more about how kaocha works? trying to debug something, like the paths being traversed or see the effect of :focus-meta? or maybe build some tooling on top, like show it in an emacs buffer?

rickmoynihan09:03:26

Ok, so backing up a bit my direct use case is refactoring large test suites that take a long time to run. We have several test suites with various focus-meta / skip-meta’s defined, and I’d like to know what tests are going to be executed in each suite (or execution of the cli with focus opts etc), without actually running them all. So an abbreviated output of the plan, that showed the order of execution and the nesting of namespaces / test vars. Also it’s possible in complex suites to have overlapping tests, such an output would be useful to identify the overlap and potentially the cause

rickmoynihan09:03:32

starter for 10 on a humanized output format that might be useful:

rickmoynihan09:03:52

--- unit (clojure.test) ---------------------------
csv2rdf.metadata.properties-test ^:foo ^:bar 
  child-test ^:baz
  set-table-group-parent-references-test
  children-test
  
csv2rdf.metadata.inherited-test ^:foo ^:baz
  null-value-test ^:bar
    string
    array
    array with invalid values
    invalid type

csv2rdf.http-test
  parse-link-header-test
    Valid header URI only
    Valid header with parameters
    Invalid URI
    Malformed

rickmoynihan09:03:59

i.e. similar to the document output just missing testing blocks as printing those will mean executing the tests. Could also be useful to augment ns’s and vars with their metadata keys… though we might want to elide standard clojure metadata e.g. ^:doc / ^:line etc.

rickmoynihan14:03:40

(clojure.edn/read-string {:default tagged-literal} "{#'foo :bar}") raises:

Util.java:  221  clojure.lang.Util/runtimeException
            EdnReader.java:  552  clojure.lang.EdnReader$DispatchReader/invoke
            EdnReader.java:  757  clojure.lang.EdnReader/readDelimitedList
            EdnReader.java:  680  clojure.lang.EdnReader$MapReader/invoke
            EdnReader.java:  145  clojure.lang.EdnReader/read
            EdnReader.java:  111  clojure.lang.EdnReader/read
            EdnReader.java:   67  clojure.lang.EdnReader/readString
                   edn.clj:   46  clojure.edn/read-string
                   edn.clj:   37  clojure.edn/read-string
                      REPL:   15  mut.loader-test/eval98441
                      REPL:   15  mut.loader-test/eval98441
             Compiler.java: 7177  clojure.lang.Compiler/eval
             Compiler.java: 7132  clojure.lang.Compiler/eval
                  core.clj: 3214  clojure.core/eval
                  core.clj: 3210  clojure.core/eval
    interruptible_eval.clj:   87  nrepl.middleware.interruptible-eval/evaluate/fn/fn
                  AFn.java:  152  clojure.lang.AFn/applyToHelper
                  AFn.java:  144  clojure.lang.AFn/applyTo
                  core.clj:  665  clojure.core/apply
                  core.clj: 1973  clojure.core/with-bindings*
                  core.clj: 1973  clojure.core/with-bindings*
               RestFn.java:  425  clojure.lang.RestFn/invoke
    interruptible_eval.clj:   87  nrepl.middleware.interruptible-eval/evaluate/fn
                  main.clj:  437  clojure.main/repl/read-eval-print/fn
                  main.clj:  437  clojure.main/repl/read-eval-print
                  main.clj:  458  clojure.main/repl/fn
                  main.clj:  458  clojure.main/repl
                  main.clj:  368  clojure.main/repl
               RestFn.java:  137  clojure.lang.RestFn/applyTo
                  core.clj:  665  clojure.core/apply
                  core.clj:  660  clojure.core/apply
                regrow.clj:   20  refactor-nrepl.ns.slam.hound.regrow/wrap-clojure-repl/fn
               RestFn.java: 1523  clojure.lang.RestFn/invoke
    interruptible_eval.clj:   84  nrepl.middleware.interruptible-eval/evaluate
    interruptible_eval.clj:   56  nrepl.middleware.interruptible-eval/evaluate
    interruptible_eval.clj:  152  nrepl.middleware.interruptible-eval/interruptible-eval/fn/fn
                  AFn.java:   22  clojure.lang.AFn/run
               session.clj:  202  nrepl.middleware.session/session-exec/main-loop/fn
               session.clj:  201  nrepl.middleware.session/session-exec/main-loop
                  AFn.java:   22  clojure.lang.AFn/run
               Thread.java:  834  java.lang.Thread/run

onetom17:03:13

i was trying to provide a dispatch fn for the single quote, but i failed too:

$ jet --from edn --edn-reader-opts "{:readers {(symbol \"'\") (fn [& args] args)}}" <<<"#'asd"
Exception in thread "main" clojure.lang.EdnReader$ReaderException: java.lang.RuntimeException: No dispatch macro for: '
...

rickmoynihan09:03:15

Yes IIRC the issue is that dispatch macros are essentially reader macros and only implementable by clojure/edn, i.e. in clojure and edn you can’t extend the reader itself; only tags. ' is a reader macro for vars… others include things like { for sets etc. AFAIK ' isn’t included in the edn subset of clj.

👍 3
plexus07:03:06

yeah the output of --print-test-plan isn't readable, making it unsuitable for its intended purpose: an interface that other tools/scripts can consume. Either we should add print handlers so things like vars/ns/atoms etc print as tagged literals, or we should do a pre-processing pass to filter them out or replace them with placeholders. Not sure if there's already an issue for this, if not feel free to create one. A PR for this would be more than welcome, perhaps integrating https://github.com/lambdaisland/data-printers . I'm quite partial to the tagged literals approach, my only concern with that is that Kaocha then modifies the global environment. But maybe we can only load the print handlers when calling --print-test-plan or --print-result. (`--print-config` is also in this list but is less likely to contain unprintable objects)

plexus08:03:12

Thank you 🙏:skin-tone-2: