Fork me on GitHub
#kaocha
<
2020-07-24
>
andrea.crotti08:07:25

I was trying to extract all the tests from the test-plan, which is a quite nested data structure

andrea.crotti08:07:31

and this seems to work

(def plan (kr/test-plan))

(def all-test-names
  (->> plan
       :kaocha.test-plan/tests
       (mapcat :kaocha.test-plan/tests)
       (mapcat :kaocha.test-plan/tests)
       (map :kaocha.var/name)
       sort))

andrea.crotti08:07:36

I wonder if there is an easier way though

plexus08:07:52

There's kaocha.testable/test-seq, or you can use clojure.walk

andrea.crotti08:07:29

ah yes better thanks.

andrea.crotti08:07:46

seems like I was getting the same result at least

andrea.crotti08:07:51

apart from the skip maybe

plexus08:07:45

you are encourage to use hierarchy/leaf? to check if a testable is a an actual test as opposed to a grouping like a namespace or a test suite

plexus08:07:57

different test types may have more or fewer than two levels of nesting

plexus08:07:34

also :kaocha.var/name is specific to clojure.test, whereas all testables have a :kaocha.testable/id

andrea.crotti08:07:34

ah nice thanks

andrea.crotti08:07:12

well what I have to do is to actually split tests with the circleci tool

(def all-test-names-2
  (->> (kaocha.testable/test-seq plan)
       (map :kaocha.var/name)
       (remove nil?)))

(def split
  (-> (sh/sh "circleci tests split --split-by=timings"
             :in (clojure.string/join "\n" all-test-names-2))
      :out))

andrea.crotti08:07:21

and then run them with kaocha

andrea.crotti08:07:33

something like (kr/run "--focus" (first all-test-names-2)) for example

andrea.crotti08:07:39

but yeah running all the ones returned by the split

andrea.crotti08:07:02

but yeah I guess I can use the id as well

plexus08:07:15

yeah so in this case you definitely want to use kaocha.testable/id, that's what you pass to focus. it just happens to correspond to the var name for clojure.test

andrea.crotti08:07:52

ah yeah but don't think circleci would handle that

plexus08:07:26

I'm not familiar with this circleci tool. What does it do?

andrea.crotti08:07:56

so it kind automatically split tests by timing, parallelising automatically

andrea.crotti08:07:08

but still keeping only one job

andrea.crotti08:07:26

we had that working for clojure.test, I just wanted to convert to kaocha

plexus08:07:35

interesting, I'm guessing it gets the test information via junit.xml?

andrea.crotti08:07:49

from the circleci agent from previous runs I guess

plexus08:07:56

in which case it will have the testable ids

andrea.crotti08:07:07

ah ok I'll try

plexus08:07:25

I mean if you're only using clojure.test it comes down to the same thing

plexus08:07:38

and then (kaocha.repl/run {:focus [id1 id2 id3]}) I think

andrea.crotti08:07:32

I'm always confused about the arguments to pass to kr/run

finops.ci-tests> (kr/run {:focus smaller-subset})

0 tests, 0 assertions, 0 failures.
#:kaocha.result{:count 0, :pass 0, :error 0, :fail 0, :pending 0}
finops.ci-tests> (kr/run "--focus" (first smaller-subset))
[(...........................................................................................................................................................................................................................................................................................................................................................................................................................................

andrea.crotti08:07:45

so it works fine if I pass things same args I would pass to the cli

andrea.crotti08:07:52

but not with configs

plexus08:07:07

hmmm that's surprising

plexus08:07:54

but that "--focus" version only works by accident, kaocha.repl does not take command line options like that

plexus08:07:48

kaocha.run takes testable-ids, and optionally a config map as last arg, so you can just do (apply kaocha.repl/run subset)

andrea.crotti08:07:11

ahok thanks let's see if

(apply kr/run
         (conj (tests-to-run)
               {:plugin :kaocha.plugin/junit-xml
                :junit-xml-file "test-results/kaocha/results.xml"}))
works

andrea.crotti08:07:19

maybe I can add an example in hte kr/run docs

andrea.crotti08:07:32

it's not the first time I get confused even after looking at docs and code

plexus08:07:35

that'd be great! doc improvements are always most welcome

plexus08:07:57

in general a writeup of this parallel testing with circle would be very cool!

andrea.crotti09:07:16

ah yeah as soon as I get it working

andrea.crotti09:07:54

Yeah I didn't even know there was this feature, but it's quite a good way to get some free parallelism without having to split yourself creating multiple jobs

andrea.crotti09:07:10

still doesn't work but it's because of something I'm passing wrong to the circleci command, but well almost there

andrea.crotti10:07:44

mm so splitting didn't work yet, but the other weird thing is

..............................................................................................................................................................................................................................................................................................................................................................................)]
1 tests, 2800 assertions, 0 failures.

andrea.crotti10:07:58

so doing it like this seems to group all the tests into one

andrea.crotti10:07:09

which is maybe expected I guess and maybe not a big deal

andrea.crotti10:07:12

just seems a bit strange

andrea.crotti11:07:35

also any idea how to pass the --reporter documentation config in that config map?

andrea.crotti11:07:51

:reporter :documentation or :reporter "documentation" don't work

andrea.crotti11:07:37

also I was trying to use leaf? but I find nothing if I do this for example

(defn fetch-tests
  [test-plan]
  (w/postwalk
   (fn [t]
     (when (kh/leaf? t) t))
   test-plan))

plexus12:07:25

I think that's because of how postwalk works, you're removing the parents so the chidren are never reached

plexus12:07:51

kaocha.repl/config is your friend to see what the results are of your config options

plexus12:07:27

(:kaocha/reporter (kaocha.repl/config))
(:kaocha/reporter (kaocha.repl/config {:reporter kaocha.report/documentation}))