Fork me on GitHub
#kaocha
<
2019-05-23
>
Alex Whitt19:05:18

Sorry if this has been mentioned already, but when using tools.deps, using --watch from the command line just hangs forever. If I set :watch? true in tests.edn, it works.

Alex Whitt19:05:46

Oh, @bbrinck, I see you've found this tool as well. After our conversation on #expound the other day, I decided to try tools.deps after all, and I think I like it. kaocha seems to be the best option for a test runner with tools.deps, but I'm struggling to get instrumentation failures to print properly with my expound printer. It looks like kaocha and orchestra both re-bind s/*explain-outat different levels, and I'm pretty confused as to what's happening exactly. The closest I can get is the following (some stolen and modified from your test_utils, some taken from https://gist.github.com/kennyjwilli/8bf30478b8a2762d2d09baabc17e2f10):

(ns vertiv-common.demo-test
  (:require [clojure.spec.alpha :as s]
            [clojure.spec.test.alpha :as stest]
            [clojure.string :as str]
            [clojure.test :as test]
            [expound.alpha :as expound]
            [orchestra.core :refer [defn-spec]]
            [orchestra.spec.test :as orchestra]))

(alias 'stc 'clojure.spec.test.check)

(defn report-results [check-results]
  (let [checks-passed? (->> check-results (map :failure) (every? nil?))]
    (if checks-passed?
      (test/do-report {:type    :pass
                       :message (str "Generative tests pass for "
                                     (str/join ", " (map :sym check-results)))})
      (doseq [failed-check (filter :failure check-results)]
        (let [r       (stest/abbrev-result failed-check)
              failure (:failure r)]
          (test/do-report
           {:type     :fail
            :message  (binding [s/*explain-out* (expound/custom-printer
                                                 {:show-valid-values? true
                                                  :theme :figwheel-theme})]
                        (expound/explain-results-str check-results))
            :expected (->> r :spec rest (apply hash-map) :ret)
            :actual   (if (instance? #?(:clj Throwable :cljs js/Error) failure)
                        failure
                        (::stest/val failure))}))))
    checks-passed?))

(def default-opts {::stc/opts {:num-tests 50}})

#?(:clj
   (defmacro defspec-test
     ([name] `(defspec-test ~name (stest/checkable-syms) nil))
     ([name sym-or-syms] `(defspec-test ~name ~sym-or-syms default-opts))
     ([name sym-or-syms opts]
      (when test/*load-tests*
        `(defn ~(vary-meta name assoc :test
                           `(fn [] (report-results (stest/check ~sym-or-syms ~opts))))
           [] (test/test-var (var ~name)))))))

(defn check-spec-assertions [test-fn]
  (binding [s/*explain-out* (expound/custom-printer {:show-valid-values? true
                                                     :theme :figwheel-theme})]
    (s/check-asserts true)
    (test-fn)
    (s/check-asserts false)))

(defn instrument-all [test-fn]
  (binding [s/*explain-out* (expound/custom-printer {:show-valid-values? true
                                                     :theme :figwheel-theme})]
    (orchestra/instrument)
    (test-fn)
    (orchestra/unstrument)))

(test/use-fixtures :once
  check-spec-assertions
  instrument-all)

(defn-spec do-it-wrong boolean?
  [a int?] 1)

(defn-spec test-a-thing boolean?
  [thing int?]
  (do-it-wrong false)
  true)

(defspec-test gen_test-a-thing `test-a-thing)

Alex Whitt19:05:55

But that gives me:

FAIL in vertiv-common.demo-test/gen_test-a-thing (demo_test.cljc:21)
== Checked vertiv-common.demo-test/test-a-thing 

  (vertiv-common.demo-test/test-a-thing -1)

 threw error

#error {
 :cause "Call to vertiv-common.demo-test/do-it-wrong did not conform to spec:\ndemo_test.cljc:68\n\n-- Spec failed --------------------\n\nFunction arguments\n\n  (false)\n   ^^^^^\n\nshould satisfy\n\n  int?\n\n-------------------------\nDetected 1 error\n"
 :data {:clojure.spec.alpha/problems [{:path [:args :a], :pred clojure.core/int?, :val false, :via [], :in [0]}], :clojure.spec.alpha/spec #object[clojure.spec.alpha$regex_spec_impl$reify__2509 0x2d8a45a "[email protected]"], :clojure.spec.alpha/value (false), :clojure.spec.alpha/args (false), :clojure.spec.alpha/failure :instrument, :orchestra.spec.test/caller {:file "demo_test.cljc", :line 68, :var-scope vertiv-common.demo-test/test-a-thing}, :orchestra.spec/var #'vertiv-common.demo-test/do-it-wrong}
 :via
 [{:type clojure.lang.ExceptionInfo
   :message "Call to vertiv-common.demo-test/do-it-wrong did not conform to spec:\ndemo_test.cljc:68\n\n-- Spec failed --------------------\n\nFunction arguments\n\n  (false)\n   ^^^^^\n\nshould satisfy\n\n  int?\n\n-------------------------\nDetected 1 error\n"
   :data {:clojure.spec.alpha/problems [{:path [:args :a], :pred clojure.core/int?, :val false, :via [], :in [0]}], :clojure.spec.alpha/spec #object[clojure.spec.alpha$regex_spec_impl$reify__2509 0x2d8a45a "[email protected]"], :clojure.spec.alpha/value (false), :clojure.spec.alpha/args (false), :clojure.spec.alpha/failure :instrument, :orchestra.spec.test/caller {:file "demo_test.cljc", :line 68, :var-scope vertiv-common.demo-test/test-a-thing}, :orchestra.spec/var #'vertiv-common.demo-test/do-it-wrong}                                                                                                                                                                                                                                                                             
   :at [orchestra.spec.test$spec_checking_fn$conform_BANG___4644 invoke "test.cljc" 115]}]
 :trace
 [[orchestra.spec.test$spec_checking_fn$conform_BANG___4644 invoke "test.cljc" 115]
  [orchestra.spec.test$spec_checking_fn$fn__4650 doInvoke "test.cljc" 126]
  [clojure.lang.RestFn invoke "RestFn.java" 408]
  [vertiv_common.demo_test$test_a_thing invokeStatic "demo_test.cljc" 68]
  [vertiv_common.demo_test$test_a_thing invoke "demo_test.cljc" 66]
... 
expected: boolean?
  actual: nil
1 tests, 1 assertions, 1 failures.

Alex Whitt19:05:09

It's like the output from the printer is getting captured and wrapped in another exception

bbrinck23:05:59

@alex.joseph.whitt I haven’t dug into it too deeply, but my guess is that the issue is w/ expound, not Kaocha. I would guess we could find a repro that didn’t include koacha, but only requires that in the process of running stest/check, we run into a case where we run into an instrumentation failure

bbrinck23:05:44

This type of bug with nested failures have occurred in expound before

bbrinck23:05:02

If you can file a bug in expound, i’ll see if I can solve it in the next version