Fork me on GitHub
#test-check
<
2020-12-05
>
gfredericks13:12:58

`:reporter-fn`
    A callback function that will be called at various points in the test
    run, with a map like:
      ;; called after a passing trial
      {:type            :trial
       :args            [...]
       :num-tests       <number of tests run so far>
       :num-tests-total <total number of tests to be run>
       :seed            42
       :pass?           true
       :property        #<...>
       :result          true
       :result-data     {...}}
      ;; called after the first failing trial
      {:type         :failure
       :fail         [...failing args...]
       :failing-size 13
       :num-tests    <tests ran before failure found>
       :pass?        false
       :property     #<...>
       :result       false/exception
       :result-data  {...}
       :seed         42}
    It will also be called on :complete, :shrink-step and :shrunk. Many
    of the keys also appear in the quick-check return value, and are
    documented below.
☝️ an arg to clojure.test.check/quick-check , which you could monkey-patch if you aren't actually calling it directly

andy.fingerhut15:12:10

Thanks for that tip.

andy.fingerhut15:12:42

I know that shrink-loop in my case is not finishing within 10,000 iterations, since I put in some logging and let it run for a while.

andy.fingerhut15:12:30

I guess I might let it run for a day or so, out of curiosity to see if it actually finishes. In the mean time, it does seem like some kind of option to force shrink-loop to return early would be useful in some cases, as I think you may have mentioned earlier.

andy.fingerhut15:12:10

Or perhaps there is one, and I haven't read the docs yet.

gfredericks15:12:42

I would have added that feature a long time ago, but I got stuck being unable to decide on a coherent way of configuring test.check given the >=2 different usage styles

gfredericks15:12:17

also once you start returning from shrinks early people will want a feature for resuming shrinks, and that's even more complicated

gfredericks15:12:26

looks like there's no error-handling on the reporter-fn , so technically you could just throw an exception from there 😄

andy.fingerhut15:12:51

That is certainly one way to do it. Let me try that.

gfredericks15:12:49

it might not be trivial to get it to accurately return a "smallest failing case so far"

gfredericks15:12:10

shouldn't be hard to get it to return a "most recently failing case"; maybe that's the same thing, I'm not quite sure 🙂

andy.fingerhut15:12:19

I see current-smallest

andy.fingerhut15:12:34

which seems to be passed to reporter-fn whenever a new one is found