Fork me on GitHub
#test-check
<
2018-04-10
>
Joe Lane16:04:09

Is there an api in test.check to get the collection of inputs and outputs used when exercising a test? For example, when running

(defspec first-element-is-min-after-sorting ;; the name of the test
         100 ;; the number of iterations for test.check to test
         (prop/for-all [v (gen/not-empty (gen/vector gen/int))]
           (= (apply min v)
              (first (sort v)))))
getting a collection of
({:input [1] :output [1]},
 {:input [2 1] :output [1 2]}
 ...
I suspect it the data I’m looking for is :args and :result-data or :result from the quickcheck function in check.cljc. Is there a blessed way to get access to this information?

ghadi16:04:03

you can recreate it using the random seed that quickcheck gives you

Joe Lane16:04:44

@ghadi I can get the data literals using the seed? I know I can re-run the test again, i’m curious about the data itself though

ghadi16:04:37

right, I'm not sure the exact process

ghadi16:04:53

the seed is there so that you can get reproducibility

nberger16:04:59

@joe.lane may I ask what do you plan to do with that data? I'm truly curious, mostly because a while ago I wanted to add a way to get a glimpse on what's the distribution of the generated values

gfredericks16:04:53

args is the generated data

gfredericks16:04:36

The return value isn't given, you can just call your function yourself once you have the args

ghadi16:04:19

@gfredericks can you do something like sample-seq while providing the seed?

gfredericks17:04:53

It could exist, but might not be immediately useful since you couldn't easily pick out the failing args. Do you have another use case?

Joe Lane16:04:45

I had another instance of a recurring dream last night. The word cloud looks like this: clojure.spec generative testing + consumer driven contracts (like http://pact.io) except for libraries, not web apis + spec-ish type-system-ish code inspection/analysis (e.g. (-> (makes-random-int) expects-only-odds) would be analyzed and the system would show there is a problem) + service for publishing specs to a public repository. The big thing here is consumer driven contracts for function apis. A library author could know the expectations of its consumers based on properties they provide, and then know if they make a breaking change or not.

Joe Lane16:04:19

There are a LOT of conflated ideas here, please dont take this as a proper description of all goals.

Joe Lane16:04:13

I suppose the ideas are 1. consumer driven contracts for public library functions 2. analysis of clojure programs using spec to determine areas for potential bugs 3. maven for specs.

Joe Lane16:04:35

The reason I’m asking is about inputs and outputs is to create a system like haskell’s hoogle (https://www.haskell.org/hoogle/) with spec.

grzm16:04:12

@joe.lane The way something like that could work is if consumers published specs that the upstream author could consume, and those specs would have to be more constrained versions of the specs the author already publishes. The upstream author could swap those more specialized specs in their own tests. That reminds me of "specialization by constraint" in Date and Darwen's "Databases, Types, and the Relational Model". I'm not well-versed in type-theory in general, so it might have other names, but that's where I came across it.

grzm16:04:28

The idea there is that any proper subtype is only a subtype if it satisfies the definition of its parent type plus some additional specialization constraint.

Joe Lane16:04:11

That’s exactly how http://pact.io does it, except with json and regex, not something more expressive like spec + edn.

grzm17:04:44

How would "maven for specs" be different from maven (or similar tool) in general? Specs are code and just like when you use a library, you still need to know the names of the functions (or specs) once you have it.

Joe Lane17:04:46

It was a bit of a mischaracterization of what I’m going after ( which, is admittedly, a moving target). It’s more about a place for consumer “contracts” to be placed. In addition to that, the idea of a central place that I can look up what functions take this input and produce this output could be really exciting for a variety of reasons. Not so much maven for specs. Sorry for the confusion.