Clojurians
# test-check

This page is not created by, affiliated with, or supported by Slack Technologies, Inc.

alexmiller 20:12:41

@gfredericks hey, I have a probably too broad question re test.check generators. There are a number of circumstances where random inputs can get “too big” (usually via either collection sizing or nesting) if you do enough tests. Surely this is an issue that quickcheck has already encountered - do they have generic ways to do thinning or something else? (I will do some of my own research but figured you might already know about it)

gfredericks 20:56:43

@alexmiller you're asking about something more sophisticated than the normal size control techniques?

alexmiller 20:58:47

well… kind of. this is really about controlling size at a higher level, which as I’ve been thinking about it more is really connected to controlling generation over a greater composite structure (which is the thing we’re producing out of a composite spec)

alexmiller 20:59:14

I went and looked at some of the quickcheck stuff more and it seems like there are all those knobs for size, scale, etc

alexmiller 20:59:21

but they assume pretty localized and intentional use

alexmiller 21:00:45

my real goal here is: how can we assure that an automatically created composite generator does not produce a value so big that it kills testing. and maybe the answer is really being smarter in spec about how we use test.check

alexmiller 21:01:56

some more specific problem areas are: collection size, recursive/nesting size, and individual value size (for things that use variable memory like strings, keywords, symbols)

alexmiller 21:02:32

really these are mostly easy to manage in isolation but are in some cases harder in combination

alexmiller 21:03:03

sorry, stepping away for a few

gfredericks 21:05:07

Yes definitely; sizing is one of the biggest headaches I've encountered in maintaining test.check -- I've kept wanting to redo the size parameter so that it has some kind of linear relationship to the (comp count pr-str) of the value generated

gfredericks 21:08:49

I thought about maybe having each generator have knowledge about the size of things it generates, but that doesn't work so well for gen/one-of or gen/frequency, and not at all for gen/bind

gfredericks 21:08:53

even gen/fmap would be hard

gfredericks 21:09:37

so I think the only other approach to a general solution would be to have some sort of logical post-processing that prunes things when the whole structure is too big

gfredericks 21:09:41

I haven't thought about that one too much

gfredericks 21:10:51

My guess is that this is the biggest sizing pitfall for spec: https://dev.clojure.org/jira/browse/TCHECK-106

gfredericks 21:11:55

it's possible that solving that would be enough from spec's perspective

alexmiller 22:22:20

yes, I think that is the highest priority issue

alexmiller 22:22:58

Rich suggested that maybe there was some notion of “thinning” that could be applied as you generate - reducing the scale as you get deeper in a structure

gfredericks 22:25:12

@alexmiller yeah, that's something that the recursive-gen already does (else it'd likely generate infinite structures)

gfredericks 22:26:17

I certainly considered it -- it had the feel of a breaking change, so I was probably hesitant on that count, but there might have been some more subtleties;

I'll give that some more thought and try it out

gfredericks 22:27:02

one approach I tried with recursive-gen was to take the current size and partition it up randomly among the elements of the collection to be generated

gfredericks 22:27:14

I think that's a lot harder with recursive-gen than it would be with the regular collection generators

gfredericks 22:27:20

so maybe there is hope

alexmiller 22:28:43

yeah, I saw some interesting stuff in the quickcheck papers about the recursive gen aspects

alexmiller 22:29:07

and spec is doing some of its own things there as well

gfredericks 22:29:33

yeah I noticed that and wasn't sure what I thought about it

gfredericks 22:30:44

I'm not sure that there are any serious backwards-compatibility concerns about changing the sizing behavior, so I'll consider this a plausible solution

gfredericks 22:30:59

do you have a guess about whether there will be other serious sizing issues remaining?

alexmiller 22:31:23

collection size and recursion seem to be at the root of the majority of issues I see

gfredericks 22:31:44

does spec use recursive-gen at all?

alexmiller 22:31:55

don’t know

gfredericks 22:32:03

I think the biggest thing I noticed was that it didn't even with naturally recursive specs

alexmiller 22:32:43

yeah, the gen namespace doesn’t pull in recursive-gen so I’d guess nothing is using it

gfredericks 22:32:56

it might not be a good fit anyhow

alexmiller 22:33:14

I don’t know if that was a conscious choice or not (although with Rich I typically bet on the side of conscious choice)

alexmiller 22:33:43

stepping away again, maybe for a while, sorry :)

gfredericks 22:33:54

no problem