Clojurians
# test-check

This page is not created by, affiliated with, or supported by Slack Technologies, Inc.

gfredericks 15:46:12

I'm working on a biginteger generator for test.check, presumably to be used by spec as well (directly or indirectly); the main decision to make is what the distribution should be. Here's what I've got so far, interested in any comments.

https://gist.github.com/gfredericks/b6b59f1c531dc36017e45f2f0beeff9e

bbloom 21:39:31

let’s say i want to test some stateful thing by generating a sequence of actions to perform

bbloom 21:39:42

how do i use state to inform future generated actions?

bbloom 21:41:06

simple example: let’s say i want to test a growable array class with three methods: getLength, append, and getNth

bbloom 21:41:24

that was easy to test, but as soon as i wanted to test setNth, i was at a loss

bbloom 21:41:40

how do i make it so that setNth can check getLength first?

bbloom 21:42:22

my current hack is to just do modulo math at the time of applying the action - but that only works for this simple example & i have more complex stuff i want to test

bbloom 21:45:06

my best guest is to try defining generators recursively, deferred by an fmap or bind or something - but wasn’t sure if that was a hack or recommended or what

bbloom 21:45:44

ie parameterize them based on a model of the state

alexmiller 21:46:40

the general approach is to first build a (random) model, then produce the action list via fmap/bind

bbloom 21:47:44

not 100% sure i follow. is there a super simple example somewhere?

gfredericks 21:49:28

There's a whole lib for this kind of thing I think

bbloom 21:50:00

i’m looking at two such libraries now, trying to see if they clarify things for me: stateful-check and states

gfredericks 21:50:19

I haven't used either, but I've reviewed stateful-check a bit and it seems solid

bbloom 21:52:43

there’s a non-trivial amount of code in this lib. i’ll study it, but i’m hoping to identify the essence of it.

gfredericks 21:54:40

@bbloom do you understand the idea of modeling the whole interaction ahead of time?

alexmiller 21:55:21

it’s been a while since I’ve looked at collection-check but I know it does a lot of this operation kind of thing (can’t remember if it’s stateful though)

bbloom 21:56:38

so i haven’t used test-check in anger at all, but i did successfully roll my own generative/simulation testing thing in go for a wire protocol. but that was a hierarchical temporal marckov model, i didn’t do shrinking, and i did all the validation later on the log file - worked mostly as a stress test for a long running system

bbloom 21:57:28

to get multiple tests, i didn’t have a backtracking generator type thing - instead just ran a bunch in parallel, since they were also testing limited hardware resources (ie physical devices)

gfredericks 21:57:41

I think the shrinking model is the biggest reason that the generate-everything-up-front approach works best

bbloom 21:57:41

so i have a bit of a grasp on the concepts, but no knowledge of the specific apis

bbloom 21:58:03

sooo now back to your question: i’m not sure what you mean “modeling the whole itneraction ahead of time”

gfredericks 21:58:11

the only viable alternative I've seen is the python/hypothesis approach which is extremely imperative and wildly different

bbloom 21:58:16

@alexmiller: i’m looking at collection-check, as what i’m doing is quite similar. thanks

gfredericks 21:59:19

@bbloom I mean the generator generates the entire intended interaction, e.g.

  • insert a
  • insert b
  • read 0, expect a
  • read 1, expect b
  • etc.
gfredericks 21:59:32

I believe this is essentially the idea behind stateful-check

gfredericks 21:59:44

it does all the wiring-up for you

bbloom 22:00:10

yeah, so i’m trying to test some java code that is stateful & i’m successfully generating a sequence of actions & applying them

bbloom 22:00:44

the tricky bit is the bit that stateful-check seems to address: using the state of the model in order inform the generators

gfredericks 22:01:48

"Why can't I call generators in the middle of my test?" is a common thing people run into; currently test.check doesn't try to give you a way to do that, but I'm not 100% convinced it can't be done

gfredericks 22:02:00

I've thought about it a lot; the shrinking is the hard part

bbloom 22:02:38

the fact that it’s asked for a lot leads me to wonder: is it a misunderstanding that leads to it being wanted? or it just hard to provide?

bbloom 22:02:44

ie should i be thinking about it differently?

alexmiller 22:03:32

We've done a number of sim testing projects at Cognitect that pushed this kind of thing really far

alexmiller 22:04:05

But there you don't really care about shrinking at all

bbloom 22:04:51

glad to hear that - since i had no use for shrinking in my sim test and wondered if i was just missing out

alexmiller 22:05:15

I'd ask @luke about some of those ideas - he has some libs that I think are oss that do statistical / stateful generative stuff

bbloom 22:05:22

this is the first time i wanted to test something that felt complex enough to justify test check and not complex enough to justify distributed sim testing

gfredericks 22:05:25

it should be possible to write a thingamajigger to call generators in the test deterministically, without doing any shrinking

alexmiller 22:07:42

In general with sim testing you want some simple (ish) model how users use the system such that you can generate realistic (ish) streams of random activity

bbloom 22:08:27

yeah - in my case, i simulated a police officer on patrol. turning cameras on and off, watching videos, driving to and from places with or without wifi, etc

bbloom 22:08:39

worked nicely

alexmiller 22:09:00

Yeah, exactly

alexmiller 22:09:17

Our domains were a lot more complicated :)

bbloom 22:09:29

i’d imagine

alexmiller 22:09:54

Cognitect does arch consult gigs, just saying :)

bbloom 22:10:21

:slightly_smiling_face:

alexmiller 22:11:27

architecture

gfredericks 22:11:32

@gfredericks uploaded a file: Untitled and commented: @bbloom something like this probably works (I haven't run it); if you want to use it and prefer it to be published somewhere, I could add it to test.chuck.

gfredericks 22:11:45

like...CPU chips? big buildings?

gfredericks 22:11:53

large-scale computer systems?

alexmiller 22:12:12

Really any scale :)

alexmiller 22:12:23

Architecture review

bbloom 22:13:53

i don’t work on that any more, but the team seems quite happy with the architecture i built for them - which is largely clojure/hickey-inspired, thank you very much :wink:

bbloom 22:16:13

@gfredericks i don’t quite understand the api well enough yet to know if that code is useful to me

gfredericks 22:16:26

@bbloom I added an example

gfredericks 22:16:44

the idea is you generate one of these fs and then you can call it from the body of your property as many times as you want with arbitrary generators

gfredericks 22:17:35

it's a stateful function, whose state is randomness derived from the normal test.check source of randomness, so the whole thing should be deterministic as long as you don't call the function in nondeterministic ways

bbloom 22:18:12

i’m not sure i need something that sophisticated

gfredericks 22:19:30

so e.g., if you have your stateful java collection and want to generate a valid index, you'd call (f (gen/choose 0 (dec (.getLength thingamajig))))

gfredericks 22:20:00

and that call would return an index between 0 and (dec (.getLength thingamajig))

gfredericks 22:20:18

the sophistication is mostly to satisfy my own ideals of reproducibility

gfredericks 22:20:40

if you couldn't care less about that you can just call gen/generate and not bother with any of this

bbloom 22:25:10

so i don’t actually need to call .getLength on the real stateful object during test execution

bbloom 22:25:19

i just want to track some state in the model & use that to inform which actions to generate

bbloom 22:25:51

looking at collection-check, they seem to use clojure’s types as a model

bbloom 22:26:15

but zach does basically what i mentioned earlier: he generates wild indexes for nth/assoc and then fixes them up during execution

bbloom 22:28:00

how much am i expected to know about the rose tree stuff in order to make effective use of test-check? this stateful-check thing manipulates it quite a bit, plus has some gen-do monadic bind syntax etc that seems redundant w/ fmap, gen/let, etc

gfredericks 22:28:41

users don't normally need to care about the rose tree

gfredericks 22:29:28

my guess is that stateful-check's low-level internals are an attempt to shrink better than the naive shrinking you get by making a pile of binds

gfredericks 22:29:46

gen/bind is pretty dumb about shrinking

gfredericks 22:30:11

and stateful-check might be able to make assumptions that bind can't

gfredericks 22:30:53

maybe a simple example of generating a model with actions would be helpful

gfredericks 22:31:28

gimme a minute

bbloom 22:31:43

thanks for your help. greatly appreciated

bbloom 22:31:47

you too @alexmiller

bbloom 22:34:05

i’m still reading and trying to make sense of stateful-check, but some things concern me

bbloom 22:34:29

for example, command argument generation seems to attempt to implicitly and recursively lift values to be generators

bbloom 22:34:45

which seems like magic i don’t want, at least not before i am comfortable w/ the monadic api

bbloom 22:35:20

meanwhile, i’m failing to extract the essence from the core of the generate-commands* routine

gfredericks 22:37:04

yeah I've never liked the values-can-be-generators idea

gfredericks 22:37:23

in practice it's probably not too confusing

gfredericks 22:37:41

....maybe

bbloom 22:37:56

….maybe with a type system? :stuck_out_tongue:

gfredericks 22:38:07

yeah :slightly_smiling_face:

gfredericks 22:40:13

@gfredericks uploaded a file: simple model action generation thinger and commented: @bbloom

gfredericks 22:41:00

could fancy it up to add the intermediate state at each step, or generate random assertions if you don't want to just compare the whole map

gfredericks 22:41:12

note that the gen/lets are where bind is happening

gfredericks 22:42:12

there might be stack problems with this; maybe that's another reason for stateful-check's complexity

gfredericks 22:42:26

e.g., if you wanted to generate 10000 actions you might have problems

bbloom 22:42:33

yeah - so i tried something like this and got a stackoverflow

bbloom 22:42:59

but thanks, this example is much closer to my intuition

gfredericks 22:43:20

I just generated 20,000 of these up to the normal max-size of 200 and got no exceptions

gfredericks 22:43:49

looks like the sizes don't get above 20 though

gfredericks 22:43:52

so that might be bad

gfredericks 22:44:34

that's probably why it's not SOing :slightly_smiling_face:

bbloom 22:44:44

:stuck_out_tongue:

gfredericks 22:44:50

to get larger you'd use gen/frequency instead of gen/one-of and tweak the weights

gfredericks 22:45:02

alternately you could generate a size up front and pass that through the recursion

bbloom 22:45:02

well you had equal assoc/dissoc frequency

bbloom 22:45:25

i wasn’t doing any dissoc to reduce the size, so it grew faster

gfredericks 22:46:02

I think stack problems would be related to action-count, not the size of the state

bbloom 22:46:36

oh, hmm that’s right

bbloom 22:46:47

not sure what i did, since it’s like 5000 undos ago :slightly_smiling_face:

bbloom 22:48:10

with respect to shrinking: how well does that work as things grow more complex?

bbloom 22:48:35

i’m somewhat skeptical it can work well for stateful tests - since each removed operation can potentially invalidate the remainder of the operations

gfredericks 22:48:42

that's exactly the problem

gfredericks 22:49:15

if you're using the naive bind structure in my example, what would happen is if test.check tries to shrink an earlier action it will generate a totally new set of following actions

gfredericks 22:49:28

it might be that in practice this isn't as bad as it sounds

gfredericks 22:49:44

I get almost 0 complaints about this

bbloom 22:50:30

which of course could mean 1) it’s perfect 2) no body is using it or 3) people don’t understand it well enough to complain about it without embarassing themselves like i’m doing now :slightly_smiling_face:

gfredericks 22:50:31

I made a ticket about it at least, but don't know if it will lead anywhere https://dev.clojure.org/jira/browse/TCHECK-112

gfredericks 22:50:43

yeah exactly

gfredericks 22:50:57

2 isn't plausible; people are definitely using bind, at least via gen/let

gfredericks 22:51:08

in fact gen/let tends to encourage over using bind

gfredericks 22:51:41

i.e., using bind to combine two independent generators in a way that you could do with gen/tuple

bbloom 22:52:23

does tuple do “parallel” bind? ie it shrinks from any position fairly?

bbloom 22:52:39

and bind is “left biased”?

gfredericks 22:53:02

that...might be right

gfredericks 22:53:08

tuple shrinks each thing independently

gfredericks 22:53:19

you can always rewrite tuple with bind but not vice versa

bbloom 22:53:22

gotcha - so tuple is applicative

gfredericks 22:53:32

that's probably true

bbloom 22:53:58

ok - so i guess i’ve confirmed 1) i understand this and 2) i still have no idea how to use the API :stuck_out_tongue: thanks

bbloom 22:54:11

i’ll keep playing with it

bbloom 22:54:21

will let you know how it turns out

gfredericks 22:54:24

oh well; do let me know if you have more questions or accusations

bbloom 22:55:09

heh, i hope the accusations thing was a joke and/or not about me. let me know if i failed to politely convey my gratitude!

gfredericks 22:55:33

oh yeah sorry, definitely a joke

bbloom 23:36:05

so shrinking only removes elements from the reproduction, right? no rewriting in any way?

bbloom 23:39:50

i’m attempting a “loose” approach to action generation - will leave state out of that part & use state in the code that “runs” the actions. that will screw up shrinking probably, but i’ll deal with that when i have trouble pinpointing an issue in the future

bbloom 23:41:56

so like if i were testing that random access stack example from before, actions may just be like “do 5 pushes, then randomly swap elements 8 times, then do two pops, etc….”

bbloom 23:42:00

and see how that does for me