Fork me on GitHub
#test-check
<
2022-03-08
>
dnolen19:03:37

maybe I'm not looking in the right places - but there seems to be not much information about writing customer generators?

dnolen19:03:30

Not knowing that much about custom generators, after auditing some repos I have a trivial stateful generator for producing commands to drive some test

dnolen19:03:48

it works by recursion by feeding the augmented state to produce the next command

dnolen19:03:17

by doing so then it becomes easy to make the command sequence more meaningful, by i.e. decreasing the frequency that the last command will be chosen, etc. etc.

dnolen19:03:46

but before proceeding any further I did a quick check (ba-dum) and of course it did not shrink

hiredman19:03:35

instead of generating a single command generate a list of commands, and statefully process each list, with a fresh state for each list

hiredman19:03:46

basically you want to generate programs, instead of generating a single command and building programs out of the generated command

hiredman19:03:00

if you generate a whole program then the whole program can shrink

dnolen19:03:22

the recursive thing does indeed produce a list of commands - but that is what doesn't shrink

hiredman19:03:46

ah, I thought you where doing the recursive thing outside of generating

dnolen19:03:29

looking at the built in generators - it appears that shrinking might be a bit manual

dnolen19:03:37

like you have to make the shrink tree yourself?

hiredman19:03:27

I've never had to manually do shrinking

hiredman19:03:10

I usually generate programs then feed them into some interpreter, and that interpret might no-op some commands in certain states, instead of feeding the state through the generator

dnolen19:03:13

hrm really, in the (repo docs) the first thing that it says is that if you build a stateful generator via bind - it will not shrink very well

hiredman19:03:47

oh yeah, I've had to massage things around to get good shrinking

hiredman19:03:12

because bind is opaque

gfredericks19:03:16

I feel like @dnolen described this to me at a conference like five years ago

hiredman19:03:36

I think anything other than bind being a simple maping kind of operation (take a vector of numbers and produce a java.util.Date object) isn't going to shrink well

dnolen19:03:09

@gfredericks haha we probably talked about it - but now I actually have a reason to use this approach at work

gfredericks19:03:12

I feel like maybe the best thought I had on this was that it'd be useful to have a general framework for this style of generator Something that could be implemented once against the lower level primitives

dnolen19:03:15

only tangentially related but I found this interesting paper - https://proper-testing.github.io/papers/icst2018.pdf

gfredericks19:03:19

I assume the style is that you have a model of the state of the system at a given time, and can generate commands based on the current state, and each command transforms the current state to the next state?

dnolen19:03:52

PropEr is a open source Erlang framework for testing stateful systems, and they have been implementing things to make it easier to test stateful system

dnolen19:03:14

both by providing a framework, but also integrated simulated annealing to control the search

1
hiredman19:03:32

I've done that sort of thing, but without feeding the state through command generation, and then ignoring invalid commands in the interpreter

dnolen19:03:00

I don't really care one way or another about whether test-check provides such affordances

dnolen19:03:26

but rather some guidance on writing custom generators that might be influenced by above tactics

dnolen19:03:05

so if I recur on this generator, do I need to manually populate the rose tree w/ children for the previous state

dnolen19:03:08

so it can shrink later?

gfredericks19:03:59

Custom shrinks aren't available at all in the main user facing api I don't think

dnolen19:03:27

yes I know that 🙂

dnolen19:03:05

but the non-doc'ed but not private internal helpers can be used as far as I can tell (to accomplish the goal, not future-proof)

gfredericks19:03:01

By "manually popular the rose tree" do you mean something like the haskell quickcheck api where you provide a function X -> [X] X being a list of commands

dnolen19:03:14

(oh yeah, re: using test-check, I think that was about testing stateful UIs - barely remember that so long ago)

dnolen19:03:57

@gfredericks can you not just gen-fmap and create new rose w/ what you know should be the children?

Alex Miller (Clojure team)19:03:59

if you build out of fmap and bind, it will shrink

Alex Miller (Clojure team)20:03:47

right? looks at gfredericks

gfredericks20:03:37

This kind of thing can definitely shrink, but often it's degenerate

gfredericks20:03:43

If you generate 200 commands

gfredericks20:03:59

And each is logically dependent on all the earlier ones

gfredericks20:03:23

Then shrinking command #7 invalidates the other 293 of them

gfredericks20:03:55

So they have to be replaced with freshly generated commands based on the new state after the new command 7

gfredericks20:03:13

Because TC can't do better than that generically

gfredericks20:03:52

But dnolen with his domain knowledge can describe more particularly when it's valid to shrink or delete intermediate commands

gfredericks20:03:00

And so can get much better results

gfredericks20:03:04

So the choice is A) add a new feature for this situation that lets users express that domain knowledge B) dnolen does this particular case by hand

gfredericks20:03:14

(as I see it)

gfredericks20:03:20

Writing your own shrink tree can be laborious because it's hard to take advantage of the built in shrinking for the atomic types you're using

gfredericks20:03:47

I also had an idea for a user-enhanced shrinking, but not sure that's a comprehensive solution here

dnolen20:03:26

@gfredericks design is hard ... I'm curious as to why the shrinking stuff that's there is not just exposed and documented

dnolen20:03:54

it seemed like one of the novel things about test-check was that shrinking was decoupled - but then somehow not a part of the api

gfredericks20:03:52

I don't think that was too intentional I mean you could ask Reid if you want historical context I largely took the approach of not changing things without a really strong reason So I think it's totally valid to suggest exposing it

gfredericks20:03:39

I think the hope was that the built in shrinking was high enough quality that users wouldn't have to think about it Probably not realistic, but it works much better than the QuickCheck approach I think

dnolen20:03:07

I think you don't really hit this problem until you want to write stateful generators

dnolen20:03:33

then the domain starts to dominate and the defaults just aren't that useful - because it's mostly junk

dnolen20:03:21

I will continue poke around at what's there for now - still barely know how to do anything so will probably say marginally more intelligent things later 😉

gfredericks20:03:41

Oh hey, I wrote something about this

dnolen20:03:43

yes already read that 🙂

dnolen20:03:46

several times

hiredman20:03:31

it is iterate (unfold) for generators

gfredericks20:03:17

@dnolen does the idea there under "general problem" seem like it would cover your use?

hiredman20:03:15

not entirely sure, but it seems like https://github.com/advancedtelematic/quickcheck-state-machine takes the approach of generating events without regard to state, but has you define preconditions which say if some event + state is well formed

hiredman20:03:02

(I can tell the precondition is a thing, but my haskell is not good enough to immediate tell if the precondition stuff is somehow used as part of generation or just as a filter afterwards)

dnolen20:03:55

basically the problem that I've encountered so far is as @gfredericks alluded - steps depend on previous steps

hiredman20:03:59

> The pre-conditions are used while generating programs (lists of actions).

dnolen20:03:24

as the command grows expecting to randomly line anything up is unlikely

dnolen20:03:50

instead we want to look at the previous state to make intelligent but not totally predetermined steps

dnolen20:03:02

at the same time, if something is found, we want to it to shrink

dnolen20:03:45

ok manually managing the rose tree seems like it leads to better results here

dnolen20:03:55

@gfredericks I do see that one irritation w/ the rose tree versions is that mixing things (on accident) leads to very cryptic problems

hiredman21:03:13

Another option would be, given a state machine description S, generate a vector of numbers, and then in a map bind, from your start state, take the sorted possible transitions, mod the first integer and take the nth, then iterate that for each state and integer

hiredman21:03:02

Which would shrink based on vectors and int generators, which might not be the most intelligent shrinking, but avoids recursive generation passing the state through

dnolen21:03:06

hrm, the problem has more constraints then it may appear, so I don't see that approach as being less cumbersome?

dnolen21:03:25

in truth what I'm interested in is simulation generation

dnolen21:03:33

that is there is some actor A, and then they do something

dnolen21:03:45

well then maybe there is corresponding action for actor B

dnolen21:03:54

and then also a corresponding action for actor C

hiredman21:03:54

Sure, and I guess you'd get shrinking, but the behavior would be very odd

dnolen21:03:13

because frequencies is dynamic - I'm also interested in using the state at each step to dynamically bind new likelihoods

dnolen21:03:35

so the tools in test.check at the moment seem sufficient to accomplish what I want, without going elsewhere

dnolen21:03:22

I think probably only the command choice function (which generates the liklihoods and the command arguments) will be of any real interest

dnolen21:03:27

the rest will probably be boring

Alex Miller (Clojure team)21:03:04

sounds like you are really doing simulation testing. we did several projects like this at Cognitect with Simulant (and other Simulant like things). there are a couple other open source things we made for those

dnolen21:03:32

but we are not actually testing a whole system - we don't care about time

dnolen21:03:46

and we're going to drive anything that would be external

dnolen21:03:49

we don't even care about concurrency

hiredman21:03:52

state is time 🙂

dnolen21:03:12

we really don't care about time

dnolen21:03:48

we have a purely functional system - for what we want to test - no externalities are of interest

dnolen21:03:19

value to value only - including all storage of interest (Datomic and custom data structures for cryptographic purposes)

dnolen21:03:33

thus test.check is attractive

hiredman21:03:41

sure, but if you have a succession of states, that is a ticking clock (how things like event clocks work)

dnolen21:03:58

but it's not of any interest at all

dnolen21:03:11

in fact what we are interested is only the cryptographic results at the end of sequence

dnolen21:03:19

they must hold - nothing else really matters

dnolen21:03:21

the key is actually we are in fact looking for the shortest sequences that would break some invariant - I don't think the simulation stuff can do that @alexmiller ?

Alex Miller (Clojure team)21:03:39

no, that's different use case