Clojurians Log v2

maybe I'm not looking in the right places - but there seems to be not much information about writing customer generators?

Not knowing that much about custom generators, after auditing some repos I have a trivial stateful generator for producing commands to drive some test

it works by recursion by feeding the augmented state to produce the next command

by doing so then it becomes easy to make the command sequence more meaningful, by i.e. decreasing the frequency that the last command will be chosen, etc. etc.

but before proceeding any further I did a quick check (ba-dum) and of course it did not shrink

instead of generating a single command generate a list of commands, and statefully process each list, with a fresh state for each list

basically you want to generate programs, instead of generating a single command and building programs out of the generated command

if you generate a whole program then the whole program can shrink

the recursive thing does indeed produce a list of commands - but that is what doesn't shrink

ah, I thought you where doing the recursive thing outside of generating

looking at the built in generators - it appears that shrinking might be a bit manual

like you have to make the shrink tree yourself?

there is https://github.com/clojure/test.check/blob/master/src/main/clojure/clojure/test/check/generators.cljc#L1641 which might help

I've never had to manually do shrinking

I usually generate programs then feed them into some interpreter, and that interpret might no-op some commands in certain states, instead of feeding the state through the generator

hrm really, in the (repo docs) the first thing that it says is that if you build a stateful generator via bind - it will not shrink very well

oh yeah, I've had to massage things around to get good shrinking

because bind is opaque

I feel like @dnolen described this to me at a conference like five years ago

I think anything other than bind being a simple maping kind of operation (take a vector of numbers and produce a java.util.Date object) isn't going to shrink well

@gfredericks haha we probably talked about it - but now I actually have a reason to use this approach at work

I feel like maybe the best thought I had on this was that it'd be useful to have a general framework for this style of generator Something that could be implemented once against the lower level primitives

only tangentially related but I found this interesting paper - https://proper-testing.github.io/papers/icst2018.pdf

I assume the style is that you have a model of the state of the system at a given time, and can generate commands based on the current state, and each command transforms the current state to the next state?

PropEr is a open source Erlang framework for testing stateful systems, and they have been implementing things to make it easier to test stateful system

both by providing a framework, but also integrated simulated annealing to control the search

I've done that sort of thing, but without feeding the state through command generation, and then ignoring invalid commands in the interpreter

I don't really care one way or another about whether test-check provides such affordances

but rather some guidance on writing custom generators that might be influenced by above tactics

so if I recur on this generator, do I need to manually populate the rose tree w/ children for the previous state

so it can shrink later?

Custom shrinks aren't available at all in the main user facing api I don't think

yes I know that 🙂

but the non-doc'ed but not private internal helpers can be used as far as I can tell (to accomplish the goal, not future-proof)

actually

By "manually popular the rose tree" do you mean something like the haskell quickcheck api where you provide a function X -> [X] X being a list of commands

(oh yeah, re: using test-check, I think that was about testing stateful UIs - barely remember that so long ago)

@gfredericks can you not just gen-fmap and create new rose w/ what you know should be the children?

You can

if you build out of fmap and bind, it will shrink

right? looks at gfredericks

This kind of thing can definitely shrink, but often it's degenerate

If you generate 200 commands

And each is logically dependent on all the earlier ones

Then shrinking command #7 invalidates the other 293 of them

So they have to be replaced with freshly generated commands based on the new state after the new command 7

Because TC can't do better than that generically

But dnolen with his domain knowledge can describe more particularly when it's valid to shrink or delete intermediate commands

And so can get much better results

👍

So the choice is A) add a new feature for this situation that lets users express that domain knowledge B) dnolen does this particular case by hand

(as I see it)

Sometimes you can get by with tricks like this one https://clojurians.slack.com/archives/C0JKW8K62/p1646769032795009

Writing your own shrink tree can be laborious because it's hard to take advantage of the built in shrinking for the atomic types you're using

I also had an idea for a user-enhanced shrinking, but not sure that's a comprehensive solution here

@gfredericks design is hard ... I'm curious as to why the shrinking stuff that's there is not just exposed and documented

it seemed like one of the novel things about test-check was that shrinking was decoupled - but then somehow not a part of the api

I don't think that was too intentional I mean you could ask Reid if you want historical context I largely took the approach of not changing things without a really strong reason So I think it's totally valid to suggest exposing it

I think the hope was that the built in shrinking was high enough quality that users wouldn't have to think about it Probably not realistic, but it works much better than the QuickCheck approach I think

I think you don't really hit this problem until you want to write stateful generators

then the domain starts to dominate and the defaults just aren't that useful - because it's mostly junk

I will continue poke around at what's there for now - still barely know how to do anything so will probably say marginally more intelligent things later 😉

Oh hey, I wrote something about this

https://github.com/clojure/test.check/blob/master/doc/old-confluence-notes.md#stateful-generators

yes already read that 🙂

several times

Hah

it is iterate (unfold) for generators

@dnolen does the idea there under "general problem" seem like it would cover your use?

not entirely sure, but it seems like https://github.com/advancedtelematic/quickcheck-state-machine takes the approach of generating events without regard to state, but has you define preconditions which say if some event + state is well formed

(I can tell the precondition is a thing, but my haskell is not good enough to immediate tell if the precondition stuff is somehow used as part of generation or just as a filter afterwards)

basically the problem that I've encountered so far is as @gfredericks alluded - steps depend on previous steps

> The pre-conditions are used while generating programs (lists of actions).

as the command grows expecting to randomly line anything up is unlikely

instead we want to look at the previous state to make intelligent but not totally predetermined steps

at the same time, if something is found, we want to it to shrink

ok manually managing the rose tree seems like it leads to better results here

@gfredericks I do see that one irritation w/ the rose tree versions is that mixing things (on accident) leads to very cryptic problems

Another option would be, given a state machine description S, generate a vector of numbers, and then in a map bind, from your start state, take the sorted possible transitions, mod the first integer and take the nth, then iterate that for each state and integer

Which would shrink based on vectors and int generators, which might not be the most intelligent shrinking, but avoids recursive generation passing the state through

hrm, the problem has more constraints then it may appear, so I don't see that approach as being less cumbersome?

in truth what I'm interested in is simulation generation

that is there is some actor A, and then they do something

well then maybe there is corresponding action for actor B

and then also a corresponding action for actor C

Sure, and I guess you'd get shrinking, but the behavior would be very odd

because frequencies is dynamic - I'm also interested in using the state at each step to dynamically bind new likelihoods

so the tools in test.check at the moment seem sufficient to accomplish what I want, without going elsewhere

I think probably only the command choice function (which generates the liklihoods and the command arguments) will be of any real interest

the rest will probably be boring

sounds like you are really doing simulation testing. we did several projects like this at Cognitect with Simulant (and other Simulant like things). there are a couple other open source things we made for those

https://github.com/candera/causatum

but we are not actually testing a whole system - we don't care about time

and we're going to drive anything that would be external

we don't even care about concurrency

state is time 🙂

we really don't care about time

we have a purely functional system - for what we want to test - no externalities are of interest

value to value only - including all storage of interest (Datomic and custom data structures for cryptographic purposes)

thus test.check is attractive

sure, but if you have a succession of states, that is a ticking clock (how things like event clocks work)

but it's not of any interest at all

in fact what we are interested is only the cryptographic results at the end of sequence

they must hold - nothing else really matters

the key is actually we are in fact looking for the shortest sequences that would break some invariant - I don't think the simulation stuff can do that @alexmiller ?

no, that's different use case