This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2022-03-08
Channels
- # aleph (2)
- # announcements (2)
- # asami (50)
- # babashka (39)
- # beginners (17)
- # calva (61)
- # cider (9)
- # clj-kondo (5)
- # clojure (37)
- # clojure-europe (52)
- # clojure-nl (1)
- # clojure-norway (14)
- # clojure-uk (5)
- # clojurescript (28)
- # cursive (3)
- # datahike (11)
- # datomic (28)
- # deps-new (11)
- # events (3)
- # fulcro (18)
- # google-cloud (1)
- # graphql (8)
- # introduce-yourself (4)
- # jobs (2)
- # leiningen (7)
- # lsp (15)
- # pathom (9)
- # re-frame (6)
- # reagent (35)
- # reitit (17)
- # releases (1)
- # shadow-cljs (20)
- # specter (1)
- # test-check (106)
- # tools-deps (8)
- # uncomplicate (1)
- # vim (29)
maybe I'm not looking in the right places - but there seems to be not much information about writing customer generators?
Not knowing that much about custom generators, after auditing some repos I have a trivial stateful generator for producing commands to drive some test
by doing so then it becomes easy to make the command sequence more meaningful, by i.e. decreasing the frequency that the last command will be chosen, etc. etc.
but before proceeding any further I did a quick check (ba-dum) and of course it did not shrink
instead of generating a single command generate a list of commands, and statefully process each list, with a fresh state for each list
basically you want to generate programs, instead of generating a single command and building programs out of the generated command
the recursive thing does indeed produce a list of commands - but that is what doesn't shrink
there is https://github.com/clojure/test.check/blob/master/src/main/clojure/clojure/test/check/generators.cljc#L1641 which might help
I usually generate programs then feed them into some interpreter, and that interpret might no-op some commands in certain states, instead of feeding the state through the generator
hrm really, in the (repo docs) the first thing that it says is that if you build a stateful generator via bind - it will not shrink very well
I feel like @dnolen described this to me at a conference like five years ago
I think anything other than bind being a simple maping kind of operation (take a vector of numbers and produce a java.util.Date object) isn't going to shrink well
@gfredericks haha we probably talked about it - but now I actually have a reason to use this approach at work
I feel like maybe the best thought I had on this was that it'd be useful to have a general framework for this style of generator Something that could be implemented once against the lower level primitives
only tangentially related but I found this interesting paper - https://proper-testing.github.io/papers/icst2018.pdf
I assume the style is that you have a model of the state of the system at a given time, and can generate commands based on the current state, and each command transforms the current state to the next state?
PropEr is a open source Erlang framework for testing stateful systems, and they have been implementing things to make it easier to test stateful system
both by providing a framework, but also integrated simulated annealing to control the search
I've done that sort of thing, but without feeding the state through command generation, and then ignoring invalid commands in the interpreter
I don't really care one way or another about whether test-check provides such affordances
but rather some guidance on writing custom generators that might be influenced by above tactics
so if I recur on this generator, do I need to manually populate the rose tree w/ children for the previous state
Custom shrinks aren't available at all in the main user facing api I don't think
but the non-doc'ed but not private internal helpers can be used as far as I can tell (to accomplish the goal, not future-proof)
actually
By "manually popular the rose tree" do you mean something like the haskell quickcheck api where you provide a function X -> [X] X being a list of commands
(oh yeah, re: using test-check, I think that was about testing stateful UIs - barely remember that so long ago)
@gfredericks can you not just gen-fmap
and create new rose w/ what you know should be the children?
You can
if you build out of fmap and bind, it will shrink
right? looks at gfredericks
This kind of thing can definitely shrink, but often it's degenerate
If you generate 200 commands
And each is logically dependent on all the earlier ones
Then shrinking command #7 invalidates the other 293 of them
So they have to be replaced with freshly generated commands based on the new state after the new command 7
Because TC can't do better than that generically
But dnolen with his domain knowledge can describe more particularly when it's valid to shrink or delete intermediate commands
And so can get much better results
:thumbsup:
So the choice is A) add a new feature for this situation that lets users express that domain knowledge B) dnolen does this particular case by hand
(as I see it)
Sometimes you can get by with tricks like this one https://clojurians.slack.com/archives/C0JKW8K62/p1646769032795009
Writing your own shrink tree can be laborious because it's hard to take advantage of the built in shrinking for the atomic types you're using
I also had an idea for a user-enhanced shrinking, but not sure that's a comprehensive solution here
@gfredericks design is hard ... I'm curious as to why the shrinking stuff that's there is not just exposed and documented
it seemed like one of the novel things about test-check was that shrinking was decoupled - but then somehow not a part of the api
I don't think that was too intentional I mean you could ask Reid if you want historical context I largely took the approach of not changing things without a really strong reason So I think it's totally valid to suggest exposing it
I think the hope was that the built in shrinking was high enough quality that users wouldn't have to think about it Probably not realistic, but it works much better than the QuickCheck approach I think
then the domain starts to dominate and the defaults just aren't that useful - because it's mostly junk
I will continue poke around at what's there for now - still barely know how to do anything so will probably say marginally more intelligent things later 😉
Oh hey, I wrote something about this
https://github.com/clojure/test.check/blob/master/doc/old-confluence-notes.md#stateful-generators
@dnolen does the idea there under "general problem" seem like it would cover your use?
not entirely sure, but it seems like https://github.com/advancedtelematic/quickcheck-state-machine takes the approach of generating events without regard to state, but has you define preconditions which say if some event + state is well formed
(I can tell the precondition is a thing, but my haskell is not good enough to immediate tell if the precondition stuff is somehow used as part of generation or just as a filter afterwards)
basically the problem that I've encountered so far is as @gfredericks alluded - steps depend on previous steps
instead we want to look at the previous state to make intelligent but not totally predetermined steps
@gfredericks I do see that one irritation w/ the rose tree versions is that mixing things (on accident) leads to very cryptic problems
Another option would be, given a state machine description S, generate a vector of numbers, and then in a map bind, from your start state, take the sorted possible transitions, mod the first integer and take the nth, then iterate that for each state and integer
Which would shrink based on vectors and int generators, which might not be the most intelligent shrinking, but avoids recursive generation passing the state through
hrm, the problem has more constraints then it may appear, so I don't see that approach as being less cumbersome?
because frequencies
is dynamic - I'm also interested in using the state at each step to dynamically bind new likelihoods
so the tools in test.check at the moment seem sufficient to accomplish what I want, without going elsewhere
I think probably only the command choice function (which generates the liklihoods and the command arguments) will be of any real interest
sounds like you are really doing simulation testing. we did several projects like this at Cognitect with Simulant (and other Simulant like things). there are a couple other open source things we made for those
we have a purely functional system - for what we want to test - no externalities are of interest
value to value only - including all storage of interest (Datomic and custom data structures for cryptographic purposes)
sure, but if you have a succession of states, that is a ticking clock (how things like event clocks work)
in fact what we are interested is only the cryptographic results at the end of sequence
the key is actually we are in fact looking for the shortest sequences that would break some invariant - I don't think the simulation stuff can do that @alexmiller ?
no, that's different use case