Fork me on GitHub
#test-check
<
2017-04-27
>
gfredericks00:04:45

Yeah, test.check hopes that the public API is sufficient

gfredericks00:04:16

I assume you see the bug in your code?

gfredericks00:04:58

Or, why that function doesn't/ can't work

bbloom00:04:41

other than my typo with the wrong variable? and the lack of the “AST interpreter” that would drive that protocol? no, not really

bbloom00:04:48

at least i assume you have some insight i’m missing

gfredericks00:04:31

In the fmap str example, you'll be passing a string to a shrinker that expects integers

gfredericks00:04:09

value is a string, (shrink x ...) wants an integer

bbloom00:04:17

i was thinking the value would be whatever value was being wrapped

gfredericks00:04:20

Somebody somewhere will have to parse the string

bbloom00:04:20

not the generated value

bbloom00:04:38

i you’d have a tree of alternating generate/shrink calls

bbloom00:04:17

the idea was that fmap wouldn’t do any shrinking at all

bbloom00:04:23

or at least no extra shrinking

bbloom00:04:29

beyond what the wrapped generator does

gfredericks00:04:52

That's ideal, I'm just not sure how you wrap this up into the test runner in an arbitrarily composable way

bbloom00:04:11

i’m not even convinced that it makes sense to combine generating and shrinking one-to-one like this

bbloom00:04:23

i just assumed that shrinking would be simpler than generating in that the tree would be a subset of the generator tree

bbloom00:04:35

and only isomorphic to it if most of the nodes were the identity-shrinker

bbloom00:04:02

for example if you generate an arbitrary number and then filter the even ones, you only want one number shrinker, not two

bbloom00:04:31

similarly, for the stateful testing case with shrinking, you almost only want to remove actions, not actually alter the contents of any of those actions

bbloom00:04:02

way beyond just generative testing, i’m not a fan of monadic apis in general…. i’d much rather build a generator out of transparent data and then interpret (or compile) that - especially so that i could add more methods beyond just generate/shrink….

bbloom00:04:20

but this is way off topic, my apologies

gfredericks00:04:58

I'm not even sure what sort of tree you're imagining

bbloom00:04:44

the tree of generators that is returning a pure value (that is, of course, tree shaped)

gfredericks00:04:20

The shape of generator composition doesn't necessarily mirror the shape of the data

bbloom00:04:02

sure, it’s not 1-to-1, but it has some similarity in general, no?

gfredericks00:04:16

I've thought about generators-as-data, but seems hard to do bind

bbloom00:04:30

bind (well, lexical scope in general) is always the hard part

gfredericks00:04:29

So test.check has an approach where bind at least works 😛

gfredericks00:04:08

So you're imagining that the user-facing API for your generators is the protocol of two functions? If so I'm still not seeing how you get around the fmap problem

bbloom00:04:35

i’m not trying to propose a new solution

bbloom00:04:48

i’m saying that it didn’t match my intuition - which has now been corrected

bbloom00:04:08

but the larger problem i have is that, even understanding how it works, i’m not sure how to make effective use of it

bbloom00:04:20

let’s take the stack example again

bbloom00:04:01

if i have random push/pop operations and it’s going to do shrinking, it’s going to waste a bunch of time pushing or popping smaller integers, when really i want to try fewer pops or fewer pushes

gfredericks00:04:44

It will shrink each integer straight to 0

gfredericks00:04:05

Only if that doesn't work does it try other things

gfredericks00:04:14

So N steps total to accomplish that

gfredericks00:04:28

But it could also try removing things first, I'm not sure

gfredericks00:04:06

That depends on which step bind tries first

gfredericks00:04:24

One downside to this structure is you can't give shrink hints

gfredericks00:04:08

I think shrinking efficiency is only an issue when generating really large compositions

gfredericks00:04:45

I suppose I shouldn't say there's no way to give hints; for your example, if it was a problem, you could wrap the integer generator in gen/no-shrink

gfredericks00:04:35

test.check does this itself for the builtin UUID generator, on the assumption that there's not much to be gained from shrinking a UUID

gfredericks00:04:01

w.r.t. the earlier question on large-integer boundary testing, that's a more general issue I'm aware of but haven't been sure how clever it should try to be

gfredericks00:04:16

I've considered rebooting the generators namespace to clean up the API; maybe there could be a partition into lower-level generators and higher-level clever ones

gfredericks00:04:49

the low level ones do simple easy to understand things, the high level ones do lots of ad-hoc stuff to try to break your code

bbloom00:04:52

interesting

bbloom00:04:59

i’m gonna have a think on all this

bbloom00:04:19

thanks again

nberger02:04:25

Hey guys, great discussion here. I think the announcement of hedgehog, a new quickcheck-like library for haskell is very relevant: https://www.reddit.com/r/haskell/comments/646k3d/ann_hedgehog_property_testing/. One of its selling points is that it "uses integrated shrinking, so shrinks obey the invariants of generated values by construction [...] an approach to property testing which is also used by QuiviQ's Erlang QuickCheck, Hypothesis, and test.check"

gfredericks02:04:18

Oh nice - I forgot about that

nberger02:04:27

I see references to a shrink tree in the code. But can't say much more 🙂

nberger02:04:00

@gfredericks: btw I made some progress on the async quick-check. I pushed a draft to my repo today: https://github.com/nberger/test.check/compare/test.check.refactor...refactor-allow-async. I'm working on a defspec-async macro now that would use that stuff and would make the tests much easier to write (for example it "knows" when to call the done from cljs.test/async, which is on :succeeded or :shrunk) so the :step-fn that I'm using in the tests would be gone

gfredericks02:04:55

@nberger someone made a ticket asking for that

nberger02:04:09

yep, saw it. I've been thinking about this for a long time, the ticket made me go to the keyboard and try to make it work, and I'm quite happy with the result so far

nberger02:04:47

I'm hoping to have a more rounded version soon

gfredericks02:04:57

cool, let me know and I'll take a look