Clojurians Log v2

Yeah, test.check hopes that the public API is sufficient

I assume you see the bug in your code?

Or, why that function doesn't/ can't work

other than my typo with the wrong variable? and the lack of the “AST interpreter” that would drive that protocol? no, not really

at least i assume you have some insight i’m missing

In the fmap str example, you'll be passing a string to a shrinker that expects integers

value is a string, (shrink x ...) wants an integer

i was thinking the value would be whatever value was being wrapped

Somebody somewhere will have to parse the string

not the generated value

i you’d have a tree of alternating generate/shrink calls

the idea was that fmap wouldn’t do any shrinking at all

or at least no extra shrinking

beyond what the wrapped generator does

That's ideal, I'm just not sure how you wrap this up into the test runner in an arbitrarily composable way

i’m not even convinced that it makes sense to combine generating and shrinking one-to-one like this

i just assumed that shrinking would be simpler than generating in that the tree would be a subset of the generator tree

and only isomorphic to it if most of the nodes were the identity-shrinker

for example if you generate an arbitrary number and then filter the even ones, you only want one number shrinker, not two

similarly, for the stateful testing case with shrinking, you almost only want to remove actions, not actually alter the contents of any of those actions

way beyond just generative testing, i’m not a fan of monadic apis in general…. i’d much rather build a generator out of transparent data and then interpret (or compile) that - especially so that i could add more methods beyond just generate/shrink….

but this is way off topic, my apologies

I'm not even sure what sort of tree you're imagining

the tree of generators that is returning a pure value (that is, of course, tree shaped)

The shape of generator composition doesn't necessarily mirror the shape of the data

sure, it’s not 1-to-1, but it has some similarity in general, no?

I've thought about generators-as-data, but seems hard to do bind

bind (well, lexical scope in general) is always the hard part

So test.check has an approach where bind at least works 😛

So you're imagining that the user-facing API for your generators is the protocol of two functions? If so I'm still not seeing how you get around the fmap problem

i’m not trying to propose a new solution

i’m saying that it didn’t match my intuition - which has now been corrected

but the larger problem i have is that, even understanding how it works, i’m not sure how to make effective use of it

let’s take the stack example again

if i have random push/pop operations and it’s going to do shrinking, it’s going to waste a bunch of time pushing or popping smaller integers, when really i want to try fewer pops or fewer pushes

right?

Somewhat

It will shrink each integer straight to 0

In one step

Only if that doesn't work does it try other things

So N steps total to accomplish that

But it could also try removing things first, I'm not sure

That depends on which step bind tries first

One downside to this structure is you can't give shrink hints

I think shrinking efficiency is only an issue when generating really large compositions

I suppose I shouldn't say there's no way to give hints; for your example, if it was a problem, you could wrap the integer generator in gen/no-shrink

test.check does this itself for the builtin UUID generator, on the assumption that there's not much to be gained from shrinking a UUID

w.r.t. the earlier question on large-integer boundary testing, that's a more general issue I'm aware of but haven't been sure how clever it should try to be

I've considered rebooting the generators namespace to clean up the API; maybe there could be a partition into lower-level generators and higher-level clever ones

the low level ones do simple easy to understand things, the high level ones do lots of ad-hoc stuff to try to break your code

¯\(ツ)/¯

interesting

i’m gonna have a think on all this

thanks again

Hey guys, great discussion here. I think the announcement of hedgehog, a new quickcheck-like library for haskell is very relevant: https://www.reddit.com/r/haskell/comments/646k3d/ann_hedgehog_property_testing/. One of its selling points is that it "uses integrated shrinking, so shrinks obey the invariants of generated values by construction [...] an approach to property testing which is also used by QuiviQ's Erlang QuickCheck, Hypothesis, and test.check"

Oh nice - I forgot about that

I see references to a shrink tree in the code. But can't say much more 🙂

@gfredericks: btw I made some progress on the async quick-check. I pushed a draft to my repo today: https://github.com/nberger/test.check/compare/test.check.refactor...refactor-allow-async. I'm working on a defspec-async macro now that would use that stuff and would make the tests much easier to write (for example it "knows" when to call the done from cljs.test/async, which is on :succeeded or :shrunk) so the :step-fn that I'm using in the tests would be gone

@nberger someone made a ticket asking for that

yep, saw it. I've been thinking about this for a long time, the ticket made me go to the keyboard and try to make it work, and I'm quite happy with the result so far

I'm hoping to have a more rounded version soon

cool, let me know and I'll take a look

will do