Fork me on GitHub
#test-check
<
2017-01-02
>
paulspencerwilliams20:01:47

Hi. I’ve a question about property based testing and test.check and their suitability to the following integration validation. If I wanted to check that function

(first-available-timeslot {:attendees 3 :starts 2017-02-03 :ends 2017-02-03 :duration 60})
returns the first 1 hour slot not conflicting with a generated set of random meetings in the same calendar. The properties in english would be “result doesn’t overlap with other meetings”, “time slot is in a meeting room with at least a capacity of :attendees" and somehow “the result is the first suitable result. Am I thinking about this the right way?

gfredericks20:01:47

I assume the function can fail if the constraints aren't satisfiable

gfredericks20:01:50

Are you planning on just testing with that piece of example data you showed, or generating that input as well?

paulspencerwilliams20:01:19

Correct, and generating input also.

gfredericks20:01:02

I think the first two properties are good

gfredericks20:01:30

The third has slightly more of the "duplicating the business logic" flavor

gfredericks20:01:55

But might be easy enough as to be worth it

paulspencerwilliams20:01:42

okay. I’ll have a bash. My concern was that integration tests are expensive, and having many tests with large datasets even more so, although they obviously offer far more confidence.

gfredericks20:01:16

Expensive meaning dev time? Waiting for tests?

paulspencerwilliams20:01:12

Waiting for tests. Further, the query might operate against a db, or possibly google calendars api so there’s timeouts and error conditions complicating things.

gfredericks20:01:54

I would try to mock out things you don't control

gfredericks20:01:23

Unreliable tests are even more annoying in PBT

gfredericks20:01:48

Shrinking can be almost meaningless

paulspencerwilliams21:01:44

Yeah, that’s what I was thinking. I actually wonder whether syncing a copy of the calendar into a clojure owned data structure before querying might allow better reasonability. It’s only a small dataset.

gfredericks21:01:41

Or generate the calendar

gfredericks21:01:04

A property I just thought of that is a corollary of the "earliest time slot" requirement:

gfredericks21:01:28

"Given two reservation requests where the first is strictly easier to satisfy than the second, processing them in order should never result in the second being scheduled before the first

gfredericks21:01:22

Meaning they should have the same date range, and the second should involve at least as many people and minutes

paulspencerwilliams21:01:12

Just parsing that...

paulspencerwilliams21:01:59

Got it, yes. It’s these property identifications that make it difficult, and require a lot of experience.

gfredericks21:01:00

I'm about to sacrifice my phone for the sake of providing entertainment to my offspring, so that may be all the pondering I can do for now

paulspencerwilliams21:01:27

@gfredericks thanks for the help. I will explore this further tomorrow and enjoy the rest of your evening!

mattly21:01:19

is there any way to instrument a set of generators for a test to track performance?

mattly21:01:00

I've got what, to me, seems like a relatively simple generation set that takes 20+ seconds per iteration for some reason

mattly21:01:46

there's no recursion, and the only such-that checks that a basic keyword produces something not part of a previously-generated set

hiredman21:01:00

that definitely will slow things down

hiredman21:01:23

depending on what you are doing, you could generate a set, then use fmap to remove an element and return a tuple of [removed-element set] to get a similar effect without thrashing such-that

mattly22:01:36

that could work

mattly22:01:54

as it is I'm using a modified version of test.chuck's testing that will output things on each iteration

mattly22:01:03

I'm modifying that to display gen vs. test times

mattly22:01:59

and then doing simple one-off tests with each of my generators

gfredericks22:01:02

I doubt thrashing such-that would have such severe consequences unless you manually raised the iteration limit

gfredericks22:01:12

It gives up after 10 tries

mattly22:01:21

yeah, it's not that so far as I can tell

mattly23:01:47

heh, as usual, my takeaway is "don't use generators for things you don't need to test the properties of"

mattly23:01:12

replacing (gen/map gen/keyword some-other-gen {:min-elements 1 :max-elements 20}) with

mattly23:01:06

reduced the time spent running generators to about an eighth of the origional

gfredericks23:01:14

That's unfortunate

gfredericks23:01:29

But sounds plausible

mattly23:01:26

the keys don't really have any special properties here, it's mostly, I'm testing the map of some-other-gen here and how that effects the code I'm testing

mattly23:01:49

but it literally went from

mattly23:01:03

...this being the output of my runner

mattly23:01:34

I have another project where I did something similar

mattly23:01:01

it's a parser of sorts and I needed a bunch of word-looking text

mattly23:01:53

I started with (gen/not-empty (gen/string-alphanumeric)) and ended up switching to (gen/elements a-set-of-words-from-lorem-text) because it was a similar magnitude faster

mattly23:01:43

so I've started having to ask on these things, ok, what am I actually testing the properties of here?

mattly23:01:56

and what can I get away with not generating