Clojurians Log v2
clojure-spec
Ok so here's a scenario I'm not sure how to approach:
I've changed some specs and introduced a bug. Now my generative tests don't work because "Couldn't satisfy such-that predicate after X tries." I'm pretty sure I've identified the spec that contains the bug but I can't tell exactly why it's wrong.
My first idea was to generate a config (even if it doesn't satisfy all the predicates) and then use explain-data on it and navigate :clojure.spec/problems. However, :clojure.spec/gen and :clojure.test.check.generators/sample refuse to even show me the generated data if it's invalid. Is there an alternative that would work in this case?
have you tried generating values directly from that "possibly broken" spec instead of the whole config?
I usually debug such problems by starting from the simplest case which generates successfully and then gradually add more restrictions until I find the spec/predicate, which fails to pass the data
Sure but I was wondering if spec could assist with that. The culprit was a custom predicate that's part of the property tests. It's used to ensure that the generated data satisfies certain properties (e.g. make sure that a key only appears once in a vector of maps). However, in the process it introduced a regression (it used concat on a vector, which turned it into a list). After some digging I figured it out and fixed it; I was just wondering if there was a less manual way to do it, with spec, to save me some time in the future. As in: 1. generate the data 2. run explain-data 3. get back a map with s/problems 4. see something like ":v should be a vector, not a list"
There has been some work in test.check to provide better reporting and eventually I would like to help surface some of that through spec for cases like this
That's good to know, thanks 🙂 To be honest I'm a bit surprised there's no way to get the generated data if it doesn't satisfy all the predicates but I'm sure there's a reason it's like that. I mean I understand it's an all or nothing situation but on the other hand having the incomplete state would be useful for debugging.
there's good reason to verify generated values match the spec, but there is a general problem with not getting example failures if no matching generated value can be found
with spec, is it good idiomatic usage/taste to say that for example a reduce over collection should satisfy a predicate? in real terms, lets imagine we have a spec of a Security. And we build a holding. So each security, that is included in holding has a weight. And naturally all weights must sum up to a 100. Is this too convoluted and going too far with spec? Goal is to use spec not only to validate inputs, but also generate artificial holdings, that make sense.
You can have arbitrary predicates composed with s/and , it’s very idiomatic IMO
e.g. your holding spec can be something like (s/and (s/coll-of ::security) holdings-add-up-to-100?) where holdings-add-up-to-100? is your predicate
nice. good to know... now off to write a generator 😂
yeah, that 😄
that's the fun part 🙂
but I would say it’s all idiomatic and good taste
especially the generator
yeah, sounded like a good idea, but it is easy to get overexcited and go in directions one was not meant to travel in...
that depends on where your specs are going to be used. if it’s for testing the code that relies on “holdings add up to 100” invariant, then it makes sense to invest in generator. If that code is not of super importance, you can have your generator to be a set of examples as a starting point…
yeah, ideally we would use this extensively in unit tests. First, to catch edge cases that real data does not expose easily. and second to get rid of huge and obnoxius fixtures. 🙂
but a good point, generator writing is optional.