Any good tools for debugging slow generators? We have lots of specs with tens of s/and preds & custom gens associated. I'd like to know which predicate is causing a such-that to fail often.

I bet you could alter-var-root such-that to default to a lower max-tries

looks like spec just uses the default (10) as far as I can tell

so if you change that to, e.g., 2, you might get more failures

but maybe that's not useful if the "telling you which s/and failed" feature isn't wired up yet

> isn't wired up yet Is a feature like this on the roadmap?

Good idea on the such-that max-tries thing. That should help a bit. Though, the such-that errors are usually pretty obtuse, hiding the actual name of the predicate behind an anonymously named test.check pred.

spec's roadmap, yes, as far as I know; that feature couldn't exist prior to the 0.10.0 versions of test.check

> the such-that errors are usually pretty obtuse that's exactly the problem that the aforementioned up-wiring would address

Ah, very cool. Debugging these issues has been a huge time sink for us. It seems like it should be a fairly easy change to spec. What feature in 0.10.0 allows this better up-wiring?

@kenny 95% of the time the issue with slow generators is generating large (and particularly nested large) collections

setting :gen-max 3 in s/coll-of, s/map-of etc is often a big difference

This structure is highly nested & has lots of colls. We have these opts set:

:coll-check-limit 10
:coll-error-limit 10
:fspec-iterations 10
:recursion-limit  1

none of those affect the gen'ed count

with a 3 level coll, you could have 10x10x10 nested elements

Oh, right - that param is on the actual generator. Lemme see if we have those set.

well, recursion limit might affect the depth (although I think there are some scenarios where it is not getting applied right now)

if you have fspecs, that's another possible issue where you might want to consider simpler preds

like ifn?

We have considered that a number of times simply due to the massive amount of time debugging generators take. Every time, however, we always decide not to because those fspec generators catch valuable bugs haha

the eternal test.check conundrum

another testing tip is that gens will get run 1000 times in check and size/complexity increases

so gen/sample by default (20) will not expose generator issues

but you can gen/sample with 1000 and will see those

if you separate the args spec for an fdef into its own spec, it's easy to test that

(s/fdef foo :args ::args-spec), then (gen/sample (s/gen ::args-spec) 1000)

I searched for :gen-max in this service and didn't find any so it may not be used. Though, there are tons of custom gens which use the gen/* functions which don't take in :gen-max, I think. So that's the usual way to debug the slow gens -- something like this (do (doall (gen/sample (s/gen ::db/db-control-args) 1000)) nil). Then that will hang forever and that's where the pain starts.

I would try just blindly adding :gen-max 3 to all the collection specs

@kenny what version of test.check are you using?

there is a ticket specifically for changing this default (or exposing a global default) btw

Will try that. We're using 0.10.0-alpha3.

Yeah, that would help to figure out if the problem is collection gen or something else.

okay, that should avoid some exacerbatory issues with the 0.9.0 release

Haha, yeah. I remember a huge perf bump when switching from 0.9.0

oh good

Another thing that slows down debugging slow gens is that the check tests still run in the background even though I interrupted the current op in the REPL.

I often will need to restart the REPL to get it to stop.

is that a pmap thing?

We're not using that. Does test.check?

I think spec does

but could be wrong

pmap is in the check impl.

Maybe a call to shutdown-agents would help?

Getting a frequencies map returned after generating a spec with s/and would also be super helpful. The keys would be the forms of each s/and pred.

well shutdown-agents will shutdown the pool never to restart again, so you'd still need to bounce your repl

pmap is lazily parallel

so when you stop using the result, up to n-1 (where n is number of processors) may still be working to compute the next few results in the lazy seq

Hmm. It seems to go on forever. I’ll add prints to some functions to debug the slow gens and it never stops printing after interrupting.

Perhaps because of the doall in this line? (do (doall (gen/sample (s/gen ::db/db-control-args) 100)) nil)

This is helpful :

(let [such-that gen/such-that]
  (with-redefs [gen/such-that (fn
                                ([pred gen]
                                 (such-that pred gen 1))
                                ([pred gen opts]
                                 (such-that pred gen 1)))]
    (gen/generate (s/gen ::db-control-args))))

Actually, @gfredericks the problem with taking the above approach is simple generators sometimes need the such-that. e.g.

(s/def ::int+-interval
  (s/and (s/tuple ::m/int+ ::m/int+)
         (fn [[x1 x2]] (>= x2 x1))))

I suppose you could write a custom gen for an interval.

@kenny yeah the general rule with such-that is that the predicate needs to be highly-likely to succeed; so for something like an interval you'd want to write your own generator, which is pretty easy

I mean, it is highly likely to succeed in the above case.

isn't it just 50%?

I think so

more precisely what you want is "astronomically unlikely to fail 10 times in a row"

and something that only passes 50% of the time doesn't have that quality

Fair enough

as such-that retries, it's also incrementing size, so often it's good enough that the probability of failure goes down with larger size

e.g., gen/not-empty relies on that

It almost seems like you should be able to enable a warning that tells you when your specs use s/and and don't have a custom gen.

Writing the generator for an interval, like the above is tricky without such-that because you don't know how to generate a value that will match the spec. I was originally thinking it'd just be a gen/bind but that doesn't work because I don't know how to generate a ::m/int+ such that the second one is greater than the first without using such-that.

one option: before doing a such-that add something like this https://github.com/bfabry/specify-it/blob/master/src/specify_it/bst_spec.clj#L268 that just reports on how often the such-that would succeed. then tune your inputs until it’s a reasonable frequency

Well this is a reasonable frequency but it's certainly not "astronomically unlikely to fail 10 times in a row"

It's fairly unlikely though.

in the case that you’re going to throw away generations that don’t match your predicate, I would put “reasonable frequency” at like 50%

That seems right. This would go back to the original problem then where, in practice, overriding such-that doesn't really work as a debugging technique.

a test.check generator mode where all of the various predicates print out their form line number and % success rate at the end would be cool (and quite hard to do)

That's exactly what I want too 🙂

@kenny w.r.t. avoiding such-that, how about fmap with sort?

Ooo, I like it!

fmap can be used to avoid such-that in a lot of cases another example is here: https://clojure.org/guides/test_check_beginner#_generator_combinators

@gfredericks That cleaned up this code nicely 🙂 Thanks for the idea! https://github.com/Provisdom/math/blob/488573a16947eab78ecd096c7ef71a33f94cd0d7/src/provisdom/math/intervals.clj#L14-L55

I would generate the starting value and the range size, then fmap to get [start (+ start size)]

I don't think that'd work because you'd need to know how to generate the range.

For example, we have a "prob-interval" where the interval's lower and upper bounds must be between 0 and 1.

One issue with @gfredericks approach, however, is if you have a strict-interval. You have a chance of hitting the such-that then.

Sort+inc?

Can't be that simple because of the above interval case.

"prob-interval", that is

You could add Double/MIN_VALUE in that case. But if the interval is integers only, that won't work.

Well, even adding Double/MIN_VALUE may not work because that may go outside the range.

Math/nextUp 🙂

I think that's still problematic for the same reason though.

Why isn't generator? public in spec's gen ns?

Might be an oversight, might be a reason it won’t work with dynaload

Same with vector-distinct-by

Might be able to add them to our own libs by using lazy-combinators

Won't work because lazy-combinator isn't qualified 😞 https://github.com/clojure/spec.alpha/blob/5228bb75fa10b7b13ea088d84f4d031cce74a969/src/main/clojure/clojure/spec/gen/alpha.clj#L92

Could s/coll-of take a new option for :distinct-by?

I need something like (s/tuple (s/tuple ::a) (s/tuple ::b)), but should allow values like [["a" "d"] ["b"]] (ignore "extra" tuple values)