Fork me on GitHub
#test-check
<
2022-12-13
>
skylize14:12:40

I don't understand why this causes "Couldn't generate enough distinct elements!"

(defn foo [gen]
  (gen/vector-distinct gen {:num-elements 10}))

(defspec foo-test
  (prop/for-all [x (gen/let [n (gen/resize 10 gen/nat)]
                     (foo (gen/vector gen/any n)))]
    true))

(foo-test)
Works fine if I replace the gen/let wrapper with an inline constant.
(defspec foo-test
  (prop/for-all (fgen/foo (gen/vector gen/any 5))
    true))
Is there a better way to do this? or some way to work around the problem?

gfredericks14:12:58

Does it happen when n is zero?

skylize14:12:16

This was stripped way down to highlight the issue. Here is real code. unary, shown for easier explanation of the concept, takes gens for x and y, and generates a function f mapping x -> y, (returning all 3 in a map). n-aries is the same basic idea. But the number of xs is variable (i.e. generating n-arity of function instead of unary) and the number of xs -> y mappings is also variable, so the generated function can return different ys depending on specific inputs. The arity (or count of xs) should be determined by xs-gen (which is the problematic n generated in the spec). The count of xs-> y mappings is determined by count-gen. An empty xs should be a valid result from xs-gen. But xss needs to be a list of unique xs.

;; fgen.cljc

; unary

(defn- x-y->map [X Y]
  {:x X
   :y Y
   :f (fn [x]
        (if (= X x)
          Y
          (throw (ex-info "generated function called with incorrect arguments"
                          {:expected X
                           :actual x}))))})

(defn unary [x-gen y-gen]
  (gen/let [x x-gen
            y y-gen]
    (x-y->map x y))

; n-aries

(defn- xss-ys->map [xss ys] 
  (let [xss-ys (zipmap xss ys)]
    {:xss-ys xss-ys
     :f (fn [& xs]
          (if (some #(= % xs) xss)
            (xss-ys xs)
            (throw (ex-info "generated function called with incorrect arguments"
                            {:expected {:one-seq-of xs}
                             :actual xs}))))}))

(defn n-aries [count-gen xs-gen y-gen]
  (gen/let [n count-gen
            xss (gen/vector-distinct xs-gen {:num-elements n})
            ys (gen/vector y-gen n)]
    (xss-ys->map xss ys)))
; fgen_test.cljc

(defspec n-aries
  200
  (prop/for-all [{f :f {xs :xs ys :ys} :xs-ys}
                 (gen/let [n (gen/resize 10 gen/nat)]
                   (fgen/n-aries
                    (gen/resize 10 gen/nat)
                    (gen/vector gen/any n)
                    gen/any))]
    (or (util/empties? xs ys)
        (= ys (map f xs)))))

gfredericks14:12:02

Why is uniqueness important?

skylize14:12:29

Because the goal is to generate a function f, which has deterministic behavior y based on its input value(s) xs.

gfredericks14:12:11

Then why is generating a minimum number of things important?

skylize14:12:11

Additionally, I am representing that determinism internally as a map where xs is a key on a map of xs -> ys.

skylize14:12:48

Who said anything about a minimum number of things?

gfredericks14:12:11

Your gen-vector-distinct call did

skylize14:12:51

The gen-distinct call asks for a specific number of things, where that number is generated by a user-provided generator. The user could choose to provide (gen/return 0) or gen/nat. That is not for me to decide.

gfredericks14:12:21

What does n represent there, in the context of generating functions?

skylize14:12:35

n in n-aries (not causing problems and replaced by 10 in the foo example) is the number of mappings from xs-> ys. n in the spec (which is causing problems) represents the arity of the target generated function. i.e (defn foo [x y z]) is (= n 3), and (defn foo []) is (= n 0). Again this will ultimately be provided by the user, when they define an xs-gen to pass into n-aries.

gfredericks14:12:27

Do you at least agree that the error in your simplified example makes sense?

skylize14:12:47

Yes. I see it now. In the case my test gens (= n 0) , foo cannot generate distinct data.

skylize14:12:39

But that still leaves me at a loss how to put n-aries through its paces, when (= n 0) for should be a valid possibility. 😞

skylize15:12:21

Hmm. I think with that in mind, the issue is not the test. Somehow n-aries needs to be smarter about comparing the result of count-gen with the results of xs-gen to avoid nonsensical attempts at uniqueness.

gfredericks15:12:28

A 0-ary pure function can only have one input/output pair

gfredericks15:12:44

It's effectively a constant

skylize15:12:34

Yes. I don't know what that though. The fact a 0-arity pure function is a constant does not obviate the need for someone testing a higher-order function check how it handles being given a 0-arity function, among others.

gfredericks15:12:39

I haven't internalized the details of the situation, but if you're getting a "can't generate enough distinct elements" error because of a trivial generator, then you should probably either question the need for distinctness or minimum elements, or else consider that your api allows users to ask for impossible things

skylize04:12:40

The problem was just the location of let. It should be wrapping the generator passed to foo to keep generating new nvalues, rather than binding a single n ahead of calling foo.

(defspec foo-test
  (prop/for-all [x (foo (gen/let [n (gen/resize 10 gen/nat)]
                          (gen/vector gen/any n)))]
    true))
... instead of ...
(defspec foo-test
  (prop/for-all [x (gen/let [n (gen/resize 10 gen/nat)]
                     (foo (gen/vector gen/any n)))]
    true))

gfredericks14:12:33

If I'm reading correctly, in that scenario you're asking it to generate ten distinct empty vectors

gfredericks14:12:29

I can't tell what you're actually trying to do, so it's not clear what would count as a fix