The docs suggest using stest/abbrev-result to get more info about what happened when it failed, https://clojure.org/guides/spec#_testing. Have you tried looking at that info?

if i do as per docs (stest/abbrev-result (first (stest/check mean`) i do not see any more relevant information. There is the sample case that "failed", but it is correct input and if i give it to mean or its validators/helpers everything seems fine :/

What is the failing sample?

it is a bit random. sometimes it says spec failed and collection is [ 1.0000004768371582 1.000000238418579 2.147483648E9 ] and calculated mean is 7.158278833333336E8 other times there is nothing inside, other than the message in exception "additional data must not be null".

stacks point at explainInfo --- Contents: 0. clojure.spec.test.alpha$explain_check 1. invokeStatic 2. "alpha.clj" 3. 390

https://clojurians-log.clojureverse.org/clojure-spec/2019-08-02

found some old @U064X3EF3 explanations about similiar thing... but unsure if that still applies...

one thing that is suspicious is that mean is called in the validation function. I would try to narrow down where the problem might be. Examples: • run stest/check with just the :args specification • run with just the :ret • run with just :fn • run with just :fn and only check within-range? • etc

yeah, that mean call maybe sth... i changed the avg-unchanged-by-order? to be a dummy that always returns true and now 1000 tests pass...

thanks 🙂

I think there’s a way to write those sorts of checks, but I’m not sure how to do it off the top of my head.

They’re definitely supported by test.check

i would think calling function under test is ok... it is not like recursion...

yeah, all pass if i return true, but still do all the calcs and logic. so just calling mean seems to not be the issue

Another thing that just popped into my head is that your test expects the order of the numbers not to make a difference in the calculated mean, but that's not true for all floating point numbers!

https://stackoverflow.com/questions/21373865/different-results-when-adding-same-doubles-in-different-order

not sure why that didn't occur to me earlier :doh:

You saved my bacon sir! Swapped the spec to be ints, not doubles, and behold it is rock solid now. That mysterious error pops up when the spec is non-deterministic, which Alex has said years ago more less. So a better error message would be nice 🙂

So a new question arises, what now? What would be best way to move forward? 1. drop the invariant (yikes...) 2. replace the generator in spec to gen ints, even though we say doubles... 3. use a tolerance delta in the invariant <= my favourite i think... but adds so much noise ;/

It depends on your goals. If you just want a function that calculates the mean of a sequence of numbers correctly, then I would use a library. However, there might be other goals (like learning spec) where you might do something different.

There are lots of fiddly trade offs when working with floats. Most libs that work with floats use a number of tricks like comparing within some tolerance for tests. Another trick is to pre sort the numbers before summing to improve accuracy and consistency. I’m not an expert here which is why I would just use a library.

yeah, the goal is to use spec to learn. i am not implementing new statistical package... yet :D

i went with tolerance delta. The biggest takeaway is: if your spec is non-deterministic, you will get the mysterious exception mentioned above. Easy trap to fall into and burn, but once burned unlikely to happen again. And seeing that error immediately points in right direction. Typical newbie problem...

I actually haven't used s/fdef . For these types of things, I usually go straight to https://clojure.org/guides/test_check_beginner. The nice thing is that you can create separate tests for each property (eg. within-range, avg-unchanged-by-order?) and run them independently for however long it takes to feel comfortable that your implementation works. For avg-unchanged-order?, doing one shuffle only tests a small part of the test space. I would probably write it to use https://github.com/clojure/math.combinatorics to test a broader part of the input space.

For test.check, there's a way to get the RNG seed to make tests consistent and reproducible. There's probably also a way to do that with s/fdef so that avg-unchanged-order? is consistent/reproducible and avoids the problem you were running into.

Thanks for ideas. Will think on that. One point intrigued me especially. You say: _I actually haven't used `s/fdef` . For these types of things, I usually go straight to <https://clojure.org/guides/test_check_beginner|test.check>. The nice thing is that you can create separate tests for each property (eg. within-range, avg-unchanged-by-order?) and run them independently for however long it takes to feel comfortable that your implementation works._ One could add `s/fdef` to helpers as well too. It seems to mee then, that desired test "pyramid" would be: 1. s/fdef for most important and public facing parts 2. test.check for these as well, 3. exaple based classic unit tests, to present few cases to readers. It gets hard to get a glance of "shapes" involved without examples.

will try to roll with this and see how my perceptions change 😄

i do feel there is this a bit awkward dichotomy with spec/gen and test.check/gen i.e. some things are wrapped, but sooner or later (rather sooner) one HAS TO land in test.check land and learn it good.

OTOH, using test.check for double generator, where there is a nice s/double-in in spec feels wrong... so i end up doing things twice... and have extra pass => can i do that using spec namespace only...

I still use stuff like s/def and s/double-in. You can get the generator to use with test.check via s/gen.

in official docs they do mention that one should use test.check as last order of biz. as that will force that dependency on prod build. So it seems like the spirit of what i should do is try to avoid test.check in my prod namespaces, and reach for that only in testing namespaces. Although i have to say, i do not really follow the argument there, as spec depends on it anyway... so who cares whether i include that explicit or it is sucked in in deps of deps...

spec dynamically loads it so you can load spec without test.check

yes it does. so i do not fully understand this: There are three ways to build up custom generators - in decreasing order of preference: 1. Let spec create a generator based on a predicate/spec 2. Create your own generator from the tools in clojure.spec.gen.alpha 3. Use test.check or other test.check compatible libraries (like https://github.com/gfredericks/test.chuck) The last option requires a runtime dependency on test.check so the first two options are strongly preferred over using test.check directly.

test.check bytecode ends up in my prod slim and shiny build anyway... What is the perspective here that i miss? Maybe some more intricate interactions when designing libs based on libs...

Or just a rule of thumb to make code as future proof as possible as this will be the direction?

I don't think test.check should show up unless you explicitly include it.

maybe some other dependency is pulling it in?

hmm... i might have misunderstood the wording then. i assumed that spec is using test.check namespace internally

i have not inspected the build image btw, just to be clear

but what you say makes more sense for sure 🙂

and clears my confusion, so this is 99% correct i think

It does use test.check , but only for functionality starting at generators and later, https://clojure.org/guides/spec#_generators

which you won't usually need at runtime

spec generators rely on the Clojure property testing library https://github.com/clojure/test.check. However, this dependency is dynamically loaded

yes, to that was the part that set my mind on it

so, i hack away happily using spec namespace, but ... it will load in test.check at some point even in prod. even without me saying it has to

so we go back to the 1.2.3. above. why would that matter?

anyway. i can live with not knowing for now. Thank You for help 🙂

If your prod code doesn't use generators or things that need them, then you don't need test.check as a prod dependency and it won't be included

https://github.com/clojure/spec.alpha/blob/ad06cdc7407c11990c7e93206133fb14eb62cacf/src/main/clojure/clojure/spec/gen/alpha.clj#L55

That's what all the delay and dynaload stuff is doing.

and if test.check isn't included and you try to use generators, you'll get an exception, https://github.com/clojure/spec.alpha/blob/ad06cdc7407c11990c7e93206133fb14eb62cacf/src/main/clojure/clojure/spec/gen/alpha.clj#L40

nice. 👍

I'm using metosin data specs and have a datastructure like this:

[{:type "foo" :message "foo-msg"}]

(def errors-a
  {:foo "foo-msg-a"
   :bar "bar-msg-a"})

(def errors-b
  {:foo "foo-msg-b"
   :bar "bar-msg-b"})

(defn keyset [m]
  (->> m keys (map name) (into #{})))

(defn ->error-spec [errors]
  [{:type (s/spec (keyset errors))
    :message string?}])