Fork me on GitHub
#clojure-spec
<
2019-02-08
>
butterguns00:02:19

That's a good point. I think this perhaps exposes a weakness in my understanding of generative testing. My brain says "what will this really test that can't be better accomplished with targeted unit testing"

butterguns00:02:37

wacky strings is often my go-to answer

butterguns00:02:03

But this is a larger question, somewhat off topic

seancorfield00:02:57

My experience so far leads me to think that the power of generative testing is more in the combinations of data it uses to exercise code rather than the specific values of an individual item.

butterguns00:02:12

Can you elaborate? To that point: I feel like that is what makes gen testing so difficult. E.g. I may have a "Command" that has two parameters. But only a very narrow combination of those two parameters actually make sense. Otherwise the "Command" is just gibberish.

butterguns00:02:40

And when I start using (gen/elements ["very" "small" "number" "of" "valid" "inputs"]) it sorta doesn't feel like generative testing anymore

seancorfield00:02:02

If you have interdependent parameters, that's certainly a bit trickier to test. You'd need to write a spec for the possible combinations that were valid, and make sure it's generatable.

seancorfield00:02:47

An example from one of our apps. We have a spec for the set/sequence of valid interactions a user (an admin in this case) can have with a sequence of data. Given that (complex!) spec, we can generate arbitrary but valid combinations of operations the user could run. We use those generated combinations of operations to drive an HtmlUnit test to verify that a) they are actually possible and don't produce errors and b) that the end state for the transformed sequence of data is still valid (using its own set of specs).

seancorfield00:02:35

Automated UAT of a web app, via generative testing 🙂

butterguns00:02:07

Ah, ok this makes a lot of sense! I think I'm getting hung up on the idea that I have to check the result of applying these operations, in literal sense. Like, "after a generated operation, I should inspect the operation and foo should == bar". It sounds like you're saying "after a generated operation, make sure everything conforms to spec, plus no errors thrown, and then just move on"

seancorfield00:02:49

It's really going to depend on your specific system under test. In another part of our system, we have certain pairs of functions that are inverses of each other, so for those we'll do full-on property-based testing to ensure that, for arbitrary "valid input", calling (= (inverse-fn (some-fn some-input)) some-input).

seancorfield00:02:19

And then in other situations, you might want to specifically test handling of bad data so you might write a spec for some of the things that your input cannot be, and then run tests to ensure that for arbitrary "bad input" you get some sort of appropriate failure response (i.e., that given bad input you don't get a success response).

seancorfield00:02:57

(that might help you detect combinations of bad input that you unexpectedly allow through your app -- a gap in your validation)

ericnormand01:02:19

john hughes (creator of quickcheck and now property-based testing consultant) always talks about how prop-based testing is exploratory

ericnormand01:02:41

you model the system you are checking, and often the tests don’t pass

ericnormand01:02:02

you investigate and find that your test was missing something

ericnormand01:02:13

so you modify it and run it again

ericnormand01:02:28

it fails again, it was missing something else

ericnormand01:02:19

my point is that property based testing always requires work to model the system

ericnormand01:02:37

there is no “one way” to test

ericnormand01:02:50

i think property based testing is a good name. you’re looking for algebraic properties to test for

ericnormand01:02:15

@seancorfield’s example of inverses is a good property

ericnormand01:02:31

another is idempotence

seancorfield01:02:48

Yup, you often start with a property you believe should hold and sometimes you discover it doesn't -- so either the property is wrong or your system doesn't preserve when it should 🙂

ericnormand01:02:04

commutativity, associativity, identity, zero

ericnormand01:02:17

all good properties to start with

ericnormand01:02:35

then you can get creative

ericnormand01:02:55

as in “it’s commutative under these conditions”

ericnormand01:02:17

or it’s commutative under this comparison operator

ericnormand01:02:29

(instead of =)

ericnormand01:02:19

as for generators, i have made use of three

seancorfield01:02:36

This output value should always exceed this input value. The output should always be an ordered sequence (regardless of its actual contents). The output value(s) should always be between these bounds (possibly based on input values). Etc.

👍 5
ericnormand01:02:49

always valid data, junk data, and almost valid data

ericnormand01:02:58

the almost valid is the hardest to model

ericnormand01:02:50

for me at least

ericnormand01:02:14

it’s almost as if you need to start with valid data and break it

ericnormand01:02:21

in a random way

ericnormand01:02:57

those are to test error conditions as sean was saying

seancorfield01:02:08

Mutation testing.

ericnormand01:02:45

does such a thing exist in clojure?

seancorfield01:02:30

In theory, you could build such a system (to mutate code). Easier to mutate input data and see if your system breaks in unexpected ways 🙂

ericnormand01:02:15

if d doesn’t pass spec, f should throw an exception

ericnormand01:02:41

it’s hard to test that thoroughly

ericnormand01:02:23

to feel confident you’ve found all of the ways you could do the data wrong

ericnormand01:02:54

it would be interesting to make a generator that could mutate any other value randomly

ericnormand01:02:08

add or subtract 1 from numbers

ericnormand01:02:21

add a char to a string

ericnormand01:02:37

drop an element from a list

borkdude15:02:25

It’s probably an issue with not fully qualifying something in a macro

Alex Miller (Clojure team)15:02:06

I fixed it, then broke it again :) I’ve reverted my last changes

borkdude15:02:29

it works now, except for some error messages that look strange (see issue). I think we’ve now seen the first time that all speculative specs can be instrumented. 🙂

borkdude15:02:19

I have to port respeced to spec-alpha2 before I can run the test suite, I’ll do that sometime soon

Alex Miller (Clojure team)15:02:55

I’m not getting what’s weird about the errors

borkdude15:02:46

ok.

(atom 1 {:validator 1})
Execution error - invalid arguments to clojure.core/atom at (test.clj:129).
{:validator 1} - failed: keyword? at: [:options :clojure.spec-alpha2.impl/k]
The failed keyword should be something like :atom/validator

Alex Miller (Clojure team)15:02:02

spec 1 version of the first one is the same

Alex Miller (Clojure team)15:02:05

user=> (atom 1 {:validator 1})
Execution error - invalid arguments to clojure.core/atom at (REPL:1).
{:validator 1} - failed: keyword? at: [:options :clojure.spec.alpha/k]

Alex Miller (Clojure team)15:02:33

this may be a pre-existing weakness of keys*

borkdude15:02:34

interesting, I’ll try

Alex Miller (Clojure team)15:02:48

what’s weird about the dissoc one?

borkdude15:02:48

On spec 1 I get:

user=> (dissoc 1)
Execution error - invalid arguments to clojure.core/dissoc at (REPL:1).
1 - failed: map? at: [:map :clojure.spec.alpha/pred] spec: :speculative.specs/map
1 - failed: nil? at: [:map :clojure.spec.alpha/nil]
On spec 2 I get:
user=> (dissoc 1)
Execution error - invalid arguments to clojure.core/dissoc at (test.clj:129).
1 - failed: map? at: [:map :clojure.spec-alpha2/pred]
1 - failed: nil? at: [:map :clojure.spec-alpha2/nil]

borkdude15:02:11

so it’s clear which of my own specs the argument is violating

Alex Miller (Clojure team)15:02:25

so this bit: spec: :speculative.specs/map

borkdude15:02:31

I guess so.

borkdude15:02:27

Maybe it would also be good for expound to test this behavior, since it relies on things like this (@bbrinck)

Alex Miller (Clojure team)15:02:07

yeah, I’m sure there are some subtle things like this that have broken, particularly in regex world

borkdude15:02:54

I don’t have specific demands for this, but expound probably has. Now’s the chance to get it fixed 🙂

borkdude15:02:02

Are spec objects that are defined in the registry backward compatible with spec1? I don’t mean how you are defining them, but the actual result of defining a spec?

bbrinck15:02:21

@borkdude Good idea! I will start a spec2 branch soon for expound to see what works and what needs changes.

Alex Miller (Clojure team)15:02:45

for the moment they are the same protocol, but that protocol is in a different ns so not compatible

Alex Miller (Clojure team)15:02:24

migration/release is another whole thing - we’ve talked about it at length but not going to worry about it for now till we’re much closer

borkdude15:02:30

@alexmiller if that were true, I think there would be a more smooth upgrade path. spec2 would just register it in both the old and new registry and old consumers would still work

borkdude15:02:38

ah ok, makes sense.

bbrinck15:02:08

I’m wondering if expound may need to depend on spec1 and spec2 at the same time. Even if expound has different namespaces that use spec1 and spec2, presumably a lib could upgrade to spec2 but still use a lib that uses spec1.

Alex Miller (Clojure team)15:02:04

that should be fine (presuming you have a version of clojure aware of both registries, etc)

Alex Miller (Clojure team)15:02:56

the clojure integration aspects will require a clojure release - lots of open questions about exactly what that will cover still tbd

borkdude15:02:30

I’ve thought about supporting two versions of the lib, one with a -spec2 suffix in the version. Could work.

bbrinck16:02:27

That makes sense, as you think about migration/release it would be useful for lib authors to come up with some practices as well, informed by your decisions. I suspect many libs will be in the situation where they want to be spec2 compat but must assume that some deps use spec1.

borkdude16:02:29

but then again, there’s only one or two places right now where I needed to change something? the finicky difference is the namespace declarations

borkdude16:02:13

anyway, now it probably not the time > but not going to worry about it for now till we’re much closer 🙂

bbrinck16:02:38

If all spec-libs require both spec1 and spec2, then at least the incremental cost of using any spec-lib won’t be that much :)

borkdude16:02:32

And then we have CLJS in the mix as well, so if you write CLJC there is that to consider too

gklijs18:02:37

Is there a cljs way to statisfy inst? I used it in a cljc spec and ran into trouble reading the java Instant, then just switched to number, but there might be a nicer way?

lilactown19:02:35

are you asking how to make a value that is an inst? AFAIK it's just a (js/Date.)

gklijs19:02:24

I should explain in more detail. I have a list of images in de back-end (Clojure) with just an id of the image and the time it was uploaded for now. I want to spec and use this list in the front-end (Clojurescript). I already have some (de)serialize code based on the spec to not repeat the keys everywhere. So I think I could get away with transforming the Clojure Inst to a string in such a way it's easy for Clojurescript to make it into a js/Date. Which gives me my answer by using a DateTimeFormatter.ISO_INSTANT to format the Instant to string, it should be possible to use Instant.parse(x) to get back the Instant while at the same time in Clojurescript I can use #inst x to get the js/Date.

lilactown19:02:20

if I understand correctly, what you are trying to figure out is how to transfer the instant over the wire?

lilactown19:02:25

from the back-end to the front-end?

lilactown19:02:41

FWIW, you can't call #inst x where x is a var or some other binding; #inst is a literal, so you'd have to write the date literally #inst "1980-01-01"

lilactown19:02:36

what I would suggest is using something like EDN or Transit to communicate between your back end and front-end, so that they can transfer the inst literally without having to format to a string and then re-parse the string

gklijs19:02:15

I already use EDN, but that goes wrong, because there is no Instant object in cljs. You where right about the literal, I need (js/Date. x) to create it.

lilactown19:02:33

are you reading the EDN in?

lilactown19:02:06

you might just need to provide the correct arguments to the reader

gklijs20:02:43

Thanks, might work better. I actually use reader/read-string now..

Frank Henard19:02:42

Hey everyone... question about using s/merge with s/multi-spec: https://stackoverflow.com/q/54599149/59439 I put it on SO for easier findability

borkdude22:02:35

@alexmiller I now ported my testing library to spec-alpha2, this works. Now I’m running the tests of speculative and I ran into a bug which I can reproduce as follows:

user=> (require '[clojure.spec-alpha2.gen :as gen])
nil
user=> (gen/sample (s/gen any?))
Execution error (IllegalArgumentException) at clojure.spec-alpha2.protocols/eval189$fn$G (protocols.clj:11).
No implementation of method: :gen* of protocol: #'clojure.spec-alpha2.protocols/Spec found for class: clojure.core$any_QMARK_

borkdude22:02:27

I can probably workaround this by using the test.check generator directly, but just fyi.

borkdude22:02:28

This does work: (gen/sample (gen/gen-for-pred string?))

borkdude22:02:28

This too:

user=> (gen/sample (s/gen (s/spec string?)))
("" "I" "" "4Cv" "" "Cs" "" "864w" "4" "w8TIh7")

seancorfield22:02:41

user=> (gen/sample (s/gen (s/spec any?)))
([] nil [] () {[] ()} () nil nil {} ([{\M :nK21+I.s*-J_ce9P+.-B!ll?._0?!07zMs*.k4N-y_A.?T?b.V*!mC/c-m-?k7, \q .z?} [:?57*-_?*x0:wg!!*v66+:+!Nwv:?*:Vti3*78I7:2_EVDZ8Dij:1:xC0 -1.375]]))
user=> 

seancorfield23:02:06

Because any? isn't a symbol, so you need s/spec now.

seancorfield23:02:19

This seems expected to me, given the spec1 -> spec2 changes.

borkdude23:02:40

I’m not 100% clear on this. E.g. it says in https://github.com/clojure/spec-alpha2/wiki/Differences-from-spec.alpha > Symbolic specs consist only of: … > Qualified symbols (predicate function references) If I’m reading that, I would say that

(gen/sample (s/gen `any?))
should work since it’s a qualified symbol referencing a predicate function?

borkdude23:02:33

It might as well be an accidental breaking change. I’ll wonder what Alex thinks. I’m off to bed now.

seancorfield23:02:29

Given that (s/valid? any? :foo) etc all reject any? (and string? etc), I think it's pretty clear it's deliberate @borkdude

seancorfield23:02:59

any? isn't a spec, it's a predicate. You can define a spec in terms of a predicate. So (s/def ::foo any?) is valid and then (s/exercise ::foo) will be valid but (s/exercise any?) is not.

seancorfield23:02:39

And then (s/spec any?) produces a spec from a predicate. So (s/exercise (s/spec any?)) is valid.

seancorfield23:02:33

The line between specs and predicates is very blurred in spec1.