Hi

does anyone have an example showing how they wired generative tests in their test suite?

I'm using exercise-fn and instrument at the repl but wanted to know if there was a standard way of wiring that in tests

I guess there are a few ways to do this: one would be just to call instrument on "normal" tests, then another could be replacing your test values with generated versions of it, which might or might not be difficult depending on what it is.

I haven't done any of it so far personally. Well actually just the instrumentation part on 1 project.

clj jdbc and alia both have instrumentation on in the test suite for instance

cool, will look at that, thanks!

so, as an alternative of calling s/explain on all pre or post conditions, what about calling s/assert rather than s/valid ?

e.g.

(defn new-web-server [config]
  {:pre [(s/assert ::config config)]}
  (-> (map->WebServer config)
      (using [:routes])))

i know the semantics aren't very clean (:pre is supposed to return true/false rather than an exception), but it does contain a lot more information

@mpenet Integrating with the output of check works well

How do you go about generating fixed size arrays quickly, usually?

@pyr gen bible: https://github.com/clojure/test.check/blob/master/doc/cheatsheet.md

depends if you mean primitive array or vector/list

primitive in this case, but the cheatsheet will help, thanks again

you can generate a list of ints then feed it in byte-array in a gen/fmap for instance

there are probably easier ways

(tgen/resize 64 (gen/bytes)) does the trick

nice

@mpenet (actually not at all, so I'm left with a gen/bind on top of a vector)

I had something like this in mind: (gen/fmap byte-array (gen/vector gen/byte 10))

@pyr ping

probably very similar to what you have

a gen/bytes* with a size argument would be nice tho

@mpenet this is what I have, yes

(except there's no gen/byte)

there is in clojure.test.check/gen

ah

https://clojure.github.io/test.check/clojure.test.check.generators.html#var-byte

indeed, that's much better

then what you have works out of the box and is much shorter than what I had

clojure.spec aliases some of test.check but not all of it

I guess the idea is to (maybe) someday promote most of it in spec itself, but I could be wrong

test.chuck is also awesome for more gen utils

string-from-regex for instance was quite the time saver

A last question for now, is it possible to specify custom generators only for clojure.spec.test/check

It's only a matter of visual preference, to be able to have simple specs in my main namespaces and specs with generators for tests

Yup, I think so, test/check takes a number of options for this

I think that means you want to use overrides

@gfredericks yes, but I only see support for those in exercise and exercise-fn

i'd like c.s.t/check to use them

wait

i misread the doc

sorry

it seems as though :gen does this for check

Ah, so the map given to :gen in c.s.t/check is shallow

i.e: the generator overrides are only taken into account for the called fn. If the fn depends on other functions, they won't use the provided generators

like for stubbing in particular?

@gfredericks not sure I understand

I just don't know why else, when using c.s.t/check, functions other than the one being tested would use generators

you're right i'm not articulating my pb properly

it's when generators rely on other overrided generators that i run into the issue

on of my generators does:

(def map-gen (gen/bind (s/gen (s/keys :req [::a ::b ::c]) mangle-and-return))

if i do (c.s.t/check map-test :gen {::map mag-gen ::a a-gen ::b b-gen})

the generation of ::a and ::b in map-gen will not use overrides

so I have to provide overrides in the s/gen call in map-gen too

spec uses functions that return generators in a lot of places

Which can defer resolution, I assume. But maybe that can't help here.

it's not a big thing

i just need to specify my overrides in a couple of places instead of once.

that seems unfortunate

@gfredericks @mpenet thanks for your help today

I described my approach here: http://spootnik.org/entries/2017/01/09_an-adventure-with-clocks-component-and.html

whoa, great write-up!

I struggled to decouple the generators from the spec because otherwise it makes things harder to follow. My first iteration had with-gen on the specs themselves and that made it much harder to explain what was going on

so, if i have a defrecord for which I want to fdef and instrument certain functions defined within that record, how am I supposed to be referring those functions ?

(i think this is more related to clojure in general, and what the symbol names for functions inside records are)

AFAIK all record functions must be specified with protocols, so you can spec the protocol methods

@lmergen behavior is implemented on top of records through protocols, so as @schmee mentioned, you can specify protocol signatures as you would do functions

hmmm ok

(defprotocol Encoder (encode [this])) (defrecord A [name] Encoder (encode [this] (str name)))

if you were to have the above protocol and record

you could specify encode with (spec/fdef encode :args ... :ret ... :fn ...)

but that would do it for all instances of Encoder, right ? not just of A ?

@lmergen in this case, (spec/fdef encode :args (s/cat :encoder #(satisfies? Encoder %)) :ret string?)

indeed, this is for Encoder in general

but that's how protocols work

ok

so my concrete problem at the moment is Component

and ensuring that the start function of certain components return a specific type

so if i have 5 different components (all using defrecord and the Lifecycle protocol), i want 5 different fdefs

for those 5 different start function implementations

the expected behavior of start in Lifecycle is that you return the same record possibly augmented with additional fields

so you could write a generic spec

(def component? #(satisfies Lifecycle %))
(spec/fdef start :args (s/cat :component component?) :ret component? :fn (= (.getClass (:component %)) (.getClass (:ret %))))

or something to that effect

unless you're doing something funny with component

If you are really intent on spec'ing per-type behavior for your protocol, there is another much kludgier way. extend (http://clojure.github.io/clojure/clojure.core-api.html#clojure.core/extend) takes a map of protocol implementation for types, which takes fns as args. You could implement Lifecycle this way and then spec the individual functions you have in the map

But again, it seems as though this should not be necessary for Lifecycle

ok

i understand

i'm probably solving this problem at the wrong level

and should probably verify whether my entire system satisfies a certain spec, after it has been initialized

I don’t think it was mentioned above, but you cannot spec protocol functions

why?

I suspect that it's an implementation issue, since protocol functions have very different function signatures

http://dev.clojure.org/jira/browse/CLJ-1941?focusedCommentId=43084&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-43084

and previous comments have some discussion of why that is

basically certain kinds of "function" calls(including protocols) are compiled differently and spec just handles the generic case

what the recommended workaround?

wrap it in a regular function and spec that instead?

that's what I ended up doing yes

Am I missing a cleaner way to write this. Goal is to check that the options list has the required keys which are also declared in the data)

(s/def ::Select.props
  (s/and
    (s/keys :req-un [::value ::options ::value-key ::display-key ::on-change]
            :opt-un [::placeholder ::disabled ::loading])
    #(every? (fn [option] (contains? option (:value-key %))) (:options %))
    #(every? (fn [option] (contains? option (:display-key %))) (:options %))))

This won't produce very specific errors

From today, you can easilly share your clojure.spec related code snippets on klipse - it loads pretty fast

http://app.klipse.tech/?cljs_in.gist=viebel/cd7b4c26cf5a9fcc2d53e21021a25df0&eval_only=1

@olivergeorge so ::Select.props describes a map, one of whose keys is :options, which is a list of maps, each of which must contain values referred to by keys named :value-key or :display-key, which are in ::Select.props -- is that right?

Correct

So I really want to confirm the options based on the map data

Like this movie inception

@kenny I gave it a try at simplifying your macro and came up with this:

(defmacro defspec-test
  ([name sym-or-syms] `(defspec-test ~name ~sym-or-syms nil))
  ([name sym-or-syms opts]
   `(t/deftest ~name
      (let [check-results#  (clojure.spec.test/check ~sym-or-syms ~opts)]
        (doseq [result# check-results#]
          (t/is (nil? (:failure result#)) (clojure.spec.test/abbrev-result result#))
          (when (nil? (:failure result#))
            (println "[OK] - " (clojure.spec.test/abbrev-result result#))))))))

You want to use do-report, not println otherwise you'll break integrations with clojure.test

why? I dont know much about clojure.test actually 😕. I figured that the call to t/is already took care of the integration. doesnt it?

The other important difference is that this is counting every single generative test instead of grouping them as one

is will handle it but you will get confusing results for when tests fail

ah I see what you mean. true the output is not as clean as with a normal test since you get the generated symbol but I still find the output better since in the previos macro I was only getting the stacktrace of an error. No idea which function failed nor other info. I dont know if it was a problem in my project though

@olivergeorge unfortunately, I think what you have is about as straightforward as you can get with what you want. While spec can nest sequential and associative structures easily, there is not another way (that I'm aware of) to relate various levels other than another predicate which is what you've done. If anything, I might combine the two additional predicates into one function, though you may find it gives you an undesirable level of detail:

(fn [{:keys [value-key display-key options]}]
      (every? #(and (contains? % value-key)
                    (contains? % display-key))
              options))