Using spec, is there a way to specify something like a future or deferred that should contain a map with a set of keys? For instance, I'm using manifold/futures to make some calls to AWS; I'd like to write specs for these, but if they only represent that the functions return instances of manifold/futures, I'm not sure how valuable that would be.

Per the mailing list, it seems like this is similar situation to trying to spec protocol methods which seems to be discouraged in favor of wrapping protocol methods with functions and speccing those.

Yeah, it makes more sense to spec the functions involved that use the underlying data structures.

Thanks 😄

is there a way to remove everything that is not speced in a datastructure ? i.e the datastructure is valid but the legacy system behind it won't like extra fields to be present.

@yenda You need to use something like select-keys yourself

the problem with this solution is that it is repeating what is already in the spec, also it is hard/not-really-readable to recursively pick the keys in the structure

@yenda This is often requested here. Keep an eye on http://dev.clojure.org/jira/browse/CLJ-2112 for easy extraction of the keys. Also, spec-tools and spex might have something: https://github.com/metosin/spec-tools https://github.com/mpenet/spex

spex doesn't have this

spec-tools does I think

thanks

@mpenet I just like to link to your lib and see you apologize for it not being complete :)

🙂 I should add a notice in the readme

Got a somewhat general question here — I am trying to do a spike to see what it would be like to convert one of our services from prismatic schema to clojure spec. There is a clojure.test-compatible fixture in schema which runs all the functions in a test suite through a validator, and throws an error if their arguments and return values don't match their declared schemas. It strikes me that there's probably no way to do this easily with spec -- that the reason this works in schema is that the functions themselves were created using schema/defn, so they were already set up for this kind of thing. And that the only way to do something similar using spec is if you can find or write all the necessary generators to run the function through its paces in generative tests. (Which in our case seems like a daunting task as -- for better or worse -- there is a lot of hairy Java and Scala interop, futures, server responses, etc. that is all painstakingly mocked out in the tests.) Does this sound about right? I think this points to what @seancorfield was saying in a recent post, which is that the most leverage in spec comes from making schemas for data, not functions. At least not all the functions.

You can instrument (turn on spec checking) when you run tests but, if you have higher-order functions spec'd, you'll get generative testing on those. And instrument only checks arguments not return values. The idea is that instrument is for checking that functions are called correctly, not that they behave correctly. Generative testing is for behavior and is considered to be a separate type of task to "unit testing".

Alex was talking last night (spec unsession at Clojure/West) about an experimental generative test runner Cognitect are working on which would run continuously and watch for source changes, and analyze which functions (and therefore which specs) need to be exercised as a result of each source change -- rather than running all tests each time.

That sounds awesome

At World Singles, we've definitely focused on spec'ing data structures (and, for us, particularly using spec for validation and conformance of REST API parameters). Spec is very strong for that.

The generative testing is also amazing but, yeah, if you've got a lot of interop, it's really hard to generate those things. But the normal flow of running (unit) tests doesn't sit well with generative testing: unit tests run very quickly, generative tests often run slowly. We haven't yet settled on a good workflow for running those (we run a few generative tests from unit tests, where they run quickly, but we run them manually -- via check for example -- "from time to time").

@seancorfield Interesting, thanks for the responses. Yeah, I’ve been playing around with spec and test.check for a while in toy projects and thinking, this is awesome, we should use them on the codebase at work! But our actual production code base is another story. Would probably be easier if we were thinking in terms of spec and generative tests when it was originally being written.

We’re basically writing Scala in Clojure, is what’s happening.

Yeah, following a TDD / BDD workflow leads to nicely testable code. Adding tests later can be much harder.

yup

@seancorfield Do you know what is the rationale behind having instrument not be able to check the return value? That’s basically the core of my original question, I think.

Also, can you turn on instrument for all functions for which specs exist, or do you have to instrument the functions one by one?

You can (instrument) to turn on checks for all functions that are currently loaded (so it won't instrument namespaces that haven't yet been required!).

Or you can (instrument sym-or-syms) for specific functions.

that sounds good

The args vs ret thing is philosophical: Clojure/core folks believe that verifying behavior of a function is best done by generative testing based on stated properties. rather than test-by-example. Whereas instrumentation is something you want in place while you're developing / exercising code under test, to check calls are being made correctly.

(I hope that accurately captures what @alexmiller has been communicating about why there's this dichotomy in spec?)

Yeah, that makes sense. So instrumentation should be thought of as a dev tool, not something that you run in your automated test suite.

Yes. And I'll note that opinions on this vary even in the core team. :)

@alexmiller ha ha. And by that do you mean that opinions and dev practices vary, or that the dev practices are constrained because of the current state of the tools, and some people are of the opinion that that state should change so that they could use different dev practices?

The latter

interesting

I think it would be useful to have a way to instrument with ret checking

You know this Slack record is being recorded. 🙂

No worries :)

When I started spec'ing clojure.java.jdbc, I set it up so when the test suite runs, all functions are instrumented. We don't do that consistently at World Single (but we don't have a huge number of functions spec'd -- see above about our focus on data structure specs).

I agree with the philosophy above but I do not think that stest/check is the only possible kind of test where ret checking would be useful

(I was vocal and upset when :ret-checking stopped being done via instrument -- but I accepted Rich's explanation of why that was changed)

It used to be there and it was removed?

Yup, in an early alpha.

Do you have a link to any posts detailing the explanation you’re referring to?

If not, no worries

I think one of my main issues is probably something that it’s not the Clojure team’s job to fix, which is that I work in a place dominated by Scala, and it’s a lot easier to justify things that bear some resemblance to the way they do things, like being able to easily validate return types in example tests. But that’s not Clojure’s problem!

@seancorfield and @alexmiller , thank you very much for taking the time to answer my questions.

I think the most promising avenue will be new forms of test runners that are spec-aware and therefore some new workflows.

@seancorfield Yes, I’ve been thinking the same thing. That there is an amazing base available now, on top of which some tools could be built.

It's something I'll probably explore with Expectations, now that I maintain that...

Maybe I’ll contribute!

We have an #expectations channel and input is always welcome on both the test library itself and the supporting tooling ecosystem!

cool!