Can anyone recommend any good resources on clojure spec, or property based testing concerning side effects (like reading, writing from files). I just watched “A Deep Specification for Dropbox - Benjamin Pierce” : https://youtu.be/Y2jQe8DFzUM and have to assume the ideas from it were a motivation for clojure spec (given the timing). In the video, their is an argument that that deep specifications lead to more reliable software, which i agree with. He also argues, that this requires a lot of time and effort, which, also makes sense. My experience with spec so far has been that its hard to see how you can adapt them to handle side effects (writing to files, calling apis, dbs, etc…) In the talk, the side effects are handled by observing them, and checking if the observations follow a specification. In general, is the approach you need to take if you want a specification over side effects? My intuition is that spec is more about modeling the non side effect parts of your application. But it feels like so much work is generally done as side effects, that the cost is fairly high to create the specs and generators that “stub” out the side effect. Or really, what an “effective” stub would be.

that youtube talk is more a long the lines of using test.check directly, not using spec

I am sure some people are using spec for that kind of thing, and you might use spec as a nice dsl for creating generators for that, but you wouldn't use spec's instrumentation stuff to do that kind of testing

in general, when I am trying to test a whole process, I use test.check to generate operations, feed them in to the process, then compare the result with what my model says it should be. spec (at least how I use it, and there are a lot of parts to spec so it can be used in a lot of different ways) is more about describing data that is moving around in the system. and I end up using those descriptions in assertions to make sure what I expect is there, to parse/recognize which of some alternative data a function was passed, or to generate other descriptions of the data for other tools (schemas, serializers, etc). I pretty much never use spec to annotate a function and then instrument that function.

I think generative testing splits in to two classes. Class 1 is what you see in simple examples on line, you have a pure function that does something to a list of integers, so you generate a list of integers and pass it in and assert some property. Class 2 is like what you see in that dropbox video (which is very good), you end up generating a program (a sequence of operations) that you run on the system you are testing and then also against some reference implementation (say for a distributed key value store, the ref impl might be an in memory hashmap), and compare the results.

spec's instrument and generative stuff seems mostly aimed at class 1

class 2 is definitely a big investment in testing

@hiredman thanks, I saved your comment and I'll get around to reading it in a bit 🐕