Fork me on GitHub
Daniel Hines16:08:52

I'm writing a web app that writes to our CRM. We have a bunch of third party plugins installed on this CRM that I'm not allowed to touch. One function of these plugins is to validate incoming user-inputted data; however, somewhere in the validation, the plugins are occasionally hitting fatal exceptions, but they give no clues as to what it might be. The third party won't provide any documentation, and decompiling their code is expressly forbidden in their EULA. Through significant trial and error, I've narrowed down the cause to specific combinations of input values, which appear completely arbitrary to me. There are about 100 different inputs, some of which have hundreds of possible values, and the order the inputs seems to matter, making this very difficult to address by hand. Would it make sense to use spec in this situation to systematically explore combinations of data cause errors?


I've hit an issue with instrumentation wherein the instrument macro generates and returns a vector of symbols of those fns instrumented which is too large for the JVM to appreciate. So, I see Method code too large! failing the compilation instead. This seems like something spec will need to address, in the long term, as more people end up spec'ing more fns. Right now, looks like this project is around 300 spec'd fns and the issue is arising. Both Clojure and ClojureScript seem to be subject to the issue.


can you give a repro for the case @jeaye?


Yup. I'll have one up in a bit.


A repro case in 20 lines or so. Couldn't get it going with Clojure, but it certainly happens with ClojureScript. When I look at the implementation of instrument for each, it'd seem both would be subject, but perhaps not.


100% repro here though.


@d4hines That sounds very much like the scenario Stu Halloway was talking about in one of his presentations on spec...


I don't know. I don't think I attended SL in 2016. May have watched that online tho'. I thought it was a talk from one of the Conj's or West's...

Daniel Hines17:08:51

Ok, I'll look for it. Thanks!


Keep in mind that exercise will generate values that match the spec, but to get the power of shrinking (which is what you want, I expect), you’ll want to combine spec with test.check


I really like the test.chuck library for this, you can write something like the following (totally untested and written in slack, so beware 🙂 )

(deftest find-bug
     "all valid inputs don't cause exception"
     [args (s/gen ::args-spec)]
     ;; replace 'string?' with whatever the actual expected return type is
     (is (string? (apply third-party-api/some-fn args)))))    


IIRC there might be a talk he gave about writing specs for a Java charting library and finding a (JVM?) bug by "fuzzing" it with random spec checks?


@ghadi Is that something which you can repro?


Has anyone encountered an open-source HL7-FHIR spec library? I have been creating one for work but I’d be interested in pulling it out into an open-source project if others wanted to collaborate on it.


we would find that useful @dadair


@jeaye haven't had a chance to try yet, but off-hand: try to get rid of the nested do


i think the compiler special case's top-level do's but not nested ones


That macro's just making the fns though. That macro doesn't cause the error; the error is with instrumentation.


If you comment out the (stest/instrument) form, it compiles just fine.


Is there anyway to decompose a spec definition? I want to use a s/keys spec to define a database table


oh using s/describe