Fork me on GitHub
#clojure-spec
<
2019-03-05
>
robertfw00:03:31

We have an ID value that uses a luhn check digit. The error was being caused by line 27 - where it is using subs, it was using drop-last which ended up returning the sequence of characters

robertfw00:03:11

Is the expected behaviour to see what spec failed? e.g., hone in on luhn-check-digit being the failing fdef?

jeaye00:03:06

What is regex-spec?

jeaye00:03:21

Seems like that would be the problem to me.

robertfw00:03:03

ah sorry I missed including that. It is...

(defmacro regex-spec
  [regex]
  `(s/with-gen (s/and string? #(re-matches ~regex %))
     #(gen'/string-from-regex ~regex)))

robertfw00:03:26

I do find that there are quite a few exception messages that could be a little more helpful. For example, right now I'm hunting down a problem with a misbehaving generator, and all I have to go off of is

Execution error (ExceptionInfo) at clojure.test.check.generators/such-that-helper (generators.cljc:320).
Couldn't satisfy such-that predicate after 100 tries.

robertfw00:03:38

It's made a little more manageable through workflow (e.g., I know what spec I'm working on), but it would still be very helpful to get more details on what went wrong to help debugging

robertfw00:03:22

(That's a general spec/test.check comment, not so much orchestra)

jeaye00:03:12

I feel like you have a lot going on there, so it's not quite a minimal repro case at all.

jeaye00:03:15

To figure out the source of that string sequence issue, keep removing code until the issue goes away. 🙂 If it goes away, add that code back and remove some other code until it goes away again. Keep repeating until the only code left is necessary for the reproduction.

jeaye00:03:56

Namely, that spec error likely has nothing to do with generators or your custom macros or even all of that business logic around luhn digits.

robertfw00:03:53

As mentioned earlier - I already fixed the actual issue with the code, my question is more about debugging flow and finding the culprit. In this case the error I was getting back was not very helpful in narrowing down where to look, and I was wondering if there was something I was doing wrong to impact that.

jeaye01:03:02

Oh, that's my mistake. I came in last and must've missed that. I think the one thing I'd say we should keep in mind is that specs themselves should be simple enough to not really need debugging. To me, specs are for debugging, so it's awful to need to debug the debugging code. If you're stuck on weird issues with your specs, perhaps your approach can be simplified.

jeaye01:03:35

Note, I also never use generators, since Orchestra + good unit and functional test coverage is everything I care about. However, especially given that this functionality (ret + fn instrumentation) has been removed from Clojure core and a lot of the spec press has been around generative testing, I may not be among the majority.

👍 4
robertfw01:03:13

We're still fairly fresh with it - we have one other clojure project in our repos but it is pre-spec. The generative testing has been useful so far, it's already caught a few issues that we probably wouldn't have spotted with our usual approach. But yeah.. it's definitely tricky going at times with more complex specs, especially for relatively new clojure devs like myself

jeaye01:03:59

We have thousands of lines of specs and hundreds of spec'd functions in our code (at my company). I don't recall the last time I spent a noticeable amount of time debugging an issue with one of my specs. This is likely because they're all very simple. "This thing requires these keys. This is a number, this is a non-blank string, this matches this regex." Not too much more than that, for our entire front-end and back-end.

robertfw01:03:17

The bulk of ours are like that and haven't given us any issue. Todays fun was caused by the more complicated spec for the luhned ID, and some specs defining incoming requests on our API that have some internal consistency properties

robertfw01:03:16

Thanks for taking a look! I appreciate it. Orchestra was a great find for us, having the :ret and :fn checking on instrument is really simplifying our workflow

jeaye01:03:30

I'm happy you're enjoying it. 🙂

robertfw01:03:42

We were wondering why that wasn't part of the standard instrument

robertfw01:03:51

It seems like a natural thing to do

jeaye01:03:26

It was and then it was removed. My understanding is that the Clojure team didn't like to emphasize that usage of spec and noted the performance implications. To me, that is the usage of spec and the performance implications are negligible compared to the safety benefits.

jeaye01:03:47

I wouldn't want to misrepresent them and I haven't actually spoken with them about it. Would love to sit down with Rich and talk it out, but who wouldn't love to sit down with Rich and talk anyway?

robertfw01:03:18

Interesting. Yeah, no kidding! Definitely a brain to pick

seancorfield01:03:53

It's because instrument is for checking calls pass the right data -- the :args key -- and check is for checking the functions themselves behave correctly (given conforming :args, does the output of the function satisfy :ret and :fn).

seancorfield01:03:11

Orchestra complects those two, very different, types of checks.

seancorfield01:03:48

(which is fine as long as folks understand that is what's going on and they are comfortable making that choice to go against how spec is designed)

jeaye01:03:18

There's the official opinion. 🙂 Thanks, Sean. Fortunately, people now have the choice.

seancorfield03:03:35

I wouldn't say "official", coming from me -- I'm just repeating what I understand the Cognitect folks to have said about function specs 🙂

alexmiller04:03:28

ret specs are probably going to change a lot in spec 2 as well

jeaye05:03:29

Hopefully not in a way which is incompatible with this form of instrumentation, I hope, Alex.

jeaye05:03:59

Either upstream or via a soft fork such as Orchestra.

seancorfield05:03:02

@jeaye At work we have a branch of our code running on Spec2 -- there were several substantive changes (beside the "obvious" renaming of namespaces in :require clauses). They weren't big changes, but they were breaking changes.

jeaye05:03:53

Good to know, Sean. Thanks for the info.

jeaye05:03:05

I'll wait and see how things are, when the dust settles. The libs, spec and spec2, aren't so large as to be very difficult to either work into this case or replace with something which does. There's superb thought work going into spec2, no doubt, and I have no interest in competing with that. I just want to make sure my team and other Orchestra users will be able to take advantage of the new spec features along with the instrumentation.

prnc10:03:35

Clojure 1.10.0
user=> (require '[clojure.spec.alpha :as s])
nil
user=> (s/def ::foo :bar/this)
Syntax error compiling at (REPL:1:1).
Unable to resolve spec: :bar/this
user=> (s/def ::foo (s/keys :req-un [:bar/this]))
:user/foo
user=> (s/def ::foo (s/keys :req [:bar/this]))
:user/foo
Could someone offer a good way of thinking about the above ^^^? I.e. it is possible to refer to 'non-existent' spec inside map spec (through keys) but it's not possible at the "top level" s/def. I guess my confusion is coming from the fact that semantically those uses seem equivalent to me--both use ns-qualified keyword as a way to refer to spec--one fails the other doesn't.

alexmiller13:03:27

Aliased specs are not as delayed as they should be in this case, which is a known issue. I have been working on fixing it in spec 2.

prnc14:03:21

Amazing, thanks @alexmiller mario-star