Fork me on GitHub
Jakub Holý (HolyJak)11:10:03

@alexmiller I have a tiny improvement to gensub to provide a meaningful error when such-that fails, is it meaningful for send a patch to Essentially (gen/such-that #(valid? spec %) g 100) -> (gen/such-that #(valid? spec %) g {:max-tries 100, :ex-fn (fn [{:keys [max-tries]}] (ex-info (str "Couldn't satisfy such-that predicate after " max-tries " tries.") {:path path}))}) (Since path is the only thing that looks meaningful, contrary to test.check's gen, pred and spec's spec, form) I have updated the issue, let me know there whether to send a patch. Thank you!

Alex Miller (Clojure team)12:10:06

test.check 0.10.0 has some new hooks for this case too

Alex Miller (Clojure team)12:10:27

if you have a patch, go for it, but I'm not sure whether we're going to go back and patch anything on spec.alpha. this is tbd, but my current thought is that once we get close to a spec 2 release I would reassess all the spec tickets (I know some have been fixed) and try to do a clean-up wave. this will require re-working any of the patches as I'd expect none of them to apply to spec 2.

👍 4
Jakub Holý (HolyJak)15:10:11

Thanks, @alexmiller! Question 2: I discovered that it is even more useful to include an example of why the spec failed on sample data - knowing which spec failed to match is good but knowing why is even better. In my case the spec failed was something like ::person because of invalid :person :address :zip. So, to troubleshoot my issue, I include this in the thrown error:

{:max  max-tries
 :path path
 :sample-explain (->> (first (gen/sample g 1))
                      (explain-data spec)
- which was extremely useful for me but not sure whether it would be an appropriate general solution.


@holyjak YES. The failing args and ret should be easily shown in the error message. I don't think that particular approach will work because there's no guarantee you'll hit the failing case with one sample but I like the idea.

Alex Miller (Clojure team)15:10:30

showing an example which isn't the one that actually failed seems more confusing to me


Yes - it needs to be the failed one.

Jakub Holý (HolyJak)15:10:44

ALL the examples fail


We have internal code that does this already but it's a hack on gensub.

Jakub Holý (HolyJak)15:10:53

Well, in theory you are right. What we call is `(gen/such-that #(valid? spec %) g {:max-tries 1000 ...}) - none of the 1000 generated samples failed. What is the chance that the one we generate in the error does not fail?


Depends on how you're controlling the size.

Jakub Holý (HolyJak)15:10:29

Of course I could add a check and include :sample-explain only if it really fails. Would that be satisfactory? If not - what is a good way of capturing some generated, failing data?


IIRC, gen/sample does not use all 200 size values.


Using quick-check or st/check will.

Jakub Holý (HolyJak)15:10:55

I am not sure how such-that invokes the generator but I would assume that it does so in a way very similar to sample?

Jakub Holý (HolyJak)15:10:06

Anyway, what do you think about the "best effort" approach - generate a sample, if it fails the spec, include :sample-explain in the error. I think that would work in 99.9% cases?


Anyway, the piece I find valuable is being able to easily see why a check failed, whether it be by generator failure or check failure. When a generator fails, I would like to see the failed samples. This usually lets me figure out which predicate is causing the issue pretty quick. In the case the check fails, I'd like to see the args and the ret.


You should know which samples failed while generating @holyjak, no need to generate new ones.

Jakub Holý (HolyJak)15:10:02

Well, I do not generate anything, it is such-that that does it - and in its failure-handling function I have no idea what values were actually tried. Or? BTW here is the code


Our hack (not acceptable for a patch but let's us debug generators without going crazy) is to store invalid values in an atom and display that in the error message.


So we just change the such-that predicate to do that.

👍 4
Jakub Holý (HolyJak)15:10:09

Smart! The question is: What approach would be acceptible for a patch (and useful for the users!)?


I don't know. It definitely feels too hacky for a patch. It may be the only way to do it though. When I was looking at this, I don't think I saw any test.check hook to grab the failed values.


Just speculating but I imagine it'd be somewhat easy to add that hook to test.check if it isn't already there and is the proper solution.


I would definitely be in favor of adding some sort of patch to spec1 alpha given we don't know when spec2 is coming. Debugging generators is super painful as it is.


Check the such-that docstring in the latest version


Doesn't appear to have anything that stores failed samples.


Would you want all of them? Just the latest?


Having the generator means it should be easy to generate more


Probably all.


If I have all, I can easily only have the latest if I want.

Jakub Holý (HolyJak)15:10:34

I believe a single value is enough (preferably the 1st as it is the simplest one)?


Keeping all could be a GC issue

Jakub Holý (HolyJak)15:10:18

> Having the generator means it should be easy to generate more That is what I do in my solution (generate a new value using sample) but it has been argued that using one of the actually failed would be better.


I guess I only ever work with one of the failed samples anyway.

Jakub Holý (HolyJak)15:10:13

@gfredericks As I explained above, this popped up when working with specs - a custom generator generated value that did not conform to the spec. I can get the name of the spec but want to see why the custom gen failed to provide anything valid. There, having a sample value and running s/explain-data on it is sufficient and very useful.


Looks easy to add in such-that-helper.

Jakub Holý (HolyJak)15:10:47

I am creating an issue on test.check to pass a sample failed value to ex-fn. Stop me if I should not 🙂


You shouldn't not

👍 4
Jakub Holý (HolyJak)15:10:04

Conclusion: I create a patch for Spec to include the spec name (or rather path) in the such-that failures and we wait for test.check to expose a sample failed value to the :ex-fn before adding explain-data of it.

Jakub Holý (HolyJak)15:10:14

@alexmiller is there any point in providing also a similar patch for Spec 2 or is it too much in flux?

Alex Miller (Clojure team)15:10:41

I'm not going to do the sample thing, so not interested in a patch for that

Jakub Holý (HolyJak)15:10:02

no, not that, just including path

Alex Miller (Clojure team)15:10:14

I don't think there's any point for spec 2

👍 4
Alex Miller (Clojure team)15:10:39

one possible outcome is that we repackage everything in spec 2 to remove alpha designation, which will (again) break any patches

Alex Miller (Clojure team)15:10:03

I think in test.check, while generating in such-that, it would be possible to just retain the first or last gen'ed every time (before the pred check) and report that if you hit the retry limit. that way you're reporting an example that was actually tried.


An example generated after the fact is just as meaningful, philosophically

Alex Miller (Clojure team)15:10:01

if only a percentage of them fail, and you happen to generate a sample that doesn't fail, that seems strongly less useful than one of the actual examples that didn't pass the predicate

Alex Miller (Clojure team)15:10:07

so I'ma disagree with you on that


I was assuming you'd filter on failure 😛

Alex Miller (Clojure team)15:10:43

well you failed to mention that :)


Which, ironically, could fail :face_with_rolling_eyes: though presumably is unlikely to


But that might be a good enough reason not to do it


Adding the size to the failure data would assist with that though

Jakub Holý (HolyJak)15:10:31

@gfredericks Will you consider adding a sample failed value to the arguments of such-that's :ex-fn or some alternative improvement?


@holyjak I'm not going to be doing active work on test.check for the foreseeable future, so that'll be up to somebody else; but a jira ticket is definitely the right way to make sure it gets looked at

Jakub Holý (HolyJak)16:10:13

Hm, I was about to create it but you told me I should not I guess it was just misunderstanding.

Jakub Holý (HolyJak)16:10:41

I'd be happy to provide a patch to test.check - and even happier to get any pointers regarding the best solution.


No I meant you should create the ticket


Sorry, I was being cute with English double negatives, it was probably a bad idea

Jakub Holý (HolyJak)16:10:49

I see 🙂 My bad, I should have read more carefully. Also, I believed that English normally doesn't permit double negatives. Always learning 🙂

Jakub Holý (HolyJak)16:10:33

Any pointers regarding the best way to implement this? Is failed-value a good key name to pass to :ex-fn?

Jakub Holý (HolyJak)16:10:03

FYI I have created Provide sample failed value to such-that's ex-fn for better error messages

👍 4
Jakub Holý (HolyJak)16:10:17

FYI I have sumbitted a patch for ☝️


What is the proper way to lawfully copy/paste source code from core.specs.alpha?


What I’d like to do is copy ::defn-args and its dependencies into my own project.


Do I just put the copy right information above the code I want to copy?


Also, my project is MIT licensed, so I’m not sure how all that works.


> What are my obligations if I copy source code obtained from and licensed under the Eclipse Public License and include it in my product that I then distribute? > Source code licensed under the EPL may only be redistributed under the EPL.


I’m guessing I need to handwrite my own version of spec.


@noprompt I'm curious as to how you're depending on specs from that namespace since they are intended to support clojure.core macros?


(and, yes, you're going to be on some pretty shaky ground if you copy'n'paste that code into your system and then distribute it)


@seancorfield Imagine you want to write a macro that has the form as defn.


And imagine that I want to use conform to parse ::core.specs/defn-args.


Thats my situation.


Ah, so your code was relying on the internal names used in the those specs?


I’m noticing you’re using the word “internal”.


Only two top-level specs changed their names.


If you conform, you have to rely on the implementation of the spec.


Yeah but specs are globally public…


We've used alpha versions of Clojure libs in production for over eight years and sometimes Cognitect change stuff while it's alpha and break your code. That's the risk of using alpha libs.


I hope you can appreciate the irony in this.


To me its a bit surprising given the critique of SemVer, etc.


So suddenly calling it “alpha” makes it okay to make breaking changes?




They've been very clear that alpha means "can change/break". Things are only guaranteed additive/fixative once the alpha label goes away.


Its not on the README.


Where were they clear?


They've said it repeatedly publicly. Pretty sure even Rich has mentioned that in talks...?


Rich says it in the same talk where he contends SemVer is broken. Search for alpha in Funny thing though, Rich also says something like:

But, that is not to say just leave your thing 0.0.967. At a certain point, you are going to have users, and whether you change it to 1.0 or not, they are going to be depending on your stuff.
And that is where spec is going right now IMHO. But his Maybe Not talk and Alex Miller's blog eased my fears of spec remaining in this state forever.


Like on Twitter?


I rarely, if ever, watch talks.


The README and the source code are far more public.


The disclaimer should be there.


what would it say


What Sean said.


How about > Alpha means this library “can change/break” do not rely on it.


Because I obviously misunderstood the situation.


I'm sorry you've missed it being discussed on the mailing list and in Rich's talks and here on Slack (where it has come up several times in several situations).


I just want answers to my questions.


Can/How do I copy the source code?


Not legally into anything you are distributing under an incompatible license.


If you want to use EPL code in your OSS project, consider changing your license to EPL so it's compatible.


Okay, that’s what I gathered by reading the EPL website.


Its fine, I can write the specs myself in this case. I was just hoping I could take a short cut.


Given the day I've had working on our Spec 2 branch, I do sympathize (even if it may not sound like it).


Honestly, it didn’t feel that way but you saying that makes me feel like you do.


I just want to get my users and teammates out of a broken state quickly.


I’ve given up on trying to supply feedback or contribute to anything in the clojure repo due to conversations that go the way they do regarding these kinds of frustrations.


Yeah, I can detect a fair amount of frustration/anger...


Transparency would be nice on these topics; one can’t be expected to keep up with every feed of data.


The repo and source code is about as public and as inclusive as any place to explain semantics.


I’ve made the effort in Meander to call out the versioning semantics I employ so there is no misunderstanding.


When Rich introduced spec in 2016, he was pretty clear that it was alpha and subject to change (I just checked the transcript of one of his talks from back then). He also called out alphas as "potentially breaking" in his Spec-ulation talk (which is where his main criticism of SemVer was made.


SemVer itself is broken tho'. While you're in alpha, you can make (breaking) changes. Once you have a non-alpha, you should only make accretive/fixative changes. SemVer doesn't help with that.

👍 4

That makes this assumption that people will be exposed to those things or, if you have my memory, remember them.


Just slap a phrase on the README that says, upfront, what the version semantics are instead of relying on them being implicitly understood.


What I took from Rich was if I’m going to make a breaking change, I make a new namespace, artifact, etc.


And that works fantastic.


If you're not in alpha, yeah.


Its a greek letter.


It was an important caveat in his talk 🙂


Rich doesn’t get to decide what that means.


He can’t have it both ways.


(and it's been the practice in Clojure overall for years)


Saying that “alpha” can break is just weird thing.


None of these versions can break but this one can.


But, hey, if the semantics were with the source, fine, whatever, those are Rich’s rules. Cool.


My point is they’re not transparent.