This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2019-10-02
Channels
- # aleph (3)
- # announcements (2)
- # babashka (4)
- # beginners (74)
- # calva (21)
- # clj-kondo (30)
- # cljs-dev (7)
- # cljsrn (42)
- # clojure (121)
- # clojure-dev (13)
- # clojure-europe (23)
- # clojure-losangeles (2)
- # clojure-nl (2)
- # clojure-norway (7)
- # clojure-spec (140)
- # clojure-uk (58)
- # clojuredesign-podcast (9)
- # clojurescript (49)
- # clojutre (2)
- # cursive (32)
- # datascript (2)
- # datomic (59)
- # duct (7)
- # figwheel-main (6)
- # fulcro (18)
- # graphql (5)
- # jackdaw (1)
- # joker (6)
- # juxt (7)
- # leiningen (9)
- # off-topic (1)
- # pedestal (14)
- # quil (2)
- # re-frame (3)
- # reitit (8)
- # shadow-cljs (78)
- # sql (8)
- # timbre (3)
- # vim (69)
@alexmiller I have a tiny improvement to gensub
to provide a meaningful error when such-that
fails, is it meaningful for send a patch to https://clojure.atlassian.net/browse/CLJ-2097? Essentially (gen/such-that #(valid? spec %) g 100)
-> (gen/such-that #(valid? spec %) g {:max-tries 100, :ex-fn (fn [{:keys [max-tries]}] (ex-info (str "Couldn't satisfy such-that predicate after " max-tries " tries.") {:path path}))})
(Since path
is the only thing that looks meaningful, contrary to test.check's gen, pred
and spec's spec, form
)
I have updated the issue, let me know there whether to send a patch. Thank you!
test.check 0.10.0 has some new hooks for this case too
if you have a patch, go for it, but I'm not sure whether we're going to go back and patch anything on spec.alpha. this is tbd, but my current thought is that once we get close to a spec 2 release I would reassess all the spec tickets (I know some have been fixed) and try to do a clean-up wave. this will require re-working any of the patches as I'd expect none of them to apply to spec 2.
Thanks, @alexmiller! Question 2: I discovered that it is even more useful to include an example of why the spec failed on sample data - knowing which spec failed to match is good but knowing why is even better. In my case the spec failed was something like ::person
because of invalid :person :address :zip
. So, to troubleshoot my issue, I include this in the thrown error:
{:max max-tries
:path path
:sample-explain (->> (first (gen/sample g 1))
(explain-data spec)
:clojure.spec.alpha/problems)}
- which was extremely useful for me but not sure whether it would be an appropriate general solution.@holyjak YES. The failing args and ret should be easily shown in the error message. I don't think that particular approach will work because there's no guarantee you'll hit the failing case with one sample but I like the idea.
showing an example which isn't the one that actually failed seems more confusing to me
ALL the examples fail
Well, in theory you are right. What we call is `(gen/such-that #(valid? spec %) g {:max-tries 1000 ...}) - none of the 1000 generated samples failed. What is the chance that the one we generate in the error does not fail?
What do you mean?
Of course I could add a check and include :sample-explain
only if it really fails. Would that be satisfactory? If not - what is a good way of capturing some generated, failing data?
I am not sure how such-that
invokes the generator but I would assume that it does so in a way very similar to sample
?
Anyway, what do you think about the "best effort" approach - generate a sample, if it fails the spec, include :sample-explain
in the error. I think that would work in 99.9% cases?
Anyway, the piece I find valuable is being able to easily see why a check failed, whether it be by generator failure or check failure. When a generator fails, I would like to see the failed samples. This usually lets me figure out which predicate is causing the issue pretty quick. In the case the check fails, I'd like to see the args and the ret.
You should know which samples failed while generating @holyjak, no need to generate new ones.
Well, I do not generate anything, it is such-that
that does it - and in its failure-handling function I have no idea what values were actually tried. Or? BTW here is the code https://gist.github.com/holyjak/8cadc0d939c8e637ef6bf75b070d28b4#file-clojure-spec-alpha-clj-L17
Our hack (not acceptable for a patch but let's us debug generators without going crazy) is to store invalid values in an atom and display that in the error message.
Smart! The question is: What approach would be acceptible for a patch (and useful for the users!)?
I don't know. It definitely feels too hacky for a patch. It may be the only way to do it though. When I was looking at this, I don't think I saw any test.check hook to grab the failed values.
Just speculating but I imagine it'd be somewhat easy to add that hook to test.check if it isn't already there and is the proper solution.
I would definitely be in favor of adding some sort of patch to spec1 alpha given we don't know when spec2 is coming. Debugging generators is super painful as it is.
Check the such-that docstring in the latest version
Would you want all of them? Just the latest?
Having the generator means it should be easy to generate more
I believe a single value is enough (preferably the 1st as it is the simplest one)?
Keeping all could be a GC issue
> Having the generator means it should be easy to generate more
That is what I do in my solution (generate a new value using sample
) but it has been argued that using one of the actually failed would be better.
@gfredericks As I explained above, this popped up when working with specs - a custom generator generated value that did not conform to the spec. I can get the name of the spec but want to see why the custom gen failed to provide anything valid. There, having a sample value and running s/explain-data
on it is sufficient and very useful.
I am creating an issue on test.check to pass a sample failed value to ex-fn. Stop me if I should not 🙂
Conclusion: I create a patch for Spec to include the spec name (or rather path
) in the such-that failures and we wait for test.check to expose a sample failed value to the :ex-fn
before adding explain-data
of it.
@alexmiller is there any point in providing also a similar patch for Spec 2 or is it too much in flux?
I'm not going to do the sample thing, so not interested in a patch for that
no, not that, just including path
one possible outcome is that we repackage everything in spec 2 to remove alpha designation, which will (again) break any patches
I think in test.check, while generating in such-that, it would be possible to just retain the first or last gen'ed every time (before the pred check) and report that if you hit the retry limit. that way you're reporting an example that was actually tried.
An example generated after the fact is just as meaningful, philosophically
if only a percentage of them fail, and you happen to generate a sample that doesn't fail, that seems strongly less useful than one of the actual examples that didn't pass the predicate
so I'ma disagree with you on that
I was assuming you'd filter on failure 😛
well you failed to mention that :)
but that would be fine
Which, ironically, could fail :face_with_rolling_eyes: though presumably is unlikely to
But that might be a good enough reason not to do it
Adding the size to the failure data would assist with that though
@gfredericks Will you consider adding a sample failed value to the arguments of such-that's :ex-fn
or some alternative improvement?
@holyjak I'm not going to be doing active work on test.check for the foreseeable future, so that'll be up to somebody else; but a jira ticket is definitely the right way to make sure it gets looked at
Hm, I was about to create it but you told me I should not https://clojurians.slack.com/archives/C1B1BB2Q3/p1570030324189400 I guess it was just misunderstanding.
I'd be happy to provide a patch to test.check - and even happier to get any pointers regarding the best solution.
No I meant you should create the ticket
Sorry, I was being cute with English double negatives, it was probably a bad idea
I see 🙂 My bad, I should have read more carefully. Also, I believed that English normally doesn't permit double negatives. Always learning 🙂
Any pointers regarding the best way to implement this? Is failed-value
a good key name to pass to :ex-fn
?
FYI I have created https://clojure.atlassian.net/browse/TCHECK-156 Provide sample failed value to such-that's ex-fn for better error messages
FYI I have sumbitted a patch for ☝️
This patch has caused me some headache: https://github.com/clojure/core.specs.alpha/commit/938c0a9725a284095baa2387dff1a29c3f1e26ac
> What are my obligations if I copy source code obtained from http://Eclipse.org and licensed under the Eclipse Public License and include it in my product that I then distribute? > Source code licensed under the EPL may only be redistributed under the EPL.
@noprompt I'm curious as to how you're depending on specs from that namespace since they are intended to support clojure.core
macros?
(and, yes, you're going to be on some pretty shaky ground if you copy'n'paste that code into your system and then distribute it)
@seancorfield Imagine you want to write a macro that has the form as defn
.
Ah, so your code was relying on the internal names used in the those specs?
Only two top-level specs changed their names.
If you conform
, you have to rely on the implementation of the spec.
We've used alpha versions of Clojure libs in production for over eight years and sometimes Cognitect change stuff while it's alpha and break your code. That's the risk of using alpha libs.
They've been very clear that alpha means "can change/break". Things are only guaranteed additive/fixative once the alpha label goes away.
They've said it repeatedly publicly. Pretty sure even Rich has mentioned that in talks...?
Rich says it in the same talk where he contends SemVer is broken. Search for alpha in https://github.com/matthiasn/talk-transcripts/blob/master/Hickey_Rich/Spec_ulation.md Funny thing though, Rich also says something like:
But, that is not to say just leave your thing 0.0.967. At a certain point, you are going to have users, and whether you change it to 1.0 or not, they are going to be depending on your stuff.
And that is where spec is going right now IMHO. But his Maybe Not talk and Alex Miller's blog eased my fears of spec remaining in this state forever.I'm sorry you've missed it being discussed on the mailing list and in Rich's talks and here on Slack (where it has come up several times in several situations).
Not legally into anything you are distributing under an incompatible license.
If you want to use EPL code in your OSS project, consider changing your license to EPL so it's compatible.
Its fine, I can write the specs myself in this case. I was just hoping I could take a short cut.
Given the day I've had working on our Spec 2 branch, I do sympathize (even if it may not sound like it).
I’ve given up on trying to supply feedback or contribute to anything in the clojure
repo due to conversations that go the way they do regarding these kinds of frustrations.
Yeah, I can detect a fair amount of frustration/anger...
Transparency would be nice on these topics; one can’t be expected to keep up with every feed of data.
The repo and source code is about as public and as inclusive as any place to explain semantics.
I’ve made the effort in Meander to call out the versioning semantics I employ so there is no misunderstanding.
When Rich introduced spec in 2016, he was pretty clear that it was alpha and subject to change (I just checked the transcript of one of his talks from back then). He also called out alphas as "potentially breaking" in his Spec-ulation talk (which is where his main criticism of SemVer was made.
SemVer itself is broken tho'. While you're in alpha, you can make (breaking) changes. Once you have a non-alpha, you should only make accretive/fixative changes. SemVer doesn't help with that.
That makes this assumption that people will be exposed to those things or, if you have my memory, remember them.
Just slap a phrase on the README that says, upfront, what the version semantics are instead of relying on them being implicitly understood.
What I took from Rich was if I’m going to make a breaking change, I make a new namespace, artifact, etc.
If you're not in alpha, yeah.
It was an important caveat in his talk 🙂
(and it's been the practice in Clojure overall for years)