somethign else tht would be nice in spec would be to gnerrate recursive structures breadth first, and when we hit a certain node count, only take non recursive branches

is there a spec option to limit the size of coll-of ?

where is the code which shows how coll-of are generated?

I'm looking at github/clojure/clojure, which sees to show checking of coll-of, but not generating of coll-of

:gen-max is the option to limit coll-of

does clojure spec support things like parameterized spec? I want to be able ot say things like "this is a ::tree-of ::widget" and "this is a ::tree-of ::gadget" instead of havint to define ::tree-of-widget and ::tree-of-gadget (or one for each 'type')

No.

you can however use s/multi-spec to create an “open” thing in the middle

Ah yes, I need to into multi-spec some day 🙂

@alexmiller: I have no idea what this multi-spec technique is. Can you point me at a tutorial?

http://clojure.org/guides/spec has an example

but basically you use a multimethod to choose which spec to use based on the data

This question probably was raised already. I’ve played with spec in ClojureScript a little, created a spec for application state for validation purpose. But I found it annoying to use fully-qualified keys with maps, because it requires the namespace whenever I need to get or destruct a map. This is what stopping me from using spec in production code. Am I missing something? I feel like there should be a reasonable explanation for this.

@roman01la there are some new syntaxes for namespaced keywords

@gfredericks do you mean namespace alias? I’m using them.

for constructing maps whose keys all have the same namespace

also {::keys [foo bar baz]} I think

when destructuring

the map syntax is #:foo{:bar 12} and #::baz{:bang 12} I think

you should see it at the repl when it prints a map with namespaced keys

@roman01la: Not sure if this is what you mean by "require" you don’t need to :require and load a namespace to use keys that are fully-qualified

but you do to alias

yes

roman01la: also not sure if you know but s/keys has :req-un and :opt-un to let you bridge between unnamespaced keys in maps and namespaced keys in specs…

@gfredericks that’s great, thanks! #::baz{:bang 12} doesn’t work in ClojureScript, probably not implemented yet

@rickmoynihan yes, thanks

@roman01la: if the clojurescript compiler runs in clojure 1.9 does it work?

@gfredericks didn’t try it on 1.9 yet

if cljs still uses clojure's reader then that'd be a prereq for sure

@gfredericks tried with 1.9.0-alpha14, still doesn’t work. I guess ClojureScript has it’s own reader. Anyway, thanks for sharing the new syntax, this will save me some time!

@mpenet can you expand on what you mean by "One of the thing I like is togglable asserts in pre/post" ?

@pyr using :pre/:post in fn definitions, mostly s/valid? calls. and having the abitily to switch it on/off depending on assert value at compile time

*assert*

ex with a profile with :global-vars {assert false} for prod

you can do the same with s/assert, inline, but I find pre/post kind of nice for this stuff

s/assert has the advantage that you can toggle in on/off at runtime

yup ok

all of this is handy for the cases when instrumentation is either too intrusive or when you want to actually run this stuff in prod

@mpenet thanks, i'll play with that later today to see how I fare

I didn't know :pre/:post validation asserts could be en/disabled at compile time

I found that old post talking about it, old stuff 🙂 http://blog.fogus.me/2009/12/21/clojures-pre-and-post/

I was reading that yes, still first google link to show up 🙂

I guess it could make it relatively easy to write some sugar on top of it to have Schema like defn with annotations.

But personally I am good with what we have

yes, this is also enough for my needs

is there a good way to test that an fdef with a generator will throw an exception somewhere? I'm not having success

can you expand that question? not understanding it

Sure thanks Alex. I'm testing a function with stest/check, an fdef and a generator that is guaranteed to cause an exception. I can't figure out what :ret I can use in the fdef to check for the exception

you can’t

:ret is for non-exceptional returns

there is no way to spec exceptions

Ah ok makes sense then

what is the rationale behind clojure.spec/or returning map pairs? I was hoping to be able to use it to validate what is essentially a sum type, but seems like I will have to do more processing to handle that. I guess my other question is, is there anything suitable for validating a sum type (i.e, simply has a bunch of predicates, and is valid if one of the predicates returns true)?

happy to get pointed to docs; apologies if this has been asked many times here

I do see the doc-string says > Returns a destructuring spec that returns a map entry containing the key of the first matching pred and the corresponding value. Thus the 'key' and 'val' functions can be used to refer generically to the components of the tagged return.

I sense the rationale is lurking in there, but I suppose this speaks to my larger ignorance of how to use spec idiomatically

something something use maps more something something

Maybe I’m oversimplifying your question @ddellacosta , but when you conform using s/or, how would you know which predicate matched without some way to identify it?

the idea is that the tag is used to identify how spec parsed the value (and also used to identify problems in explain errors)

@joshjones why do I care if it’s a valid value of one of the types I’m expecting?

@alexmiller I guess I don’t see how to easily use that when composing it with another spec

many specs conform to a structure that describes how they were parsed

all of the regex specs, etc

yeah, there seems to be something basic I’m missing here

two options for not having this issue specifically with s/or...

1. write a function that accepts all the options, so instead of (s/or :e even? :p pos?), do #(or (even? %) (pos? %))

that makes the choice opaque (which may be good or bad depending on your needs)

2. use the currently undocumented and possibly to-be-removed spec s/nonconforming: (s/nonconforming (s/or :e even? :p pos?))

if you just want to check whether a value is valid, use s/valid?. if you want to know why it’s valid, use s/conform. If you want to know why it’s invalid, use s/explain.

@alexmiller thanks for all of that

I guess part of the problem is that I’m using s/conform and expecting it to more or less simply return the value I’m testing, except in specific cases (changing vector to set, in this case I think)

and part of that is maybe that I don’t yet understand the use-case for s/conform

and how one would use something like s/or effectively

within that context I mean

yeah, s/conform returns a different value than the original value specifically to tell you why/how it’s valid

if you don’t care, don’t use conform

right, okay…that is a helpful way to think about it

note also that you can use s/unform to retrieve the original value from the conformed value too

I guess thinking about it a bit more, I see two distinct possible values that could be returned from conform: one is the value coerced to a new value, and the other has added spec metadata —it seems to me right now that s/conform conflates these two somewhat

for example, converting a vector to a set is the former

whereas the output of s/or is the latter

am I off here? I still feel like I don’t get the broader implications of spec, or how we might be able to use it

(if (s/valid? ::my-spec some-value) some-value (handle-error (s/explain-data ::my-spec some-value))) seems to be what you’re looking for @ddellacosta ?

(or some similarly structured setup)

We use both s/valid? and s/conform — we use the latter when we want a result that “conforms” to a spec but may start off slightly differently (which we use for our API inputs, to produce domain model values). And as Alex said, when we don’t care about any potential change in the data, we use the former.

@seancorfield thanks, will see if that does it. Per your second point, ideally I’d like to use s/conform in the same way—but if s/conform returns data that has been changed to represent how a value conforms to a spec, doesn’t that defeat the purpose?

Well, it will only do so under the rules of the spec itself.

@seancorfield: i would be very interested to learn how you use spec in real world projects

we're introducing it right now, running alpha 14

and it's a bit tough to find the right "pattern" that makes sense to us

For example, we have API input specs that accept either a date, or a string that can be parsed to a date using two allowed formats. s/valid? on such a string will just return true (and we still have a string; or false for an illegal string). s/conform on such a string will return a date (or ::s/invalid for an illegal string).

so in this case is it accurate to say that you are using spec to actually transform the data? (string -> date)

Another use case: we allow members to search within specific distances (dating site). If they are a subscriber, they can specify distances of 20, 30, 40, 50. All members can specify distances of 150, 500, 1000, and -1 (”any”). We use a spec with s/or to get back either [:basic distance] or [:platinum distance] so we can validate both the actual value and which category it belongs to.

@joshjones Yes, some coercions. As Alex often cautions, that conforming / coercing means all clients of those specs get the coercion if they use s/conform but in this case that’s what we want.

to me that seems like a spec is somewhere in between a value and a statement about a value

I find this pretty confusing, fundamentally

seems like it makes s/conform of limited use

or at least, limited in the case of using s/or

yes yesterday @alexmiller said: “spec works best when you primarily use it to speak truthful, assertive statements about what your data is and not to transform your data or make statements about what your data is not” https://clojurians.slack.com/archives/clojure-spec/p1484765212007116 https://clojurians.slack.com/archives/clojure-spec/p1484765279007117

again, I’m talking about s/conform here @joshjones

and it seems to violate exactly what those comments are saying, in fact

If you need to know which “path” of conformance your value takes, s/or is very useful. Since otherwise you’d need to code up both the validation logic and the parsing logic in order to determine that.

I’m not referring to anything you’re talking about @ddellacosta , only to what @seancorfield said about his use of spec

@joshjones ah, apologies—misunderstood

no problem, I am also seeking a better understanding of best practice :thumbsup::skin-tone-2:

yeah, along those lines this is all really helpful—thanks for linking to those comments @joshjones

It’s taken us a while to settle on this approach — given that Alex has repeatedly cautioned against specs that coerce data 🙂 — and we also have the added complication that for a given API, some of our input parameters must be looked up and “conformed” dynamically (i.e., we have a database table listing possible inputs and each row identifies a spec that should be applied). So we can’t just write static specs for input arguments, we have to have somewhat generic specs and conform each input argument based on a lookup.

(and we’re still wading through cleaning that up)

We could probably simplify this by dynamically generating specs directly from our metadata at this point — and we may well do so.

so question for you, @seancorfield —why not simply write something explicit that does not rely on spec to add the metadata about your distance (platinum vs. basic)? It seems to me that this is a value, not a spec.

I’m asking selfishly to clarify my understanding of spec, to see how you are thinking about this, as I still don’t fully get it

Because using spec allows us to remove a lot of logic that we had before.

(s/def ::platinum-distance #{20 30 40 50})
(s/def ::basic-distance    #{150 500 1000 -1})
(s/def ::distance (api-spec ->long (s/or :basic ::basic-distance
                                         :platinum ::platinum-distance)))

Having the declarative specs means we can show those to business and they can “read the code” without having to actually read real code full of conditionals and transformations.

We can also have the business rules in one place (for the most part) and they drive not only our actual business logic but can also be used to generate test data (and test functions extensively).

okay, I’ll have to think about that a bit. Thanks a lot @seancorfield , this has been very helpful

how would one spec varargs?

could I just use the vararg's name and use coll-of?

scriptor I think the idea is to spec the args as a sequence with spec/cat and use spec/? for the optional args

@scriptor yes, compose the var args the same way you’d compose any sequence.. only it doesn’t have to be coll-of … can be k/v pairs, any sequential spec. for example:

(s/def ::someopt nat-int?)
(s/def ::vargs (s/cat :optargs (s/keys* :opt-un [::someopt])))

(defn myfunc [& optargs] nil)
(s/fdef myfunc
        :args ::vargs)
(stest/instrument `myfunc)
(myfunc :someopt 55 :otheropt 99)

can be a coll-of, use spec regexes, anything … the above just shows a common use case for var args: k/v pairs in a sequence (as opposed to a map)

so :rest indicates the following spec is for the optargs parameter?

well, not indicates, you're just using the name :rest

yes, should probably have called that optargs .. will change it for clarity

perfect, thanks @schmee and @joshjones

@scriptor I need to clarify something after doing another example — my example above works but I wanted to use another common case because I was wrong about just using coll-of for cases where you have one required and then an optional argument. for example:

(s/def ::vargs (s/cat :req string? 
                      :optargs (s/* int?)))

if you use coll-of for :optargs it will not work because you’re giving it multiple items, not an actual collection

ah, interesting, I guess it's because of the regexy nature of s/cat

well, s/cat gives the actual argument vector spec. but inside that, you then have several elements, and one of those is not a collection if it’s for example: [”abc” 42 43 44]

the s/* greedily consumes the rest of the arglist, which as you said is multiple separate items and not a single collection

what you have there is a string, an int, an int, and an int .. no collections

yes that’s correct. i feel my answer above was misleading so i wanted to clarify

another example that will match (fn [s & opts]):

[”abc”]
[”abc” 3/4]
[”abc” 3/4 42.9]
[”abc” 3/4 33.3 10.5 99.9]

(s/def ::vargs
  (s/cat :req string? 
         :optargs (s/? (s/alt :ratioonly ratio? 
                              :ratio-double (s/cat :s ratio?
                                                   :i (s/+ double?))))))

wanted to show that the optional argument portion of the sequence can be more than just a simple homogeneous concat of elements

you should really use s/keys* for this - that is it’s reason for existence

(s/def ::varargs (s/cat :req string? (s/keys* :opt-un [::ratioonly ::ratio-double]))) etc

@alexmiller that was in my first example above — but doesn’t that require k / v pairs?

yes, isn’t that what you’re specing?

oh, I see you have a :ratio-double with multiple vals

no, in this case i was showing a spec that would match all of the following:

(myfunc “required string”)
(myfunc “required string” 4/3)
(myfunc “required string” 4/3 33.2 99.9)

I’m going to go out on a limb and say that’s weird :)

and you shouldn’t write a function signature that way in the first place :)

yes it is — my first example showed what you said — i was just showing how the optional arguments portion of a function need not be a homogeneous group of items .. not endorsing it, just demonstrating the versatility 🙂

ok

saying “hey, if it can do this, it can definitely do something much simpler” :thumbsup::skin-tone-2:

is there a way to do multiple specs at once like you have a bunch of things that should be a string so instead of doing (s/def ::foo string?) (s/def ::bar string?) you can just say [::foo ::bar :baz ::fname ::lname ::email] string?

has somebody done a richer explain function yet? i find that when you have some ors or alts w/ larger maps involved, the output becomes unwieldy

@gdeer81: no

@bbloom Drogalis started porting some of his onyx stuff to work over spec explain-data (and using fipp) - really more of a poc stage

cool

🙂

I actually picked that up and hacked on it a bit but found that explain-printers are missing some crucial data

namely, the original root value and root spec

that’s missing from the explain-data map

I have a pending enhancement + patch to add that

ah yes, those things and the immediate parent or some chain of anscestors should be passed down

this way explainers can be context sensitive

everything else is discoverable within the explain-data path

ah right, can just pop on the path

that makes sense

but you don’t have the root value

(get-in root (pop path))

gotcha. well looking forward to that 🙂

thanks

https://github.com/onyx-platform/empathy is Mike’s stuff

lol @michaeldrogalis love the name

:)

this brings up one other issue that i’m running in to now

Heh. Never did get around to finishing that.

in fact, it’s one that i ran in to in the past

I’ve done some more work on it

but not in a public repo

https://github.com/plumatic/schema/pull/134

I’ve shelved it for the moment until the patch above gets through as then it will be easier to demo

i’m running in to a situation now where i have a recursive spec that is giving me useless explain output b/c it’s producing LOTS of output

b/c the recursive spec fails to match

one thing you could do with explain-data is find the CLOSEST match

@alexmiller Curious as to how you attempted it. Took me ages to do it with Schema, not really looking forward to trying it again with Spec.

I seem to recall Colin made something similar in the macro grammar stuff he did

^^ @bbloom

yeah, it’s kinda critical i think

otherwise spec falls down on ASTs and other recursive structures after data gets too big

@michaeldrogalis well, I mostly brought it up to more recent spec and extended it just slightly further

at least explain does

Ah, got it.

unfortunately the literature on error detection and recovery in parsers seems to be kinda ad-hoc

@bbloom I also have a patch that sorts the longest errors first, which generally is a good strategy

yeah, that’s a good start

except with recursion when you probably want the reverse :)

heh

is that longest path to an error?

yeah

or just BIGGEST error?

ok

i asked about this on twitter a while ago in the context of text parsing

https://twitter.com/jordwalke/status/805921251004821504

reference manual is here: http://gallium.inria.fr/~fpottier/menhir/manual.pdf

chapter 10 is about error handling, which may be worth studying

there seems to be some attempt at a sensible system for producing good error messages

I think cleaning up fanout is another big area for improvement

For removing repetition

you mean like representing the output as a tree?

Nah, just reporting all errors at common path with single context

ah ok

An ide might do a tree though

another idea is a sort of two column view

ie pretty print (or just indent) on the left

and messages on the right

I suspect recursion will just suck regardless :)

yeah, well recursion always sucks when it comes to printing stuff 🙂

as a man who does a lot of recursion and a lot of printing, believe me, i know

heh

whatever happened to that data structure visualizer? is that still built in? lol

it is :)

it still doesn’t work very well :)

yeah, that’s the thing about text based UIs

when they are bad, they are kinda bad

vs guis

when they are bad, THEY ARE VERY BAD

gotta run, later

cya