in the cognicast espisode rich mentions a new syntax for default namespaces for keywords in a map - do we know what that looks like yet?

@shaunxcode: this jira item has some info http://dev.clojure.org/jira/browse/CLJ-1910?page=com.atlassian.streams.streams-jira-plugin:activity-stream-issue-tab

@alexmiller It's a shame that conform does not (and probably won't) flow into map-of and coll-of 😞. I have a use case where performance is mostly irrelevant and inner values should be conformed. It seems the functionality is there, but instead i'll have to resort to some error-prone, hairy tree transformation. Would you perhaps consider a dynamic var or an optional argument to map-of and coll-of to enable such transformations (at the expense of performance)?

@moxaj there actually is 'coll-check-limit' now

Defaults to 100 but you could set it higher

I'm not sure whether that changes what flows though

@alexmiller It does not, currently. Just to be clear, only the size restriction is settled on, the flowing stuff might be subject to change?

Let's just say I'm not the one making the decision :)

a conforming coll spec would not have the same name - either you are exhaustive/conform or neither, not going to be a dynamic magic thing

Is there a way to specify in spec, that a key is the focus of my validation, (or where the path of the failure would be), but take other values from the map? Useful for having two fields that match for example.

you can write a custom predicate that validates anything you like

like #(= (:foo %) (:bar %))

not sure if I'm answering your question though

@dominicm: I guess you want something like (s/and (s/keys :req [::password ::password-confirmation]) #(= (::password %) (::password-confirmation %)))

@alexmiller @wilkerlucio Do these methods set the paths? So that path would/could be set to [::password-confirmation]

@dominicm: sorry, I don't understand what you mean by set the paths, can you please clarify?

@wilkerlucio: In the failure, you get a path when you do a key.

well, you always have to be composing, so it may depend where the failure is

if it's a key with a problem, it will point directly to it

if it's on the last checker (the match one) it will point to the map and say that the fail comes from the map and point the predicate out

does that answer your question?

@dominicm: I think is a good idea to try the (explain) on the REPL so you see if the explanations of the errors met your needs

@wilkerlucio: Yeah. It answers my question. I'm trying to figure out how I can that the predicate after failure, actually comes from ::password-confirmation when doing explain.

(s/def ::a boolean?)
(s/def ::b boolean?)

(s/def ::thing (s/keys :req [::a ::b]))

(s/explain
  ::thing
  {::a true
   ::b "foo!"})
;; => In: [:foo.core/b] val: "foo!" fails spec...

I want to be able to set that In for say password validation. So that it's set to ::password-confirmation

To clarify, I want this:

(s/def ::a boolean?)
(s/def ::b boolean?)

(s/def ::thing (s/and (s/keys :req [::a ::b])
                      #(= (::a %) (::b %))))

(s/explain
  ::thing
  {::a true
   ::b false})

@dominicm: not sure if there is an easy way to do that, you might get some insight by checking the sources for (s/keys)

@wilkerlucio: Looks like it works on reification. Fun fun fun.

is there any way to tell what spec was used to conform a value?

no?

I was looking for a way to define a function to transform conformed output

so if I had a way of extracting the keyword that defined the spec I could use a multimethod to do that - probably misthinking it

sure I can find another way, thought it was worth asking anyway. thanks @alexmiller?

@brabster: if you would like to customize a spec's conform function, you can do it by implementing the Spec protocol: https://github.com/nwjsmith/datomic-spec/blob/master/src/datomic_spec.clj#L286-L293

Although I'm not sure that's the best way to go about it, I just wanted my Datomic queries to conform the same whether in map or list form

I think you're treading on impl details there that can change without warning, fyi

😓

I'm going to try the multi-spec route to see if I can get similar results. Maybe I don't want conformed map and list queries to be the same.

@nwjsmith: thanks for offering up the suggestion anyway 🙂

We're having a meetup about spec and are wondering about the output in this snippet:

(s/explain-data (s/cat :cat1 keyword?
              :cat2 (s/& (s/cat :cat1 integer?
                                :cat2 integer?)
                         #(> (:cat1 %) (:cat2 %)))
              :cat3 keyword?)
       [:foo 42 43 :baz])

having issues just getting specs going... lein repl complains about spec not being defined

@fenton have you tried lein clean?

What is the output of lein deps :tree?

thank you...i had it over-ridden in ~/.lein/profiles.clj ! doh!

@borkdude: what are you wondering?

the fact that it reports the error at :baz ?

that’s because :cat2 has eaten the ints, then failed its secondary pred. Currently & can’t know that its preds won’t be made happy by some future input, so it’s still working on the & and then can’t eat :baz

currently regex ops can communicate that they can succeed w/o more input, but not that they are ‘full’, which is what would be required for & to bail out

that particular case could probably be handled though

what does (s/cat :a :b) mean? It compiles, but matches only empty lists

to me it does not seem a valid spec, it should complain

yeah the :b seems like the potentially invalid part

well though

I’m asking because I had smth like (s/cat :url ::url) and I forgot to define ::url above

keywords are predicates

so it's probably being treated that way

but arguably it would be worth not treating them that way since you can't used namespaced keywords as predicates anyhow

though if it matches empty lists then I'm probably mistaken about the treating-it-like-a-predicate part

(s/valid? :a {:a 1}) throws an exception

could there be spec-enabled versions of fn and defn?

@richhickey: thanks!

writing a function (defn age-plus1 [{:keys [::age]}] (inc age)) already defines something about it’s inputs, the specs could be extracted from the source?

@borkdude: next alpha will do this ^

thanks for the report

@ikitommi: If I understand correctly, the reason for fdef and explicit instrumentation is to prevent default instrumentation of functions.

@richhickey: In the cognicast you mentioned a bit about the possibility of generating specs because they're data. It's not obvious to me how I would be able to do that, could you point me the right direction?

they are data as code is data, you’d generate them as you would any other code

oh that's what you meant! thanks 🙂

call s/form on a spec for an example

that's very useful... I'm playing with a (simple) heuristic that would produce specs from example data

hard to get right, but maybe I can come up with something that's good enough

specs for spec will help, e.g. the spec for s/cat:

are those somewhere in the code already, or still unreleased?

not yet released

but that's the 2-stage generate you mentioned in the cognicast

I have to go through them with @alexmiller still

right, that’s for testing mostly. Working from example data is interesting, but not possible in the general case

testing and parsing I should say

although now with unform, the toolchains get interesting

I'm inspired by F#'s type providers... if they can do it to a useful extent, I think the idea has some mileage

if not regex-requiring sequences, not so bad. keysets mostly and leaf scalar types

still you won’t be able to tell required from optional, etc

yeah figuring out regexes would take some crazy sequence alignment algorithms (as in biology)

well, if a key appears in only half the maps, we can assume it's optional... I'm thinking this could be a cool way to inspect data

sounds like fun

we'll see how far I can take it! Imagine applying this to functions while exercising an existing codebase to figure out signatures.

As I understand from the docs multi-spec is meant for tagged maps, but would be appropriate for specs that apply to either maps or sequences?

An example:

(defmulti query-form (fn [query] (if (map? query) :map :list)))

(defmethod query-form :map [_]
  (s/keys :req-un [::find] :opt-un [::with ::in ::where]))

(defmethod query-form :list [_]
  (s/cat :find (s/cat :find-kw #{:find} :spec ::find-spec)
         :with (s/? (s/cat :with-kw #{:with} :variables (s/+ ::variable)))
         :in (s/? (s/cat :in-kw #{:in} :inputs (s/+ ::input)))
         :where (s/? (s/cat :where-kw #{:where} :clauses (s/+ ::clause)))))

(s/def ::query
  (s/multi-spec query-form (fn [g _] g)))

What do I lose by dropping the re-tagging in multi-spec?

you can do any discrimination you can with a multimethod - tag keys in maps is just an example. Also note that retag could be polymorphic

Saw an earlier comment that instrumenting protocol implementations doesn't always result in the spec being validated. Will this be supported at some point? Or does it even make sense, i.e. would it be "better" to spec a map with function specs instead of using protocols?

multi-spec is for data-dependent specs, generally

spec + protocols tbd. Certainly protocol behind API fn is good practice and supported (spec the API), but many people expose their protocols directly

is there a way to disallow non-mentioned keys? (s/valid? (s/keys :opt-un [::foo ::bar]) {:foo 1 :baz 2})

An and with a map-of spec and your keys spec will do it I think

map-of is for homogenous maps

Yes, but you can use map-of's kpred to ensure the keys are the ones you accept

ah, you mean put a set there?

yes

indeed, it works

thanks

you can use s/and with s/keys as well

(s/and (s/keys …) some-further-constraints)

map-of not nearly as powerful, won’t do per-key validation etc

map-of is for map-as-collection, keys for map-as-information-object

I see

now I’m trying to support :opt in this case

you should always question why you are limiting keys

just creating brittleness

well, I’m just exploring the possibilities

but people keep asking for this, maybe we’ll have a :closed option to keys

this works:

(s/valid? (s/and (s/keys :req-un [::foo] :opt-un [::bar])
                   (s/map-of #{:foo :bar} nil))
            {:foo 1})

map-of samples, will not test every key, so a set not a good predicate

but keys will

A use case I can think of is where you might want to limit keys on a map you expose as a data structure outside your code boundary: for example something you convert to JSON to return from a web service or when you insert! into a SQL database (using clojure.java.jdbc for example, where additional columns would lead to an exception). But I’d question whether those are places where you should rely on something like clojure.spec to catch the error (instead of a more explicit approach).

We have select-keys for that sort of scenario, right?

yeah

@dryewo: keys is open, so will let other keys through, and map-of won’t necessarily catch them, it just samples for type consistency, not particular values

you could use #(every? #{:foo :bar} (keys %)) as the second pred

I tried this: (s/valid? (s/map-of #{:foo :bar} nil) {:foo 1 :baz 2}) and figured that map-of limits possible keys to those that satisfy the set. But you say it’s something that can change in the future?

the solution with every? looks quite ultimate

I found *coll-check-limit* in the sources, that explains

neither coll-of nor map-of do exhaustive checking, nor conforming. We will make the docs clearer

they are ok for very large collections

thanks, that makes sense

i don't want closed sets of keys without a compelling example

haven't seen one yet

enough people seem to want exhaustive conforming version so we’ll likely have both

@ghadi: well, that’s why open is how it is, I agree 🙂

the only thing that comes to mind for me is the principle of being conservative about the data you return from an API

you just engender brittle code on the other end

oh if the closedness is publicized then yeah

I was just imagining using it for checking that you aren't accidentally leaking extra stuff

well as @seancorfield said, you can call select-keys

but I suppose specs are intended to be publicized

right

cool, I'm on board

there’s a bit in the podcast interview about change, while maybe not made clear yet, this kind of thing is an important part of the design

because it means adding new keys is backwards compatible?

there are separate semantics for accept/return, but they align with co-contravariance, and the keyset and regex system will let us say when things accept/return compatible things

how will predicate equality work in that case? check that the forms are the same?

i.e. :req less on what you accept is compatible, :req (essentially guaranteeing) more on return compatible, etc

regexes can be tested to see that they new accepts everything old does (accept) and old accepts everything new does (return)

base predicates presumed immutable, yes forms

what I hope people find is that at the fine granularity of spec, things rarely change, and if the do they must compatibly

and it could get fancier (generative testing to do the non-spec preds)

richhickey: I asked on the ML about tactics for moving libraries forward; in the podcast you mentioned something about adding numbers at the end of function names, do you imagine that being a sort of last resort?

stu also seemed to mention creating entirely new namespaces, so I'm curious how common you imagine each of those tactics being

how often do you change the sigs of public fns? I think for algorithmic things rarely and for informational things only in spec-compatible ways. It’s the breaking change that;s the last resort, the numbering of fn names is just being honest with your users

The problem with libraries is that a lib has 50 fns and when you add 2 new ones it gets a different ‘version’ and no one knows what’s different. What they actually depend upon are the individual fns that they call, not the lib. Stable specs will show the true granularity of change (usually additive)

I’d much rather deal with foo-2 (which btw can co-exist with old foo) than ‘version 2’ of foo.

I can move to foo-2 when I please

yeah, I definitely appreciate that

there are a handful of minor warts in the test.check API that feel hard to improve without breaking

for some value of "breaking"

at least if "the API is surprising and hard to sort through" is part of the problem, as then adding a foo-2 might maintain those downsides

so I suppose I'm having to accept not being able to "fix" things in that way

for big changes you can also do test.check2

as the lib name, or namespace name?

lib/ns same difference

then people can move when they please, specs will be in different nses, so they haven’t changed

test.check2.11

yep

richhickey: thanks

@gfredericks: I would like to be able to convey explanation data through such-that somehow

richhickey: alex mentioned something about having a fancier such-that to make the generator for s/and more robust, do you know if that's a different issue from what you just mentioned?

(just so I can keep track)

maybe a (compatible!) extra arg which is a fn to call on failing val to get exception text)

not sure if that’s what @alexmiller was talking about

what he said exactly was: "I think there is some desire to have a better (programmatic) way to have a conversation about filtering with such-that than just boolean yes/no"

yes, so right now the pred fails and test.check can’t say anything useful because the pred is boolean

re: open/closed keysets -- I haven't had the need to ensure a closed set, but when working with an API not under my control, it is useful to detect added information

but spec knows what the conditions were

richhickey: okay so he was talking about the same thing? I interpreted his statement to imply some super-fancy way of giving such-that hints about other things to try

@ghadi: yes, that’s different, and has been requested - (keys-not-in spec map)

@gfredericks: nope, just better failure reporting, because the failure to gen could be in a nested spec we’d like to at least tell the user which one

richhickey: is this a fair summary? http://dev.clojure.org/jira/browse/TCHECK-107

could something along the lines of keys-not-in work at any path? use case is detecting a fully compatible change made on an external API

richhickey: after thinking about it a bit, I'm imagining a function that's expected to return a throwable, that way you can customize the error text and the ex-data

I guess that means the stack trace starts in a weird place though, which I was trying to avoid :/

Big thanks to everyone who contributed to clojure.spec. Spent two days building an internal library on it, and it’s, so, so, so, so, so, so… enjoyable.

@ghadi: another use case i just ran into in a (non-clojure) project: warning on configuration errors

had a config.toml file and somebody had a rogue [Section] and a Key = true line was getting put in to the wrong section, so the config setting wasn’t taking hold

i used a feature of our toml parser to print all the unrecognized keys to the logs