Fork me on GitHub
#clojure-spec
<
2016-06-29
>
Alex Miller (Clojure team)02:06:01

I’ve updated the guide for 1.9.0-alpha8 although there will be some additional updates http://clojure.org/guides/spec

martinklepsch09:06:08

I have a question about spec'ing maps: lets say I have two keys that should follow the same spec, is the intended way to define proxy specs which I then use as map keys?

Alex Miller (Clojure team)11:06:03

You can register the map keys to point to either the same spec definition or to another registered spec. I think doing this will be common. One layer of registered base "types" and another layer of domain attributes.

martinklepsch12:06:14

- [changed] explain-data - output is now a vector of problems with a :path element, not a map keyed by path
Whats the reasoning behind this change? Could there be multiple problems referencing the same path?

richhickey12:06:10

@martinklepsch: yes, multiple problems at same path happens with collections (was already happening with ‘keys' which jammed them all into same pred, now doesn’t)

robert-stuttaford14:06:50

@martinklepsch: on your question from 11:20 today, do you mean like what i've done with ::pos-int here? https://github.com/robert-stuttaford/mtg/blob/master/src/mtg.clj

martinklepsch14:06:50

if you want to use a generic spec in keys without using it's key you need a proxy like this

Alex Miller (Clojure team)14:06:22

@robert-stuttaford: there is a pos-int? predicate now btw in core

Alex Miller (Clojure team)14:06:34

that ::types spec you have can now be accomplished with coll-of in alpha8 too: (s/coll-of ::type :kind set? :min-count 1)

Alex Miller (Clojure team)14:06:38

that version has the benefit of automatically gen’ing too

Alex Miller (Clojure team)14:06:07

And I think your ::spell, ::creature, ::planeswalker, etc can also be handled by s/merge now

robert-stuttaford14:06:41

how far, as a %, would you say the core.spec slated for 1.9 is?

robert-stuttaford14:06:07

it seems like you've still got a lot of meat on the bone, given all the stuff that gets added with every alpha

richhickey14:06:07

@robert-stuttaford: not a lot of big things - assertions, perf tweaks, reconciling regexes and vectors, some refinement of keys*, then details

richhickey14:06:02

then moving to infrastructure around test-running etc, the whole model implied by instrument+test which is different from current practice

ghadi14:06:24

On the Complexity and Performance of Parsing with Derivatives -- PLDI 2016 https://www.youtube.com/watch?v=WPde6DDLtXg

ghadi14:06:41

iteration.next of matt might & co's work

robert-stuttaford14:06:48

fantastic, rich! very exciting set of tools you're building, here. fundamental, like the data-oriented approach and immutability are. going to take some time to soak in

robert-stuttaford14:06:40

i realise this may well be answered with 'maybe :-)', but i'm super curious how you plan to leverage spec in Datomic 🙂

richhickey14:06:25

@ghadi: thanks for the links

robert-stuttaford14:06:54

Datalog is an obvious one, as is transaction data. wondering how else it may improve the experience of programming with it

richhickey14:06:09

@robert-stuttaford: still TBD (but lots of obvious opportunities), first things first, and that’s a stable clojure.spec

richhickey14:06:46

I can say we are getting a lot out of using it in the development and testing of Datomic already

robert-stuttaford14:06:34

i can't think of a better way to dogfood it

richhickey14:06:40

the new stubbing support is something to check out - you can now take a version of your wire interface, spec it, and get a no-wire stub for free from spec

robert-stuttaford14:06:45

ah - if i understand you correctly, you mean inferring a spec from an existing data set?

robert-stuttaford14:06:00

if so, thanks .. i was planning to do this as a learning exercise!

richhickey14:06:15

no, let’s say you have a data-driven wire protocol that talks to a service - you spec the protocol, instrument its namespace saying :stub in the overrides, and then run tests of code that consumes the protocol. Now those tests don’t talk to a service at all but get checking of their payloads and generated returns

ghadi14:06:18

aside: I really like the syntax of (s/alt :foo x :bar y). That is something I didn't get quite right in pex, my vm-based PEG parser; I made capturing things too verbose. Definitely going to adopt that approach

richhickey14:06:41

you can selectively override gens to generate ‘stateful’ things like connections

richhickey14:06:52

@ghadi: yeah, the path system is something I think people resist initially, but it keeps paying dividends

robert-stuttaford14:06:19

man. i feel like i'm going to need a spec sabbatical.

robert-stuttaford14:06:33

so many new toys, but at such a fundamental place in the language.. it's going to touch everything!

ghadi14:06:47

curious, why do people resist the path system?

richhickey14:06:25

sometimes you don’t care about the labels or want to see them in the destructuring

richhickey14:06:04

but they let us talk about the structure of a spec all the way down, which is huge

robert-stuttaford14:06:06

not seeing anything about stub in there, yet

ghadi14:06:49

my plan for better dev-time experience in the PEG library was to provide a tracing mode on all the patterns attempted

ghadi14:06:25

since backtracking is likely in a failure, and most of the ways to handle it were through heuristics like longest match

Alex Miller (Clojure team)14:06:47

@robert-stuttaford: I think those are lagging an alpha or two, I will get them updated

richhickey14:06:18

oops, didn’t realize API docs were behind

bhauman15:06:31

alright, I'm running into an issue in alpha7

(s/def ::string  (s/or :string string?))
  
  (s/def ::myid ::string)
  
  (s/explain (s/and
              (s/keys :opt-un [::myid])
              (s/keys :req-un [::myid]))
             {:myid "asdf"})

bhauman15:06:13

Results in

In: [:myid] val: [:string "asdf"] fails spec: :strictly-specking.test-schema/myid at: [:myid :string] predicate: string?

bhauman15:06:55

it appears to be validating against the conformed value

bhauman15:06:13

I'll check against alpha8, in a bit, i have to port some stuff

bhauman15:06:55

and it's strange because this works

(s/def ::myid string?)
  
(s/explain (s/and
              (s/keys :opt-un [::myid])
              (s/keys :req-un [::myid]))
             {:myid "asdf"})

bhauman15:06:51

and that makes sense because the conformed value is just a string

richhickey15:06:10

‘and’ does flow conformed values - "Successive conformed values propagate through rest of predicates."

richhickey15:06:10

that may change for ‘merge’, which is what you should switch to for this application

richhickey15:06:29

there may also be and and and-> variants since sometimes you need this (e.g. when preceding pred is a regex) and sometimes not

bhauman15:06:31

great, looking at merge now

bhauman15:06:36

the above works with merge right now and inspection reveals that the conformed values aren't flowing right now.

richhickey15:06:39

@bhauman: merge does flow through conform, not explain (that’s broken)

richhickey15:06:37

but no reason for it to flow through conform, that will change. The big benefit of merge is that it will gen where ‘and’ cannot for this

bhauman15:06:38

my use case is taking an existing keys spec and constraining it by moving some keys into :req

richhickey15:06:54

I understand, and that will be common

richhickey15:06:09

also various intersections

robert-stuttaford15:06:26

just so i'm sure i understand what you're both talking about, are you talking about what i'm trying to do here? https://gist.github.com/robert-stuttaford/e1bc7bfbcdf62020277dda1a277394ca

richhickey15:06:26

so no flow is the right thing

robert-stuttaford15:06:51

where i want to gen (s/and (s/keys ...) (s/keys ...)) automatically?

robert-stuttaford15:06:16

seems like s/merge may be what i need to use instead

richhickey15:06:27

yes, that’s precisely the point of merge

richhickey15:06:39

it gens map unions

richhickey15:06:11

subject to the (temporary) flow problem @bhauman identifies above

robert-stuttaford15:06:05

i've just seen http://dev.clojure.org/jira/browse/CLJ-1910 and http://dev.clojure.org/jira/browse/CLJ-1919. it's going to be terribly tempting to rewrite everything!

robert-stuttaford15:06:38

i guess there'll be ways to add or remove namespaces from these maps?

richhickey15:06:39

this only affects reading, not the map

richhickey15:06:38

i.e. it’s not a lingering property of the map

robert-stuttaford15:06:02

ah, so if i wanted to add namespaces to a non-nsed map, i'm going to do so to the keys in the map myself

richhickey15:06:28

this is only reading/printing

seancorfield16:06:39

https://clojurians.slack.com/archives/clojure-spec/p1467211082001261 I hope that whatever comes out of this doesn’t make it harder to use other testing libraries (like Expectations) harder.

seancorfield16:06:39

(but, yeah, great changes in Alpha 8… I updated java.jdbc to accommodate enough changes to get it running again but I haven’t dug deep into redoing the specs to leverage the new features)

jannis16:06:05

An optional piece of data inside a tuple is not supported, I assume? (s/def ::my-tuple (s/tuple (s/? keyword?) symbol?)) claims ['foo] is invalid: [foo] fails spec: :workflo.macros.command/my-tuple predicate: (= (count %) 2)

jannis16:06:48

My current "workaround" is to use (s/and vector? (s/cat :optional (s/? ...) :mandatory ...))

jannis16:06:10

Which is ok, it does mean that you have to write a custom generator for it though.

jannis16:06:55

Also, is there something like s/tuple for fixed-size non-vector sequences with a specific order of subspecs?

Alex Miller (Clojure team)16:06:05

You can use s/cat, s/? etc for things like this

Alex Miller (Clojure team)16:06:09

If you have optionality regex is definitely the right choice over tuple

Alex Miller (Clojure team)16:06:36

We are still looking at options to also enforce vector on regex in a good way

Alex Miller (Clojure team)17:06:25

s/and is sufficient as you saw but needs custom gen as you said

jannis17:06:18

@alexmiller: Ah, s/cat didn't work because I forgot s/spec around it to make it a subspec and require the content of s/cat to be in a child vector or sequence.

uwo18:06:53

my colleagues would like to create a single source of truth to generate both datomic schema and specs. Is this advisable? I was under the impression that perhaps spec would be written mostly by hand and not codegen

Alex Miller (Clojure team)18:06:31

seems low on the crazy scale to me

uwo18:06:36

haha. thanks

Alex Miller (Clojure team)18:06:50

code is data and all that :)

tyler20:06:36

Is clojure.spec.test/test meant to be used with clojure.test?

tyler20:06:50

I guess more specifically, what is the idiomatic way to set up generative tests with clojure.spec is there an equivalent of defspec from test.check for clojure.spec?

Alex Miller (Clojure team)20:06:05

fully utilizing the instrumentation and test facilities in clojure.spec.test is a different kind of process from the existing clojure.test and I don’t think we will provide a linkage like defspec.

Alex Miller (Clojure team)20:06:57

we have more to say about this and I expect we will have more guidance to provide before 1.9 is released

tyler20:06:30

Gotcha, is it expected to be a developer concern? Or will there be additional functionality added to integrate it with clojure.test? The return seems more data-centric than clojure.test.

Alex Miller (Clojure team)20:06:57

I don’t currently expect anything that integrates it with clojure.test. The returns are more similar to the results from test.check as that’s what we’re actually invoking.

Alex Miller (Clojure team)20:06:17

there are levels of question here - some around what’s provided in core, another level around runners, and another level around build tooling

tyler20:06:24

I suppose the core of my question is around whether or not clojure.spec.test/test will integrate into the existing test tooling or is it expected that new test tooling will need to be built? instrument seems to fit into the current ecosystem but it seems like clojure.spec.test/test will require new tooling.

puzzler20:06:44

Is there a difference between (s/def ::type vector?) and (s/def ::type (coll-of ::s/any :kind vector?)) ?

martinklepsch20:06:56

When a required key is missing from a map the error looks like this:

{:pred [(contains? % :to) (contains? % :subject)],
   :val {},
   :via [:forms.app/email-form],
   :in [],
   :path []}
Is there a way to get an easier to parse description that will allow me to (programatically) figure out which keys are missing from a map?

angusiguess20:06:04

You can get that data structure with s/explain-data

wilkerlucio21:06:30

@tyler: if you want to run generative tests and have report of it in clojure.test you can use the clojure.spec.test/test like this: (is (true? (-> (clojure.spec.test/test 'my.ns/some-fn) :result)))

tyler21:06:42

Thanks @wilkerlucio. I had been playing around with that a little bit but the output isn’t very informative for failures for that. I think I’m going to go the route of experimenting with developing my own runner. Just wanted to make sure I wasn’t duplicating work/effort.

martinklepsch21:06:50

@angusiguess: right, what I pasted is part of what s/explain-data returns, was just wondering if there's a better way to get a list of missing keys

wilkerlucio21:06:03

@tyler: try using something like this:

wilkerlucio21:06:39

not super pretty, but might be a good start 🙂

tyler21:06:51

Thanks I’ll give that a go

puzzler21:06:22

Is there an explanation somewhere of how the sampling of every works? I'd like to get a sense for the probability that an error is detected if I switch from coll-of to every.

Mathieu Corbin21:06:20

Hello, i play with clojure-spec and i don't understand this :

Mathieu Corbin21:06:09

Why explain returns Success ? The second condition in the "s/and" should return false with a string parameter

leongrapenthin21:06:49

@mcorbin I believe the lambda pred is always invoked with a vector, i. e. [:string "Hello Clojurians"]

bhauman21:06:13

@mcorbin: theres a discussion of this from earlier today ...

bhauman21:06:51

see my posts above

Mathieu Corbin21:06:47

@leongrapenthin: you win, thanks @bhauman i will look, thx

Alex Miller (Clojure team)21:06:52

@puzzler: if you mean “how it samples” - no there intentionally is not, as it may change in undefined ways in the future

Alex Miller (Clojure team)21:06:11

all you should count on is that it will sample no more than *coll-check-limit* elements

seancorfield22:06:23

@tyler: FYI, we use Expectations to drive test.check stuff and ended up with a pattern like this /cc @wilkerlucio

(expect (more-of {:keys [result num-tests]}
                 true result
                 100 num-tests)
        (tc/quick-check 100
                        (prop/for-all [s gen/string]
                                      (= 0 (ip->long (str/replace s #"[0-9]" "."))))))

puzzler23:06:40

Suggestion: Right now it looks like every and coll-of check for distinctness with (apply distinct? ...). Would recommend (or (set? ...) (apply distinct? ...)) to short-circuit when a set is passed in to a function that expects a collection of distinct items.