Fork me on GitHub
#clojure-spec
<
2017-02-15
>
seantempesta01:02:35

What’s the best practice for validating data submitted to a server with spec (for an API)? Should I use just check with s/valid? and use code to return an error? Or maybe use an assert and throw an exception?

Oliver George01:02:54

Your first suggestion sounds sensible to me.

Oliver George01:02:12

(In general people try to avoid relying on exceptions in pure functional code since it breaks the input->output contract. )

seantempesta02:02:26

Right. That makes sense. Thanks.

ag02:02:43

I need to have either bool or string, why this generates single value wrapped in a vector? (gen/generate (s/gen (s/alt :s string? :b boolean?)))

ag02:02:33

eh, nevermind. I forgot that there’s s/or

smogg11:02:21

Howdy. Assuming I'm writing a spec looking like this:

(s/def ::labels (s/coll-of string?))
(s/def ::values (s/coll-of integer?))
(s/def ::chart (s/keys :req-un [::labels ::values]))
how would I go about validating that both :labels and :values are of equal length in :chart?

dergutemoritz11:02:57

@smogg You could do it like this (s/def ::chart (s/and (s/keys :req-un [::labels ::values]) #(= (count (::labels %)) (count (::values %)))))

smogg11:02:37

ok, so a custom fn

cryptorat14:02:37

So today I am trying to generate a list. (s/def ::team (list? (s/+ ::team-member))) I have verified that ::team-members can be generated. What obvious thing am I missing this morning?

cryptorat14:02:00

hmm.. Looking at what smogg wrote (s/def ::team (s/coll-of ::team-member))

cryptorat14:02:06

Seems to work for generating.

smogg14:02:42

@cryptorat yeah, if you want a coll of something you'd use coll-of

cryptorat14:02:22

Does that mean there is no way to determine if it is a list vs a map?

cryptorat14:02:54

Or is a map not considered a collection? hmm…let me read up on coll-of

smogg14:02:09

if you want a map you can use s/keys to specify the keys

smogg14:02:45

if you want to specify what collection you want to get with s/coll-of you'd pass a :kind

smogg14:02:02

for example: (s/coll-of integer? :kind vector?)

cryptorat14:02:37

:kind yeah. I just found that.

cryptorat14:02:08

Fantastic. It is all coming together nicely. Thank you for the help.

lsnape16:02:01

Interesting semantic differences between s/merge and clojure core merge:

(s/def :foo/bar string?)
(s/def :baz/bar nat-int?)

(s/def ::bar-map (s/merge (s/keys :req-un [:foo/bar])
                          (s/keys :req-un [:baz/bar])))
I would expect :baz/bar to override :foo/bar, but AFAICT it ands the specs together for that key.

Yehonathan Sharvit16:02:54

s/merge behaves like s/and

Yehonathan Sharvit16:02:04

The term merge might be misleading

Yehonathan Sharvit16:02:34

merge
macro
Usage: (merge & pred-forms)
Takes map-validating specs (e.g. 'keys' specs) and
returns a spec that returns a conformed map satisfying all of the
specs.  Unlike 'and', merge can generate maps satisfying the
union of the predicates.

lsnape16:02:05

Thanks! What exactly is meant by union of the predicates here?

Yehonathan Sharvit16:02:33

It means: union of required keys, union of optional keys

Yehonathan Sharvit16:02:00

And I guess that there is some rule for handling cases where a key appears in one spec as required and in the other as optional

Yehonathan Sharvit16:02:12

Could you test it?

lsnape16:02:36

yup I reckon

Yehonathan Sharvit16:02:06

(Sorry for the long url)

joshjones16:02:54

interesting, i did not think merge behaved this way because i haven’t used it much

lsnape16:02:13

It looks like the :req-un wins

joshjones16:02:00

(s/def :foo/bar #(< 1 % 10))
(s/def :baz/bar #(< 5 % 20))

(s/def ::bar-map (s/merge (s/keys :req-un [:foo/bar])
                          (s/keys :req-un [:baz/bar])))

(s/valid? ::bar-map {:bar 2})
=> false
(s/valid? ::bar-map {:bar 11})
=> false
(s/valid? ::bar-map {:bar 7})
=> true

lsnape16:02:25

(require '[clojure.spec :as s])

(s/def :quux/flib int?)

(s/def ::bar-map (s/merge (s/keys :req-un [:quux/flib])
                          (s/keys :opt-un [:quux/flib])))

(s/valid? ::bar-map {}) ;; false

Yehonathan Sharvit16:02:49

No matter what’s the order inside the merge?

lsnape16:02:40

Order doesn’t matter. I guess this behaviour is consistent with the anding of the value specs.

lsnape16:02:18

Yeah, good to know 🙂

Alex Miller (Clojure team)16:02:58

Merge means: the value must pass each merged spec

Alex Miller (Clojure team)16:02:15

The specs are not combined or unions though

lsnape16:02:30

What I was hoping for was someway to relax an existing spec that I’ve defined. For example, a new map that has blank strings initially ready for a user to fill out.

lsnape16:02:11

The spec on the server requiring that these strings be filled out in the map.

vikeri17:02:20

Is there a spec that specifies a sorted-map?

spieden18:02:22

@vikeri not built in i don’t think, but you could just have a type checking predicate i suppose

vikeri18:02:56

@spieden Alright thanks, in the end I decided to use a vector instead, better when serializing

spieden18:02:24

yeah that may make your life easier overall

Alex Miller (Clojure team)19:02:55

@vikeri you can do something like (s/map-of int? string? :kind sorted?)

vikeri19:02:56

@alexmiller That sure looks like it, didn’t think of that you could use :kind for broader things than vector?, list? etc.

Alex Miller (Clojure team)20:02:08

it’s an arbitrary function

Alex Miller (Clojure team)20:02:30

however, it does affect gen - if you use, either it should gen or you should also use :into

Alex Miller (Clojure team)20:02:07

I’m not sure there is some way to get the sorted map constraint in there and make map-of gen (without supplying a custom gen for the overall coll)

spieden20:02:37

i seem to be hitting an infinite recursion trying to gen for a part of my spec

spieden20:02:45

it’s not the only recursive definition i have — other one works fine

viesti20:02:11

hum, trying to figure out this:

(s/exercise (s/keys :req #{:lol/id} :opt #{:lol/name}))
ExceptionInfo Couldn't satisfy such-that predicate after 100 tries.  clojure.core/ex-info (core.clj:4725)

hiredman20:02:19

somewhere in the specs for id and name or in a spec they rely on you are using s/and

hiredman20:02:50

and somewhere in that s/and is a predicate that is very picky

viesti20:02:18

where :lol/id is int? and :lol/name is

(s/and string? #(<= (.length %) column_size))

hiredman20:02:27

the way generates work for s/and is it uses the generator for the first predicate and then filters the output using such-that

hiredman20:02:59

right, so you should hook up a custom generator

viesti20:02:36

hum, s/exercise works individually for both specs but not for the key spec

viesti20:02:08

should I write a custom generator for the key spec

hiredman20:02:14

that is going in to the weeds a little bit on what happens as you compose test.check generators

hiredman20:02:49

for :lol/name

viesti20:02:29

first thought that the problem was strings limited by length, but it’s probably combination of :req and :opt, since having both key specs as :opt passes s/exercise

hiredman20:02:29

because you can generate random strings for a long time before you get one that is less than or equal to column_size

viesti20:02:48

since this seems to work:

(s/exercise (s/keys :req #{} :opt #{:lol/name :lol/id}))
([{} {}] [{} {}] [{} {}] ...)

hiredman20:02:31

this is the nature of randomness, sometimes things work that don't always work

viesti20:02:51

have to take a shower now, introducing some randomness 🙂

hiredman20:02:42

I don't recall the details of how the generator for s/keys works, but it may ignore :opt most of the time, or some large enough amount of the time for it to mostly work when you move the keys to opt

spieden20:02:33

any thoughts on infinite recursion during spec generated gen? only difference i can see is the “call depth” where the recursion happens — it’s deeper in the case that’s going into the infinite loop

tbaldridge21:02:35

@spieden you have to always give the generator a "way out"

spieden21:02:56

ah like a terminal node in the recursion?

tbaldridge21:02:09

so one of the s/or or s/keys needs to have a nil, or fully optional params, etc.

spieden21:02:11

i thought maybe there’d be a way to control depth

tbaldridge21:02:51

there is, but the generator needs something to bottom out at, if you say "go 2 levels deep", what's it going to gen at the bottom level? Has to be a empty map, or a nil, or something.

tbaldridge21:02:03

or basically a scalar value.

spieden21:02:24

ok thanks, i’ll take a look at my specs again

spieden21:02:49

seems like there ought to be, actually

spieden21:02:15

yeah there’s an s/or on the path that has both a recursive and non recursive case

tbaldridge21:02:38

any hint from the stacktrace where the problem is?

spieden21:02:46

there’s no stack trace, it just spins endlessly and eats CPU when i try to sample one from it

spieden21:02:00

here’s the code in case you feel like taking a look: https://gist.github.com/spieden/49b6df8c2dae9f8e659edac3b974bb94

spieden21:02:25

it’s ::sgr/parallel that’s the problem while ::sgr/condition is fine

spieden21:02:51

::sgr/parallel -> ::sgr/branches -> ::sgr/states -> ::sgr/state -> ::sgr/parallel is the only cycle i can find

tbaldridge21:02:12

yeah, I don't think that's the problem

tbaldridge21:02:20

if you have a loop like taht you'd get a stack overflow

tbaldridge21:02:33

the problem here is that spec creates the generators very eagerly.

tbaldridge21:02:28

and so I've seen really long times to create generators.

spieden21:02:05

hmm i’ll try giving it longer to see if it ever completes

spieden21:02:59

nope just cooks my cpus

Alex Miller (Clojure team)21:02:03

one thing worth trying is to add :gen-max 3 to all of your coll-of generators

Alex Miller (Clojure team)21:02:35

that’s frequently a problem for me in composite gens as the default is 20 and it doesn’t take much nesting for that to be big

spieden21:02:21

ok thanks, i’ll try that!

spieden21:02:44

boom, results

spieden21:02:06

yeah i guess the branching was just going out of control

Alex Miller (Clojure team)21:02:31

I have a pending enhancement to change the default to 3 :)

spieden21:02:55

sounds like a sensible default =)

qqq21:02:48

I'm wondering if it's possible to statically analyze, for each non-trminal, the smallest structure it can generate -- then, after gen hits a certain size, it tells everything to use it's smallest choice

qqq21:02:22

thus, instead of controlling depth, gen merrily does its thing until it hits say 1000 nodes, then it goes "oh -%&#$&" and it tries to finish as quickly as possible

ag22:02:17

hey guys, I forgot how do you select random multiple elements out of a vector? something like gen/elements but for a random number elements instead of just one

ag22:02:34

ehm, nevermind, I guess I could combine (gen/sample) and (gen/elements)

ag22:02:56

what a spec would look like for a map where values depend on each other? e.g. date range with keys :from and :to where :to always should be bigger that :from?

ag22:02:21

is it possible at all?

spieden22:02:50

@ag using a predicate anything is possible!

spieden22:02:33

i.e. (s/and (s/keys …) (fn my-pred [the-map] …))

ag22:02:06

oh, right… let me try that

ag22:02:03

wow… Spec truly is amazing!

spieden22:02:32

so satisfying when spec finds allthebugs =)

spieden22:02:50

from hundreds of lines of test code to three

gfredericks23:02:01

@ag don't use sample to build new generators

gfredericks23:02:51

(gen/vector (gen/elements ...)) should do what you were asking