Fork me on GitHub
#clojure-spec
<
2017-01-06
>
uwo00:01:02

how would you recommend handling this? The map I’m validating has an attribute whose value is an entity. Depending on the context, at times that entity will have only one required attribute, and at other times it will have many required attributes. So, given two forms:

(s/def ::form1 (s/keys :req [::attr1]))
(s/def ::form2 (s/keys :req [::attr1]))

;; what we want in form 1                       
(s/def ::attr1 (s/keys :req [::attr-a]))
;; what we want in form 2
(s/def ::attr1 (s/keys :req [::attr-a ::attr-b]))
It seems to me, we’ll have to create another version of ::attr1, but that seems problematic because we use the ::attr1 key to mean the same entity across the project.

bbloom01:01:20

there are a few things you can do that depends on your situation

bbloom01:01:39

the simplest is just make the “sometimes required” fields optional and call that good enough, if it meets your validation needs

bbloom01:01:19

another thing you can do is to have decorated and undecorated versions of the data, and just put them at two differently named keys

bbloom01:01:21

so instead of (update m :foo assoc :bar 123), you do (assoc m :foo-bared (assoc (:foo m) :bar 123)

bbloom01:01:37

then use foo vs foo-bared as appropriate

bbloom01:01:06

i’ll leave more complex solutions for others to discuss 🙂

joshjones04:01:10

@uwo The part that interests me is the "depending on the context" -- you might be tempted to do something like this:

joshjones04:01:27

(s/def ::attr-a any?)
(s/def ::attr-b any?)

(s/def ::entity (s/or :multiple (s/keys :req [::attr-a ::attr-b])
                      :single (s/keys :req [::attr-a])))

(s/def ::form (s/keys :req [::entity]))

joshjones04:01:12

however, this does not do you any good, as in a context where you need multiple the single will still match ... can you be more specific on "context"?

uwo04:01:52

@bbloom thanks. I’ve got to be strict about required versus optional because I’ve got to bark at the user if they don’t provide a field in a particular context. I’ll have to mull over what the decorated versus undecorated means

uwo04:01:04

@joshjones thanks. the “context" is two separate forms, one of the forms has a subset of the fields in the other. Both forms describe the same entities. It’s just one form requires more than the other. I’m loath to change to name (or ns) of the attribute, because it’s really the same entity

bbloom04:01:30

@uwo is your form necessarily hierarchical?

uwo04:01:45

yeah, it gets very nested

bbloom04:01:42

hm, yeah, so this is something was talking about with @gfredericks i believe last week or so: parameterized specs

uwo04:01:01

I haven’t played it out fully yet, but I think the s/or may work @joshjones

uwo04:01:32

@bbloom yeah, I was wondering if there was some avenue to parameterization

joshjones04:01:55

so, what is the criteria, in plain english, for choosing map spec one, versus map spec two? It sounds like you are saying "map spec one has these keys" and "map spec two has these keys" ... but the presence or absence of keys is not really an appropriate way to spec it. If you have a key itself, something like {:data {...} :type "A"} .. then you can identify which "version" of the map you have on hand

joshjones04:01:37

said another way, you have to bark at the user if they don't provide a field, in WHAT context? how do you know?

joshjones04:01:49

the s/or won't work, for the reason i described ... the single will always match, regardless of your context

uwo04:01:09

@joshjones there are two separate forms on different pages. I was simply going to validate against a different aggregate spec on each. Do I understand your question?

bbloom04:01:17

as a total NON SUGGESTION, but just for funsies: i bet you could hack a (very bad) solution with s/conformer and s/and… idea is you assoc in a parameter and then use arbitrary code to look for it — oh man, that’s an evil hack, i kinda wanna try it.... for science....

joshjones04:01:53

@bbloom we've established you're a heretic, please cease your evil ways

uwo04:01:15

@joshjones oh oh. lol reading comprehension fail. I just looked at your example the first time around. yeah, you’re right that’s no good 😊

uwo04:01:53

for science...

joshjones04:01:59

are the keys in the maps namespace-qualified? or no?

bbloom04:01:10

if you use un-qualified keys, you can build up a library of specs for non-recursive parts and then manually recreate all the recursive bits

bbloom04:01:17

but really, there’s probably a better representation

bbloom04:01:24

especially a non-recursive one

uwo04:01:30

hehe. using qualified keys. They’re the same names we use in datomic

joshjones04:01:25

i suggest two specs then .. it's the most straightforward, and models your use case

joshjones04:01:56

are you checking the spec as a function arg, or what?

bbloom04:01:05

i’d like to know more about your use case, maybe we can propose a simplification

uwo04:01:05

Do I misunderstand spec? Can I have two specs for the same (namespaced) key?

bbloom04:01:27

no, you can’t, which is an intentional design decision

bbloom04:01:30

and a pretty good one at that 😉

uwo04:01:45

cool. that’s what I assumed

joshjones04:01:51

are you checking the spec as a function arg?

bbloom04:01:58

the challenge is that specs aren’t really context sensitive in any useful way

bbloom04:01:10

also probably on purpose, but questionable if you actually do care about context

uwo04:01:11

@joshjones sorry not sure I follow.

bbloom04:01:20

if i were you, i’d probably use spec in a context-insensitive way

joshjones04:01:20

how do you plan to use this spec?

bbloom04:01:23

ie make everything optional

bbloom04:01:26

and then check required independently

bbloom04:01:34

ie don’t use spec for required’s take

bbloom04:01:51

if your required fields are context sensitive, then the fields are optional from the perspective of spec

bbloom04:01:00

you can conform and then ALSO traverse yourself to check required

uwo04:01:00

ah. grab the underlying data structure from the form and then verify it against the spec

joshjones04:01:26

and something in the data structure identifies it as one version, or the other?

uwo04:01:34

@bbloom hmm. yeah, that’s a possibility

bbloom04:01:14

my apologies for my goofy hacky mood .... this is real advice: don’t try to force spec to do 100% of the work. use it for whatever % you can get away with, and write your own code to do the rest

bbloom04:01:59

clojure is still really good at recursive functions 😉

uwo04:01:06

@joshjones not really no. one form is a proper subset of the other. And, as forms go, validations run while they’re partially filled out, so there’s really no way to tell a difference from the specs perspective. Of course, I know what form I’m on, and so could call with an appropriate spec

joshjones04:01:24

how do you know what form you're on?

uwo04:01:36

@bbloom heh. thanks. we already have an implementation that doesn’t leverage spec very much. I was just wondering if I could improve what we have

uwo04:01:07

@joshjones I think we may be passing around a form key that further identifies it, now that I think about that 😊

joshjones04:01:17

one of two things should happen: 1) since you know at this point in your code which type of data you're dealing with, then you should have a different spec for each, and validate against that spec. 2) your map itself should contain a key which specifies which type of map this is. in this case, use a multi spec

bbloom04:01:37

you can use conform at the leaves of your own recursive validation function

joshjones04:01:55

is the form key a clojure keyword, or something else?

joshjones04:01:57

doing the spec, give me a sec

joshjones05:01:02

(s/def ::attr-a any?)
(s/def ::attr-b any?)

(s/def ::id-key keyword?)
(defmulti map-type ::id-key)

(defmethod map-type ::single-key-version [_]
  (s/keys :req [::attr-a]))
(defmethod map-type ::multi-key-version [_]
  (s/keys :req [::attr-a ::attr-b]))

(s/def ::some-map (s/multi-spec map-type ::id-key))

(s/conform ::some-map {::id-key ::single-key-version ::attr-a 42 ::attr-b "abc"}) ; valid
(s/conform ::some-map {::id-key ::single-key-version ::attr-b "abc"})             ; invalid
(s/conform ::some-map {::id-key ::multi-key-version ::attr-a 42 ::attr-b "abc"})  ; valid
(s/conform ::some-map {::id-key ::multi-key-version ::attr-a 42})                 ; invalid

uwo05:01:30

I can see the multispec approach working. So, I’d need an additional key in the model, which I’m guessing would be transitory (I wouldn’t persist it to the db)

uwo05:01:18

thanks, by the way. I really appreciate your time!

bbloom05:01:30

yeah, so multi-spec works, but you need to eliminate context for nested stuff

joshjones05:01:46

you are welcome @uwo

bbloom05:01:52

one way to do that is to walk the tree and add a key to each node in the tree with the type

bbloom05:01:00

basically pre-load all the context, so that the spec can be context-free

bbloom05:01:23

i’m working up an example of that for my own sake - almost ready

uwo05:01:16

too bad I can’t key a multispec off of a type key in a parent map. Because there are so many (nested) entities on some of the forms, I’d have to annotate each one of them with a dispatch key for the multispec.

bbloom05:01:44

yeah - that’s what i was saying about spec being context-insensitive

bbloom05:01:54

it’s also what i was saying about the hack with conformers 😉

uwo05:01:57

since it’ll be the same key though, I could, like you said, just walk the tree and toss the context around

uwo05:01:10

(yeah that hack went over my head, sorry 😄 )

bbloom05:01:23

basically using conformer to automate the annotation

bbloom05:01:27

it’s a dirty dirty hack 😛

bbloom05:01:40

conformer lets you change the value that flows through spec

bbloom05:01:17

so you could do something like (s/and (s/conformer (fn [x] (assoc x :context foo))) ::node)

bbloom05:01:22

now ::node can see :context

bbloom05:01:16

there, that seems to work

bbloom05:01:17

i quite like that

bbloom05:01:22

make-fancy basically adds the context to each node, such that the specs no longer need be context sensitive

uwo05:01:35

thanks, I’ll have to mull that over. The form I’m working with isn’t recursively structured, but that’s still useful

bbloom05:01:02

recursive/nested/whatever

bbloom05:01:13

sending data down the tree

uwo05:01:24

works with pre/post walk 😄

uwo05:01:48

well thanks again, both! I’m out for the night.

joshjones05:01:11

one last thought @uwo -- take a step back, and ensure that any added complexity is helpful rather than harmful. nite 🙂

joshjones05:01:40

as complexity is the enemy of good software, and of clojure itself 😉

bbloom05:01:41

lol yes, that

carocad15:01:48

@kenny are you sure that your defspec-test macro works? I tried it with my specs and it always succeeds which is weird because if I use s/check some tests fail 😕

lmergen15:01:09

could someone elaborate what exactly the difference between :ret and :fn are with fdef ?

lmergen15:01:31

:fn feels more like a higher level qualifier, while :ret describes a type

lmergen15:01:33

is that correct ?

lmergen15:01:05

when i look at the spec documentation, it uses a ranged-rand example, and i see this: > The :ret spec indicates the return is also an integer. Finally, the :fn spec checks that the return value is >= start and < end. sounds like the :fn spec passing implies the :ret spec passing ?

lmergen15:01:22

so then why define :ret as well ?

Alex Miller (Clojure team)15:01:26

:ret describes the return value

Alex Miller (Clojure team)15:01:56

:fn receives the conformed values of both args and return and can thus validate more complicated relationships between inputs and output

lmergen15:01:32

ahhh, i see now

Alex Miller (Clojure team)15:01:36

In that example, :ret can only say that the return value is an integer, but :fn can say how it relates to the args

lmergen15:01:54

this makes sense

lmergen15:01:00

and feels extremely powerful

lmergen15:01:04

thanks alex

carocad17:01:45

has anyone tested the defspec-test macro that is pinned on this channel? For some reason it always succeeds even if I give it garbage specs 😞

kenny18:01:32

@carocad It works. Can you give an example?

carocad19:01:34

@kenny the following should break but it doesnt:

(s/def ::lat (s/and number? #(<= -90 % 90)))
(s/def ::lon (s/and number? #(<= -180 % 180)))

(def RADIOUS 6372800); radious of the Earth in meters
(defn haversine
  [^double lon-1 ^double lat-1 ^double lon-2 ^double lat-2]
  (let [h  (+ (Math/pow (Math/sin (/ (- lat-2 lat-1) 2)) 2)
              (* (Math/pow (Math/sin (/ (- lon-2 lon-1) 2)) 2)
                 (Math/cos lat-2)
                 (Math/cos lat-1)))]
    (* RADIOUS 2 (Math/asin (Math/sqrt h)))))

(s/fdef haversine
  :args (s/cat :lon-1 ::lon :lat-1 string?
               :lon-2 ::lon :lat-2 ::lat)
  :ret ::dist)

(defspec-test test-haversine      [haversine] {:clojure.spec.test.check/opts {:num-tests 50}})

carocad19:01:02

when I run lein test it says run 4 test containing 4 assertions. 0 failures, 0 errors

carocad19:01:18

am I doing something wrong?

carocad19:01:57

I omited the namespaces to make it short

kenny19:01:24

@carocad The parameters passed to it should be the same as the parameters passed to clojure.spec.test/check -- you need to pass a fully qualified symbol.

carocad19:01:00

@kenny as I mentioned, I ommited the ns in this case only for brevity. The real test have the fully qualified symbol. see https://github.com/carocad/hypobus/blob/master/test/hypobus/basic_test.clj#L54

kenny19:01:34

I just tested your code and it fails with a ClassCastException 🙂

kenny20:01:11

@carocad You need to quote your symbols

kenny20:01:23

(defspec-test test-haversine      [`hypobus.basics.geometry/haversine] {:clojure.spec.test.check/opts {:num-tests 50}})

kenny20:01:11

Params are exactly the same params you'd pass to clojure.spec.test/check 🙂

carocad20:01:12

oh I see @kenny. Thanks a lot 🙂 I took the sample code from http://stackoverflow.com/questions/40697841/howto-include-clojure-specd-functions-in-a-test-suite, and since it was the same code I assumed that I was correct. My mistake

carocad20:01:22

I didnt know there was a gist as well. I added a comment on how to use it for those with not much macro understanding like me 🙂

schmee20:01:13

what does the retag argument do in multi-spec?

schmee20:01:17

> retag is used during generation to retag generated values with matching tags. retag can either be a keyword, at which key the dispatch-tag will be assoc'ed, or a fn of generated value and dispatch-tag that should return an appropriately retagged value.

schmee20:01:58

says the docs, but I think I need an example to understand what that means

schmee20:01:11

ahh, okay, now I see it 🙂

carocad21:01:48

@kenny if you dont mind ... could you help me a bit further. I get a weird java.lang.ClassCastException when running lein test. It seems to be a compilation error though 😞

kenny21:01:29

Try changing failure# to (throw failure#) (line 20 in the gist)

carocad21:01:10

I still get java.util.concurrent.ExecutionException: java.lang.ClassCastException: clojure.lang.AFunction$1 cannot be cast to clojure.lang.MultiFn, compiling:(clojure/test/check/clojure_test.cljc:95:1)

Alex Miller (Clojure team)21:01:33

maybe you’re running into the lein monkeytest problem

Alex Miller (Clojure team)21:01:04

just add this to your lein project.clj: :monkeypatch-clojure-test false

Alex Miller (Clojure team)21:01:28

it’s a conflict with how test.check and lein are both modifying clojure.test

carocad21:01:27

@alexmiller I just found that bug report in github. Indeed I just tried that and it worked 🙂

Alex Miller (Clojure team)21:01:49

btw, it is fixed in test.check too for next release of that lib

carocad21:01:58

@alexmiller , @kenny You saved a very frustrated man, thanks for the help and the heads up

Alex Miller (Clojure team)21:01:31

comes up here about once a month :)

schmee22:01:48

can I declare “dependencies” between keys in a map?

schmee22:01:16

for example,

(def m {:url ""
        :top-level "com"})

(= (:top-level m) (-> (str/split (:url m) #"\.") last))
can I have this in the key specs somehow?

schmee22:01:28

or is this the business of function specs?

schmee22:01:18

the reason I’m asking that I want to generate data with this property

bfabry22:01:20

@schmee I think that's exactly what s/and is for

schmee22:01:21

I don’t see how s/and helps here, since keys have to be speced separately

bfabry22:01:35

(defn top-level-matches-url? [m] (= (:top-level m) (-> (str/split (:url m) #"\.") last))) (s/def ::my-map (s/and (s/keys :req-un [::url ::top=level]) top-level-matches-url?))

bfabry22:01:46

or maybe I'm misunderstanding what you're saying

schmee22:01:13

ooooooohhhhh

schmee22:01:27

I never though about using s/and with s/keys that way 😄

schmee22:01:06

I’m gonna try that out, thanks!

schmee22:01:28

I guess it will be tricky to write generators for stuff like this

schmee22:01:50

it’s fine if you have one relation like this in the map, but if you have 10 or more...

bfabry22:01:16

oh totally as soon as you spec something like that you're going to need to do custom generators

schmee22:01:17

I guess I can write separate generators for the special relations, then generate a sample from the spec itself and merge all the special cases onto that

schmee22:01:00

that way I can still use most of the default generator and just "add on” the exceptions

bfabry22:01:11

I don't know much about it sorry. I'm holding out for beta before we switch to 1.9 and start using it in anger. there was a screencast about custom generators though ¯\(ツ)

schmee22:01:56

once you get in the mindset they’re quite fun to write actually

schmee22:01:22

it always feels like magic when you get to see it in action 😄

gfredericks22:01:47

Dependencies in a map can likely be generated by wrapping the map generator in gen/fmap

crimeminister23:01:10

I was looking for a way to serialize a clojure.spec to return from an API and stumbled across this nifty-looking library: https://github.com/uswitch/speculate

crimeminister23:01:08

Any idea if similar functionality is likely to ship with spec itself?

schmee23:01:30

crimeminister if I interpret this correctly, that seems unlikely: http://clojure.org/about/spec#_code_is_data_not_vice_versa

crimeminister23:01:03

Thanks for the link @schmee, it was instructive