Fork me on GitHub
#clojure-spec
<
2017-06-29
>
harrigan10:06:36

Is the following behavior for s/& expected?

(s/conform (s/* int?) []) ;; => []
(s/conform (s/& (s/* int?)) []) ;; => nil

harrigan10:06:05

AFAIKT, (s/& (s/* ...) ...) []) always returns nil no matter what predicates are passed to s/&.

gmercer12:06:18

@harrigan not sure if it is relevant but there was an earlier discussion regarding the difference between s/& and s/and

gmercer12:06:00

(s/conform (s/and (s/* int?)) []) ;; => []

harrigan12:06:42

I don’t think this is due to the difference between s/& and s/and. The actual instance that tripped me up was:

(s/conform (s/& (s/+ int?) #(= 3 (count %))) []) ;; => :clojure.spec.alpha/invalid
(s/conform (s/& (s/* int?) #(= 3 (count %))) []) ;; => nil

harrigan12:06:41

I expected s/conform to return invalid in both cases.

gmercer12:06:32

looking back, it was your discussion ..

gmercer12:06:33

(require '[clojure.spec.alpha :as s])
(s/conform (s/& (s/* (fn [i] (prn "pred of " i ) (int? i))) (fn [x] (prn "exec test of " x) (= (count x) 1))) [1])
(s/conform (s/and (s/* (fn [i] (prn "pred of " i ) (int? i))) (fn [x] (prn "exec test of " x) (= (count x) 1))) [1])
(s/conform (s/and (s/* (fn [i] (prn "pred of " i ) (int? i))) (fn [x] (prn "exec test of " x) (= (count x) 0))) [])
(s/conform (s/& (s/* (fn [i] (prn "pred of " i ) (int? i))) (fn [x] (prn "exec test of " x) (= (count x) 0))) [])

gmercer12:06:20

@harrigan curiouser and curiouser ... 😿

Alex Miller (Clojure team)13:06:06

This is a bug with s/& where it won't check the extra preds if the empty coll passes

Alex Miller (Clojure team)13:06:17

There is a ticket with a patch waiting to go in

gmercer13:06:55

thanks Alex - any idea about the double 'exec test' on the

(s/conform (s/& (s/* (fn [i] (prn "pred of " i ) (int? i))) (fn [x] (prn "exec test of " x) (= (count x) 1))) [1])

Alex Miller (Clojure team)13:06:45

Can happen - there is no guarantee on how many times a predicate may be called

Alex Miller (Clojure team)13:06:13

I'd have to spend more time to explain why here

Alex Miller (Clojure team)13:06:26

To understand why that is

gmercer13:06:28

nw - but interesting!

stathissideris15:06:09

has anyone noticed multi-spec being slow to generate values? it also sometimes fails randomly

stathissideris15:06:01

I’m doing a gen/sample with a size of 1, and I get ExceptionInfo Couldn't satisfy such-that predicate after 100 tries.

stathissideris15:06:00

the spec is a multi-spec that contains maps with strings, nothing fancy

stathissideris15:06:12

one of the keys is a multi-spec itself

gfredericks15:06:23

the such-that exceptions happen when you have an s/and somewhere and the generator for the first clause isn't likely to satisfy the remaining clauses

gfredericks15:06:21

in the worst case the problem is solved by writing a custom generator for the (s/and ...)

gfredericks15:06:42

(the slow generation is probably a different symptom of the same problem)

stathissideris15:06:10

the only suspect thing in there was this:

(def email-re #"[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,63}")
(s/def ::email (-> (s/and string? #(re-matches email-re %))
                   (s/with-gen #(gen'/string-from-regex email-re))))

stathissideris15:06:26

but I removed it and still got the same behaviour

gfredericks15:06:34

well that looks like it already has the custom generator

gfredericks15:06:07

is it possible you're using some higher-level spec that expands to an s/and?

stathissideris15:06:47

hm, I think s/merge might

gfredericks15:06:33

:thinking_face: that sounds strange

stathissideris15:06:45

I think I know what it is: I’ve been using this macro to disallow “unknown” keys (I know it’s discouraged):

(defmacro only-keys
  [& {:keys [req req-un opt opt-un] :as args}]
  `(s/merge (s/keys ~@(apply concat (vec args)))

            (s/map-of ~(set (concat req
                                    (map (comp keyword name) req-un)
                                    opt
                                    (map (comp keyword name) opt-un)))
                      any?)))

stathissideris15:06:06

I don’t think it plays well with generation

stathissideris15:06:27

removing it fixed my problem

gfredericks15:06:51

I would've expected s/and to be used there instead of s/merge 😕

gfredericks15:06:18

if that's equivalent, I bet the s/and would actually generate just fine

gfredericks15:06:25

since I don't think s/keys generates any extra keys

Ed15:06:05

I'm having a problem with macro expansion and specs ... sometimes the specs are not checked

Ed15:06:17

has anyone else seen this?

gfredericks15:06:47

specs on macros?

Ed15:06:52

i'm having trouble working out the circumstances where it works or doesn't

gfredericks15:06:12

these are specs about the compile-time forms, or the runtime data?

Ed15:06:23

i've narrowed down this test case:

Ed15:06:11

which fails if i use lein test

Ed15:06:28

but passes if I load the code with C-c C-k in cider

Ed15:06:49

@gfredericks compile-time forms

gfredericks15:06:05

"load the code in cider" means actually running the test, not just compiling it?

Ed15:06:09

no ... load the code in cider means running cider-load-buffer

Ed15:06:41

and then either running the tests with C-c C-t C-n ... or running the tests at the repl

stathissideris15:06:50

@gfredericks thanks, I’ll give it a try

Ed15:06:52

with run-tests

gfredericks15:06:11

@l0st3d okay, so if you add (is false "dangit") as a second assertion in your test, it will fail there but not with the first assertion?

gfredericks15:06:07

okay, I have no idea then

Ed15:06:28

it reports that the test was run, but that succeeded ...

Ed15:06:50

ok ... @gfredericks ... thanks for your help

Ed15:06:20

i'm stuck then

lwhorton15:06:33

spec is my new addiction. how did i ever do front-end development without you, my love?

lwhorton15:06:36

if i need to build a feature but the backend team is busy and won’t get to an api data endpoint in time, no worries: spec can auto-generate a static api response for me and I can go on my way building the html.

gfredericks16:06:49

@l0st3d if I were you I would try to reproduce the problem without cider; I assume you've tried restarting the cider process?

Ed16:06:51

yes ... the 3 files I posted above always fail to check macro expansion when run from the terminal using lein

Ed16:06:23

cider seems to fix it ... which is how i didn't notice at first 😉

gfredericks16:06:12

I think I read the whole thing backwards

gfredericks16:06:23

due to ambiguity of "fail" and "pass" in this case

gfredericks16:06:31

so it's leiningen that has the problem, not cider

gfredericks16:06:35

in that case I'd try reproducing the problem using just java -cp ... clojure.main <file>

Ed16:06:54

yeah ... started doing that ... what's the minimal classpath that you need to start a repl with 1.9.0-alpha17?

Ed16:06:45

i've clearly got something missing

gfredericks16:06:10

you could try running lein classpath and using that

gfredericks16:06:32

which reminds me that if you have deps in your ~/.lein/profiles.clj then it can be helpful to turn that stuff off via LEIN_NO_USER_PROFILES=1

Ed16:06:56

fair point

Ed16:06:14

there's nothing in my lein profile

Ed16:06:03

ah ....

Ed16:06:08

it's the ~'s

Ed16:06:21

zsh is expanding the first ~ but not the rest

Ed16:06:50

so it's not finding the jar on the class path ... ~/.m2 is not a dir

gfredericks16:06:38

computers are terrible

Ed16:06:40

yeah ... and the minimal case there works fine

rickmoynihan16:06:23

are there any plans to attach doc strings to s/defs?

mpenet17:06:19

In theory you could hack your own doc registry, but it d be nicer to have this built-in

mpenet17:06:44

Proper metadata support would be nice, not just docs imho

rickmoynihan17:06:35

one other small issue I have is that if I have two namespaces foo.bar.baz and foo.bar.baz.impl and I want to add a spec alias for foo.bar.baz so I can do ::baz/blah inside foo.bar.baz.impl there doesn’t appear to be a way to do it when foo.bar.baz requires …impl. Basically I’d like a version of alias that doesn’t require the namespace to be loaded.

rickmoynihan17:06:02

not sure if there’s a solution to that one yet… other than fully qualify the keywords

rickmoynihan17:06:51

I guess that would also allow you to alias keywords to namespaces which don’t exist — which would also be useful as keywords don’t have to map to namespaces

Alex Miller (Clojure team)17:06:49

@rickmoynihan @mpenet yes, Rich, Stu, and I all want this, just haven’t gotten to it yet (and I think it’s trickier than it appears). would be happy to see someone dig in and suggest alternatives to move it forward

rickmoynihan17:06:24

@alexmiller: doc strings or aliasing, or both?

Alex Miller (Clojure team)17:06:32

that was about doc strings

rickmoynihan17:06:13

and other metadata?

Alex Miller (Clojure team)17:06:16

on aliasing, Rich has some thoughts on an approach to making keyword aliasing better, but I have not gotten info on it yet

mpenet17:06:21

What's so tricky?

Alex Miller (Clojure team)17:06:37

@rickmoynihan maybe. needs someone to dig in to see.

Alex Miller (Clojure team)17:06:14

a spec can be a keyword reference to another spec - in that case, there is nowhere to hang meta

mpenet17:06:10

Could be a simple metadata registry, atom with a map from kw -> metadata no?

Alex Miller (Clojure team)17:06:54

I don’t think it’s useful to do so here in this chat, but someone needs to evaluate what the options are. if you want to write something up in the ticket or elsewhere, please do.

mpenet17:06:35

Sure, I can do that tomorrow morning (cooking dinner atm :))

Alex Miller (Clojure team)17:06:46

off the top of my head some options include: explicit support in the spec protocol, hanging meta off the values in the spec registry, wrapping the values in the spec registry, creating a separate registry for docs or meta

Alex Miller (Clojure team)17:06:04

these all have various pros and cons

Alex Miller (Clojure team)17:06:56

they need to be checked against all the things that can be specs - sets, pred functions, anon functions, keyword refs to other specs, things that implement the Spec protocol, etc

Alex Miller (Clojure team)17:06:28

if other kinds of meta, what are some examples? would some meta be set automatically by spec or is it all custom per use?

Alex Miller (Clojure team)17:06:56

all of this stuff needs to be answered

rickmoynihan18:06:34

yeah it seems like quite a big job - but spec has a pretty big surface area so not surprising

lwhorton19:06:08

i’ve looked around at s/conformer and browsed a bit for conforming data with spec… but it seems like a not-great idea to use spec as a sort of “api parser”. is this correct?

lwhorton19:06:56

i.e. if a service provides me some json, i can’t really write a spec on the client saying “this is my shape”, then somehow use spec to convert json -> my shape, can I?

seancorfield19:06:04

The general advice is don’t coerce/transform data via s/conform unless you intend all potential clients of your spec to need that conversion!

seancorfield19:06:19

At World Singles, we do some coercion with our API specs, but we already have JSON converted to Clojure data structures as input (using JSON middleware for Ring), and so we only coerce strings to keywords and a few other “minor” things in our specs.

seancorfield19:06:03

So I’d recommend keeping the JSON -> Clojure data step separate from the Clojure data -> conformed input step.

lwhorton19:06:33

okay, so considering that advice I guess i will write custom (parse-*-entity) which converts edn (xformed in a handler) from the wrong shape to edn of the client shape.

lwhorton19:06:29

would be really neat if there’s a tool similar to spec that handles this, though i’m not really sure what it would look like. the real answer is have a properly shaped api response, but beggars cant be choosers

wilkerlucio21:06:09

lwhorton: I made a tutorial where I provide one solution to have coercion on top of specs (without affecting the conformance), check the section "Coercing results with spec" at https://medium.com/@wilkerlucio/implementing-custom-om-next-parsers-f20ca6db1664

lwhorton23:06:34

thanks for the link. this is a good read regardless of coercing because i don’t know om very well

wilkerlucio21:06:08

@lwhorton besides the simple version of coercions on the article, I have code for a more sophisticated one (but no explanations) at: https://gist.github.com/wilkerlucio/08fb5f858a12b86f63c76cf453cd90c0