Fork me on GitHub
#clojure-spec
<
2016-06-16
>
seancorfield00:06:21

(t/check-var #’foo) — it accepts a Var.

seancorfield00:06:31

Similarly (s/fn-spec #’foo) — although the result of that is less useful to print since it is an object.

seancorfield00:06:45

You could also do (t/check-fn foo (s/fn-spec #’foo))

seancorfield00:06:44

check-fn is so you can provide an (s/fspec :args … :ret … :fn …) for an arbitrary function without fdef.

leifp01:06:50

Hi, all. I hacked up specs for most fns in clojure.core and 1/3rd of the macros (WIP). I don't know much about clojure.spec (of course), though, so it wasn't the good kind of hacking,. Hold your nose if necessary: https://gist.github.com/leifp/abe50082f6baa0063f8b7840e80657af

jannis09:06:59

Shouldn't ((instrument #'my-fun) :foo) throw an exception with an explanation if my-fun has a spec defined for it with fdef and if the returned value does not conform to the spec of my-fun?

jannis09:06:15

Instead it just returns :clojure.spec/invalid.

jannis10:06:09

Oh, that's because that's what my-fun returns in this case. So it doesn't even fail, it just returns its value, even if that doesn't conform to the function spec.

patrkris10:06:39

I see clojure.spec uses "named arguments" in several places (e.g. (clojure.spec/fdef :args ... :ret ....)). Is there any convention as to when to use named arguments vs. a map? Maybe named arguments are primarily used for macros?

pithyless11:06:10

Hi, all. Has someone come across the need to have an s/keys that is more strict (does not allow extra keys)?

jrychter11:06:45

@pithyless: I have. I am guessing Rich thought it isn't a good idea, as in the long term it could hurt expandability and composability. But I still think we should be able to do it in some cases.

pithyless11:06:42

I'm sure it's been brought up internally; I wonder how likely it would be to get an :only option added to s/keys.

jrychter12:06:42

As I suspected — "spec embraces the idea of open maps". But there are cases where you don't want open maps. A good example is the current world of lein-cljsbuild, where I regularly pull my hair out because it isn't clear if a particular setting is in the right place. Extra keys do no harm, so people tend to put config options all over the place, and you end up with a terrible mess. Figwheel recently started doing some great work towards fixing this — and this kind of config checking should really be "these keys, and these keys only".

pithyless12:06:38

I've got some half-baked solution with clojure.set/difference and keys, but I wish it were simpler.

jrychter12:06:22

I am hoping someone will write a validations library similar to bouncer, based on clojure.spec.

jannis12:06:14

Things are getting a little meta over here... I'm spec'ing out a data format and a parse function that first uses clojure.spec/conform to validate and normalize the input and then passes it on to a transformation function. For that transformation function the input data format is the output of conform, so I now have a spec for what conform returns as well. 🙂

pithyless12:06:52

@ghadi: Thanks for linking to the archives. I understand why the default is what it is, my use case falls clearly in the "don't want to accidentally be leaking stuff" camp. But it's a minor thing... clojure.spec is nice 🙂

dominicm13:06:37

@jrychter: As someone working on this: spec isn't suited to the task.

jrychter13:06:25

@dominicm: oh 😞 I was hoping that one could build a validation system based on what explain-data returns…

dominicm13:06:55

That's the easy bit.

jrychter13:06:34

I currently use bouncer, but I have to work around its problems, and as I migrated from schema to spec, I was hoping I could get rid of it entirely.

dominicm13:06:37

@jrychter: When you do user registration validation, you need to check email uniqueness. So that requires 2 things: 1. Data to validate, the email 2. Sideband data, the database (connection). Spec has a funnel for 1. Not one for 2.

ghadi13:06:37

spec has nothing to do with integrity checks (#2)

dominicm13:06:38

Your only options for validating in that way, and neither of these are particularly spec-y, are dynamic vars, or the weird macro I started working on, and gave up on because it was a big ball of mud.

jrychter13:06:44

As for (2) above, I currently use global state encapsulated using mount.

dominicm13:06:06

@jrychter: You may not have as much trouble.

dominicm13:06:23

However, I land on the weavejester side of the camp. 😛

jrychter13:06:34

Meaning, some of my bouncer validations actually use database or even (gasp) network connections.

dominicm13:06:45

Yep, it's a necessity.

dominicm13:06:55

@ghadi: What kind of check does spec do?

jrychter13:06:58

EU VAT numbers are an example.

dominicm13:06:12

> weavejester side of the camp I should clarify this means that I use component.

dominicm13:06:10

@ghadi: I'm mostly interested in the terminology and reading on the subject.

seancorfield15:06:15

As I recall, the discussion about closed maps ended with consensus that using select-keys was appropriate for "not leaking additional keys", but there wasn't much consensus on validation of closed maps.

seancorfield15:06:21

I got the impression @richhickey is "considering" adding support for it due to repeated requests from the community but he doesn't consider it a good idea.

seancorfield15:06:33

I think designing systems to simply ignore extra keys is more robust, but I also accept there are cases where that is difficult (the example I gave is clojure.java.jdbc/insert! where a map is converted to a SQL insert statement without reflection and therefore extra keys become extra columns and will be rejected by the database; the alternative is to introspect the database and ... well, do what? Silently ignore the extra columns? Throw a different exception?)..

seancorfield15:06:23

And there in lies the issue: the behavior for extra keys is not knowable at that level - it would have to be an application-level decision (and the application has the means to do that reflection and call select-keys if it wants).

seancorfield15:06:26

Which is why I side with Rich that it's not clojure.spec's job to directly support restricting keys, since it shouldn't be the norm, and you can do it via custom predicates already if you really have one of those odd cases where you really do need to check.

donaldball15:06:22

Goes back to my point that if you want that level of validation, just specifying that the db arg to insert! is a Connection or a Connectable or whatever is not sufficient; it would need to specify that the database in question has such and such a schema

seancorfield15:06:06

Right, and that's a whole different thing. java.jdbc provides metadata/reflection APIs but doesn't use them internally (for performance mostly),

bhauman16:06:55

A strict key set constraint can make a lot sense depending on the situation. This depends on the api of course, but I can't imagine a higher level api, that third parties are going to rail against, not having tighter key set constraints.

bhauman16:06:11

That being said, you can compose a strict map key set and description with spec fairly handily.

bhauman16:06:55

Spec is beautiful.

wilkerlucio17:06:56

@pithyless: if you really want to get just the strict keys from a map, I would suggest using select-keys as people mentioned before, as a plus, I wrote a simple utility that can extract the keys from a keys spec, it's still a naive implementation (it doens't consider req-un or opt-un) but can be starting point if you want to go further on the idea:

wilkerlucio17:06:24

so I believe if extra keys are harmful on your case, you can use this kind of trick to remove unwanted ones, so we need to force the restriction of the keys, but just eliminate the ones you can't deal with (and only if you can't silently pass it on)

seancorfield17:06:48

@wilkerlucio: The s/def for ::name and ::email don’t do anything in that example…?

seancorfield18:06:10

Are you doing something else to check the values of the ::my-aggregate map conform to those specs, by convention (based on the key names)?

wilkerlucio18:06:51

@seancorfield: the idea here was just to show an example of how to do a select-keys while reusing the specs, you are supposed to call s/valid? yourself before doing the strict-keys on this case

seancorfield18:06:44

OK, so you’re relying on convention that the (qualified) key names are the same as the specs that apply to their respective values?

wilkerlucio18:06:24

but like I said, this is a naive implementation, if you really wanna rely on it, will need more work there

seancorfield18:06:33

FYI, instead of (->> (s/form spec) (next) (partition 2) (map vec) (into {})) you could just do (->> (s/form spec) next (apply hash-map))

wilkerlucio18:06:03

@seancorfield: thanks for that, I'll update the snippet 🙂

bfabry18:06:42

has anyone come up with a good way of speccing non-keyword maps with known keyval pairs?

zane18:06:46

Does spec just not play well with tools.namespace?

zane18:06:57

Or with the REPL?

zane18:06:45

dev=> (defn f [x] (inc x))
#'dev/f
dev=> (s/fdef f :ret pos?)
dev/f
dev=> (s/instrument #'f)
#'dev/f
dev=> (f 1)
2
dev=> (f -3)
-2

bfabry18:06:27

instrument doesn't check :ret or :fn in the latest alpha

bfabry18:06:28

only :args

zane18:06:41

Is that going to change?

zane18:06:51

Did it check :ret and :fn in previous alphas?

zane18:06:24

… Huh.

bfabry18:06:32

the reasoning is that :ret and :fn are for checking whether the function is correct, which should happen when you're using the functions in clojure.spec.test. :args are for checking the function was invoked correctly

zane18:06:15

So, we have a function that reads environment variables. That function has a :ret spec on it that validates that required environment variables are set and have valid values.

zane18:06:58

My understanding now is that instrument will not help me here and I should use s/conform?

bfabry18:06:04

yeah, s/conform or s/valid? or whatever explicitly. there's also an s/conform-ex coming iirc

zane18:06:34

Where should I look for info on s/conform-ex?

bfabry18:06:50

in a subsequent release 😆

bfabry18:06:27

I'm sure it's coming soon, they've been evolving very rapidly

zane18:06:38

Understood.

zane18:06:53

Do you know what s/conform-ex going to do?

bfabry18:06:10

conform or throw an exception on failure I assume

zane18:06:35

Ah, I see.

leifp18:06:05

As mentioned before, I've (roughly) spec'ed a good chunk of clojure.core. I made that an actual repo in case someone wants to test and/or beautify them: https://github.com/leifp/spec-play

zane18:06:40

@bfabry: When is the :ret argument to fspec ever used, then?

bfabry18:06:22

when clojure.test.* functions run generative tests

zane18:06:23

Got it. So only for generative testing.

zane19:06:05

That seems like a very weird design choice to me.

bfabry19:06:01

I'm not real sold on it either

bfabry19:06:26

but I haven't used spec on anything big enough to be confident

robert-stuttaford19:06:54

so, i'm new to test.check in general. anyone know how i might generate a set of keywords, from a known set of possible keywords?

robert-stuttaford19:06:30

e.g. i have #{:a :b :c :d :e} and i want generated subsets of same

bfabry19:06:20

(s/exercise #{:foo :bar}) => ([:bar :bar] [:bar :bar] [:bar :bar] [:bar :bar] [:foo :foo] [:foo :foo] [:bar :bar] [:foo :foo] [:bar :bar] [:foo :foo])

bfabry19:06:34

this is probably better

bfabry19:06:35

(s/exercise (s/coll-of #{:foo :bar} #{})) => ([#{} #{}] [#{} #{}] [#{} #{}] [#{:bar :foo} #{:bar :foo}] [#{} #{}] [#{:bar :foo} #{:bar :foo}] [#{:bar :foo} #{:bar :foo}] [#{:bar} #{:bar}] [#{:bar :foo} #{:bar :foo}] [#{:bar :foo} #{:bar :foo}])

robert-stuttaford19:06:19

i'm modelling Magic the Gathering cards as an exercise

bfabry19:06:31

haha, nice

robert-stuttaford19:06:40

(s/def ::type #{:land :creature :artifact :enchantment :sorcery :instant :planeswalker})
(s/def ::types (s/with-gen
                 (s/and set? (s/+ ::type))
                 #( ? )))

bfabry19:06:03

anyway yeah s/exercise generates data that suits a spec, a spec of "sets of these keys" is (s/coll-of #{:keys} #{})

robert-stuttaford19:06:38

::types works, but it can't be generated because of how s/and generators work: generate for the first and discard anything that doesn't satisfy the rest of the ands

bfabry19:06:42

I think you just want (s/coll-of ::type #{})

robert-stuttaford19:06:36

indeed, thank you

bfabry19:06:08

(s/def ::type #{:land :creature :artifact :enchantment :sorcery :instant :planeswalker})
=> :kafka-google-connector.runner/type
(map first (s/exercise (s/coll-of ::type #{})))
=>
(#{}
 #{:planeswalker}
 #{:artifact}
 #{:land}
 #{:creature :planeswalker}
 #{:sorcery}
 #{:land :planeswalker :sorcery}
 #{:instant :enchantment :land :planeswalker}
 #{:instant :enchantment :creature :land :planeswalker :sorcery}
 #{:land :sorcery})

bfabry19:06:12

man.. that's pretty neat

robert-stuttaford19:06:26

ok. my next question (which is the thing i really want to solve, now that i've softened you up 🙂 ) is how might i write a generator when i'm using s/and on two s/keys specs?

robert-stuttaford19:06:45

(s/def ::base-card (s/keys :req-un [::name ::types ::metadata]
                           :opt-un [::sub-type ::legendary? ::world?]))

(s/def ::cost string?) ;; todo

(s/def ::spell (s/and ::base-card (s/keys :req-un [::cost])))

bfabry19:06:09

I've not actually looked into generators sorry

robert-stuttaford19:06:17

ah 🙂 worth a try!

robert-stuttaford19:06:35

it's a super-interesting problem to solve

wilkerlucio19:06:39

@robert-stuttaford: just a suggestion, since you are modeling something new, maybe would be better to use the namespaced keys instead of clear ones, with namespaced keys you can for example validate a map keys even if you don't know the aggregate name for it

bhauman20:06:46

Spit-balling on strict keys just for the fun of it. I would love some feedback.

leifp20:06:21

bhauman: What are the advantages of this vs. just (s/& (s/keys ...) #(only-these-keys % [:k ...])) ? Or a macro that expands into that.

bhauman20:06:09

@leifp: the only interesting thing here is the explain data

bhauman20:06:29

where it points exactly to the key that failed

bhauman20:06:37

In: [:there] val: :there fails spec: :howdy/fine at: [:there] predicate: #{:builds :server-port :server-ip :http-server-root}

bhauman20:06:05

it will create explain data for all the keys that failed

bfabry20:06:43

@robert-stuttaford: I wrote this which works. I don't know how sane it is. my guess is "not very"

(defmacro extend-keys [spec-name & {:keys [req-un opt-un]}]
  (let [spec-m (apply hash-map (rest (s/form spec-name)))]
    `(s/keys :req-un ~(into (:req-un spec-m) req-un)
             :opt-un ~(into (:opt-un spec-m) opt-un))))
=> #'kafka-google-connector.runner/extend-keys
(s/def ::spell (extend-keys ::base-card :req-un [::cost]))
=> :kafka-google-connector.runner/spell

leifp21:06:43

bhauman: Hmm... I guess there is no explain equivalent of conformer or with-gen, so I can't really think of another way to do it than reifying Spec. Your impl. looks fine, but it doesn't seem to explain the extra keys if one of the required keys fails its spec.

bhauman21:06:48

@leifp: I haven't looked at that, must be because of the s/and

bhauman21:06:03

btw I've iterated on it a bit

bhauman21:06:10

@leifp: what do you mean by extra keys? you mean in the explain data?

leifp21:06:03

@bhauman:

user=> (s/explain (strict-keys :req [::r]) {::r "bad" ::extra 2})
In: [:user/r] val: "bad" fails spec: :user/r at: [:user/r] predicate: number?
In: [:user/extra] val: :user/extra fails at: [:user/extra] predicate: #{:user/r}  ;; <<< expected, not present

leifp21:06:32

That output line was expected and not present, I mean.

bhauman21:06:38

oh yeah it short cuts

bhauman21:06:15

s/and short cuts

bhauman21:06:29

so that makes sense

bhauman21:06:47

I would need to compose over the keys-spec to get that behavior