Fork me on GitHub
#clojure-spec
<
2016-07-21
>
gfredericks01:07:37

@alexmiller here's an interesting question about spec naming conventions in namespaces that exist primarily to provide specs: https://github.com/gfredericks/schpec/pull/1/files#r71633898

gfredericks01:07:57

i.e., is there a difference between the real specs and the specs describing args to the functions to make specs?

gfredericks01:07:23

I'm imagining maybe namespacing the function arg/ret specs one level deeper, using the name of the function as the last component of the namespace

gfredericks01:07:59

which of course immediately brings up the old "aliasing a non-code namespace"

donaldball01:07:22

If s/keys allowed anonymous specs for :req-un and :opt-un, that would be the obvious choice, but absent that, using the fn name as the last component of the spec namespace is p reasonable

gfredericks01:07:18

oh hi donaldball

donaldball01:07:33

howdy, stranger

gfredericks01:07:29

donaldball: would you say the main point of these specs is combining range-checks with type-agnosticism?

donaldball01:07:31

I think of them as more describing domains in the math sense, but we’re probably saying the same thing

bhauman01:07:32

sooner or later we are going to have a record like map spec that doesn't require namespaced keys

donaldball01:07:47

I’ve written a few score specs now and am tentatively concluding that Rich’s encouragement towards namespaces keys is tremendously valuable for describing domain model maps, and not very useful, maybe even a bit abrasive, when describing function option maps or kwargs.

bhauman01:07:13

I agree completely, did you see my suggestion on the PR?

donaldball01:07:36

That’s an interesting idea, I haven’t tried it in that way yet

bhauman01:07:53

a bit verbose but it feels like it fits the problem better

donaldball01:07:07

Yeah, it’s more in the spirit

bhauman01:07:14

it allows duplicate keys but ...

donaldball01:07:34

If you had a bunch of fns that shared many of the same options, maybe a spec registry would be called for

bhauman01:07:57

absolutely, in fact it would be shame not to

donaldball01:07:42

kwargs do allow duplicate keys anyway, so your formulation may be more correct

donaldball01:07:12

(even though it’s probably a usage bug when it happens)

bhauman01:07:45

well its straightforward enough to add a check for that

bhauman01:07:33

but a bit cumbersome ... really asking for a macro at that point

gfredericks01:07:51

schpec.bhauman

bhauman01:07:26

the notorious bad boy ns

bhauman01:07:40

oh you're using that ...

bhauman01:07:40

all the gen functions return (repeat 'figwheel)

gfredericks02:07:21

bhauman: so the whole point of your comment code is to avoid the namespacing-requirement of keys*?

gfredericks02:07:47

I bet that spec conforms a lot more awkwardly than keys* does :/

bhauman02:07:13

but I'm just trying to use the whats available to avoid registering in situations that don't merit it

gfredericks02:07:56

we could make a macro that names them in the background with gensyms or something

bhauman02:07:24

but you could also make a macro that doesn't require the registration

gfredericks02:07:48

by using keys* or by avoiding it?

bhauman02:07:43

by avoiding it

bhauman02:07:06

but I don't really see whats wrong with the regex formulation

gfredericks02:07:27

less readable, conforms weird

gfredericks02:07:29

I think that's it

gfredericks02:07:50

well more readable in one sense I guess

gfredericks02:07:59

keys* requires you to go somewhere else for the rest of the spec

bhauman02:07:15

and a macro could fix that

bhauman02:07:36

the conforming, and readability ... but this isn't a big deal at all

bhauman02:07:57

I'm just interested because this has come up a bunch for me

bhauman02:07:19

I've been writing a spec for leiningen

gfredericks02:07:31

you can change the conforming with a macro?

gfredericks02:07:46

I'm still hazy on all the details of clojure.spec

bhauman02:07:52

you mean the final conformed output?

bhauman02:07:02

with a conformer yes

gfredericks02:07:21

man I gotta learn this stuff better

bhauman02:07:56

(conformer (fn [x] (if (= x "one") 1 ::spec:invalid))

gfredericks02:07:55

so you could take the sequence of pairs that your style of regex creates and conform that further into a map?

bhauman02:07:08

yes sireee

bhauman02:07:17

you can do ANYTHING

gfredericks02:07:06

this has been another episode of Turing Completeness with Bhauman

gfredericks02:07:50

I'm interested in how useful conformers are for dealing with transforming things in and out of json and jdbc

gfredericks02:07:08

I'd like to be able to precising describe the idiosyncratic shape of some funky json at the same time as describing the pristine clojure-friendly equivalent and automatically get functions to translate between them

gfredericks02:07:19

ditto for goofy jdbc types

bhauman02:07:24

well it can be done

gfredericks02:07:37

that's all I wanted to hear

bhauman02:07:41

and lets face it ... it should be done and done now

gfredericks02:07:14

schpec.bijections

gfredericks02:07:41

s/precising/precisely/

seancorfield03:07:14

clojure.spec FTW: as we’re specifying more of our domain model and rewriting our validation logic on top of s/conform, we’re discovering some interesting edge cases that we hadn’t considered before, as well as uncovering inconsistencies between how we treat certain pieces of data, in different parts of our application. It’s very enlightening… and it’s also making us feel a lot more confident in our updated validation code!

seancorfield03:07:55

Not all of our validation is suitable for clojure.spec tho’: we have several places where the validation is contextual, based on current data from the environment, but all of the basic "shape" validation is much nicer now.

bbloom06:07:10

surely clojure.spec vNext will have context sensitive logic variables 😉

bbloom06:07:35

unrelated: I’m all for the openness of maps etc, but I’ve run in to some real bugs where typos or misplaced data gets ignored. Is there anything currently or planned for identifying “extra” data? I’ve found it incredibly useful to log unknown data in the past.

bbloom06:07:19

As an example from a Go program I worked on recently, we were parsing a Toml config file and our tester lost a few hours to a flag that was in the wrong section. So I added a call to this “Undecoded” method: https://github.com/BurntSushi/toml/blob/99064174e013895bbd9b025c31100bd1d9b590ca/decode_meta.go#L113 — logged all the undecoded keys and he never had this problem again

seancorfield06:07:17

@bbloom: I think the only solution right now is s/and with a set operation on the keys of the map?

bbloom06:07:56

@seancorfield: gotcha, thanks. was hoping not to treat it as invalid only to discover that it exists

seancorfield06:07:04

Yeah, if you're dealing with just :req keys you're OK but once you get into optional key territory it's tricky 😞

bbloom06:07:04

something like s/conform but like s/unconformed that returns paths to all the data that was ignored

bbloom06:07:35

that Go code just returns a collection of paths

mpenet06:07:55

another case like this in our context: maps as database records, an invalid map-entry could cause a query execution exception, invalid column etc (a map missing a key because of the typo is valid too, which doesn't help)

mpenet06:07:54

@seancorfield: do you have an example of this keys+and code?

mpenet06:07:22

I guess this can be done with map-of as well

mpenet06:07:49

(map-of #{:foo :bar} ...)

bfabry06:07:29

@mpenet: I think it'd be like (s/and (s/keys :req-un [::foo]) #(every? #{:foo} (keys %)))

mpenet08:07:58

" Instruments the vars named by sym-or-syms, a symbol or collection of symbols, or all instrumentable vars if sym-or-syms is not specified."

mpenet08:07:05

I guess you need to call ns-publics and then pass the coll to instrument, or just call instrument without arg

mpenet10:07:39

maybe this is something that changed recently

mpenet11:07:03

(s/explain-str (s/cat :statements (s/+ string?)
                                  :type keyword?)
                    [["q"] :logged])
--> "In: [0] val: [\"a\"] fails at: [:statements] predicate: string?\n"

mpenet11:07:12

any idea of what I am doing wrong here?

mpenet11:07:52

this is part of my attempt to debug https://github.com/mpenet/alia/blob/feature/specs/modules/alia-spec/src/qbits/alia/spec.clj#L296-L299 (the rest of the specs seem fine so far)

mpenet11:07:00

replacing (s/+ string?) by (s/coll-of string? :min-count 1 or (s/spec (s/+ string?)) works, but I am not sure why "+" failed here

mpenet11:07:39

"When regex ops are combined, they describe a single sequence. If you need to spec a nested sequential collection, you must use an explicit call to spec to start a new nested regex context." okay

gfredericks13:07:30

@bbloom: regarding discovering typos/etc in open maps, I've been pondering the idea of a pair of schemas -- the normal one, for which violations are a bug/error-condition/whatever, and a more restrictive one, for which violations are merely suspicious, and might warrant logging, etc.

gfredericks13:07:59

E.g. "this string is probably a uuid", "this map probably has no unknown keys", ...

gfredericks13:07:12

@bbloom: your conform/unconform idea suggests that you could just roundtrip the data and do a diff, but that would only work with map keys and not general mistakes I think

mpenet13:07:38

@gfredericks: something that could be nice for speck would be a schema->spec converting function/macro. When you have large'ish Schemas (ex multi-level maps) it get tedious/verbose fast. I might actually give it a try if I get some free time this week end

gfredericks13:07:49

By "schema" you mean something like plumatic/schema's api where the schema is the same shape as the data?

mpenet13:07:05

I meant plumatic/schema schemas

gfredericks13:07:10

Yeah I definitely imagined that would show up

mpenet13:07:16

since it's so wide spread atm

gfredericks13:07:24

Oh you mean their whole API?

gfredericks13:07:37

Or as much as possible at least

mpenet13:07:48

no, I mean just an utility to convert schemas from their format to spec

mpenet13:07:58

some of their api is hard to port to spec

mpenet13:07:06

ex all the coercion stuff

gfredericks13:07:15

A 1-time dev utility or a runtime thing?

mpenet13:07:34

well I guess if you do the latter the former spawns in a couple of lines

gfredericks13:07:21

The former can be less robust

mpenet13:07:13

In my case I am mostly concerned about the ton of map schemas we have in our projects, would be nice to be able to port these without spending ages on it and laying a bug field in the process

mpenet13:07:34

I could just not bother as well, and use both in parallel

gfredericks13:07:06

Just punt on certain things and point the user to where it's broken

mpenet13:07:14

the example of nested maps is a good one I think, a very concise schema from plumatic Schema can end up being a tons of lines of mostly s/def's for k/v

mpenet13:07:00

could end up being a more concise way to write map specs as well

gfredericks13:07:22

It definitely would be

Ed13:07:52

Hi ... I've just been looking at spec (alpha10) and am confused by some results I'm seeing ...

Ed13:07:37

the docs from fdef suggest that it should instrument the function referred to

Ed13:07:45

but it doesn't seem to

gfredericks13:07:52

Not by itself

gfredericks13:07:14

You have to call c.s.test/instrument

Ed13:07:47

and that will permanently replace the var with a function that checks it's args?

Ed13:07:15

the docs for fdef say:

Ed13:07:17

Once registered, function specs are included in doc, checked by instrument, tested by the runner clojure.spec.test/run-tests, and (if a macro) used to explain errors during macroexpansion.

gfredericks13:07:37

"Checked by instrument" is a reference to the function I mentioned above

gfredericks13:07:06

The wording is confusing though since that's not obvious

bhauman13:07:10

@gfredericks: I have a strict-keys macro that uses a dynamic variable to set it's level of strictness

Ed13:07:43

ok ... so I need also need to include clojure.spec.test in my production code to have specs checked at runtime?

bhauman13:07:50

:ignore, :warn, :blowup

gfredericks13:07:13

@bhauman: I'm in the middle of pushing more customization into plumatic/schema to support this kind of thing

bhauman13:07:46

I think a better solution is to have both the dynamic variable with a spec level override to force a certain level for certain specs

gfredericks13:07:51

@l0st3d: I think the intention is for you to do more explicit checks if you want production checking

gfredericks13:07:53

@bhauman: cuz e.g. defproject is always open but certain submaps are more restrictive?

Ed13:07:10

@gfredericks: fair enough ... just thought that the docs suggested it would do those checks for me if I included the fspecs ... just trying to work out the api atm i think 😉 .. thanks for your help

bhauman13:07:13

just that you would want certain maps to never allow extraneous members

bhauman13:07:28

there is actually a really interesting case for misspellings

bhauman13:07:44

and still having an open map

bhauman13:07:20

having a level of verification that does a fuzzy-selection of the set of keys

bhauman13:07:20

so the provided keys are checked with a distance to the spec'ed keys

bhauman13:07:35

This could frankly be done all the time for something like (s/keys ) where a warning will be generated

bhauman13:07:52

but being able to set a dynamic var to adjust this level of checking seems appropriate

sundbp14:07:26

i’m struggling with a recursive map spec. what i’m trying to do is something like this:

sundbp14:07:28

(s/def ::node-content ::series)
(s/def ::node-inputs ::graph-seq)
(s/def ::graph-node (s/keys :req [::node-content ::node-inputs ::label ::stream]))
(s/def ::graph-seq (s/coll-of ::graph-node))

sundbp14:07:56

i don’t quite get how i can have a key spec referring to having a required key that is related to itself..

sundbp14:07:21

I can’t find any examples of a recursive spec of a map

Alex Miller (Clojure team)14:07:36

(s/def ::label keyword?)
(s/def ::children (s/coll-of ::node))
(s/def ::node (s/keys :req [::label] :opt [::children]))
(s/conform ::node {::label :a ::children [{::label :b} {::label :c}]})
=> #:user{:label :a, :children [#:user{:label :b} #:user{:label :c}]}

Alex Miller (Clojure team)14:07:55

there’s a simple example - I can’t quite work out what you’re trying to do

sundbp15:07:53

thanks. i’ll check it out

sundbp15:07:01

first time i tried something like that i thought i got a complaint about the spec not existing.. think was because i didn’t have ::graph-seq and ::node-inputs in the right order.

sundbp15:07:07

simple mistake

sundbp15:07:16

fine if i move those 2 around

seancorfield16:07:59

@mpenet: My understanding is that, despite the docstring, you can use instrument to turn on instrumentation for an entire namespace using that call.

mpenet16:07:26

Hmm I tried but it didn't seem to do anything.

mpenet16:07:47

it returned [], which would seem to indicate it instrumented nothing

mpenet16:07:46

I am not at work anymore, but I guess I ll check the souce next chance I get

mpenet16:07:25

seems it might have changed here: https://github.com/clojure/clojure/commit/a4477453db5b195dd6d1041f1da31c75af21c939 at least that's what the docstring would suggest

mpenet16:07:26

(didnt try it)

seancorfield16:07:37

Yeah, something definitely isn’t working right now — I changed an fdef spec and the tests still pass… investigating.

seancorfield16:07:54

It definitely used to work.

mpenet16:07:24

@alexmiller: would it be a bug or is it an intentional change?

mpenet16:07:58

^ instrument no longer working with a ns argument

Alex Miller (Clojure team)16:07:39

that’s intentional - use st/enumerate-namespace to produce a list of syms in an ns

mpenet16:07:51

ok makes sense

seancorfield16:07:33

Hmm, I missed that change in the release notes @alexmiller

seancorfield16:07:53

Although now I can’t get my fdef specs to work at all

Alex Miller (Clojure team)16:07:22

for check or instrument?

Alex Miller (Clojure team)16:07:51

I have just tracked down a problem with check

seancorfield16:07:58

When I call fdef, the spec does not subsequently show up on doc...

Alex Miller (Clojure team)16:07:19

how are you calling it? expects a fully-qualified symbol

Alex Miller (Clojure team)16:07:57

actually, it resolves, so not necessarily fully-qualified

seancorfield16:07:53

I had (s/fdef drop-table-ddl …) after referring in :all and that used to work but doesn’t now. I changed it to (s/fdef clojure.java.jdbc/drop-table-ddl …) and it works now.

Alex Miller (Clojure team)16:07:22

yeah, if you use a bare symbol for def or fdef, it will treat that as <current-ns>/sym

Alex Miller (Clojure team)16:07:34

so it’s not going to pick up refer's

seancorfield16:07:02

That changed at some point. This code used to work.

seancorfield16:07:25

(not a big deal but the silent "failure" is disturbing)

Alex Miller (Clojure team)16:07:44

fdef stuff changed around 6/7 - there were some major rewrites of it in there

Alex Miller (Clojure team)16:07:16

you can use st/instrumentable-syms to verify that things are speced maybe?

sundbp16:07:23

with these changes - if you want to run your clojure.test tests with specs instrumented and checked for fdef’s, what’s the proposed setup?

Alex Miller (Clojure team)16:07:48

you’ll need to turn on instrumentation

Alex Miller (Clojure team)17:07:03

so you can do that per-test, in a fixture, etc

sundbp17:07:58

if one would like to turn on instrumentation for “everything” in a fixture - does one have to manually enumerate the NS to then do instrumentable-syms and finally instrument? i.e. there’s nothing that just turns on instrumentation for all NS in project?

Alex Miller (Clojure team)17:07:38

just (st/instrument) is supposed to do that

Alex Miller (Clojure team)17:07:07

that’s like the old instrument-all

Alex Miller (Clojure team)17:07:31

and macro fdefs are always instrumented in macroexpansion

seancorfield17:07:03

Here’s what I ended up with in java.jdbc:

(try
  (require 'clojure.java.jdbc.spec)
  (require 'clojure.spec.test)
  (let [syms ((resolve 'clojure.spec.test/enumerate-namespace) 'clojure.java.jdbc)]
    ((resolve 'clojure.spec.test/instrument) syms))
  (println "Instrumenting clojure.java.jdbc with clojure.spec")
  (catch Exception _))

seancorfield17:07:47

And I had to qualify all the symbols in clojure.java.jdbc.spec fdef calls in order to get those working again.

seancorfield17:07:55

Life on the bleeding edge … 🙂

seancorfield17:07:57

Thanks @mpenet for the heads up on that — I hadn’t noticed it was broken!

fenton17:07:40

in my app is use GPS data. Normally my data structure looks like: {:lat 33.3 :lng -129.3}. With spec things are looking like: {:pc.api/latitude 33.3 :pc.api/longitude -129.3}. I'm finding it less fun to type that everywhere. I guess I could alias my namespace to [pc.api :as a] and change latitude and longitude to lat/lng then get {:a/lat 33.3 :a/lng -129.3}. R others doing/finding something similar? I wonder if having keywords as short as lat/lng leads to being unclear? Thoughts?

seancorfield17:07:29

You can use unqualified keywords in maps if you want @fenton

mpenet17:07:49

:) @seancorfield i looked a bit at clj.jdbc to spec parts of alia, that s how i spotted this

seancorfield17:07:16

I’m cutting 0.6.2-alpha2 with those updates.

fenton17:07:38

@seancorfield: do you mean un-qualified?

fenton17:07:02

@seancorfield: I guess I was suggesting the ns qualified keywords seem a bit of a pain to type everywhere...maybe I should look into using un-qualified keywords....not sure the implications off the top of my head... I'm thinking, how do you use unqualified keywords in other files? The way I'm using specs is to put them into a third *.cljc file that is shared by my front and backends. So dont i have to use qualified keywords in that case?

seancorfield17:07:17

Yes, sorry, typo.

fenton17:07:48

I keep my specs in an external library...does that make un-qualified an issue or can i still have them defined elsewhere?

seancorfield17:07:59

Well, you can do #::a{:lat 33.3 :lng -129.3}

fenton17:07:21

@seancorfield: okay that looks a bit better....

seancorfield17:07:30

A map does not need qualified keys in order to be used with spec.

seancorfield17:07:03

(s/keys :req-un [::lat ::lng]) will conform {:lat 33.3 :lng -129.3}

fenton17:07:11

@seancorfield: oh really? how does spec use it then?

fenton17:07:37

oh...really...cool!

fenton17:07:58

i guess i missed that, i'll go check it again...

seancorfield17:07:10

Scroll down to where it shows :req-un being used.

fenton17:07:00

i'll give that a try....that'll clean up my code nicely i think.

fenton18:07:16

@seancorfield: I'm unfamiliar with the #::a{:lat 3.3 :lng 3.3} syntax. i.e. the pulling of the namespace out in front of the map. is there some documentation about that somewhere?

Alex Miller (Clojure team)19:07:58

@jjcomer I can explain what you’re seeing now btw. spec.test/check returns a lazy sequence of test results per sym. In your test, you are not realizing those results, then immediately uninstrumenting them, then later realizing (actually running the check on the uninstrumented functions). If you wrap a doall around your two calls to check, that addresses why you’re not seeing the instrumentation.

Alex Miller (Clojure team)19:07:22

while check does document this laziness, I admit it was a surprise to me.

jjcomer19:07:09

Gotcha. Thanks for the debug :)

Alex Miller (Clojure team)19:07:26

and then the other thing was not including the sym you’re replacing in the instrument list

jjcomer19:07:34

Awesome, I'll give it another go this afternoon. Thanks again

seancorfield22:07:16

If you define a spec with a custom generator (using s/with-gen), does clojure.spec filter the generated values using the spec itself, or does it trust that the generator will only ever produce conforming values?

seancorfield22:07:26

(I ask because I have a spec with a very complex validation predicate and so I have to write a custom generator and I’m not certain whether the generator I’ve written is only going to produce conforming values by itself…)

Alex Miller (Clojure team)23:07:52

It does not trust and always re-checks the custom gen

Alex Miller (Clojure team)23:07:52

It's not filtering though - it will error if the gen produces a bad value

glv23:07:04

@alexmiller: in http://clojure.org/reference/reader#_maps, an example of the #:: behavior would be helpful. I think it’s accurate as it is, but not particularly clear.

seancorfield23:07:26

Thanks @alexmiller that’s good to know. I couldn’t produce a bad value in 10,000,000 generations so I think I’ll trust it as working for now 🙂