Fork me on GitHub
#clojure-spec
<
2019-02-10
>
borkdude11:02:10

This works in spec1 and 2: (gen/sample (s/gen (s/every number? :kind vector?))) This works in spec1 but doesn’t work in spec2. Should it? (gen/sample (s/gen (s/every number? :kind coll?)))

Alex Miller (Clojure team)14:02:39

I think so? Not sure why it wouldn’t

borkdude16:02:08

Well it doesn’t yet. I posted it in the issue

Alex Miller (Clojure team)05:02:55

I did find and fix a bug in the gen from a :kind, but this particular example should and does fail in both versions. coll? can generate any kind of collection, including maps, which will not pass with something like:

Alex Miller (Clojure team)05:02:03

user=> (gen/sample (s/gen (s/every number? :kind coll?)))
Error printing return value (IllegalArgumentException) at clojure.lang.RT/seqFrom (RT.java:553).
Don't know how to create ISeq from: java.lang.Double

Alex Miller (Clojure team)05:02:25

that’s with spec 1, but I see same on spec 2 now. You might sometimes get lucky and get enough non-maps for this to work occasionally.

borkdude11:02:14

Apart from this issue and the or-spec issue, the speculative tests pass with spec-alpha2!

borkdude12:02:23

The specs are now backward compatible with spec1. So if spec2 and spec1 would have the same namespace names, it could just be a drop-in replacement.

borkdude12:02:16

First succesful build of the spec-alpha2 branch 😄 https://circleci.com/gh/borkdude/speculative/tree/spec-alpha2

borkdude12:02:03

It just occurred to me that libs can maintain compatibility with spec1 and spec2 by doing this: (:require [clojure.spec.alpha :as s1] [clojure.spec-alpha2 :as s2]) and then define specs for each version of spec: (s1/def :foo number?) (s2/def :foo number?)

borkdude13:02:01

@jpsoares106 FWIW yada is not moving to spec for this reason. Schema will remain to be used to API boundary validation/coercion.

5
seancorfield18:02:46

@borkdude what is "this reason" in your comment above? We use spec extensively for "API boundary validation / coercion" and it works great.

seancorfield18:02:34

That doesn't answer the question -- WHY?

borkdude18:02:49

Malcolm referred to this post on SO where it says that clojure.spec is not indented for coercion: https://stackoverflow.com/a/49056441/6264 I’m not sure what I said anymore, lost track.

seancorfield18:02:51

(And, yes, I read a lot of the follow up discussion there -- none of it seems substantive. It's all opinion)

borkdude18:02:32

This opinion comes from the clojure.core team. But if it works for you, nobody will stop you I guess 🙂

seancorfield18:02:16

I think it's misrepresenting what Alex said.

seancorfield18:02:59

And it specifically omits Alex's justification, which was that it forces all clients of the spec to use the coercion. Which in this case is EXACTLY what you want here.

borkdude18:02:30

What do you mean by “it”, the SO article?

seancorfield18:02:56

Dave's answer there yes

borkdude18:02:19

I haven’t tried using spec in this way. What I want at the API level is not having junk to come into my system which makes it all the way into the database in some jsonb field. Also I want coercion at the API level (e.g. query params are always strings and some have to be numbers, array, etc.). All I know is that yada does this for me using Schema and I haven’t tried to make this work with spec. The main author of yada doesn’t want to use spec for this anymore. That’s all I know 🙂.

borkdude18:02:53

I wish there was a clear and more elaborate article on this subject so I had more clarity on this myself.

seancorfield18:02:54

When I'm not just on my phone I'll write more about this. We were an early adopter of spec and use it very heavily in production for a lot of different things. I've probably spent more time discussing this aspect with Alex than anyone else outside of Cognitect 🙃

borkdude18:02:45

> We do this for parameters in our REST API, for long, double, Boolean, date, etc – we have two specs for each: one that is a spec for the target type in the domain model (which in these cases is just defined as the appropriate built-in predicate), and one that is a spec for the API level (which accepts either the target type or a string that can be coerced to the target type). Then we use the appropriate spec at the appropriate “level” in our application. How do you deal with “extra” data coming into your API? How do you deal with preventing junk going into your persistence layer?

lilactown18:02:17

select-keys? seems like that’s something spec won’t save you from

borkdude18:02:58

yeah, a nested select keys actually which is what Schema and other tools are good at

lilactown19:02:08

curious to know how sean handles that too. We’ve been using spec for our services, but just relying on whatever reitit’s integration with spec-tools does

ikitommi19:02:43

spec-tools allows one to drop all extra keys while doing coercion.

ikitommi19:02:55

it’s automatically on in reitit & compojure-api

seancorfield19:02:39

Regarding that quote from me: we have different specs for different layers. We have a spec for the persistence layer and we have code around the persistence layer that derives the set of keys (i.e., the set of columns) from the specs and uses that to narrow the keys to just those the database will accept. I.e., we are open for extensions, as far as the maps are concerned, until we have to close the set for storing it in a system that isn't open 🙂

seancorfield19:02:03

At the API boundary, we have specs that are specific to the REST parameters -- strings as inputs (of course) and validation that does the minimal coercion in order to validate those arguments.

borkdude19:02:19

It would be useful to have a function that worked generically on a spec that selected just the data that the spec describes, without changing the spec. Is that public?

seancorfield19:02:36

Given that a REST API requires coercion, you have a limited number of choices here. You must do some coercion somewhere. You do it before, during, or after the validation step. You can't validate without some coercion. There isn't much point in doing the coercion twice -- and you don't want different code inside validation and outside it. You have the choice of doing all that coercion upfront and then validating the result, or you can do it in place via spec as part of the validation. Anyone criticizing combining that in spec should also be criticizing Schema for the same thing, IMO.

seancorfield19:02:51

The objections to doing coercion in a spec are based on a number of things. Alex has repeatedly pointed out that if you do coercion in a spec, you are forcing that coercion on all clients of that spec -- which is a concern if you're building reusable specs and I agree that you shouldn't do coercion in a general, reusable spec.

borkdude19:02:10

To be clear, what’s the problem of forcing coercion on a user of the spec?

ikitommi19:02:49

Alex talks about using conform to do coercion.

ikitommi19:02:56

it’s always on.

seancorfield19:02:06

Right. So you only want to use it in a context where you need that coercion in order to do the validation.

ikitommi19:02:20

did a gist how spec-tools (and reitit + c-api) solves this: https://gist.github.com/ikitommi/68f662a399a90e8a70308ffcd4b3e752

ikitommi19:02:56

would like to see the solution baked into spec itself, here’s the most relevant issue: https://dev.clojure.org/jira/browse/CLJ-2251

seancorfield19:02:25

I'm with Alex that spec shouldn't be used for JSON transformation/parsing 🙂 Our REST API mostly has simple string input that gets coerced as part of the validation that those strings are valid numbers, dates, booleans -- but where we have JSON input, we use a JSON library to do that aspect of parsing/coercion.

ikitommi19:02:41

sure, the json strings get parsed with Jsonista/Cheshire and then the values are coerced. I would argue that “dropping keys that are not part of the spec” happens in the coercion part. Or you do it manually, which is IMO not a good idea.

seancorfield19:02:15

We disagree on that, but that's fine.

ikitommi19:02:57

hopefully spec will add support for coercion. the current ways of doing it (conform or form walking) are kinda hacks and need to go.

seancorfield19:02:34

I doubt it, given how much and how often Alex et al have argued against coercion in spec 🙂

borkdude19:02:49

Clearly there’s a tension between some users of spec and the authors of spec on this subject (coercion/stripping). I believe a good article/guide on http://clojure.org about this topic and the recommended way to go about it would be cool.

👍 20
ikitommi19:02:56

I guess the upcoming select could be used to strip out data not defined in it?

seancorfield19:02:35

No, it's for selecting which keys to check in a given context.

seancorfield19:02:08

So that it can decomplect the overall shape of the data spec from the conformance needed in specific situations.

5
ikitommi20:02:09

@borkdude you asked about a function just to strip out just keys, just remembered that there is a existing transformer doing only that. So, this works (but requires spec-tools):

(require '[spec-tools.core :as st])

(s/def ::zip int?)
(s/def ::country keyword?)
(s/def ::address (s/keys :req-un [::zip ::country]))
(s/def ::name string?)
(s/def ::user (s/keys :req-un [::name ::address]))

(st/coerce
  ::user
  {:name "liisa"
   :TOO "MUCH"
   :address {:zip 33800
             :INFOR "MATION"
             :country "INVALID"}}
  st/strip-extra-keys-transformer)
; {:address {:zip 33800, :country "INVALID"}, :name "liisa"}

ikitommi20:02:51

and you can compose json-transformer with strip-extra-keys-transformer so it’s applied in single pass.