Hi, I have a map for which there are several possible keys the values of which I'd like to use the same spec, for. So ideally I'd do something like

(s/def :key/spec (some spec...))
(s/def :key/spec2  :key/spec)

(s/def :map/spec (s/keys :opt-un [:key/spec :key/spec2])

should work. you say “don’t seem to be able to do this” - in what way?

I'm real confused. I'm trying to spec a function, and this is the error I'm seeing in my tests now:

-- Spec failed --------------------

Function arguments

  :asur

should satisfy

  (cljs.spec.alpha/* any?)

the test output/stacktrace doesn't give the line # or file or anything. but I'm confused WHY the arguments wouldn't satisfy that spec??

oh for crying out loud: https://dev.clojure.org/jira/browse/CLJS-2793

I'm writing up my thoughts on clojure.spec, having tried to use it for several months now in an application. I'm banging my head against multiple issues. So far it seems like most of them are related to data conversion: if I removed all conformers and :into from my specs and used separate explicit functions for data adjustments, I think my life with spec would be easier. But most writeups and presentations about spec tantalizingly mention conforming data as a major advantage...

conforming != coercion

That is something that I think I missed (I don't think I'm the only one).

conforming was always intended to be an advanced tool to use in writing spec implementations, not for data transformation

it is intentionally absent from the spec guide

And yet there is :into (in the guide, too).

yes, and is useful for some use cases (+ for gen), but that doesn’t make it a generic transformation tool

https://github.com/wilkerlucio/spec-coerce is imo a pretty good approach to leveraging spec’s for the purpose of coercion

Hmm. I think there is confusion around the issue (see for example https://stackoverflow.com/questions/45188850/is-use-of-clojure-spec-for-coercion-idiomatic). I think it's worth mentioning explicitly in the guide. I also found that there is not a lot of guides for spec. The "Spec Guide" is great, but I see it as more of a walktrough. I've been looking on tips on how to use spec in apps (e.g. do I use s/* or s/coll-of?). Most online presentations or tutorials are introductory material only.

@alexmiller I have a file with my notes on spec (basically documenting the holes I've been falling into). I could E-mail it to you, would you like it? It would let you know what my (erroneous) thinking was.

the answers on that stackoverflow thread seem pretty straightforward and correct to me :)

Yes.

you’re welcome to email, but I am just about to enter the Strange Loop black hole and won’t be looking at anything for Clojure for the next week and a half

That's fine. I don't want you to answer that E-mail anyway. I'm not looking for support. I just thought that if I was the author of the spec guide, I'd want to know what some misconceptions are in readers' minds.

happy to hear those - in fact, an issue on https://github.com/clojure/clojure-site would be just as good

at the moment, I’m not looking to invest a lot of time in additional spec docs because we are starting to work on some spec changes that are likely to change some of whatever advice we would give

An issue it will be, then. Perhaps the holes I'm falling into will influence some of the thinking behind spec changes 🙂

I do have a full day of spec materials I’ve taught several times as a course now

given time, could be turned into some useful advice

@jrychter FWIW, we do use spec for some very limited coercion but, as noted in one of those SO answers, we also have two types of spec: "API (coercing) specs" and "domain specs" (non-coercing). All of our internal specs are non-coercing. And even in the API specs, we only do very limited coercion: we accept strings-that-can-be-converted-to-<T> and produce values of type T, for longs, doubles, Booleans, and date/time values. That's it.

is it possible to create a generator from a specced function (i.e. one with a fdef)?

We've found that to be extremely convenient because if you have a form field that should be a long, it's going to come in as a string so you either have a spec that attempts coercion to long but still conforms to the string and then you need an actual coercion as well, or you have a layer of coercion first followed by specs for whatever successfully coerces -- and then you have two layers in your error handling which makes for more complicated code.

@seancorfield Well, my coercions were very limited, too. Strings to keywords and collection types, basically (to keep a collection as a set or a sorted set). But even then I'm falling into traps. An example is that I though that s/valid? tells you if your data is valid according to the spec. It doesn't. It tells you if the data will be valid if you pass it through s/conform.

Right, well the pattern we use is

(let [params (s/conform ::api-spec input-params)]
  (if (s/invalid? params)
    (respond-with-error (s/explain-data ::api-spec input-params))
    (happy-path params)))

I'd argue that parsing strings to longs is not really coercion, it's parsing (my form code does it).

@seancorfield Yes, I discovered this the hard way. Problem is that if you spec your functions, then data that is not valid (but conformable) will reach your function code.

We don't spec many functions -- we mostly spec data structures.

And, like I say, these are specs at the outer boundary of our system. Inside our system we only have non-coercing specs.

I will try to remove all conformers (`s/conformer` and :into) from my specs and use explicit coercion. I'd love to be able to "attach" information about coercions to the specs, otherwise I have to manually define coercion functions and deal with nested coercions.

And our use of spec is almost all explicit and part of our production code. So we don't rely on instrumentation of functions that way.

@seancorfield That's how I see it, too, but I'm still considering instrumenting functions, and looking for possible uses of spec.

So, at a first glance, that spec-coerce library tries to do too much. I can see parsing from strings to doubles or integers. I'd rather have a way to attach custom coercion fns to specs and a way to walk a spec and apply all coercion fns that I specified.

Instrument (and check) are purely part of testing for us -- as well as using exercise to produce random conforming data for use in some example-based tests.

I think I will also remove all uses of s/coll-of ... :into and convert those into explicit (s/and (s/coll-of ::something) sorted? set?) or similar. That will let me use s/valid? to catch instances of data where something isn't a sorted set.

As an FYI, from our Clojure codebase:

Clojure build/config 48 files 2538 total loc
Clojure source 268 files 63902 total loc,
    3331 fns, 658 of which are private,
    403 vars, 42 macros, 71 atoms,
    478 specs, 19 function specs.
Clojure tests 149 files 20002 total loc,
    23 specs, 1 function specs.

I have ~45k lines of code in my app right now, so not nearly there, but close.

I think it’s useful to think of the purpose of conform as “why does this value conform to the spec?” and not “transform this value into some target value”

I do think that s/valid? is not a good name, though.

because…

Because it doesn't tell me if my data is valid.

sorry, don’t understand

64k of lines of clojure is a lot of clojure 🙂

(s/def ::stuff (s/coll-of int? :into #{}))
(s/valid? ::stuff [3 2 1])

(s/def ::other-stuff (s/and (s/conformer keyword) keyword?))
(s/valid? ::other-stuff "not-a-keyword-at-all")

Both s/valid? will return true, even though the data, as supplied to s/valid?, will cause problems in functions that expect a set and a keyword, respectively.

For the first spec, would :kind set? be more accurate?

Yes, I should remove all conformers and :into from my specs. My understanding right now is: s/valid? tells you if the data will be valid when conformed, so do not use s/valid? to check if the data is valid if you use conformers or :into anywhere in your specs.

Yes, the examples seems like concrete cases where using s/conformer for coercion creates headaches

Hence my point about s/valid? not telling me if my data is valid.

validity is only testable after conforming

conforming isn't something separate that exists outside of an abstract expression of the desired result shape

it's a predicate that also transforms

Correct, if you use conformer to do coercion (which is not recommended), the results of valid? are confusing. Also, some details of explain-data get confusing because the paths to data can change. This is one reason why coercion is not recommended, AIUI

conforming is super-handy though

but using it means you are indirectly specing the "input"

the input must be something-that-i-can-make-into-something-else

maybe think of conform as a normalization step

rather than coersion or parsing

that said I have written parsers using conform

conform is not normalization or coercion or parsing

what's left?

destructuring

?

if the value matches the spec, it will destructure the input value to tell you which alternative was chosen in specs with alternates and which parts of the data matched which components of the spec

That's how I understand it now, too.

ok, I'm talking about conformers

tags exist in s/or, s/alt, s/cat so that parts and choices can be labeled in the conformed (destructured values)

nevermind we are talking past each other

conformers exist to build complex composite spec implementations from existing pieces

like building s/keys* from s/keys

conform is not transformation, conformers are not coercion

So, I'm looking for suggestions: what is the simplest way to add my own coercion functions to certain specs and then walk a spec calling my functions? Metosin's spec-tools do this by wrapping specs, which I really do not like. Is that the only way?

https://github.com/wilkerlucio/spec-coerce

^^ is vastly preferable to spec-tools imo

@alexmiller At a first glance, this seems to try to do too much. I don't want to parse strings to integers by default. I only want to walk the spec and call specific coercion functions that I provided. But perhaps I'm missing something, I'll look deeper.

@jrychter spec really pushes you in the direction of using different keys

i.e. if you care about both the "raw" data and the "cleaned-up" (slightly-parsed, whatever) data as independent specs, you need separate keys

It seems that with spec-coerce you have to use coerce-structure to walk a complex (nested) spec, duplicating your data structure in the call.

Basically, I'm looking for something that would let me do this:

(s/def ::kw (s/and keyword? (s/coerce-fn keyword)))
(s/def ::nested (s/keys :req-un [::kw]))
(s/def ::data (s/keys ::req-un [::nested]))

(coerce ::data {:nested {:kw "x"}}) => {:nested {:kw :x}}

Is spec the right tool for validating data from JSON and db results (if so, any docs or blogs about it?), or should I stick with Schema?

spec is somewhat cumbersome right now for validating maps with unqualified keys (which tends to include cases like JSON)

the next round of spec changes will have some improvements in this area

Is there some way to refer to args spec of a function?

@roklenarcic Can you provide more context? What problem are you trying to solve?

I'm trying to generate arguments to a function that is specced

but without calling it

so I have a function and a bunch of fdefs

I mean one fdef for this particular one

and I want to generate vectors of arguments

yes, if you get a function spec, it implements key lookup so you can grab :args, :ret, :fn out of it

(-> a-sym s/get-spec :args)

cool

that's what I was looking for 🙂

have you @alexmiller used either of spec-coerce or spec-tools?

I feel both are hacks (sorry @wilkerlucio), because how spec is designed.

let’s look at an example:

(require '[clojure.spec.alpha :as s])

(s/def :db/ident qualified-keyword?)
(s/def :db/valueType (s/and keyword? #{:uuid :string}))
(s/def :db/unique (s/and keyword? #{:identity :value}))
(s/def :db/cardinality (s/and keyword? #{:one :many}))
(s/def :db/doc string?)

(s/def :simple/field
  (s/cat
    :db/ident :db/ident
    :db/valueType :db/valueType
    :db/cardinality (s/? :db/cardinality)
    :db/unique (s/? :db/unique)
    :db/doc (s/? :db/doc)))

(s/def :simple/entity
  (s/+ (s/spec :simple/field)))

(def value
  [[:product/id :uuid :one :identity "id"]
   [:product/name :string "name"]])

(s/valid?
  :simple/entity
  value)
; true

I think spec really shines here (the regex parts).

but same over json:

(def json-value
  [["product/id" "uuid" "one" "identity" "id"]
   ["product/name" "string" "name"]])

(s/valid?
  :simple/entity
  json-value)
; false

out of luck. To make a standalone transformer outside of spec, one would need to be able to parse the spec and basically rebuild the s/conform to understand what branches are being used.

spec-tools uses the s/conform and makes the spec do all the heavy lifting. but sadly, the leaf specs need to be wrapped into “conforming predicates”.

spec-coerce reads the spec forms, which works nicely for most/simple forms.

hopefully there will be a solution for this in the upcoming spec releases, I think it would be bad for the clojure / spec story not to support (or enable libs to fully support) coercion. one s/walkmethod on Specs for libs to use? protocol dispatch would be clean and much faster than parsing the forms or using dynamic binding like the current coercion libs do.

@ikitommi It seems that that is exactly what I have been asking for in that thread (a way to walk a spec).

that is something we have talked about providing, but it’s a ways down the list

It would seem that this is something that people with large apps with complex data structures and JSON interop encounter all the time.

We haven't encountered that yet 🙂

(but maybe we're deliberately keeping our data structures simple enough?)

@seancorfield You seem to be using conformers for that, judging from the code you've shown?

Only right at the edge -- where I think it's acceptable.

But that proves my point. Coercion is needed.

Not "needed". Convenient.

And spec provides what we need already.

I'd argue that it doesn't. Or my expectations are too high: I expected to be able to "spec" my data structures only once, and reuse the resulting structure for coercion. I can't do that right now.

I can see how this could be outside the scope of spec, but this is why I'm asking (and @ikitommi seems to be, too) for a way to walk the spec data, so that we can extend it and reuse the definitions.

does it seem weird to you to have a “spec” that defines your data, but then require coercion that does not conform to that spec? (it seems weird to me)

web is weird. we expect a valid edn data but get JSON instead ;)

it seems conceptually cleaner to me to separate the transformation of the crap that you get into the format you want from validating that transformed data

and I get that you’ve already defined the structure and it seems convenient to use that structure to learn about target expectations while doing the transformation

@alexmiller In theory, yes. But in practice I'm dealing with data from a JSON database, nested data structures. What you are proposing means that I will be duplicating data structure definitions: once for spec, and again for coercion. Seems like a waste.

and so I think requests that deal with using specs as data in that way make sense

asking spec to do the work seems weird to me (but a good case for a lib on top)

@jrychter had you tried using coerce-structure from spec-coerce? because thats the indented feature for it, you pass it a structure and it will use the leaf attribute specs to do the coercion

@alexmiller Actually, I do not want to involve "target expectations" in this. All I want is to reuse the structure from spec and be able to hang my own data (coercion functions in this case) on it.

well that would potentially be handled by meta support, which we’re in favor of (it’s just tricky to implement well)

@wilkerlucio Yes. I am using a variant of that now, suggested by @bbrinck. But this has a number of drawbacks.

as a stop gap, it’s not hard to build a second registry, also keyed by spec name that had that info

@alexmiller agrer that it should be made on top. But we need help from spec to make this possible. E.g. the walk

yes

@alexmiller That is exactly what I'm doing right now, and there are a number of limitations, especially with non-namespaced keys.

that may become clearer soon

spec-coerce also supports a secondary registry to specify, what I would like from spec is to be easier to traverse the spec definitions, I have to do some things that I feel are not very stable to get that

but I guess specs on the specs will solve this

maybe

this is all helpful, in particular I’m trying to clarify the problem statement so I can talk well to Rich about it

spec walking has come up in several contexts and is one potentially useful generic feature, meta is another

@alexmiller It seems to me that you think much of this is unnecessary, because invalid data points to deficiencies in transport or db. That is true in general. But please consider that even using EDN you will not get sorted sets by reading. Something needs to coerce sets into sorted sets.

and better support for unnamespaced attributes

I didn’t say it was unnecessary. I’ve built my share of apps and I get it.

I just don’t think that spec necessarily needs to be the thing providing “coercion”

Oh, absolutely. It's just that it makes sense to re-use the resulting structure.

I’m agreeing with you on that

using conformers for this stuff is what Rich calls the “meat grinder” approach - I see that turning the crank gets me from A to B

Part of the challenge is that 99% of other libraries provide "coercion" of some sort (plumatic/schema and everything in Python) and there are Real Problems with that stuff. I think it has psychologically primed us to want to tangle it together... some caution is advisable and focus on the problems

but it misses that the grinder is not the point of it

Here's a practical example from my screen right now:

(ns partsbox.build-quantity
  (:require [clojure.spec.alpha :as s]
            [partsbox.coerce :as c]))

;; A build quantity is an integer that is always greater than 0.
(s/def ::quantity pos-int?)

;; A non-empty sorted set of quantities used for build quantities and pricing quantities.
(s/def ::quantities (s/and (s/coll-of ::quantity :min-count 1) sorted? set?))
(c/defcoercion ::quantities (fn [data] (into (sorted-set) data)))

(def +default-quantities+ (s/conform ::quantities (c/coerce ::quantities [1 10 25 50 100 250 500 1000])))