Fork me on GitHub

Hi, I have a map for which there are several possible keys the values of which I'd like to use the same spec, for. So ideally I'd do something like

(s/def :key/spec (some spec...))
(s/def :key/spec2  :key/spec)

(s/def :map/spec (s/keys :opt-un [:key/spec :key/spec2])
But I don't seem to be able to do this (s/def :key/spec2 :key/spec). Ideas?


I think I've done this same thing before and it worked


hmm, then maybe it's something else. I'll double check, thanks

Alex Miller (Clojure team)15:09:18

should work. you say “don’t seem to be able to do this” - in what way?


I'm real confused. I'm trying to spec a function, and this is the error I'm seeing in my tests now:

-- Spec failed --------------------

Function arguments


should satisfy

  (cljs.spec.alpha/* any?)


I think it's complaining that :asur isn't a sequence


this is my fdef: (spec/fdef get-of :args (spec/* any?))


is get-of variadic?


(s/conform (s/* any?) :asur)
=> :clojure.spec.alpha/invalid


I tried this in a CLJ REPL with a slightly stricter spec:

user=> (spec/fdef get-of
  #_=>   :args (spec/cat
  #_=>          :key keyword?
  #_=>          :path (spec/* keyword?)))
user=> (defn get-of [key & things] key)
user=> (require '[clojure.spec.test.alpha :as stest])
user=> (stest/instrument)
user=> (get-of :asur)
user=> (get-of :asur :bsh)


I think the surrounding s/cat is important there


well that exact spec fails the same way in my tests


-- Spec failed --------------------

Function arguments


should satisfy

   (cljs.spec.alpha/* any?))


hmm, not sure. this works for me:

(defn foo [& xs] (prn xs))
(s/fdef foo :args (s/* any?))
(st/instrument `foo)
(foo 1 2 3)


right. hence my WTF


in times like these, I restart REPL/computer/get back in bed


I have restarted my REPL and everything. bed is tempting


I wonder if there's something weird going on with CLJS instrumentation


the test output/stacktrace doesn't give the line # or file or anything. but I'm confused WHY the arguments wouldn't satisfy that spec??

Alex Miller (Clojure team)17:09:58

sorry, I was unaware of that one


I'm writing up my thoughts on clojure.spec, having tried to use it for several months now in an application. I'm banging my head against multiple issues. So far it seems like most of them are related to data conversion: if I removed all conformers and :into from my specs and used separate explicit functions for data adjustments, I think my life with spec would be easier. But most writeups and presentations about spec tantalizingly mention conforming data as a major advantage...


That is something that I think I missed (I don't think I'm the only one).

Alex Miller (Clojure team)17:09:01

conforming was always intended to be an advanced tool to use in writing spec implementations, not for data transformation

Alex Miller (Clojure team)17:09:12

it is intentionally absent from the spec guide


And yet there is :into (in the guide, too).

Alex Miller (Clojure team)17:09:19

yes, and is useful for some use cases (+ for gen), but that doesn’t make it a generic transformation tool

Alex Miller (Clojure team)17:09:33 is imo a pretty good approach to leveraging spec’s for the purpose of coercion


Hmm. I think there is confusion around the issue (see for example I think it's worth mentioning explicitly in the guide. I also found that there is not a lot of guides for spec. The "Spec Guide" is great, but I see it as more of a walktrough. I've been looking on tips on how to use spec in apps (e.g. do I use s/* or s/coll-of?). Most online presentations or tutorials are introductory material only.


@alexmiller I have a file with my notes on spec (basically documenting the holes I've been falling into). I could E-mail it to you, would you like it? It would let you know what my (erroneous) thinking was.

Alex Miller (Clojure team)17:09:02

the answers on that stackoverflow thread seem pretty straightforward and correct to me :)

Alex Miller (Clojure team)17:09:39

you’re welcome to email, but I am just about to enter the Strange Loop black hole and won’t be looking at anything for Clojure for the next week and a half


That's fine. I don't want you to answer that E-mail anyway. I'm not looking for support. I just thought that if I was the author of the spec guide, I'd want to know what some misconceptions are in readers' minds.

Alex Miller (Clojure team)17:09:06

happy to hear those - in fact, an issue on would be just as good

Alex Miller (Clojure team)17:09:30

at the moment, I’m not looking to invest a lot of time in additional spec docs because we are starting to work on some spec changes that are likely to change some of whatever advice we would give


An issue it will be, then. Perhaps the holes I'm falling into will influence some of the thinking behind spec changes 🙂

Alex Miller (Clojure team)17:09:18

I do have a full day of spec materials I’ve taught several times as a course now

Alex Miller (Clojure team)17:09:49

given time, could be turned into some useful advice


@jrychter FWIW, we do use spec for some very limited coercion but, as noted in one of those SO answers, we also have two types of spec: "API (coercing) specs" and "domain specs" (non-coercing). All of our internal specs are non-coercing. And even in the API specs, we only do very limited coercion: we accept strings-that-can-be-converted-to-<T> and produce values of type T, for longs, doubles, Booleans, and date/time values. That's it.


is it possible to create a generator from a specced function (i.e. one with a fdef)?


We've found that to be extremely convenient because if you have a form field that should be a long, it's going to come in as a string so you either have a spec that attempts coercion to long but still conforms to the string and then you need an actual coercion as well, or you have a layer of coercion first followed by specs for whatever successfully coerces -- and then you have two layers in your error handling which makes for more complicated code.


@seancorfield Well, my coercions were very limited, too. Strings to keywords and collection types, basically (to keep a collection as a set or a sorted set). But even then I'm falling into traps. An example is that I though that s/valid? tells you if your data is valid according to the spec. It doesn't. It tells you if the data will be valid if you pass it through s/conform.


Right, well the pattern we use is

(let [params (s/conform ::api-spec input-params)]
  (if (s/invalid? params)
    (respond-with-error (s/explain-data ::api-spec input-params))
    (happy-path params)))


I'd argue that parsing strings to longs is not really coercion, it's parsing (my form code does it).


@seancorfield Yes, I discovered this the hard way. Problem is that if you spec your functions, then data that is not valid (but conformable) will reach your function code.


We don't spec many functions -- we mostly spec data structures.


And, like I say, these are specs at the outer boundary of our system. Inside our system we only have non-coercing specs.


I will try to remove all conformers (`s/conformer` and :into) from my specs and use explicit coercion. I'd love to be able to "attach" information about coercions to the specs, otherwise I have to manually define coercion functions and deal with nested coercions.


And our use of spec is almost all explicit and part of our production code. So we don't rely on instrumentation of functions that way.


@seancorfield That's how I see it, too, but I'm still considering instrumenting functions, and looking for possible uses of spec.


So, at a first glance, that spec-coerce library tries to do too much. I can see parsing from strings to doubles or integers. I'd rather have a way to attach custom coercion fns to specs and a way to walk a spec and apply all coercion fns that I specified.


Instrument (and check) are purely part of testing for us -- as well as using exercise to produce random conforming data for use in some example-based tests.


I think I will also remove all uses of s/coll-of ... :into and convert those into explicit (s/and (s/coll-of ::something) sorted? set?) or similar. That will let me use s/valid? to catch instances of data where something isn't a sorted set.


As an FYI, from our Clojure codebase:

Clojure build/config 48 files 2538 total loc
Clojure source 268 files 63902 total loc,
    3331 fns, 658 of which are private,
    403 vars, 42 macros, 71 atoms,
    478 specs, 19 function specs.
Clojure tests 149 files 20002 total loc,
    23 specs, 1 function specs.


I have ~45k lines of code in my app right now, so not nearly there, but close.

Alex Miller (Clojure team)17:09:32

I think it’s useful to think of the purpose of conform as “why does this value conform to the spec?” and not “transform this value into some target value”


I do think that s/valid? is not a good name, though.


Because it doesn't tell me if my data is valid.

Alex Miller (Clojure team)17:09:53

sorry, don’t understand


64k of lines of clojure is a lot of clojure 🙂


(s/def ::stuff (s/coll-of int? :into #{}))
(s/valid? ::stuff [3 2 1])

(s/def ::other-stuff (s/and (s/conformer keyword) keyword?))
(s/valid? ::other-stuff "not-a-keyword-at-all")


Both s/valid? will return true, even though the data, as supplied to s/valid?, will cause problems in functions that expect a set and a keyword, respectively.


For the first spec, would :kind set? be more accurate?

☝️ 8

Yes, I should remove all conformers and :into from my specs. My understanding right now is: s/valid? tells you if the data will be valid when conformed, so do not use s/valid? to check if the data is valid if you use conformers or :into anywhere in your specs.


Yes, the examples seems like concrete cases where using s/conformer for coercion creates headaches


Hence my point about s/valid? not telling me if my data is valid.


validity is only testable after conforming


conforming isn't something separate that exists outside of an abstract expression of the desired result shape


it's a predicate that also transforms


Correct, if you use conformer to do coercion (which is not recommended), the results of valid? are confusing. Also, some details of explain-data get confusing because the paths to data can change. This is one reason why coercion is not recommended, AIUI


conforming is super-handy though


but using it means you are indirectly specing the "input"


the input must be something-that-i-can-make-into-something-else


maybe think of conform as a normalization step


rather than coersion or parsing


that said I have written parsers using conform

Alex Miller (Clojure team)18:09:42

conform is not normalization or coercion or parsing


what's left?

Alex Miller (Clojure team)18:09:32

if the value matches the spec, it will destructure the input value to tell you which alternative was chosen in specs with alternates and which parts of the data matched which components of the spec


That's how I understand it now, too.


ok, I'm talking about conformers

Alex Miller (Clojure team)18:09:10

tags exist in s/or, s/alt, s/cat so that parts and choices can be labeled in the conformed (destructured values)


nevermind we are talking past each other

Alex Miller (Clojure team)18:09:40

conformers exist to build complex composite spec implementations from existing pieces


what would be an example of that? s/keys* does not appear to use conformers

Alex Miller (Clojure team)18:09:10

oh, I might have called out the wrong example (s/keys* uses s/&). might be remembering s/nilable (which actually has been rewritten since for performance)


ah but the & predicate is a conformer


it appears to be the only use in spec


so I think you are saying that the intended use of conformers is to act as glue to chain specs together


it's not meant to make s/conform visible alternations

Alex Miller (Clojure team)18:09:49

like building s/keys* from s/keys

Alex Miller (Clojure team)18:09:32

conform is not transformation, conformers are not coercion


So, I'm looking for suggestions: what is the simplest way to add my own coercion functions to certain specs and then walk a spec calling my functions? Metosin's spec-tools do this by wrapping specs, which I really do not like. Is that the only way?

Alex Miller (Clojure team)18:09:57

^^ is vastly preferable to spec-tools imo


@alexmiller At a first glance, this seems to try to do too much. I don't want to parse strings to integers by default. I only want to walk the spec and call specific coercion functions that I provided. But perhaps I'm missing something, I'll look deeper.


@jrychter spec really pushes you in the direction of using different keys


i.e. if you care about both the "raw" data and the "cleaned-up" (slightly-parsed, whatever) data as independent specs, you need separate keys


It seems that with spec-coerce you have to use coerce-structure to walk a complex (nested) spec, duplicating your data structure in the call.


Basically, I'm looking for something that would let me do this:

(s/def ::kw (s/and keyword? (s/coerce-fn keyword)))
(s/def ::nested (s/keys :req-un [::kw]))
(s/def ::data (s/keys ::req-un [::nested]))

(coerce ::data {:nested {:kw "x"}}) => {:nested {:kw :x}}
…without touching anything that doesn't have a coerce-fn defined.


yeah spec-coerce seems backward; I would want to opt-in to coercion, not have universal coersion predicates


Seems like a sensible suggestion to have the option to turn off the default coercions


(a suggestion for spec-coerce project, I mean)


I'll reiterate though that spec really seems to urge in the direction of more specs


i.e. a separate "edge" spec vs an internal spec


so in your example "::kw as a string encoding a keyword" is different from "::kw as a kw"


and then rename keys everywhere


(if using s/keys with namespaced maps)


@jrychter A challenging thing in your example above is that the coerce-fn is tied to something global (the qualified kw), but your data {:nested {:kw "x"}} is presumably figuring out that :kw means ::kw based on context?


I need a way to walk a spec, which doesn't seem to exist. I thought about defining multimethods dispatching on spec keys, but that won't work for unqualified keys.


@bbrinck I'm passing all information about the context as the first argument to coerce: the ::data spec knows everything.


right, but you’d have to walk the spec to know what spec :kw means


like you said above, that adds a requirement to walk the spec to know where you are in the spec


Yes, but I thought that is what spec does anyway (when, say, conforming).


correct, you’d need to duplicate this process. it’d be an interesting project, I’d need to dig more into the spec impl to understand how viable it’d be to duplicate


doesn't spec-tools have a spec visitor?


OTOH, if you could assume that the name unqualified keyword always used the same coercion, the problem is simpler


which doesn’t seem entirely crazy. Perhaps naively, I’d think that if you want to, say, convert :zip-code from string->int in one place, you probably want to do the same coercion elsewhere


maybe that’s oversimplifying the real world case 😉


Not necessarily, but I could stick with that for a while as a workaround. That would mean I could define a multimethod coercer and walk the structures myself. But then spec provides me very little value in the end.


Hm, well spec would still let you validate the result of coercion (and conform to use destructuring)


if you used the "option" keys in a consistent way, s/conform could perform the tagging for you (except for s/keys req-un and opt-un)


Note that if you don’t to have to call two different implementations of def for :kw, then you could just make a macro that does both e.g.


@jrychter (an example of a def macro that calls the normal def but also registers a message. In your case, you’d be registering a spec + coercer)


@jrychter whoops, I messed up the example in that gist. Please reload, I’ve updated. The example should have been: (coerce-structure {:nested {:kw "x"}}) ;; => {:nested {:kw :x}}


That is interesting. It's not as good as clojure.spec calling my coercions (clojure.spec would know exactly which function to call), but I guess it is a workaround.


@bbrinck Thank you for that gist. I am trying to implement that right now to see how it goes.


@jrychter np. Good luck! Definitely a workaround, but hopefully covers the 80% case. Let me know how it goes 🙂


@alexmiller Can we hope that this use case (explicit coercion using user-registered coercion functions) will be considered in future spec work?

Alex Miller (Clojure team)19:09:11

you can hope for anything you like :)

Alex Miller (Clojure team)19:09:22

not something we’re working on currently


Let's narrow it down a bit: at least a way to walk a spec, calling a function with the keyword and value at each point. That would still mean maintaining a separate registry, but that's fine.


@bbrinck I'm discovering more limitations as I go — for example, a certain spec might be redefined as s/nilable in some places under the same (when unqualified) keyword.


@jrychter Right, it’s probably worth making all coercing functions a little bit defensive i.e. if the pre-coercion type isn’t what you expect, just return the existing value


so, untested, but something like (if (string? x) (keyword x) x))


Keep in mind the role of coercion functions is not to validate the incoming data, but rather do a best-effort attempt to get the fields into a format that will be valid according the spec


Of course. And the suggestion about being defensive is a very good one.


And another limitation (as I'm going through my code): if a spec is redefined under a different name, coercion needs to be redefined for that new name, too. It won't carry over.


Very true, it’s per name, not per spec


You could do some magic looking at s/form for a spec and then look up that spec … but it’d get complicated 😉


Is spec the right tool for validating data from JSON and db results (if so, any docs or blogs about it?), or should I stick with Schema?

Alex Miller (Clojure team)18:09:26

spec is somewhat cumbersome right now for validating maps with unqualified keys (which tends to include cases like JSON)

Alex Miller (Clojure team)18:09:45

the next round of spec changes will have some improvements in this area

👍 4

Is there some way to refer to args spec of a function?


@roklenarcic Can you provide more context? What problem are you trying to solve?


I'm trying to generate arguments to a function that is specced


but without calling it


so I have a function and a bunch of fdefs


I mean one fdef for this particular one


and I want to generate vectors of arguments

Alex Miller (Clojure team)18:09:12

yes, if you get a function spec, it implements key lookup so you can grab :args, :ret, :fn out of it

Alex Miller (Clojure team)18:09:04

(-> a-sym s/get-spec :args)


that's what I was looking for 🙂


have you @alexmiller used either of spec-coerce or spec-tools?


No, but I took a look at both. Both do something different, not quite what I want, and both are way too complex.


I feel both are hacks (sorry @wilkerlucio), because how spec is designed.


let’s look at an example:


(require '[clojure.spec.alpha :as s])

(s/def :db/ident qualified-keyword?)
(s/def :db/valueType (s/and keyword? #{:uuid :string}))
(s/def :db/unique (s/and keyword? #{:identity :value}))
(s/def :db/cardinality (s/and keyword? #{:one :many}))
(s/def :db/doc string?)

(s/def :simple/field
    :db/ident :db/ident
    :db/valueType :db/valueType
    :db/cardinality (s/? :db/cardinality)
    :db/unique (s/? :db/unique)
    :db/doc (s/? :db/doc)))

(s/def :simple/entity
  (s/+ (s/spec :simple/field)))

(def value
  [[:product/id :uuid :one :identity "id"]
   [:product/name :string "name"]])

; true


I think spec really shines here (the regex parts).


but same over json:


(def json-value
  [["product/id" "uuid" "one" "identity" "id"]
   ["product/name" "string" "name"]])

; false


out of luck. To make a standalone transformer outside of spec, one would need to be able to parse the spec and basically rebuild the s/conform to understand what branches are being used.


spec-tools uses the s/conform and makes the spec do all the heavy lifting. but sadly, the leaf specs need to be wrapped into “conforming predicates”.


spec-coerce reads the spec forms, which works nicely for most/simple forms.


hopefully there will be a solution for this in the upcoming spec releases, I think it would be bad for the clojure / spec story not to support (or enable libs to fully support) coercion. one s/walkmethod on Specs for libs to use? protocol dispatch would be clean and much faster than parsing the forms or using dynamic binding like the current coercion libs do.


@ikitommi It seems that that is exactly what I have been asking for in that thread (a way to walk a spec).

Alex Miller (Clojure team)19:09:30

that is something we have talked about providing, but it’s a ways down the list


It would seem that this is something that people with large apps with complex data structures and JSON interop encounter all the time.


We haven't encountered that yet 🙂


(but maybe we're deliberately keeping our data structures simple enough?)


@seancorfield You seem to be using conformers for that, judging from the code you've shown?


Only right at the edge -- where I think it's acceptable.


But that proves my point. Coercion is needed.


Not "needed". Convenient.


And spec provides what we need already.


I'd argue that it doesn't. Or my expectations are too high: I expected to be able to "spec" my data structures only once, and reuse the resulting structure for coercion. I can't do that right now.


I can see how this could be outside the scope of spec, but this is why I'm asking (and @ikitommi seems to be, too) for a way to walk the spec data, so that we can extend it and reuse the definitions.


@jrychter there is CLJ-2251

Alex Miller (Clojure team)20:09:09

does it seem weird to you to have a “spec” that defines your data, but then require coercion that does not conform to that spec? (it seems weird to me)


web is weird. we expect a valid edn data but get JSON instead ;)

Alex Miller (Clojure team)20:09:49

it seems conceptually cleaner to me to separate the transformation of the crap that you get into the format you want from validating that transformed data

Alex Miller (Clojure team)20:09:21

and I get that you’ve already defined the structure and it seems convenient to use that structure to learn about target expectations while doing the transformation


@alexmiller In theory, yes. But in practice I'm dealing with data from a JSON database, nested data structures. What you are proposing means that I will be duplicating data structure definitions: once for spec, and again for coercion. Seems like a waste.

Alex Miller (Clojure team)20:09:57

and so I think requests that deal with using specs as data in that way make sense

Alex Miller (Clojure team)20:09:25

asking spec to do the work seems weird to me (but a good case for a lib on top)


@jrychter had you tried using coerce-structure from spec-coerce? because thats the indented feature for it, you pass it a structure and it will use the leaf attribute specs to do the coercion


@alexmiller Actually, I do not want to involve "target expectations" in this. All I want is to reuse the structure from spec and be able to hang my own data (coercion functions in this case) on it.

Alex Miller (Clojure team)20:09:32

well that would potentially be handled by meta support, which we’re in favor of (it’s just tricky to implement well)


@wilkerlucio Yes. I am using a variant of that now, suggested by @bbrinck. But this has a number of drawbacks.

Alex Miller (Clojure team)20:09:05

as a stop gap, it’s not hard to build a second registry, also keyed by spec name that had that info


@alexmiller agrer that it should be made on top. But we need help from spec to make this possible. E.g. the walk


@alexmiller That is exactly what I'm doing right now, and there are a number of limitations, especially with non-namespaced keys.

Alex Miller (Clojure team)20:09:52

that may become clearer soon


spec-coerce also supports a secondary registry to specify, what I would like from spec is to be easier to traverse the spec definitions, I have to do some things that I feel are not very stable to get that


but I guess specs on the specs will solve this

Alex Miller (Clojure team)20:09:08

this is all helpful, in particular I’m trying to clarify the problem statement so I can talk well to Rich about it

Alex Miller (Clojure team)20:09:52

spec walking has come up in several contexts and is one potentially useful generic feature, meta is another


@alexmiller It seems to me that you think much of this is unnecessary, because invalid data points to deficiencies in transport or db. That is true in general. But please consider that even using EDN you will not get sorted sets by reading. Something needs to coerce sets into sorted sets.

Alex Miller (Clojure team)20:09:20

and better support for unnamespaced attributes

Alex Miller (Clojure team)20:09:43

I didn’t say it was unnecessary. I’ve built my share of apps and I get it.

Alex Miller (Clojure team)20:09:00

I just don’t think that spec necessarily needs to be the thing providing “coercion”


Oh, absolutely. It's just that it makes sense to re-use the resulting structure.

Alex Miller (Clojure team)20:09:17

I’m agreeing with you on that

Alex Miller (Clojure team)20:09:52

using conformers for this stuff is what Rich calls the “meat grinder” approach - I see that turning the crank gets me from A to B


Part of the challenge is that 99% of other libraries provide "coercion" of some sort (plumatic/schema and everything in Python) and there are Real Problems with that stuff. I think it has psychologically primed us to want to tangle it together... some caution is advisable and focus on the problems

Alex Miller (Clojure team)20:09:18

but it misses that the grinder is not the point of it


Here's a practical example from my screen right now:

  (:require [clojure.spec.alpha :as s]
            [partsbox.coerce :as c]))

;; A build quantity is an integer that is always greater than 0.
(s/def ::quantity pos-int?)

;; A non-empty sorted set of quantities used for build quantities and pricing quantities.
(s/def ::quantities (s/and (s/coll-of ::quantity :min-count 1) sorted? set?))
(c/defcoercion ::quantities (fn [data] (into (sorted-set) data)))

(def +default-quantities+ (s/conform ::quantities (c/coerce ::quantities [1 10 25 50 100 250 500 1000])))
This is all fine (a separate registry, another line of code for defining the coercion). I would just want to be able to write a general c/coerce that would walk the spec correctly. I can only have a deficient implementation right now.