@bbloom: @ghadi yes, figwheel uses that idea to great effect in config file validation

“unrecognized key :foo under :bar, you probably want it in :baz

Hello. My CLJS apps use a lot of string keys. Primarily for performance reasons, transforming JSON has a cost. I went looking for the equivalent of the :strs feature of map destructuring, perhaps something like :req-strs and :opt-strs could be added to s/keys.

For now the workaround seems to be using a conformer...

(def strs->keys 
  "Turn map with string keys into map of keyword keys.  Without this we can't use s/keys."
  (s/conformer #(reduce-kv (fn [m k v] (assoc m (keyword k) v)) {} %)))
(s/conform
  (s/and strs->keys
         (s/keys :req-un [::id ::options]))
  {"id" :id
   "options" [{:id 1} {}]})

Am I missing anything?

user=> (s/valid? (s/spec #(contains? % "foo")) {"foo" 1})
true
user=> 

it depends on what you are looking to get out of it, I guess

@hiredman: You lose thikngs like path lookups doing that. I think that's the key feature that goes missing. Spec has a bit of a balancing act to perform making sure that via/paths are added. s/keys is a bit complicated in implementation, but I would be thinking that a similar-but-different implementation should surface for string keywords.

I've thought about string keys some too w.r.t. https://github.com/gfredericks/schema-bijections

I figured it could be handled with a bijection system that doesn't produce a spec on the other side, but that doesn't help if you're avoiding transformations for performance

Hey... I've been very pleased with clojure.spec, which looks absolutely fantastic, and a nice step up from plumatic.schema - and I am fully intending to play with it properly (I've only really had time to read the announcement, listen to Rich on the cognicast, and looked at a few examples)... Anyway, one thing I'd really like to do is understand how this fits with JSON schema... Now, clojure.spec is clearly far superior to JSON schema, but it's not used by the javascript community etc... So I was wondering if anyone has thought about how one might be able to build a JSON schema from a clojure.spec. As spec is way more expressive, I'm guessing that to do this you'd have to stick to a subset of spec (and I guess you could probably use clojure.spec to specify that subset)... The basic idea would be to write a super-spec in spec; and output JSON schemas to interoperate with other tooling/libraries in other languages etc... e.g. swagger clients etc. If this is possible from within clojure you'd then be able reuse that subset of spec (which generated JSON schemas), and layer ontop the additional expressivity and specs you'd want clojure side.

rickmoynihan: the library I linked to just above has a lot to do with that problem, but it works with plumatic/schema. I've been pondering how best to translate it to clojure.spec

the idea is it uses a set of transformations to turn an internal schema (the nice one) into a different schema (the coarser json-style schema in this case), with functions generated to transform data structures between the two

gfredericks: interesting

it's still a bit half-baked, but I used it a bunch on a previous project

in particular it doesn't do a good job of distinguishing between bijections and non-bijections (coercions, projections, whatever you might call them); I'm thinking ou could do that interestingly using test.check generators

e.g., if my JSON API can accept {"unitCount":42} as well as {"unitCount":42.0}, you might specify that in the schema, but say that the first one is preferred, so that the transformation function always uses that, but could also have a generator that would take {"unitCount" 42} and generate sometimes {"unitCount" 42} and other times {"unitCount" 42.0M}

so it would give you a way to test flexible APIs in a robust way

unrelated: if I'm reading the clojure.spec source correctly, it ought to be safe to add specs to test.check (and to clojure.core) without the danger of an infinite loop during instrumentation -- does that sound right?

does spec support coercion? I though rich said it wasn't part of its job

specs can refer to themselves via the keyword

(s/def ::tree (s/or :leaf #{:leaf} :children (s/tuple ::tree ::tree)))

example value: [[:leaf [:leaf :leaf]] [[:leaf [:leaf :leaf]] [:leaf :leaf]]]

yeah but I didn't think spec could coerce a value from "foo" into :foo - or can it?

oh geez sorry

I misread "coercion" as "recursion"

lol 🙂

(and thus thought your question was unrelated to what I was previously talking about)

okay right afaik it doesn't support coercion

I'm not sure if the "external spec" idea would work or not

e.g., with json you'd want to have string keys, which isn't natural

tbh I'm not too worried about the coercion case... I'd just like to write clojure.specs (even in a limited subset of spec) and output JSON schemas from them

I think that by itself would probably be easier, if you only support a subset of schemas

well I'm sure that would be possible too

it doesn't need to be perfect - I'd just like to define a service in clojure, and emit a JSON schema that's good enough to catch errors between service boundaries - where client services might be implemented in javascript/ruby etc... And ideally also target swagger, to give client developers access to some swagger tooling / swagger-docs on the other side. (Though that's just an extra nice to have)

I'm trying to write (for fun) a variant of defn where each arg can have a spec and you can have multiple bodies with the same arity as long as the specs are different

i.e., you can overload on spec, not just arity

interesting idea

I really like multispec

I'm guessing you just generate those when the arities are the same?

I was going to start with just one body that takes varargs and parses them

incidentally I have to parse arglists for this and spec is really good at that :D

@rickmoynihan: there is a open issue in ring-swagger to add support for spec, haven't had to to investigate yet. (Currently supports the Plumatic Schema). Could split the lib into separate submodules for the different sources models.

ikitommi: does ring-swagger include use a separate library for JSON schema; or bake its own one? I'm no swagger expert - but I'd assumed that Swagger included all of JSON schema...

I just think it'd be most useful to target JSON schema directly rather than drag in swagger/ring etc too

https://github.com/metosin/ring-swagger/issues/95

We're in the fortunate situation of not having a lot of legacy plumatic/schemas - mainly because we only started using schemas a month or two before spec's announcement... so I'm not too concerned about plumatic support for legacy reasons... But I'm curious whether anyone think's there's a future for plumatic/schema beyond just supporting legacy... i.e. does it have uses that clojure.spec can't support so well?

there's nothing I used it for that I would keep using it or

s/or/for/

even if there's something minor it could do better I feel like there's more leverage to be had using The One True Tool

I'll make a clojure.spec utility library if I have to though, for specialized stuff that clojure.spec decides not to support directly

e.g., maybe for the super succinct map syntax plumatic/schema has for defining map schemas

the only things I think plumatic/schema has over clojure.spec right now are: 1. it's perhaps a little closer to json schema - and possibly less work to bridge into json schema 2. no automated coercions

I'm wondering here about sort of generic specs, I'm thinking on the channel case, I can spec the return of a function to return a channel, maybe would be nice to be able to annotate also the expected value that will come from the channel, do you people have any thoughts on how to deal with stuff like this?

@gfredericks: having the fn return data which you throw is better - (s/keys :opt-un [:ex-message :ex-data])?

@rickmoynihan: the plumatic->json schema is done within the lib (the json-schema ns, based on protocol & supporting multimethod). Swagger (OpenAPI nowadays) only supports only a "pragmatic” subset of JSON Schema as it’s original target was the OO-languages like Java - has client code generators for those. Things like oneOf or anyOf are not supported - the requests to add those are over 2y old now. I think having a separate pure spec<->json schema would be awesome! Having it work with swagger requires some extra work, happy to do/help with that.

richhickey: yeah that's essentially what I was thinking; can you think of any args that should be passed? if we make it a one-arg function that's passed an empty map that would be better for backwards-compatibly adding more things later :) I'm wondering because if there's no need for any args then it's not clear why it even needs to be a function and not just passing the data directly

this fancy defn is fun; I suppose it's a lot like core.match

uh oh I just made it stack overflow

okay here it is -- defn+spec, where each arg can be decorated with a spec and you can overload a function by spec: https://gist.github.com/gfredericks/e4a7eafe5dcf1f4feb21ebbc04b6f302#file-defn-spec-clj-L73

there's a note in there about a stack overflow that I haven't tried to debug, that happens when I add a spec to the defn+spec macro itself

hi everyone I have 2 questions about clojure.spec, which are not that obvious from the beginning: 1. is it okay to use predicates that connect to external resources for validation? ie: I have a tree of "paths" and want to validate that it matches the filesystem or posts on some website? I don't see any techncal problems with that, but maybe there are other solutions for that? 2. for example, I have 2 data structures: new {"application": {"branches": {"release": "1.0", "snapshot": "1.1"}}} and old {"application": {"branches": {"release": "0.9", "snapshot": "1.0”}}} and I want to actually check that the new value is a "valid update" of the old one, based on some constraints (for example: "new value contains at least all branches from the previous one" and "all versions in new value are greater than the same ones in the old one"). is there a way to at least partially express that in clojure.spec and get validation + error reporting? Thanks

I think the predicates are supposed to be pure functions

I couldn't say where the first place you would run into trouble would be though

Can someone explain what I found in my snippet?

does replacing sp/alt with sp/or make it work?

Yes!

haha, i've had the same issue

someone worked out the differences. It has something to do with how sp/alt is used in regex, and sp/or is used otherwise.

So sp/alt on something which is not a regexp doesn’t work? Ok I see...

Simply put yes. Someone else on here had a really good explanation, but I didn't completely follow

He was wondering why sp/alt and sp/or were so similar, and produced the same results in some contexts

had to do with the regex distinction

so maybe we need spec for spec - I mean it’s all macros - everything can happen

haha, i'm curious to see how much spec is used 🙂

I wonder if they'll use clojure.spec on all of the clojure.core to try and get away from the java stracktraces

cat and alt says: “returns a regex”, and says: "returns a spec” - so maybe a spec is not a regex 🙂

ah ok

type systems would help rant

I was talking to someone about clojure.spec on freenode#programming, and he said clojure.spec is like contracts in racket

the idea being that they're supposed to be more powerful, since you can apply predicates

yes spec is more powerful as a type system, because you can inspect actual values at runtime

ya, the whole instrumentation is pretty neat

i've used it, but it's rather slow. I need to start using it per namespace

just use it only in dev and test

I have been, but it's even slow then. Maybe 20 times slower

makes testing rather slow

It isn't an issue if I run it once in a while, so i'll throw down instrumentation every once in a while

But I wonder if there's any room for improvement in performance, it would make it very appealing to just leave instrumentation enabled even in production

There's a reddit post comparing different schema/data validation libraries. No one has commented on it yet, and I wish someone more knowledgable would

https://www.reddit.com/r/Clojure/comments/4mxqcy/spec_performance_comparison/?ref=share&amp;ref_source=link

Than I think, you apply spec to inner loop things. I would apply it only to my public API.

ya, that makes sense

like, only validate data that comes in from an external source for production?

In production I would check data coming over wire directly without instrumentation, just in normal code.

I do this with Schema already.

I've been using the clojurescript version of clojure.spec, cljs.spec

oh nice

yeah, I converted my project i've been working on from schema to spec

didn't take that long, but it did require quite a bit of refactoring

I'm still not sure where I should put the specs. I kind of placed them at the beginning of the files, and would s/fdef after each function

I suppose that's good enough, I don't think i've tested s/fdef when the function hasn't been declared yet, so I don't think I could place them in another file?

I’m not completely sure either. But I have seen putting s/fdef before each funtion often.

oh, before?

hmm

yes it is possible

that's good to know

its like contract first and than the implementation

that makes sense

i'll have to consider moving my stuff into a separate folder

it's hard to get used to, really. I considered it an eye-sore at first

definitely useful though, I caught a lot of bugs early

opposite to schema, the annotations are not inline to the function - so the possibilities are broader - like in core.typed I think - never used it

Yeah, that would explain my confusion. At first, I had a hard time figuring out how to convert from schema

made a lot more sense once I got started

when I used schema, I had originally defined the structures in a separate file, sortof like how you would define a jsonschema

@benzap: it’s probably slower because it’s doing generative testing at runtime. Hopefully that becomes a config setting

@arohner: that's interesting, I thought it was trying to conform each function with respect to it's defined spec. If it's also throwing in generative testing, I can see that causing some performance issues

well, fspec does. checking fdef now

a somewhat related topic, there's a library in clojure called specter, which compiles itself to increase performance

I wonder if the same concept could be applied to clojure.spec, where you could pre-compile the validator for your functions

what would a spec look like that does the following: if a map has one key, it should have another key

And in plumatic schema, you can create validators ahead of time for performance.

oops wrong description of problem. if a key in a map has a certain value, then another key should be present. writing specs can be a challeng...like a whole nother language! 🙂

@fenton That is likely a multispec

Where you write a multimethod looking for the first key, and that dispatches another spec to check for the second.

@angusiguess: ok, will look into that. thx.

😅

could also be a simple s/or

(s/or map? (s/and (s/keys [::foo ::bar]))

depends on how complex it needs to be

@rickmoynihan: we use coercions which are extremely handy

If spec (or surrounding tooling) doesn’t support automated coercions, then we’ll need to keep schemas for our boundary interfaces between webapp and RethinkDB

danielcompton: Yeah - I've used coercions before too - and I don't dispute their utility at boundaries... I guess you could easily build a specialised coercion library on top of spec though... don't know enough yet how you might do that... Regardless I much prefer what I've seen of spec to schema; and don't think coercions are enough of a feature on their own to either not use spec, or use schema as well as spec... I'd definitely much prefer something that worked with spec