Fork me on GitHub
#clojure-spec
<
2016-06-09
>
arohner02:06:04

@bbloom: @ghadi yes, figwheel uses that idea to great effect in config file validation

arohner02:06:53

“unrecognized key :foo under :bar, you probably want it in :baz

Oliver George07:06:13

Hello. My CLJS apps use a lot of string keys. Primarily for performance reasons, transforming JSON has a cost. I went looking for the equivalent of the :strs feature of map destructuring, perhaps something like :req-strs and :opt-strs could be added to s/keys.

Oliver George07:06:27

For now the workaround seems to be using a conformer...

Oliver George07:06:36

(def strs->keys 
  "Turn map with string keys into map of keyword keys.  Without this we can't use s/keys."
  (s/conformer #(reduce-kv (fn [m k v] (assoc m (keyword k) v)) {} %)))
(s/conform
  (s/and strs->keys
         (s/keys :req-un [::id ::options]))
  {"id" :id
   "options" [{:id 1} {}]})

Oliver George07:06:18

Am I missing anything?

hiredman07:06:10

user=> (s/valid? (s/spec #(contains? % "foo")) {"foo" 1})
true
user=> 

hiredman07:06:21

it depends on what you are looking to get out of it, I guess

dominicm12:06:49

@hiredman: You lose thikngs like path lookups doing that. I think that's the key feature that goes missing. Spec has a bit of a balancing act to perform making sure that via/paths are added. s/keys is a bit complicated in implementation, but I would be thinking that a similar-but-different implementation should surface for string keywords.

gfredericks13:06:21

I've thought about string keys some too w.r.t. https://github.com/gfredericks/schema-bijections

gfredericks13:06:16

I figured it could be handled with a bijection system that doesn't produce a spec on the other side, but that doesn't help if you're avoiding transformations for performance

rickmoynihan14:06:00

Hey... I've been very pleased with clojure.spec, which looks absolutely fantastic, and a nice step up from plumatic.schema - and I am fully intending to play with it properly (I've only really had time to read the announcement, listen to Rich on the cognicast, and looked at a few examples)... Anyway, one thing I'd really like to do is understand how this fits with JSON schema... Now, clojure.spec is clearly far superior to JSON schema, but it's not used by the javascript community etc... So I was wondering if anyone has thought about how one might be able to build a JSON schema from a clojure.spec. As spec is way more expressive, I'm guessing that to do this you'd have to stick to a subset of spec (and I guess you could probably use clojure.spec to specify that subset)... The basic idea would be to write a super-spec in spec; and output JSON schemas to interoperate with other tooling/libraries in other languages etc... e.g. swagger clients etc. If this is possible from within clojure you'd then be able reuse that subset of spec (which generated JSON schemas), and layer ontop the additional expressivity and specs you'd want clojure side.

gfredericks15:06:11

rickmoynihan: the library I linked to just above has a lot to do with that problem, but it works with plumatic/schema. I've been pondering how best to translate it to clojure.spec

gfredericks15:06:59

the idea is it uses a set of transformations to turn an internal schema (the nice one) into a different schema (the coarser json-style schema in this case), with functions generated to transform data structures between the two

rickmoynihan15:06:21

gfredericks: interesting

gfredericks15:06:36

it's still a bit half-baked, but I used it a bunch on a previous project

gfredericks15:06:51

in particular it doesn't do a good job of distinguishing between bijections and non-bijections (coercions, projections, whatever you might call them); I'm thinking ou could do that interestingly using test.check generators

gfredericks15:06:06

e.g., if my JSON API can accept {"unitCount":42} as well as {"unitCount":42.0}, you might specify that in the schema, but say that the first one is preferred, so that the transformation function always uses that, but could also have a generator that would take {"unitCount" 42} and generate sometimes {"unitCount" 42} and other times {"unitCount" 42.0M}

gfredericks15:06:23

so it would give you a way to test flexible APIs in a robust way

gfredericks15:06:03

unrelated: if I'm reading the clojure.spec source correctly, it ought to be safe to add specs to test.check (and to clojure.core) without the danger of an infinite loop during instrumentation -- does that sound right?

rickmoynihan15:06:17

does spec support coercion? I though rich said it wasn't part of its job

gfredericks15:06:33

specs can refer to themselves via the keyword

gfredericks15:06:50

(s/def ::tree (s/or :leaf #{:leaf} :children (s/tuple ::tree ::tree)))

gfredericks15:06:02

example value: [[:leaf [:leaf :leaf]] [[:leaf [:leaf :leaf]] [:leaf :leaf]]]

rickmoynihan15:06:14

yeah but I didn't think spec could coerce a value from "foo" into :foo - or can it?

gfredericks15:06:28

oh geez sorry

gfredericks15:06:33

I misread "coercion" as "recursion"

gfredericks15:06:53

(and thus thought your question was unrelated to what I was previously talking about)

gfredericks15:06:07

okay right afaik it doesn't support coercion

gfredericks15:06:30

I'm not sure if the "external spec" idea would work or not

gfredericks15:06:48

e.g., with json you'd want to have string keys, which isn't natural

rickmoynihan15:06:50

tbh I'm not too worried about the coercion case... I'd just like to write clojure.specs (even in a limited subset of spec) and output JSON schemas from them

gfredericks15:06:10

I think that by itself would probably be easier, if you only support a subset of schemas

rickmoynihan15:06:34

well I'm sure that would be possible too

rickmoynihan15:06:43

it doesn't need to be perfect - I'd just like to define a service in clojure, and emit a JSON schema that's good enough to catch errors between service boundaries - where client services might be implemented in javascript/ruby etc... And ideally also target swagger, to give client developers access to some swagger tooling / swagger-docs on the other side. (Though that's just an extra nice to have)

gfredericks16:06:39

I'm trying to write (for fun) a variant of defn where each arg can have a spec and you can have multiple bodies with the same arity as long as the specs are different

gfredericks16:06:55

i.e., you can overload on spec, not just arity

rickmoynihan16:06:10

interesting idea

rickmoynihan16:06:39

I really like multispec

rickmoynihan16:06:09

I'm guessing you just generate those when the arities are the same?

gfredericks16:06:36

I was going to start with just one body that takes varargs and parses them

gfredericks16:06:00

incidentally I have to parse arglists for this and spec is really good at that :D

ikitommi16:06:54

@rickmoynihan: there is a open issue in ring-swagger to add support for spec, haven't had to to investigate yet. (Currently supports the Plumatic Schema). Could split the lib into separate submodules for the different sources models.

rickmoynihan16:06:25

ikitommi: does ring-swagger include use a separate library for JSON schema; or bake its own one? I'm no swagger expert - but I'd assumed that Swagger included all of JSON schema...

rickmoynihan16:06:04

I just think it'd be most useful to target JSON schema directly rather than drag in swagger/ring etc too

rickmoynihan16:06:41

We're in the fortunate situation of not having a lot of legacy plumatic/schemas - mainly because we only started using schemas a month or two before spec's announcement... so I'm not too concerned about plumatic support for legacy reasons... But I'm curious whether anyone think's there's a future for plumatic/schema beyond just supporting legacy... i.e. does it have uses that clojure.spec can't support so well?

gfredericks16:06:23

there's nothing I used it for that I would keep using it or

gfredericks16:06:03

even if there's something minor it could do better I feel like there's more leverage to be had using The One True Tool

gfredericks16:06:57

I'll make a clojure.spec utility library if I have to though, for specialized stuff that clojure.spec decides not to support directly

gfredericks16:06:20

e.g., maybe for the super succinct map syntax plumatic/schema has for defining map schemas

rickmoynihan16:06:40

the only things I think plumatic/schema has over clojure.spec right now are: 1. it's perhaps a little closer to json schema - and possibly less work to bridge into json schema 2. no automated coercions

wilkerlucio16:06:37

I'm wondering here about sort of generic specs, I'm thinking on the channel case, I can spec the return of a function to return a channel, maybe would be nice to be able to annotate also the expected value that will come from the channel, do you people have any thoughts on how to deal with stuff like this?

richhickey16:06:18

@gfredericks: having the fn return data which you throw is better - (s/keys :opt-un [:ex-message :ex-data])?

ikitommi17:06:25

@rickmoynihan: the plumatic->json schema is done within the lib (the json-schema ns, based on protocol & supporting multimethod). Swagger (OpenAPI nowadays) only supports only a "pragmatic” subset of JSON Schema as it’s original target was the OO-languages like Java - has client code generators for those. Things like oneOf or anyOf are not supported - the requests to add those are over 2y old now. I think having a separate pure spec<->json schema would be awesome! Having it work with swagger requires some extra work, happy to do/help with that.

gfredericks18:06:56

richhickey: yeah that's essentially what I was thinking; can you think of any args that should be passed? if we make it a one-arg function that's passed an empty map that would be better for backwards-compatibly adding more things later :) I'm wondering because if there's no need for any args then it's not clear why it even needs to be a function and not just passing the data directly

gfredericks18:06:28

this fancy defn is fun; I suppose it's a lot like core.match

gfredericks18:06:00

uh oh I just made it stack overflow

gfredericks19:06:39

okay here it is -- defn+spec, where each arg can be decorated with a spec and you can overload a function by spec: https://gist.github.com/gfredericks/e4a7eafe5dcf1f4feb21ebbc04b6f302#file-defn-spec-clj-L73

gfredericks19:06:53

there's a note in there about a stack overflow that I haven't tried to debug, that happens when I add a spec to the defn+spec macro itself

fxposter19:06:04

hi everyone I have 2 questions about clojure.spec, which are not that obvious from the beginning: 1. is it okay to use predicates that connect to external resources for validation? ie: I have a tree of "paths" and want to validate that it matches the filesystem or posts on some website? I don't see any techncal problems with that, but maybe there are other solutions for that? 2. for example, I have 2 data structures: new {"application": {"branches": {"release": "1.0", "snapshot": "1.1"}}} and old {"application": {"branches": {"release": "0.9", "snapshot": "1.0”}}} and I want to actually check that the new value is a "valid update" of the old one, based on some constraints (for example: "new value contains at least all branches from the previous one" and "all versions in new value are greater than the same ones in the old one"). is there a way to at least partially express that in clojure.spec and get validation + error reporting? Thanks

gfredericks19:06:16

I think the predicates are supposed to be pure functions

gfredericks19:06:46

I couldn't say where the first place you would run into trouble would be though

akiel20:06:08

Can someone explain what I found in my snippet?

benzap20:06:24

does replacing sp/alt with sp/or make it work?

benzap20:06:32

haha, i've had the same issue

benzap20:06:01

someone worked out the differences. It has something to do with how sp/alt is used in regex, and sp/or is used otherwise.

akiel20:06:26

So sp/alt on something which is not a regexp doesn’t work? Ok I see...

benzap20:06:54

Simply put yes. Someone else on here had a really good explanation, but I didn't completely follow

benzap20:06:14

He was wondering why sp/alt and sp/or were so similar, and produced the same results in some contexts

benzap20:06:26

had to do with the regex distinction

akiel20:06:08

so maybe we need spec for spec - I mean it’s all macros - everything can happen

benzap20:06:14

haha, i'm curious to see how much spec is used 🙂

benzap20:06:48

I wonder if they'll use clojure.spec on all of the clojure.core to try and get away from the java stracktraces

akiel20:06:20

cat and alt says: “returns a regex”, and says: "returns a spec” - so maybe a spec is not a regex 🙂

akiel20:06:47

type systems would help rant

benzap20:06:43

I was talking to someone about clojure.spec on freenode#programming, and he said clojure.spec is like contracts in racket

benzap20:06:14

the idea being that they're supposed to be more powerful, since you can apply predicates

akiel20:06:02

yes spec is more powerful as a type system, because you can inspect actual values at runtime

benzap20:06:26

ya, the whole instrumentation is pretty neat

benzap20:06:45

i've used it, but it's rather slow. I need to start using it per namespace

akiel20:06:13

just use it only in dev and test

benzap20:06:39

I have been, but it's even slow then. Maybe 20 times slower

benzap20:06:43

makes testing rather slow

benzap20:06:58

It isn't an issue if I run it once in a while, so i'll throw down instrumentation every once in a while

benzap20:06:40

But I wonder if there's any room for improvement in performance, it would make it very appealing to just leave instrumentation enabled even in production

benzap20:06:22

There's a reddit post comparing different schema/data validation libraries. No one has commented on it yet, and I wish someone more knowledgable would

akiel20:06:35

Than I think, you apply spec to inner loop things. I would apply it only to my public API.

benzap20:06:52

ya, that makes sense

benzap20:06:07

like, only validate data that comes in from an external source for production?

akiel20:06:36

In production I would check data coming over wire directly without instrumentation, just in normal code.

akiel20:06:07

I do this with Schema already.

benzap20:06:16

I've been using the clojurescript version of clojure.spec, cljs.spec

benzap20:06:34

yeah, I converted my project i've been working on from schema to spec

benzap20:06:48

didn't take that long, but it did require quite a bit of refactoring

benzap20:06:36

I'm still not sure where I should put the specs. I kind of placed them at the beginning of the files, and would s/fdef after each function

benzap20:06:34

I suppose that's good enough, I don't think i've tested s/fdef when the function hasn't been declared yet, so I don't think I could place them in another file?

akiel20:06:40

I’m not completely sure either. But I have seen putting s/fdef before each funtion often.

benzap20:06:48

oh, before?

akiel20:06:03

yes it is possible

benzap20:06:11

that's good to know

akiel20:06:18

its like contract first and than the implementation

benzap20:06:35

that makes sense

benzap20:06:54

i'll have to consider moving my stuff into a separate folder

benzap20:06:24

it's hard to get used to, really. I considered it an eye-sore at first

benzap20:06:37

definitely useful though, I caught a lot of bugs early

akiel20:06:57

opposite to schema, the annotations are not inline to the function - so the possibilities are broader - like in core.typed I think - never used it

benzap20:06:49

Yeah, that would explain my confusion. At first, I had a hard time figuring out how to convert from schema

benzap20:06:00

made a lot more sense once I got started

benzap20:06:42

when I used schema, I had originally defined the structures in a separate file, sortof like how you would define a jsonschema

arohner20:06:53

@benzap: it’s probably slower because it’s doing generative testing at runtime. Hopefully that becomes a config setting

benzap21:06:15

@arohner: that's interesting, I thought it was trying to conform each function with respect to it's defined spec. If it's also throwing in generative testing, I can see that causing some performance issues

arohner21:06:35

well, fspec does. checking fdef now

benzap21:06:46

a somewhat related topic, there's a library in clojure called specter, which compiles itself to increase performance

benzap21:06:19

I wonder if the same concept could be applied to clojure.spec, where you could pre-compile the validator for your functions

fenton21:06:27

what would a spec look like that does the following: if a map has one key, it should have another key

bbrinck21:06:33

And in plumatic schema, you can create validators ahead of time for performance.

fenton21:06:33

oops wrong description of problem. if a key in a map has a certain value, then another key should be present. writing specs can be a challeng...like a whole nother language! 🙂

angusiguess21:06:11

@fenton That is likely a multispec

angusiguess21:06:47

Where you write a multimethod looking for the first key, and that dispatches another spec to check for the second.

fenton21:06:00

@angusiguess: ok, will look into that. thx.

arohner21:06:23

could also be a simple s/or

arohner21:06:43

(s/or map? (s/and (s/keys [::foo ::bar]))

arohner21:06:50

depends on how complex it needs to be

danielcompton22:06:55

@rickmoynihan: we use coercions which are extremely handy

danielcompton22:06:25

If spec (or surrounding tooling) doesn’t support automated coercions, then we’ll need to keep schemas for our boundary interfaces between webapp and RethinkDB

rickmoynihan23:06:50

danielcompton: Yeah - I've used coercions before too - and I don't dispute their utility at boundaries... I guess you could easily build a specialised coercion library on top of spec though... don't know enough yet how you might do that... Regardless I much prefer what I've seen of spec to schema; and don't think coercions are enough of a feature on their own to either not use spec, or use schema as well as spec... I'd definitely much prefer something that worked with spec