Fork me on GitHub
#clojure-spec
<
2017-01-18
>
bbloom02:01:52

i could imagine such a thing being integrated with spec, but not exactly sure how

bbloom02:01:42

i’m using it to do something akin to constraint propagation

bbloom02:01:58

lots of other details to work out with it tho - like what to do if a contradiction occurs

ikitommi07:01:35

Are there any news on the next steps & roadmap for the spec? Trying to develop the runtime dynamic conforming stuff on top of spec. Would love to see Specs as types/records instead of reified protocols so that one could extend the specs easier. I'm toying now with stripping out extra keys from map specs & optionally giving errors for extra keys. Both at runtime, which is cool at api borders.

ikitommi07:01:31

e.g. clojure.spec/KeysSpec. could make it implement protocols to extract it’s conforming type & to list it’s keys. and to generate docs etc.

ikitommi07:01:35

this works now, but requires explicit conforming type hints (to tag specs):

(require '[clojure.spec :as s])
(require '[spec-tools.core :as st])

(s/def ::a int?)
(s/def ::b keyword?)
(s/def ::c string?)
(s/def ::ab (st/spec ::map (s/keys :req [::a ::b])))
(s/def ::cab (st/spec ::map (s/keys :req [::c] :opt [::ab])))

(st/conform
  ::cab
  {::c "c"
   ::ab {::a 1
         ::b :b
         ::EXTRA "SHOULD BE REMOVED"}
   ::EVIL "SHOULD BE REMOVED"}
  {::map st/remove-extra-keys})
; => #:user{:c "c", :ab #:user{:a 1, :b :b}}

souenzzo14:01:16

Hi! https://clojure.org/guides/spec#_using_spec_for_validation https://clojure.org/guides/spec#_spec_ing_functions There is a way to "integrate" the two uses?? Or I will need to specify the function s/fdef,,,,, and valid the input :pre s/valid?,,,, separately?

mpenet14:01:44

it's on my "kick tires" short list

mpenet14:01:52

but yes, an update would be nice indeed

ikitommi15:01:29

thanks @mpenet for the link. Spectrum, spec-tools and speculate are all trying to convert specs into something that is easier to mold - with custom records, walkers & ast. The upcoming(?) specs of specs will make them easier to interpret, but records would give a one way to extend them.

mpenet15:01:30

hard to know how to do something like that until spec stabilizes really

souenzzo16:01:13

Is someone getting pull-patterns from specs?

kspear16:01:20

Does spec enable transformation from one 'spec' to another? i.e. could I take a map {:name "Bruce" :email ""} and transform it to conform to another spec {:name-first "Bruce" :contact {:email ""}}? I feel like that might be possible with conform/unform or some other spec function, but I have yet to figure it out. Any help or direction would be appreciated.

seancorfield17:01:06

Yes, but… bear in mind that all users of your spec would then get the transformed result, which is why that sort of thing is generally frowned on.

seancorfield17:01:12

FWIW, we use specs that conform strings to non-strings as part of API input handling, but we try to avoid changing the shape of the data, unless there’s a very specific reason for that (and we want all spec client code to experience that change).

luxbock17:01:30

@seancorfield I took his question to mean if there might be an automated way of translating data that has the same components but different structure by specying two specs and maybe some mapping between sub-components

seancorfield17:01:38

Ah, I see what @kspear means now. TL;DR: “no” 🙂 That’s a general data transform issue tho’ — even with specs, you’d have to be able to specify exactly what parts of one spec are transformed to which parts of another.

luxbock17:01:45

if you have specs-of-specs perhaps you could automatically map certain ident-kw's to each other

lmergen17:01:17

I think then you're starting to build a type system, where you define rules how types can be casted

lmergen17:01:46

I think this is something Clojure wants to avoid, and wants to make these transformations explicit

kspear17:01:58

Yeah-- it seems like the sort of thing I could do 'easily' enough without spec at all; the thought occurred when I was playing around with conform/`unform`. I can, for example, use s/cat to say that each value in a list corresponds to some key and spec and see it in the 'specced' form with (conform ::my-first-spec my-list); then, i can unform it with a different spec to re-arrange the values into a different format (conform ::my-second-spec (conform ::my-first-spec my-list)). I thought if such a thing were possible on a list, there might be a way to do it with a map.

kspear17:01:24

I figure a bit of context might help. In this specific case I'm taking reports generated (XML and json files, typically) by third party tools (vulnerability scanners) and I want to be able transform each 'issue' in each report into a common internal format within my program. Then, I want to be able to transform issues in my internal generic format to a JSON object that I can pass to our issue tracker API (i.e. JIRA). I'm just wondering if there is a way to specify what all of those formats look like (the data I get from my various third party tools, my internal generic issue format, JIRAs create issue format) and transform between them taking advantage of spec.

lmergen18:01:50

yeah, I think that just mapping them might be better, you want to make these types of transformations explicit

Alex Miller (Clojure team)18:01:45

I’d say the purpose of spec is not a generic data transformation library

Alex Miller (Clojure team)18:01:07

and Clojure itself has plenty of tools for doing that (outside of spec)

Alex Miller (Clojure team)18:01:47

conform/unform can be used to move back and forth between the original and conformed version of data, but pushing this use into transformation leads to treating spec as a “meat grinder” (Rich’s words) rather than something that states a specification of your data. in the long run, I think that path will cut you off from some uses of spec as you are no longer using it as a specification but as a process.

mpenet18:01:45

We use spec a lot. Never to coerce data tho. We actually use specter for this

mpenet18:01:19

Now that I think of it l dont think we use coercers at all

mpenet18:01:37

I mean directly

Alex Miller (Clojure team)18:01:52

@ikitommi I’d say it is still unresolved what the final implementation form will be for specs (protocols, records, or something else) but regardless we would not encourage direct access to those internals (and in fact the likelihood of people abusing that access is considered to be one downside to records as impls). The encouraged path to move from a spec (which is data, but in a form) to more map-like data is to conform the spec’s form.

mpenet18:01:39

We did go through a phase where we did use coercers for transforms, but imho it gets messy fast

Alex Miller (Clojure team)18:01:52

spec works best when you primarily use it to speak truthful, assertive statements about what your data is

Alex Miller (Clojure team)18:01:59

and not to transform your data or make statements about what your data is not

mpenet18:01:14

One of the thing I like is togglable asserts in pre/post

hiredman19:01:24

once you've created data transformation functions, and spec'ed them, you can examine the spec registry to find transforms with matching inputs and outputs to derive larger transforms

kspear19:01:18

Thanks everyone so far for the responses. Definitely helps. In my specific case, I'm getting this third party data that, in essence, is the same data, just in proprietary representations. My assumption was that, using spec, I could author specs that define what the common data is in these different representations and leverage those to extract the common data into a generic representation. If spec isn't the answer to this problem, I have to assume there is another idiomatic approach to this type of problem in clojure already?

joshjones19:01:58

your first example seemed to imply that you’re just talking about different key names for the same basic map structure — is that accurate, or was your example oversimplified?

Alex Miller (Clojure team)19:01:42

well I think all that is fine if you’re talking about specifying the data in either the different representations or the common representation and where it is less so is when it’s about transformation

kspear19:01:32

@alexmiller Aye, ok. So realistically I should be writing transformation functions like (defn external-format-1->common-format [data] ...), and then using spec to validate the input (external format spec) and output (my common format spec).

kspear19:01:04

@joshjones Simplifying a bit. I need to rename keys, but also apply some transformation to the corresponding values (i.e. one external format might represent a date as a string with a particular ISO format that I then want to transform it to something compatible with the clj-time library). The structure, also, is going to be different between sources.

Alex Miller (Clojure team)19:01:08

yeah, that’s not spec’s job (imo)

Alex Miller (Clojure team)19:01:22

even though you could coax it do so with conformers etc

Alex Miller (Clojure team)19:01:01

but you could for example write a spec for both sides and then write code that read both specs and built the transformation between them for you

joshjones19:01:08

so in this case @alexmiller , you’re not using spec to do the actual transformation, but to describe the two pieces, and then you write something to bridge the two specs … is that kind of the idea? kind of like keeping spec “clean” and letting something else do the dirty work.

Alex Miller (Clojure team)19:01:28

right (although I object to clean/dirty connotations :)

Alex Miller (Clojure team)19:01:57

in that I think spec is good at specifying and that clojure transformation functions are good at transforming

Alex Miller (Clojure team)19:01:06

and you should use the right tool for the job

gfredericks19:01:02

man I seriously need to go take a year off in the woods and figure out how to do bijections right

Alex Miller (Clojure team)19:01:50

the last 10% is impossible :)

Alex Miller (Clojure team)19:01:01

in other words, the usual :)

Alex Miller (Clojure team)19:01:46

seriously though, bijections over spec would be a fantastic lib

Alex Miller (Clojure team)19:01:55

actually even narrowing the problem from json <-> clojure would be interesting in that it’s clearly the most common need for this and might be able to capture a bigger piece of the pie with default behavior

gfredericks19:01:37

every time I think about that I get grumpy that spec doesn't have a s/string-keys

gfredericks19:01:22

but maybe that can be done in userland

gfredericks19:01:20

the two things I hate about that ⇑ library are A) I think surjections/coercions should be first-class, hooking in to generators, and B) the composition API is terrible and I don't know how to make it better

triss19:01:38

hey all. has any one looked at generating WSDL from specs yet?

Alex Miller (Clojure team)19:01:28

I am professionally obligated to question your use of WSDL in the first place :) moving past that, I have not seen anyone mention it

triss19:01:32

hah indeed! does seem like a messy bit of tech… not sure it solves what it claims to or not. Seems to me a lot of it could be generated though

triss19:01:58

I’m just looking at it for academic/comparitive reasons

Alex Miller (Clojure team)19:01:16

from my experience, wsdl is almost always generated (which is why it’s so bad)

Alex Miller (Clojure team)19:01:53

people have primarily used it to write giant xml docs that describe an rpc call

triss19:01:46

yup. loose coupling. did that ever come really? I suppose we’ve abstracted over the lower level comms protocols at least

Alex Miller (Clojure team)19:01:56

Rich has a lot of ideas about this area, many of which are in harmony with spec he teased those in the last conj keynote, but they are worthy of a lot more time

donaldball19:01:02

Many folk are using swagger to do the job WSDL used to pretend to do, though I think it may be called openapi now

triss19:01:05

thanks @donaldball I’ll take a look.

ikitommi21:01:07

thanks @alexmiller. looking forward to the new alphas.

hiredman21:01:31

I've been using spec internally, and generating json schemas from specs for external interfaces (and I think someone else had a github repo or a pastebin doing it too), and I've been thinking about using it to generate graphql types (but I am not sure if I am going to use graphql) and the same for protobufs. Because spec is clojure data it is great for using the same spec to generate lots of other kinds of schemas. You take the same spec data and interpret it different ways.

qqq23:01:38

I have a three line spec that looks like:

(s/def ::node any?)
(s/def ::children (s/coll-of ::tree :kind vector?))
(s/def ::tree (s/keys :req-un [::node ::children])))
;; then I call generate on it via
(gen/sample (s/gen ::tree))
this is getting me stack overflow, probably due to generating infinite depth tree how cna I make generating trees work?

gfredericks23:01:26

I thought spec handled that via some depth limit

qqq23:01:56

Given how well thought out it is, it probably did, and I, asa a spec noob, simply am not using it correctly.

qqq23:01:58

How do I use depth limits?

gfredericks23:01:02

If that doesn't help, you can build a generator manually in test.check via the recursive-gen function

gfredericks23:01:34

I don't know, I'd search the two namespaces for anything with the word depth in the name

qqq23:01:47

(def ^:dynamic recursion-limit "A soft limit on how many times a branching spec (or/alt/*/opt-keys/multi-spec) ^^ I'm sorry, can you please explain how this helps with my ::children / ::tree recursion via s/coll-of ?

qqq23:01:53

I don't seem to be using any of the functions they 'limit'

gfredericks23:01:25

That's probably the problem

qqq23:01:12

How do I rewrite my spec to use those functions? 🙂

gfredericks23:01:16

So you want node values on internal nodes too?

gfredericks23:01:22

Not just leaves?

qqq23:01:33

Yes, both leaves and internal nodes can store values.

qqq23:01:53

Are you suggesting separate classes and and (s/or ::internal ::leaf) ?

gfredericks23:01:21

That might fix it, yeah

gfredericks23:01:40

Only other idea I have is the manual generator idea from earlier

gfredericks23:01:06

Separate classes is probably easier

qqq23:01:00

Spec is important enough that. I want to learn both.

qqq23:01:14

Do you know of a good tutorial for manual generators? seems like I'm going to run into it sooner or later

gfredericks23:01:16

There are some docs in the test.check repo

bfabry23:01:20

there's a screencast on manual generators

qqq23:01:15

@gfredericks : I can't get the two class approach to work; would you mind showing me sample code?

qqq23:01:22

this is what I have so far:

seancorfield23:01:02

This feels like a great use case for the new threaded messages feature in Slack...

bfabry23:01:06

that's actually the last in a series of 3 casts, they're very good

qqq23:01:17

` ::nt->node any? ::nt->children (s/coll-of ::nt :kind vector?) ::nt-leaf (s/keys :req-un [::nt->node] ::nt-internal (s/keys :req-un [::nt->node ::nt->chidlren] ::nt (s/or ::nt-leaf ::nt-internal)

qqq23:01:34

but when I try to gen on ::nt, I get some error about nt-leaf nt->children nt-leaf nt->chlidren ...

bfabry23:01:37

when does threading drop? this week right? I've been missing threading since flowdock died

seancorfield23:01:53

Refresh now -- it's enabled for this Slack already.

qqq23:01:02

this is the new ->> ?

bfabry23:01:22

seancorfield: oh sweet. aaaaand I'm disappointed. this is not low friction. oh well

seancorfield23:01:09

Yes, so you and Gary can chat in a thread about your specific issue at length and then just report a summary back to the channel -- and folks can read all the details if they want, or be spared them if they don't.

qqq23:01:47

interesting

qqq23:01:04

I'd still like an ignore button though 🙂

bfabry23:01:54

yeah... and no one will notice a conversation they can help out with casually and chip in, like I just did with the screencast link for qqq. meh. feels like if people use it low traffic slack channels will be even less useful than they are now. oh well

qqq23:01:08

resolved it: I was mis using (s/or .... )

seancorfield00:01:34

(You could / should have marked that last reply as “Also send to #clojure-spec” so it would appear back in the main channel)

bbloom23:01:51

i fear this is one of those “good ideas on paper” sort of things...

bbloom23:01:59

the slack threads, that is

qqq23:01:07

I don't like threads either; what I want is a way to "mute" people taht seem way too chatty; but the problem is, those ppl are probably not goign to voluntarily start a thread

seancorfield00:01:20

qqq: We (admins — and the community at large) can educate them to do so… Threads will just take some getting used to, like all changes 🙂

qqq23:01:22

s/"mute"/ignore

qqq23:01:55

trees via 2 classes works now

qqq23:01:08

@bfabry : going to study your video on manual generators now

bfabry23:01:05

qqq: not mine! stuart holloway's work

qqq02:01:17

just worked through taht video ; am now convinced that stuart holloway is a genius