this room might appreciate this: https://gist.github.com/brandonbloom/7a76d55ffeaf4ecc46832060d659dfc1

i could imagine such a thing being integrated with spec, but not exactly sure how

i’m using it to do something akin to constraint propagation

lots of other details to work out with it tho - like what to do if a contradiction occurs

Are there any news on the next steps & roadmap for the spec? Trying to develop the runtime dynamic conforming stuff on top of spec. Would love to see Specs as types/records instead of reified protocols so that one could extend the specs easier. I'm toying now with stripping out extra keys from map specs & optionally giving errors for extra keys. Both at runtime, which is cool at api borders.

e.g. clojure.spec/KeysSpec. could make it implement protocols to extract it’s conforming type & to list it’s keys. and to generate docs etc.

this works now, but requires explicit conforming type hints (to tag specs):

(require '[clojure.spec :as s])
(require '[spec-tools.core :as st])

(s/def ::a int?)
(s/def ::b keyword?)
(s/def ::c string?)
(s/def ::ab (st/spec ::map (s/keys :req [::a ::b])))
(s/def ::cab (st/spec ::map (s/keys :req [::c] :opt [::ab])))

(st/conform
  ::cab
  {::c "c"
   ::ab {::a 1
         ::b :b
         ::EXTRA "SHOULD BE REMOVED"}
   ::EVIL "SHOULD BE REMOVED"}
  {::map st/remove-extra-keys})
; => #:user{:c "c", :ab #:user{:a 1, :b :b}}

Hi! https://clojure.org/guides/spec#_using_spec_for_validation https://clojure.org/guides/spec#_spec_ing_functions There is a way to "integrate" the two uses?? Or I will need to specify the function s/fdef,,,,, and valid the input :pre s/valid?,,,, separately?

@ikitommi did you notice https://github.com/uswitch/speculate

it's on my "kick tires" short list

but yes, an update would be nice indeed

thanks @mpenet for the link. Spectrum, spec-tools and speculate are all trying to convert specs into something that is easier to mold - with custom records, walkers & ast. The upcoming(?) specs of specs will make them easier to interpret, but records would give a one way to extend them.

hard to know how to do something like that until spec stabilizes really

Is someone getting pull-patterns from specs?

Does spec enable transformation from one 'spec' to another? i.e. could I take a map {:name "Bruce" :email ""} and transform it to conform to another spec {:name-first "Bruce" :contact {:email ""}}? I feel like that might be possible with conform/unform or some other spec function, but I have yet to figure it out. Any help or direction would be appreciated.

Yes, but… bear in mind that all users of your spec would then get the transformed result, which is why that sort of thing is generally frowned on.

FWIW, we use specs that conform strings to non-strings as part of API input handling, but we try to avoid changing the shape of the data, unless there’s a very specific reason for that (and we want all spec client code to experience that change).

@seancorfield I took his question to mean if there might be an automated way of translating data that has the same components but different structure by specying two specs and maybe some mapping between sub-components

Ah, I see what @kspear means now. TL;DR: “no” 🙂 That’s a general data transform issue tho’ — even with specs, you’d have to be able to specify exactly what parts of one spec are transformed to which parts of another.

if you have specs-of-specs perhaps you could automatically map certain ident-kw's to each other

I think then you're starting to build a type system, where you define rules how types can be casted

I think this is something Clojure wants to avoid, and wants to make these transformations explicit

Yeah-- it seems like the sort of thing I could do 'easily' enough without spec at all; the thought occurred when I was playing around with conform/`unform`. I can, for example, use s/cat to say that each value in a list corresponds to some key and spec and see it in the 'specced' form with (conform ::my-first-spec my-list); then, i can unform it with a different spec to re-arrange the values into a different format (conform ::my-second-spec (conform ::my-first-spec my-list)). I thought if such a thing were possible on a list, there might be a way to do it with a map.

I figure a bit of context might help. In this specific case I'm taking reports generated (XML and json files, typically) by third party tools (vulnerability scanners) and I want to be able transform each 'issue' in each report into a common internal format within my program. Then, I want to be able to transform issues in my internal generic format to a JSON object that I can pass to our issue tracker API (i.e. JIRA). I'm just wondering if there is a way to specify what all of those formats look like (the data I get from my various third party tools, my internal generic issue format, JIRAs create issue format) and transform between them taking advantage of spec.

yeah, I think that just mapping them might be better, you want to make these types of transformations explicit

I’d say the purpose of spec is not a generic data transformation library

and Clojure itself has plenty of tools for doing that (outside of spec)

conform/unform can be used to move back and forth between the original and conformed version of data, but pushing this use into transformation leads to treating spec as a “meat grinder” (Rich’s words) rather than something that states a specification of your data. in the long run, I think that path will cut you off from some uses of spec as you are no longer using it as a specification but as a process.

We use spec a lot. Never to coerce data tho. We actually use specter for this

Now that I think of it l dont think we use coercers at all

+1 :)

I mean directly

@ikitommi I’d say it is still unresolved what the final implementation form will be for specs (protocols, records, or something else) but regardless we would not encourage direct access to those internals (and in fact the likelihood of people abusing that access is considered to be one downside to records as impls). The encouraged path to move from a spec (which is data, but in a form) to more map-like data is to conform the spec’s form.

We did go through a phase where we did use coercers for transforms, but imho it gets messy fast

yes

spec works best when you primarily use it to speak truthful, assertive statements about what your data is

and not to transform your data or make statements about what your data is not

One of the thing I like is togglable asserts in pre/post

once you've created data transformation functions, and spec'ed them, you can examine the spec registry to find transforms with matching inputs and outputs to derive larger transforms

Thanks everyone so far for the responses. Definitely helps. In my specific case, I'm getting this third party data that, in essence, is the same data, just in proprietary representations. My assumption was that, using spec, I could author specs that define what the common data is in these different representations and leverage those to extract the common data into a generic representation. If spec isn't the answer to this problem, I have to assume there is another idiomatic approach to this type of problem in clojure already?

your first example seemed to imply that you’re just talking about different key names for the same basic map structure — is that accurate, or was your example oversimplified?

well I think all that is fine if you’re talking about specifying the data in either the different representations or the common representation and where it is less so is when it’s about transformation

@alexmiller Aye, ok. So realistically I should be writing transformation functions like (defn external-format-1->common-format [data] ...), and then using spec to validate the input (external format spec) and output (my common format spec).

@joshjones Simplifying a bit. I need to rename keys, but also apply some transformation to the corresponding values (i.e. one external format might represent a date as a string with a particular ISO format that I then want to transform it to something compatible with the clj-time library). The structure, also, is going to be different between sources.

yeah, that’s not spec’s job (imo)

even though you could coax it do so with conformers etc

but you could for example write a spec for both sides and then write code that read both specs and built the transformation between them for you

so in this case @alexmiller , you’re not using spec to do the actual transformation, but to describe the two pieces, and then you write something to bridge the two specs … is that kind of the idea? kind of like keeping spec “clean” and letting something else do the dirty work.

right (although I object to clean/dirty connotations :)

lol

in that I think spec is good at specifying and that clojure transformation functions are good at transforming

and you should use the right tool for the job

man I seriously need to go take a year off in the woods and figure out how to do bijections right

the first 80% is easy

the last 10% is impossible :)

in other words, the usual :)

seriously though, bijections over spec would be a fantastic lib

@alexmiller did you ever see this https://github.com/gfredericks/schema-bijections

nope

actually even narrowing the problem from json <-> clojure would be interesting in that it’s clearly the most common need for this and might be able to capture a bigger piece of the pie with default behavior

every time I think about that I get grumpy that spec doesn't have a s/string-keys

but maybe that can be done in userland

the two things I hate about that ⇑ library are A) I think surjections/coercions should be first-class, hooking in to generators, and B) the composition API is terrible and I don't know how to make it better

hey all. has any one looked at generating WSDL from specs yet?

I am professionally obligated to question your use of WSDL in the first place :) moving past that, I have not seen anyone mention it

hah indeed! does seem like a messy bit of tech… not sure it solves what it claims to or not. Seems to me a lot of it could be generated though

I’m just looking at it for academic/comparitive reasons

from my experience, wsdl is almost always generated (which is why it’s so bad)

people have primarily used it to write giant xml docs that describe an rpc call

yup. loose coupling. did that ever come really? I suppose we’ve abstracted over the lower level comms protocols at least

Rich has a lot of ideas about this area, many of which are in harmony with spec he teased those in the last conj keynote, but they are worthy of a lot more time

Many folk are using swagger to do the job WSDL used to pretend to do, though I think it may be called openapi now

thanks @donaldball I’ll take a look.

thanks @alexmiller. looking forward to the new alphas.

I've been using spec internally, and generating json schemas from specs for external interfaces (and I think someone else had a github repo or a pastebin doing it too), and I've been thinking about using it to generate graphql types (but I am not sure if I am going to use graphql) and the same for protobufs. Because spec is clojure data it is great for using the same spec to generate lots of other kinds of schemas. You take the same spec data and interpret it different ways.

I have a three line spec that looks like:

(s/def ::node any?)
(s/def ::children (s/coll-of ::tree :kind vector?))
(s/def ::tree (s/keys :req-un [::node ::children])))
;; then I call generate on it via
(gen/sample (s/gen ::tree))

I thought spec handled that via some depth limit

Given how well thought out it is, it probably did, and I, asa a spec noob, simply am not using it correctly.

How do I use depth limits?

If that doesn't help, you can build a generator manually in test.check via the recursive-gen function

I don't know, I'd search the two namespaces for anything with the word depth in the name

https://github.com/clojure/clojure/blob/master/src/clj/clojure/spec.clj#L19

(def ^:dynamic recursion-limit "A soft limit on how many times a branching spec (or/alt/*/opt-keys/multi-spec) ^^ I'm sorry, can you please explain how this helps with my ::children / ::tree recursion via s/coll-of ?

I don't seem to be using any of the functions they 'limit'

That's probably the problem

How do I rewrite my spec to use those functions? 🙂

So you want node values on internal nodes too?

Not just leaves?

Yes, both leaves and internal nodes can store values.

Are you suggesting separate classes and and (s/or ::internal ::leaf) ?

That might fix it, yeah

Only other idea I have is the manual generator idea from earlier

Separate classes is probably easier

Spec is important enough that. I want to learn both.

Do you know of a good tutorial for manual generators? seems like I'm going to run into it sooner or later

There are some docs in the test.check repo

there's a screencast on manual generators

@gfredericks : I can't get the two class approach to work; would you mind showing me sample code?

this is what I have so far:

http://blog.cognitect.com/blog/2016/8/10/clojure-spec-screencast-customizing-generators

This feels like a great use case for the new threaded messages feature in Slack...

that's actually the last in a series of 3 casts, they're very good

` ::nt->node any? ::nt->children (s/coll-of ::nt :kind vector?) ::nt-leaf (s/keys :req-un [::nt->node] ::nt-internal (s/keys :req-un [::nt->node ::nt->chidlren] ::nt (s/or ::nt-leaf ::nt-internal)

but when I try to gen on ::nt, I get some error about nt-leaf nt->children nt-leaf nt->chlidren ...

when does threading drop? this week right? I've been missing threading since flowdock died

Refresh now -- it's enabled for this Slack already.

i fear this is one of those “good ideas on paper” sort of things...

the slack threads, that is

I don't like threads either; what I want is a way to "mute" people taht seem way too chatty; but the problem is, those ppl are probably not goign to voluntarily start a thread

s/"mute"/ignore

trees via 2 classes works now

@bfabry : going to study your video on manual generators now