This was something I wrote to tackle the specific case of having a more general spec (as a key in a map) that context (what kind of map it is in) would narrow

e.g. :a could be 1 2 or 3, but when it's in a map with :map-type "foo" :a must be 1 or 2

dealing with that case over and over with predicates was a hassle, so this macro (keys+) allowed adding an additional keyword->spec mapping for validity, conformance, and generator purposes

but the original "natural", "contextless" spec still had to validate

so it's not aimed at the DSL use case, this was to deal with data from a foreign system that had a kind of tagging/inheritance relationship. their fields and entites had a base definition that was overly broad, and could be narrowed without changing their structure

think xsd facets where the element names don't change

the purpose was to allow generic processors to understand everything, but particular producers/consumers to add additional constraints about what they would produce/consume

e.g. a field is cardinality-many, but some processor guarantees they only ever put one value in that field

@cjsauer you can s/or or s/alt lookup-refs spec with s/keys. Or avoid speccing maps with lookup-ref values (`{:my/attr [:db/id :db/ident]}`). Or, if you can isolate lookup-ref key-values from actual data key-values, spec lookup-refs with key-agnostic spec, something like (s/map-of qualified-keyword? lookup-ref-spec). I don't think having s/select would solve spec overload (data or lookup) in this case, because: https://clojurians.slack.com/archives/C1B1BB2Q3/p1547474311308100

Another option is to have s/or + (s/map-of keyword? not-a-look-up-value?) where you expect already looked-up values.

So I have [org.clojure/test.check "0.9.0"] in my dependencies, but when I’m trying to use generators with spec I’m still getting a clojure.test.check.generators never required error…. Anyone have an idea of what I should try? Is there a different one for cljs potentially?

have you tried requiring it manually?

@shanelester55 are you on CLJS or CLJ and which version?

I’m on CLJS. Using shadow-cljs so… not positive which version. Whichever version it is currently using, which I imagine is the latest.

Is requiring it manually just specifiying it at the top of the file where it is used?

you should be able to know the CLJS version in order to deal with known issues. can you try *clojurescript-version* in the REPL?

there were some changes around this recently, but they are only on master, not released yet, that’s why I’d like to know

Sure thing. trying that now… for some reason giving me trouble

if you don’t have a REPL, just print it the console from your app

or type cljs.core._STAR_clojurescript_version_STAR_ in the browser console

Ah that worked easily enough. "1.10.439"

ok, how do you use the generators. using clojure.spec.test.alpha?

Currently using these

ok, I imagine you want to call gen/sample to just see some generated data right?

correct

you’ll need a require to [clojure.test.check.generators] in your ns as well

Ahhh that did it.

Thank you very much

actually just [clojure.test.check] should work too

since it requires all the other needed namespaces

Makes sense. That indeed works as well.

Thanks again 🙂 Works well now

Enjoy 🙂

reposting my notes on spec-related content in the 'Maybe Not' talk (https://www.youtube.com/watch?v=YR5WdGrpoug) since the thread is buried. 1. separating out schema (shape) and selection (optionality). cool stuff! - it's still not "just data". why the departure? everything in Clojure is just data, and for good reason. the entire core lib is built around manipulating data. it's the best part of clj. - it's still place oriented, at least from what i gather so far 2. Better programmatic manipulation of specs. great! but again: why not make it data? then programmatic manipulation comes with. Imagine if the schema for:

{:weather {:weatherbitio {:key "xoxb" :default {:zip "98104"}}}
   :command {:prefix "!"}}

{:weather {:weatherbitio {:key string? :default {:zip string?}}}
   :command {:prefix string?}}

but string? is code, not data

the idea of predicates is core to spec, i dont see how spec could ever be jsut data

data for edges, predicates for leaf nodes

the symbol string? is data

the form (s/coll-of string?) is also data

that’s thin

it’s not thin, this is the whole basis of lisp

homoiconicity is about macros

If you call describe it does return a list of symbols at runtime, so spec isn't just confined to macros. Admittedly it could be hard to do anything with more complex predicates

that’s just … not true

for string? to have meaning we’d have to syntax quote it for starters, and now we’re in the clojure reader, not the edn reader

and spec forms will be even more central in spec 2

string? has meaning in the context of a namespace

so now we hve (ns) forms and are very solidly in the clojure reader, not edn reader

that’s how i udnerstand it anyway, always looking to deepen 🙂

if you have fully qualified symbols, then the edn reader is sufficient

for example, s/form always give you that

we are also moving towards a world where spec forms are a data form, but may not be the only data form

so some “map” data form could also exist

but that’s not more or less data than the list form (it may be easier to perform some kinds of transformations on)

yo’re saying (s/coll-of ...) is sugar for something deeper?

no

I’m saying that (s/coll-of string?) is data. Some map syntax {:op coll-of :pred string?} (not a real syntax) is also data. both are sufficient to describe a spec.

I don't think this matters. the power and problem is the predicates. predicates can only be data if you know what all of them mean (independently of an implementation--a well-known list of predicates)

yes, that is essential in the design of spec which from day 1 was predicate based

and that is what people mean when they say "specs are not data"

i accept that a qualified symbol is equivalent to a qualified keyword – but not a closure or fn reference, only symbols

> the idea of predicates is core to spec, i dont see how spec could ever be jsut data

that's I think what you mean @dustingetz?

certainly how I understood it

in my own thinking about the "dataness" of spec

As soon as someone puts #(and x y) as a spec pred, it’s all over, no logner data

well, that’s false and we support that now

how can you serialize that

(fn [&] ...)

its a reference

it could be jvm interop there

or you mean x and y as closed over values?

if so, then agreed

right, it could be anything

although we have some thoughts on how to allow you to do that as well

now that has my interest piqued …

if x and y are vars you’re invoking

if it’s values, then you’re in the realm of custom spec creation (which also is now more codified and easier to do)

well, maybe not that much easier, but I think we’re ready to commit to what that means

I would still say that pred references are data if it’s possible to re-establish their meaning remotely (in the context of namespaces) and var serialization / hydration is something that we’ve been slowly working on for the last year or so. some of it’s in clojure already.

Aren’t you going to run up against the halting problem

(= #(even? (inc (inc %)))
   #(even? (+ 2 %)))

we may ultimately end up going beyond var quote to define something new, but we’re still just thinking about those parts

why do you need to do that?

we’re never comparing specs for equality

because you said it’s data, so i want to do data things, otherwise doesn’t that defeat the point

this is too theoretical. what problem are you trying to solve?

(= `string? `string?)

why are you even doing that?

Well for example if specs are data then i want to store them efficiently in a database

you’re inventing problems that imho don’t exist

it is essential to hyperfiddle that datomic schema is actually data and stored in a database

we match on :db.type/string, for example, to decide what widget to draw

well you’ll be glad we’ve scoped it to just data then - symbols, sets, and forms combining them

lol

inline fns are no longer allowed?

inline fns are forms

well, looking forward to seeing whatever you come up with

....

work proceeding in https://github.com/clojure/spec-alpha2 although I have not pushed up in quite a while

debating how bad it would be to push wip stuff there, but maybe I’ll just put it on a branch

spec is data if you have eval and a full clojure runtime and possibly other required nses got it

Like for example, imagine a big honking match, like this but huge, and it tested the specs too. If specs are data that should be possible and could be very interesting.

well specs are an open protocol, so no match is big enough to cover custom extensions

could be multimethod, you get the idea

but sure

specs have a form, keyed by the op in first position (and that’s how resolution occurs in the spec 2)

what is the difference between form and data?

none

data can be understood without evaluation

that can be true for a well-defined subset of spec

as we’re talking forms above, I mean combinations of lists, symbols, keywords, and sets

> data can be understood without evaluation well, it depends, because your brain is the evaluator which "understands", and knowing format is essentially :require in brain (if POV is a person)

it depends what you mean by “understood”

arguably, string? can also be understood w/o evaluation, even w/o namespace. so yeah, it depends on definition of "understood"

yes, most derivative uses of spec rely on some implementation-independent meaning for the predicate symbols

but you need to limit that set, or else have some way to mark useful traits about a predicate; otherwise you can't "understand" it without getting a clojure runtime to run it

more specific you (want to) get – more it'll depend on actual implementation behind the symbols

that's what I mean by a well defined subset

it starts to sound like static types argument. There is value in being able to use anything clojure allows to construct a predicate. But if you want it to be understandable over the wire – use subset of what is allowed. Instead of having spec forbid e.g. java interop, because it would not be understandable in 0.0000001% cases

if you want use specs with the spec api calls (valid?, conform, explain, etc), then yes, you need implementations for all predicates mentioned in the specs

this has been explicit in the design of spec from the beginning

none of this is new or really any different

no one is arguing it's not been explicit, or even that it is bad or wrong

it's just annoying to hear that called "data"

that’s baffling to me

as this is no different than the whole “code as data” / macros thing that is part of Clojure, which presumably you are also familiar with

what is "data" to you? @favila

data makes possible a whole new universe of systems in 20 years that we barely even have the words to talk about today (thanks to Clojure) – what seems pedantic today is pretty important in hindsight, i think

just because parts of the data are linked to semantics, does not mean it’s not data

[:find ?e :where [?e :dustingetz.reg/email]]
(d/q '[:find ?e :where [?e :dustingetz.reg/email]])

One of these can be used to generate UI or compile to SQL, the other can’t

both is edn data, no? vectors, lists, keywords, symbols

well this has been fun, but I’m going to bow out here and go work on the code

Thanks Alex

I think that there is a valid desire to have specs be represented in a form that doesn't require a Clojure interpreter in order to use

we'd probably prefer to have a simpler DSL, using a subset of EDN, that would enable easier parsing and consuming

for example, how do you know what ?e is in 1st line? you need to provide "this is variable placeholder to be bound because datomic" context somehow somewhere. if you don't - this is exactly the same as 2nd line, but in a list, not vector

Once it is coupled to Clojure, it is an order of magnitude harder to build a data driven system on top of it

XML is data too

the trick is to tease out, what is the minimum set of semantics required to express a spec?

we can also turn the knob on that by limiting the ease and power of specs

e.g. right now, specs use predicates that you can easily inline. what if we disallowed that (either by convention or some other tricky mechanism)?

that would make it easier to serialize, but potentially less powerful and harder to write

"every predicate must have a qualified name"

Or just say “hey this thing is programmable, have at it”, that’s fine too, but the halfway world is pretty weird and i think leads to fragility

wonder if free monads and the interpreter pattern is useful to consider in this context. spec as a pure data AST.

(defn make-sql [data] data)
=> #'user/make-sql

(-> "[:find ?e :where [?e :dustingetz.reg/email]]"  (edn/read-string)  (make-sql))
=> [:find ?e :where [?e :dustingetz.reg/email]]

(-> "(d/q '[:find ?e :where [?e :dustingetz.reg/email]])"  (edn/read-string)  (nth 2)   (make-sql))
=> [:find ?e :where [?e :dustingetz.reg/email]]

or we could have a convention that says, any predicate must not close over any vars

> One of these can be used to generate UI or compile to SQL, the other can’t see above

(d/q (->> [:find '?e] (concat [:where ['?e :dustingetz.reg/email]]))) i can play that game all day

it has extra step, yes, but it does not make it not data

well, how do you know the 1st one can be converted to sql?

by looking at it first? you interpret while you look. your brain is repl

brain of someone w/o knowing what edn is will not be able to tell that 1st can be converted and second cant

it's a repl w/o edn/reader required yet

I don't think data = edn

yes, it is less convenient, and requires extra step. if this is the argument - I agree, there is no limit to convenience

if data is just "anything you can quote" then what is code?

that's why I asked https://clojurians.slack.com/archives/C1B1BB2Q3/p1547588438496700

when people say "this is data" they are saying it has a constellation of useful properties that are strictly less powerful than a programming language

what does it mean? in this spec conversation context, and ... in general?

e.g., being able to understand and interpret it multiple ways; draw inferences from it that are not stated explicitly; compare it to other things for equivalence; build different interpreters on top of it

you somehow need to parse/read/interpret data to "draw inferences"

I'm having flashbacks to this conversation between Rich Hickey and Alan Kay a few years ago: https://news.ycombinator.com/item?id=11945722

LOL

yes @misha, you do. You need to assign meaning to things without implementing them

this is possible for spec until you hit a predicate

otherwise how do you know what is in the string? how do you know that this is map: {:a 1}, etc.

if you have a well-known list of predicate symbols, then you can hard-code that meaning in

you have to have some context, it might be in your brain, but you need to communicate it to the system "which might not be using clojure, etc."

that's how e.g. https://github.com/arohner/spectrum can work

but once you have a custom predicate, it's meaning cannot be shared without sharing it as code

I don't think it even needs to be well-known

there's no data language that can express a new predicate

spec has parts of its language that can express new specs

Data is also secure, imagine a REST service that returned Clojure code

you could receive a spec, and then read all of the symbols from that spec to see what needs to be implemented

e.g. s/or s/and etc

but below predicates, you cannot know anything unless you have independent knowledge of its meaning

I think, that "assigning meaning w/o implementation" is a very blurry line

even executing it you can only know what it means for a specific value you give to it

hardcode that meaning where?

you cannot make any universal inferences about the behavior of a predicate without knowing its meaning

because that is just "global state", and you need to make sure system you are sending it to - is in the same context

Datomic’s :db.type/string is not ambiguous or blurry

@misha having to send it to the same runtime environment to use it sounds an awful lot like a property of code rather than data

show it to java dev and it would be pretty ambiguous to him

The success of HTML over desktop gui toolkits is also because it is data

You can ship HTML over the wire to arbitrary platform, and somehow they manage to interpret it usefully

well, then yours "well defined symbols we could just hardcode" just means "we have implementation :ruquired here in clojure and there in python, so dont worry about it"

which smells like transit

otherwise, define "well defined" :)

You can render arbitrary HTML fearlessly, you aren’t worried about getting hacked

"well defined" just means we know the list ahead of time

I know what "clojure.core/string?" means

I don't know what "foo.core/blah?" means

If I had the code I could possibly execute foo.core/blah?

html just has very small set of tokens

You do however know what github/markdown? is, you know who the maintainer is, you know where the docs are and where you might find useful code to operate with it

maybe another angle: spec can express things that are not in spec

i.e. the predicates

well, html makes no sense to my mom, neither does english, so can I say it is not data then? ¯\(ツ)/¯

Yet somehow I manage to pay my bills

her brain is a system, and it does not support html. mine does.

@favila you know the list of clojure code ahead of time too. it is just way too large.

the same way we know the set of real numbers

will foo.core/blah? return true for a real number?

one difference would be - you cant eyeball it and say "yep, it's fine, we are not getting hacked"

will the next system interpret 1 as true or int?

sometimes you have to reject invalid data if you don't know how to parse it, too

or you convert it to a less meaningful form, e.g. clojure tagged-value

or, a fully qualified predicate symbol

but then the power the spec gained by being able to use any predicate it wants is lost by any other system that wants to make sense of the spec

by making s/valid? easier something else has been made harder

@dustingetz if stuff on github - is data, code is data in exactly same way, so we are on the square 1 again

this is going to sound :male-astronaut: but: it becomes just like any other part of our system, where we need to keep a single source of truth for the semantics of that data

e.g. maybe in the future we'll be building "spec-services"

(I'm thinking of building a "spec-service")

that just sounds hard to do given predicates

@lilactown What is a spec service?

well, because serializing the spec itself requires a lot of context, what I'm thinking of doing is just having, literally, spec-as-a-service

so like an xmlns

GET valid/foo.core/blah => (s/valid? :foo.core/blah (:body req))

ah cool

for instance, we have a lot of disparate services that need to validate similar sets of data according to the same rules

and they're not all Clojure 😞

Yes, e.g. imagine internet protocols like HTTP had a formal spec written instead of English, in clojure.spec, and thus it becomes much easier to, well im not sure what i would want to think about it

It is a bit like HATEOAS

in fact it is HATEOAS

I retract that, but it is really interesting to imagine

If spec is clojure code, it gets a little harder to start throwing them around on wires though, because how can you trust if it is safe to clojure.core/eval

But there is still a trust chain, so its probably fine

But you’re saying a service, so that bypasses the trust issue

right. just like in clojure, you have a single, global spec registry

your system should have the same: a single, global registry of what your specs are

yeah that's a neat idea

pretty scary to run that server

hopefully no one manages to get a bitcoin mining predicate on there

how to make it fast?

that's a lot of data to push over the wire

(uploads of clojure data to s/valid?)

Right, the huge difference between Datomic schema and Clojure specs today, is that you can recklessly throw them around. For example it’s easier to deploy them into a validated healthcare enterprise environment

Slurp in some EDN updates, not that big a deal

Slurp in some clojure specs to pass to eval – yeah right

I can imagine a dialect of clojure/core that does not have infinite recursion, and no unsafe operators like !, that might be safe enough

You can also make the requestor pay for compute

yeah, easier to tag things with specs and send the data you want validated along with the names of specs you want it to be validated against

Here is a cool application of specs as data: [:select.ui (select-keys props [:value :class :style :on-change :read-only :on-click])] aka https://github.com/facebook/prop-types

it funny how

// An object that could be one of many types
  optionalUnion: PropTypes.oneOfType([
    PropTypes.string,
    PropTypes.number,
    PropTypes.instanceOf(Message)
  ]), ...

(d/q '[:find ?e :where [?e :dustingetz.reg/email]])