Fork me on GitHub
#clojure-spec
<
2016-12-10
>
gfredericks00:12:09

@bbloom I can't really spec test.check generators meaningfully without parameterized specs

bbloom00:12:51

@gfredericks 1) i assume that there’s existing code out there for which that’s true 2) i’d love to hear why and 3) i’m curious if you’d do things differently now to avoid that

gfredericks00:12:35

@bbloom what's a good generator for the function (defn vector "Returns a generator of vectors of items from g" [g] ...)?

bbloom00:12:15

i haven’t a clue

gfredericks00:12:20

right now I just have the args as (s/cat :g generator?) and the ret as generator?, which doesn't remotely capture the relationship between the two

gfredericks00:12:26

or even the fact that the return value generates a vector

gfredericks00:12:48

so there's two aspects to this

bbloom00:12:50

well at least now we can join the Go community in complaining that our “type system” doesn’t have generics 🙂

bbloom00:12:10

that’s interesting

gfredericks00:12:36

I can't imagine what would be done differently to avoid this

bbloom00:12:40

i’d be really curious to hear rich’s thoughts on this

bbloom00:12:52

b/c i can’t imagine he hadn’t considered it

gfredericks00:12:10

any comment I've seen about higher-order-ish things is usually along the lines of "specs are for data"

bbloom00:12:53

sure, the problem comes in once you introduce contextual specs

gfredericks00:12:03

a corollary of which I have to imagine is "specs aren't useful for things like generators", but I couldn't defend that and haven't heard it explicitly taken that farb

bbloom00:12:04

which happens instantly once you have functions

gfredericks00:12:25

yeah you can't spec map completely for the same reason

gfredericks00:12:42

so I guess you're right that they must have thought about it :)

bbloom00:12:47

yeah, i think it’s perfectly OK to only spec data “at rest” so to speak

bbloom00:12:13

in theory it could be extended to spec more things, but you’re not going to serialize a generator to edn

gfredericks00:12:40

you ultimately can't, because of bind

bbloom00:12:44

at least not from opaque closures

gfredericks00:12:58

even fmap I guess

bbloom00:12:36

i guess “spec doesn’t cover that use case” is fine - i’m just curious if that’s now & more stuff could be spec’d later with more powerful primitives, or if it’s like “eh, this approach only goes so far & you can spec some other way later if you need to"

bbloom00:12:00

like in theory you could reuse specs in another context, which i guess is what Typed Clojure will do: generate types from specs

bbloom00:12:06

or otherwise treat specs as types

bbloom00:12:39

for me, i’ve basically only bothered to spec things that i intend to pprint

bbloom00:12:07

basically: “hey spec, save me the trouble of looking at this thing and checking to make sure it looks right"

gfredericks00:12:58

so a parameterized spec is a function that takes a spec and returns a spec

gfredericks00:12:09

presumably you could make a spec for that function

gfredericks00:12:15

and test it by generating specs?

gfredericks00:12:19

this is really hazy

gfredericks00:12:45

haskell uses => to talk about this stuff doesn't it?

bbloom00:12:02

=> is used to constrain this stuff

bbloom00:12:30

the => basically means to unify things on the left with the environment

gfredericks00:12:43

oh I guess the parameterization is more basic

gfredericks00:12:00

listOfThings :: [a]

gfredericks00:12:05

is already parameterized

bbloom00:12:18

so like if you have some type a => b -> c that says you have some function of type b, returning c, and there exists some a… and you can constrain it like: Num a -> b c there exists some a such that a is tagged with the class Num

bbloom00:12:40

yeah, you don’t need to talk about constraints to get to parameters

gfredericks00:12:45

does that mean anything interesting if a doesn't appear on the right side of the =>?

bbloom00:12:58

you could have listof Things :: Num a => [a] if my syntax is correct

gfredericks00:12:12

listOfNumberishThings

bbloom00:12:25

i believe that the it’s required to be on the right side at least in haskell 87 w/o extensions

bbloom00:12:40

i used unique letters for the sake of clarity talking about parts of it

bbloom00:12:45

also, i suck at haskell, so don’t ask me 🙂

gfredericks00:12:23

hey @bbloom I think I got my lens stuck in a monad transformer can you help.

bbloom00:12:36

you’re going to have to amputate.

bbloom00:12:18

but yeah, spec obviously has specs parameterized by specs, like every

gfredericks00:12:31

are those all special?

bbloom00:12:34

or cat, or alt

bbloom00:12:03

the issue is that named specs can’t have parameters

gfredericks00:12:18

I've wondered if (generator integer?) could be a thing

bbloom00:12:20

there’s no s/fn

gfredericks00:12:40

presumably it could be implemented with the lower-level machinery

gfredericks00:12:56

even then you couldn't fully describe gen/vector

bbloom00:12:02

like in theory you could have (s/fn [val] (s/every-kv integer? val))

gfredericks00:12:19

this is the "function from specs to specs" idea

bbloom00:12:42

however, this gets in to an area where i feel the current tradition of type systems is real bad

bbloom00:12:22

positional parameters force you to provide all the values when you call it

bbloom00:12:47

types are curried in haskell (i think), but still, you don’t actually get a type until you give it all the args

gfredericks00:12:11

where does that get awkward?

bbloom00:12:27

b/c once you add a parameterized type, it’s like a virus

bbloom00:12:40

everything above it needs to be parameterized until you know a concrete type

bbloom00:12:45

same problem with monads 😉

bbloom01:12:01

or checked exceptions

bbloom01:12:11

"oh shit, this function has an effect. ooooh well, let’s change the signature of every function everywhere"

bbloom01:12:13

if i were designing a type system with generics, i’d make all generic parameters named and optional

bbloom01:12:56

so you’d have (List {}) and (List {:element String}) - in the former case, the default for :element would be Any

gfredericks01:12:56

so "the type system knows about effects" seems like a positive thing, it's just the "and I have to type it in all sorts of unrelated places" that's the bad part, right?

gfredericks01:12:03

would a more advanced inference system fix that?

bbloom01:12:05

or whatever List declared it to be

bbloom01:12:18

yeah, that’s the bad part

bbloom01:12:38

more advanced inference would help, but the badness comes from the syntax

bbloom01:12:49

haskell avoids this problem a little bit by using currying

bbloom01:12:19

if you have some function that has an effect, you just leave the monad as the last parameter & then if you change the monad, that’s fine b/c you’re using curring / point-free to avoid mentioning the monad type as much as possible

gfredericks01:12:41

the monad is the last constraint mentioned before the =>? or it's actually a physical argument?

bbloom01:12:01

the order ofthings left of the => don’t matter, as far as i know

bbloom01:12:12

the syntax is super confusing

bbloom01:12:51

the monad would be an actual argument

bbloom01:12:09

but you can leave it out of the signature or just say “hey, i return an a"

gfredericks01:12:15

is that a general thing you deal with with monad transformers?

bbloom01:12:45

monad transformers are a whole ‘nother bag of goofy ideas 🙂

bbloom01:12:15

a lot of these problems stem from the fact that they don’t have associative or open structures in their type systems

bbloom01:12:20

if you want to combine more than 1 monad without manually creating a custom monad, you can layer them and the order in which you layer them matters b/c the type of a monad stack is essentially a cons list of monad types

bbloom01:12:45

the "extensible effects" stuff comes up with a clever way to encode a set without order in to the type

bbloom01:12:19

GHC now has some kind of union types with some extension, but at some people they were like sorting types to represent a multi-set or some craziness to achieve “order independence”

bbloom01:12:33

but that still failed to address openness properly

gfredericks01:12:23

okay so static types aren't all good yet.

bbloom01:12:08

everybody in the ML/Haskell camp AND the Lisp camps were afraid to take a step back and say “hey, what happens if we have maps and sets” 🙂

cemerick14:12:33

what is the current best idea for how to spec a map with (un-namespaced) symbol keys? The best I've figured is to use every to spec map entries, not very satisfying.

gfredericks16:12:10

@cemerick I don't know of anything besides an opaque function

gfredericks16:12:26

I haven't investigated how to use the lower-level facilities to build something more first-class

cemerick16:12:06

@gfredericks having something equivalent to s/keys (s/syms? :-P) doesn't look hard, but way more work than manually :syms destructuring and conforming each part

gfredericks16:12:39

I'm more likely to need strings, personally

cemerick16:12:04

and bob needs numbers, and poor jane is dealing with maps that have namespaced and un-namespaced keywords with the same name carrying different stuff

gfredericks16:12:00

make a new system and call it spec2

cemerick16:12:05

your convo with @bbloom above is serendipitous. I was really trying to get spec to do way more than it's meant to last night, notwithstanding my cat problems.

gfredericks16:12:16

someone should build some sort of adapter to plumatic/schema for fallback needs

gfredericks16:12:56

teach spec that schemas are specs and teach schema that namespaced keywords are schemas

cemerick16:12:54

well, part of my frustration is in wanting to get useful generators out of specs, so, things like schema aren't super useful

gfredericks16:12:07

schema has generator support too

gfredericks16:12:14

it's just more wiring to do

cemerick16:12:59

insofar as I need to do extra work to get corresponding generators (esp. if schema/spec is a separate thing from the generator), it's not super useful

cemerick16:12:37

I mean, beyond doing the work to have a validating/destructuring spec and corresponding generator

gfredericks16:12:17

I'm not sure what extra work you're thinking of; I think the overhead to get to generators is comparable in the two systems

cemerick16:12:06

I'm not sure either, I guess. It's been a while since I touched schema.

cemerick16:12:52

my recollection was that I ended up bailing and hand-writing my own generators at a certain level of complexity

gfredericks16:12:02

the big unfortunality is that (every version I've used) has bad errors when there's a missing generator you need to fill in

gfredericks16:12:19

it just says "Missing leaf generator" or something of that sort and doesn't attempt to tell you which schema caused the problem

gfredericks16:12:44

but once you build up your stockholm syndrome around that issue then it goes pretty well

cemerick16:12:29

this callous is my own, there is no other like it

bbloom17:12:33

@cemerick i too suffer from instantly running in to edge cases and limits when attempting to use new things 🙂

bbloom17:12:10

gfredricks as well - we’re just those guys who instantly find the dark corners

cemerick18:12:04

I don't know that I find dark corners any more frequently or quickly than others. Probably just louder than most about it, maybe

cemerick18:12:38

@bbloom the filesystem expression problem tweet was deeply triggering

bbloom18:12:47

¯\(ツ)

cemerick18:12:51

mmm, guess that's OT here

Alex Miller (Clojure team)19:12:28

@cemerick Re your question above, did you look at s/keys with :req-un ?

Alex Miller (Clojure team)19:12:40

@gfredricks for string keys, you can use a leading conformer that converts to keywords as a workaround

Alex Miller (Clojure team)19:12:32

Assuming you have an information map with keys that are strings ala json

gfredericks20:12:01

@alexmiller was the s/keys with :req-un the same as your other two comments, or about something different?

gfredericks20:12:09

I didn't know about the leading conformer idea, so that's useful

Alex Miller (Clojure team)20:12:57

It was about the original question of maps with unqualified keys

Alex Miller (Clojure team)21:12:06

you can do lots of tricks with leading conformers, like for example calling select-keys to narrow the key set you’re validating

Alex Miller (Clojure team)21:12:17

yada yada beware of conformers yada yada

gfredericks21:12:11

@alexmiller unqualified keys are pretty basic, right? chas had been asking about symbols as keys, which I assume is the same approach as strings

Alex Miller (Clojure team)21:12:47

oh, right I heard unqualified keys in my head

Alex Miller (Clojure team)21:12:08

but you could use a leading conformer for symbol->keys too

bbloom21:12:17

may i ask: why are you using strings or symbols heterogeneously?

bbloom21:12:36

besides maybe strings ~= keys when doing interop w/ java maps or javascript objects

gfredericks22:12:27

I like using strings when talking about javascript in clojure

gfredericks22:12:34

s/javascript/json/

gfredericks22:12:09

my rule of thumb is not using keywords that don't appear in the source code anywhere

gfredericks22:12:46

in the past I've used schemas to describe both the internal clojure-fluent version of data but also the external JSON-flavored version

gfredericks22:12:52

and I found that terribly useful

gfredericks22:12:42

spec does not fit that pattern nearly as well as p**matic/schema

seancorfield22:12:20

I’m curious why you don’t use keywordize at the boundary? I find that approach far more natural.

gfredericks22:12:43

I essentially do, but I prefer it in a less blind way

gfredericks22:12:18

to be clear, these are mostly just thoughts. I've only tried this in one codebase, but couldn't figure out how to get a nice API for it. most of my thoughts are here: https://github.com/gfredericks/schema-bijections#rationale

gfredericks22:12:56

so what I want is something like ↑ that library, but less confusing/verbose, and maybe involving spec if that ends up making sense

bbloom22:12:58

yeah, i definitely support using strings for json rather than keywordize everywhere

bbloom22:12:39

does raise some questions about how spec should work for json tho

gfredericks22:12:48

I suppose that readme doesn't talk about trying to separate bijections and surjections, which I never figured out how to do

gfredericks22:12:46

I wish bijections and surjections were first-class concepts

gfredericks22:12:17

they compose well; surjections are interesting for generators

gfredericks22:12:08

if I can ever work this into a better library it probably ought to be named jections

pedroiago23:12:14

is it intentional that s/? unforms a s/cat into a list? 'e.g. (s/unform (s/? (s/cat :a #{1})) {:a 1}) ;=> [(1)]'