Fork me on GitHub
#clojure-spec
<
2018-03-19
>
mgrbyte14:03:50

I have 3 specs describing a possible "identity" for an entity. I then have an "identity" spec along the lines of (s/def ::identity (s/or :domain/id1 ::id1 :domain/id2 ::id2 :domain/id3 ::id3). one of the specs can't use the built-in generator, how would I go about providing a custom generator that works for s/or? I've managed to write some custom generators before for maps, s/and specs, but can't get my head around how to start writing one for this s/or case, and google hasn't yielded anything so far. Does anyone have any pointers pls?

mgrbyte14:03:14

I'm guessing I need to figure out how to use a combination of gen/tuple and gen/one-of :thinking_face:

bbrinck15:03:11

@mgrbyte Can you call with-gen inside the or?

bbrinck15:03:15

(require '[cljs.spec.alpha :as s])
  (require '[clojure.test.check])
  (s/def ::id1 string?)
  (s/def ::id2 pos-int?)
  (s/def ::identity (s/or :id1 ::id1 :id2 ::id2))

  (map first (s/exercise ::identity)) ;; => (1 2 "" 2 "" 2 "F91Q6p" "o6" "oet" "DE7")

  (s/def ::identity (s/or :id1 (s/with-gen ::id1 (fn [] (s/gen #{"foo" "bar"})))
                          :id2 ::id2))

  (map first (s/exercise ::identity)) ;; => (1 "bar" 2 2 2 15 31 "bar" "bar" 75)

bbrinck15:03:48

Would that work for your case?

mgrbyte16:03:49

@bbrinck Trying to keep generators in tests only, since I'm using minter.strgen/test.check stuff to help make the gen for "::id1" . I've managed to get somewhere near what I need with (gen/one-of [gen1 gen2 gen3]), but as mentioned in the spec guide, need it's a good idea to generate s/conform like example for any s/or like specs: "For r specs that have a conformed value different than the original value (anything using s/or, s/cat, s/alt, etc) it can be useful to see a set of generated samples plus the result of conforming that sample data."

mgrbyte16:03:37

So I'll need to interleave the specs and generators. but think that will do it

bbrinck16:03:44

@mgrbyte I haven’t tried it out, but I believe you can use (require '[clojure.spec.gen.alpha :as gen]) and that will be OK in dev and test

bbrinck16:03:20

(it lazy loads test.check when generator is called, not defined, but presumably you won’t be invoking generators outside of tests)

bbrinck16:03:31

Might simplify your implementation, although I don’t know if you want to keep generators in test for dependency reasons, or for other reasons like attaching different generators at different tiems

ikitommi16:03:34

Hi. Did someone make a stab at spec bijections? Any lessons learned? Bumped into a actual need: need to uncoerce path-parameters in reverse-routing into string based either on the value type or on the defined spec.

ikitommi16:03:03

With spec, I guess we could put both the encode & decode functions into spec metadata, something like:

(st/spec
  {:spec keyword?
   :description "a bijecting keyword"
   ::encode/string #(name %2)
   ::decode/string #(keyword %2)})

gfredericks16:03:42

st/spec? does that exist?

gfredericks16:03:13

I've been talking about bijections a lot; I was thinking of setting up a generic library for composing bijections, that's not spec-specific

gfredericks16:03:50

if you have ideas about how to tie them with spec, that would be interesting

ikitommi16:03:11

st/spec is from spec-tools, until spec meta-data appears.

gfredericks16:03:24

that's a 3rd-party library?

ikitommi17:03:45

does already the encode/coercion, could do the other way too.

gfredericks17:03:26

my starting point is assuming that the most common use for bijections would be where you have some internal canonical representation of something, and you want to specify how to transform it to/from other "contexts", which could be things like json, jdbc, cvs, etc.

ikitommi17:03:31

Generic library/foundation would be nice, as this is needed with Schema too.

gfredericks17:03:32

so when deciding how to integrate that with spec, one question that would come up would be whether it's worth automating the process of transforming the specs themselves so that you get context-specific versons of the specs, or merely to automate the process of writing the functions that take values in each direction

ikitommi17:03:34

yes, the name of the (`string` in ::encode/string) is the context name.

gfredericks17:03:49

the downside of actually generating specs for each context is that it's more complicated and probably more work for the programmer the upside is I think you can get more features, like validating closer to the edge, and giving validation errors based on that representation rather than the internal one

ikitommi17:03:35

I tried the context-versions of spec, but coudn’t get it working.

ikitommi17:03:56

failed with the qualified keys with s/keys.

gfredericks17:03:30

s/keys is a big barrier; especially if you want to convert the keys to strings like in a json context

ikitommi17:03:55

(s/def ::id keyword?)
(s/keys :req [::id])

ikitommi17:03:30

if you have that, you can’t have a different ::id as the full key names are exposed outwards.

ikitommi17:03:31

why would the validation be closer to the edge?

gfredericks17:03:35

yeah you'd need a new namespace for each context probably

ikitommi17:03:19

… or a different registry.

gfredericks17:03:21

e.g., if I write a public JSON API and biject my internal spec to a JSON version of that spec, I can theoretically validate the user input using the json spec, before trying to transform/coerce the data to the internal representation

gfredericks17:03:49

you can also then provide validation errors to the user in a way that's specific to the json representation, rather than the less helpful internal representation

gfredericks17:03:17

here's what I came up with a few months ago; I did a proof of concept of integrating it with some code at work https://gist.github.com/gfredericks/358c39478f104281a6364f446f5b3c6b

ikitommi17:03:37

oh, true that. Would mean multiple walks on the same model, but in most cases, the perf impact would not matter.

ikitommi17:03:25

nice draft!

ikitommi17:03:44

are there bijections in other languages?

ikitommi17:03:04

could read some theory/reasoning to understand this.

gfredericks17:03:15

I've not seen it, other than I've speculated you could typecheck them in a dependently typed language

gfredericks17:03:04

I've also wondered about separating the bijective parts from surjective parts (the times when you want to drop information, e.g., the order of keys in a query string)

gfredericks17:03:31

if the surjections are formalized, you can use them to create generators of all the variant representations of something

stathissideris18:03:27

super-interesting stuff, I think I should do some reading on this

ikitommi18:03:32

thanks for the pointer! In my test, the transforming functions take two arguments: the spec and the value. This allows the transformation to extract information from the spec to do the transformation.

ikitommi18:03:19

for example, stripping out non-defined keys from keys-spec needs the set of keys defined:

(defn strip-extra-keys [{:keys [keys]} x]
  (if (and keys (map? x))
    (select-keys x keys)
    x))

gfredericks18:03:12

I'd always approached that by building the function at compile-time

gfredericks18:03:45

I should also point out this old effort in case it hasn't been seen: https://github.com/gfredericks/schema-bijections

gfredericks18:03:05

I used that for real code at my old job, and it worked pretty good but made the code difficult to read

gfredericks19:03:07

Validating before converting isn't any extra walks compared to validating after converting I don't think

Drew Verlee16:03:59

Knowing full well this doesn’t make sense. How would you write a spec for a collection of maps where each maps keys were 1) unique 2) represents a value themselves e.g [data, data, data, …] where data is something like {5 6}

Alex Miller (Clojure team)17:03:42

well you would start with (s/coll-of (s/map-of int? int?))

Alex Miller (Clojure team)17:03:22

each individual maps keys are (by definition) unique, but I assume you mean all of the keys over all the maps are unique

Alex Miller (Clojure team)17:03:49

for checking the uniqueness constraint, s/and another predicate that checks whatever property you need - something like #(apply distinct? (mapcat keys %))

Drew Verlee17:03:36

> each individual maps keys are (by definition) unique, but I assume you mean all of the keys over all the maps are unique correct

ghadi18:03:16

you can't have duplicate keys in a map @drewverlee

ghadi18:03:02

oh I didn't see the collection of maps requirement. yeah, s/and is your friend here

roklenarcic18:03:22

I'm trying to write my own Spec protocol implementation. Is there somewhere where Spec and Specize protocols' contracts are explained?

Alex Miller (Clojure team)19:03:17

no, because they are likely to change

misha19:03:51

@mgrbyte you can override generator just for ::id1 during exercise time

(s/def ::id1 string?)
(s/def ::id2 int?)
(s/def ::id (s/or :id1 ::id1 :id2 ::id2))
(map first (s/exercise ::id 10 {::id1 #(s/gen #{"foo"})}))
;;=>  (0 -1 1 "foo" 0 -7 "foo" -1 "foo" 0)

borkdude19:03:08

I have:

(s/def ::key
  (s/and keyword?
         #(= "widget"
             (namespace %))))

(s/fdef widget
        :args
        (s/cat :opts
               (s/keys
                :req-un [::title
                         ::content]
                :opt-un [::key
                         ::icon
                         ::widget-class
                         ::settings
                         ::collapsed?
                         ::dropdowns
                         ::controls
                         ::preview?
                         ::tabs
                         ::collapse-opts
                         ::help])))
but ::key does not seem to get checked when it’s provided

taylor19:03:54

this is working for me locally, w/a dummy, instrumented widget function

taylor19:03:01

In: [0 :key] val: :foo fails spec: : at: [:args :opts :key] predicate: (= "widget" (namespace %))

borkdude19:03:16

hmm yeah… for me too when I isolate this

borkdude19:03:30

I guess some reloading glitch

cch121:03:58

Is there an idiomatic way of expressing a spec for “anything” other than (constantly true)?

cch121:03:35

I’m trying to define a spec for a value that must ultimately be usable as a key in an associative structure.

cch121:03:57

In Clojure, I think that is pretty much everything.

bronsa21:03:54

any? is what you want

camachom23:03:02

I'm trying to pass a vector to s/keys, but it keeps failing. Works great if I just write it out though. Something like this:

(def required
  [:foo/city :foo/country :foo/street])
(spec/def :foo/form (spec/keys :req required
                               :opt []))

camachom23:03:23

any ideas as to what i'm doing wrong?

alexisvincent23:03:30

Hi guys, got a bit of a modelling problem. How do you build up specs with context dependent information. E.g. Say you’re building a space for card details, say a map of the keys :card/number :card/expiry :card/name :card/cvv. Now we want this to be a nice composable spec called :card/details. But in some contexts :card/cvv is required and in others it isnt. So The :card/details spec is really a partial spec with requiredness of :card/cvv dependent of context. We might then say make two spec’s :card/details and :card/full-details. But this isn’t really scalable. It means whenever we have branching up the composition tree we need to rename things for all cases, giving an exponential explosion. Is there any way to build ‘partial’ specs where specs higher up the composition tree provide more or overriding data for specs they depend on. Similar to dependent types.