Fork me on GitHub
#clojure-spec
<
2019-06-21
>
Ben Hammond07:06:49

following on from yesterday; I think the tidiest way to achieve > generator uses id from a database, > when there is a database is to use the overrides feature of (spec/gen)

(doc spec/gen)
-------------------------
clojure.spec.alpha/gen
([spec] [spec overrides])
  Given a spec, returns the generator for it, or throws if none can
  be constructed. Optionally an overrides map can be provided which
  should map spec names or paths (vectors of keywords) to no-arg
  generator-creating fns. These will be used instead of the generators at those
  names/paths. Note that parent generator (in the spec or overrides
  map) will supersede those of any subtrees. A generator for a regex
  op must always return a sequential collection (i.e. a generator for
  s/? should return either an empty sequence/vector or a
  sequence/vector with one item in it)
=> nil

djtango11:06:03

thanks for this btw - am enjoying your writeup! Shame it'll be gone whenever the history runs out

๐Ÿ˜ 4
vemv15:06:20

I was thinking that a spec/not could make sense as exemplified here

(spec/def ::version (fn [s]
                      ...))

(spec/def ::alpha-version (spec/and ::version
                                    (fn []
                                      ...)))

(spec/def ::stable-version (spec/and ::version
                                     (spec/not ::alpha-version))) ;; <- imaginary API
has this been considered already?

vemv15:06:33

(I guess the generative part would suffer)

Alex Miller (Clojure team)15:06:33

I've written a ton of specs and have never needed an s/not

vemv15:06:52

I've also written a fair number of specs for a few years. Just now I think of not, so that kind of proof is limited

Alex Miller (Clojure team)15:06:03

you could do something like (spec/def ::stable-version #(spec/invalid? (spec/valid? ::alpha-version %)))

Alex Miller (Clojure team)15:06:30

it is probably inherently difficult to auto-gen

๐Ÿ‘ 4
Alex Miller (Clojure team)15:06:10

in any case, we have no plans to add it

vemv15:06:54

:thumbsup: this was more curiosity than anything else. One could always write his own not anyway

Ben Hammond15:06:43

I've got quite an interesting situation; I have a spec that generates a fairly complex data structure I sometimes want to overide some of the generators to make it use foreign keys from the database but it only works intermittently: I have hooked up a snitching ILookup to tell me what is happening around

(if-let [g (c/or (when-let [gfn (c/or (get overrides (c/or (spec-name spec) spec))
                                          (get overrides path))]
and what I see is that When a recompile that spec directly to REPL, then the overridy will wok When I recompile the entire namespace to REPL, it does not work Which is intriguing behavour

Ben Hammond15:06:12

so this my ILookup

(reify ILookup
        (valAt [_ k]
          (println (str "valAt*1: " k))
          (println (str "==>" (get m k)))
          (get m k))

        (valAt [_ k nf]
          (println (str "valAt*2: " k ":" nf))
          (println (str "==>" (get m k nf)))
          (get m k nf))
        )

Ben Hammond15:06:08

when it works, I see

valAt*1: :db.generators.offers/offer
==>
valAt*1: []
==>
valAt*1: :db.generators.offers/offer_headline
==>
valAt*1: [:offer_headline]
...

Ben Hammond15:06:37

when it does not work, I see

valAt*1: :db.generators.offers/offer
==>
valAt*1: []
==>

Ben Hammond15:06:12

is this ringing any bells?

Alex Miller (Clojure team)15:06:24

specs compile in their dependent specs so if you modify a spec, you need to reload any specs that depend on it. that's a likely reason you'd see different results for the two cases

Alex Miller (Clojure team)15:06:43

I'd expect "recompile the entire namespace" to give you the more accurate answer.

Alex Miller (Clojure team)15:06:25

hard for me to tell from this what the actual problem is though

Ben Hammond15:06:39

perhaps I have misunderstood usage. I looked at

(doc spec/gen)
-------------------------
clojure.spec.alpha/gen
([spec] [spec overrides])
and hoped that I could plug in some overrides that would get me proper foreign keys, and it would just work

Ben Hammond15:06:21

would you expect to re generate all of the specs to handle a overrides map?

Alex Miller (Clojure team)15:06:24

you should be able to plug in overrides that way. I was responding to

When a recompile that spec directly to REPL, then the overridy will wok
When I recompile the entire namespace to REPL, it does not work
which seemed like a pretty textbook outcome from spec compilation

Alex Miller (Clojure team)15:06:37

you haven't actually shown what you're doing, so I can't really comment

Alex Miller (Clojure team)15:06:43

can you give a full example?

Ben Hammond15:06:40

(spec/def ::offer
  (spec/keys
    :req-un [::offer_headline
             ::offer_classifiers
             ::offer_title
             ::offer_title_short
             ::offer_merchant
             ::offer_voucher_codes
             ::offer_description
             ::offer_terms_and_conditions
             ::offer_terms_and_conditions_url
             ::offer_claim_restrictions
             ::offer_presentation]
    :opt-un [::offer_pre_claim_advice
             ::offer_key_terms
             ::offer_redemption_guidelines
             ::offer_taxable_value
             ::offer_identifiers
             ::offer_images
             ::offer_approval_required
             ::offer_discount_mechanic
             ::claim_condition_msisdn_list]))


(spec/def ::offer_merchant ::ingestion/merchant_id)
and then I have an ingestion namespace that says
(s/def ::merchant_id (s/and string? #(<= 1 (count %) 64)))
is the main offer spec

Ben Hammond15:06:35

so I'm a bit suspicious about that (spec/def ::offer_merchant

Ben Hammond15:06:03

and now I want to generate a bunch off offers where the merchant ids have come out of the database

Ben Hammond15:06:55

so I write

(:offer_merchant
 (test.gen/generate
   (spec/gen :db.generators.offers/offer
     (let [m {:customer.ingestion/merchant_id (constantly (test.gen/return "OVERRIDEa"))
              :db.generators.offers/offer_merchant (constantly (test.gen/return "OVERRIDEb"))
              :offer_merchant (constantly (test.gen/return "OVERRIDEc"))}]
       (reify ILookup
         (valAt [_ k]
           (println (str "valAt*1: " k))
           (println (str "==>" (get m k)))
           (get m k))

         (valAt [_ k nf]
           (println (str "valAt*2: " k ":" nf))
           (println (str "==>" (get m k nf)))
           (get m k nf))
         )))))

Alex Miller (Clojure team)15:06:06

there is a known issue with specifying generator overrides on spec aliases

Alex Miller (Clojure team)15:06:14

in that, it doesn't work

Ben Hammond15:06:42

okay. that's a simple explanation

Ben Hammond15:06:51

is there a workaround?

Ben Hammond15:06:00

should I copy-and-paste it?

Alex Miller (Clojure team)15:06:58

I think it should work if you specify it on the aliased spec :customer.ingestion/merchant_id, but seems like you are?

Alex Miller (Clojure team)15:06:25

I would expect OVERRIDEb and OVERRIDEc to never work here

Ben Hammond15:06:54

so if I ttry to override :db.generators.offers/offer_title instead

Ben Hammond15:06:27

it still doesn'tt work though

Ben Hammond15:06:52

(:offer_title (test.gen/generate
                (spec/gen :db.generators.offers/offer
                  {:db.generators.offers/offer_title (constantly (test.gen/return "OVERRIDEb"))})))
=> "SC3T9wmvO54Q2TNx4"
`

Ben Hammond16:06:52

[clojure.test.check.generators :as test.gen]

Alex Miller (Clojure team)16:06:32

so test.check.generators expects something different than clojure.spec.gen.alpha - namely, the spec version takes generator thunks, whereas I think test.check just takes generators

Alex Miller (Clojure team)16:06:45

can you try it with spec's version?

Alex Miller (Clojure team)16:06:33

I'm not sure what test.check supports as far as override marking - I'm not sure it even knows about the attributes as that's all spec registry based

Ben Hammond16:06:11

well it looks like all logic is buried inside clojure.spec.alpha/gensub

Ben Hammond16:06:17

so perhaps I don't understand what you are asking. If I run

(:offer_title (spec.gen/generate
                (spec/gen :db.generators.offers/offer
                  {:db.generators.offers/offer_title (constantly (test.gen/return "OVERRIDEb"))})))
I get the same outcome as when ran test.gen/generate

Ben Hammond16:06:59

the good news is that I'm an idiot

Ben Hammond16:06:02

so what I'm doing to sabotage the overridee

Ben Hammond16:06:19

The declaring file goes like this

(ns db.generators.offers
...
(defmacro add-gen
  "update spec to add generator"
  [k g]
  `(spec/def ~k
    (spec/with-gen ~k
      (constantly ~g)) ))

(defmacro update-gen
  [k f]
  `(add-gen ~k (~f (clojure.spec.alpha/gen ~k))))
...
(spec/def ::offer
  (spec/keys
    :req-un [::offer_headline
...
(update-gen ::offer (fn [g] (test.gen/fmap add-msisdn-csv-bytes g)))
...

Ben Hammond16:06:01

so after the offer spec is declared I go and set its generator using with-gen to mix in an extra bit of functionality

Ben Hammond16:06:26

and the existence of this gfn means that

Ben Hammond16:06:18

clojure.spec.alpha/map-spec-impl sees

(gen* [_ overrides path rmap]
        (if gfn
          (gfn)
          (let [rmap (inck rmap id)
...

Ben Hammond16:06:44

and says >Ooh a gfn. My work here is done

Ben Hammond16:06:10

Thankyou for your help

Ben Hammond16:06:27

I'm not sure how I'm going to fix it but at least I understand it

Alex Miller (Clojure team)16:06:05

Some of this stuff really needs a rethink, itโ€™s pretty tricky

Ben Hammond16:06:48

well I thought the the next spec counts as a rethink?

Ben Hammond16:06:45

discovering where the tripwires are is tricky

Ben Hammond16:06:05

that's true of my entire time with Clojure

Ben Hammond16:06:34

you have to trip 'em to find 'em

Alex Miller (Clojure team)16:06:41

Yeah, we may get to it in spec 2

Ben Hammond16:06:25

right its the weekend in my timezone

Ben Hammond16:06:32

have a goodd weekend Alex

misha18:06:59

what is the difference between: (s/coll-of ... :kind vector?) and (s/coll-of ... :into []) ?

Alex Miller (Clojure team)19:06:43

kind adds a validation on the input

Alex Miller (Clojure team)19:06:50

into is about gen and conform (output)

Alex Miller (Clojure team)19:06:13

(but kind also impacts the starting coll for gen/conform)

misha19:06:59

so :into [] is "list/seq is fine, but generate me a vector"?

Alex Miller (Clojure team)20:06:21

And conform to vector

๐Ÿ‘ 4
misha19:06:00

thanks. had an impression, that :into is a younger replacement for :kind.

Alex Miller (Clojure team)20:06:53

No, they are different purposes

misha20:06:41

what is the best :ret spec for "predicaty" function? boolean? is insufficient because nil is falsey. and because everything but false/nil - is truthy. any?

Alex Miller (Clojure team)20:06:10

Depends on what the function returns

Alex Miller (Clojure team)20:06:48

(s/nilable boolean?) is useful sometimes

Alex Miller (Clojure team)20:06:04

Donโ€™t use sets for logically false values

misha20:06:23

function is supplied by user, and is used as predicate, but return value is not compared to true or false.

misha20:06:17

limiting it to :ret boolean? or :ret (s/nilable boolean?) would be... limiting

misha20:06:59

what is the best way to spec atom fn arg? #(instance clojure.lang.Atom %)? any??

misha21:06:04

it actually will be an atom, not just something dereferable. and fn might call swap! or reset! on this arg

Alex Miller (Clojure team)21:06:49

on the boolean question, I'd just not spec it then

Alex Miller (Clojure team)21:06:55

not spec the ret at all

Alex Miller (Clojure team)21:06:08

on atom, I'd do the first one

Alex Miller (Clojure team)21:06:23

or actually, use IAtom, the interface, not the concrete class

๐Ÿ‘ 4