Fork me on GitHub
#clojure-spec
<
2020-06-04
>
nmkip02:06:17

I have a map that has this shape {"account" {"active" true "balance" 30}} , what's the best way to spec the "account" key? I've tried these and I'm using the first one.

(s/def ::account-key (s/with-gen (s/and string?
                                        #(= "account" %))
                       #(gen/return "account"))) 
(s/def ::account-key (s/with-gen (s/and string?
                                        #{"account"})
                       #(gen/return "account"))) 

seancorfield02:06:59

You can use #{"account"} to spec a known set of string values.

seancorfield02:06:16

(and that will also generate correctly)

nmkip02:06:35

just #{"account"} that makes sense

seancorfield02:06:54

(s/def ::account-key #{"account"})

nmkip02:06:30

great, thanks! 🙂 I'll use that instead, much shorter

nmkip02:06:25

I was being redundant here:

(s/def ::account-key (s/with-gen (s/and string?
                                        #{"account"})
                       #(gen/return "account"))) 

4
seancorfield02:06:00

Inventive... but certainly "overkill" 🙂

dev.4openID13:06:48

Learner. Maybe a dumb question. I have built up a detailed spec using s/def s/valid? s/conform. It works fine. Following best practices all my keywords use :: (e.g. ::timestamp). However, my data covered JSON to map, all keywords are single colon (e.g. :timestamps) When I construct the data with two colons the valid? and conform has no problem whereas the mapped data with single colon fails. I was understanding the :: indicates the keyword in this namesapace so having :: in my spec vs. the single : in may data should not matter. Any clarifications on this? :thinking_face: <script src="https://gist.github.com/dev4openid/1becc5be61b764e330edc82d16f99eb6.js"></script>

aisamu13:06:40

The conversion from JSON will probably give you unqualified keywords, since no such concept exists there. You have to either qualify your keywords on a separate post-processing step, or make a spec for the payload that takes unqualified keys (e.g. :req-un instead of :req)

dev.4openID13:06:06

@U3UFFB420 Thanks for the explanation. But..... why is the :timestamp not found in the namespace or is it global? If global than how do I relate it to my incoming data - as it were. Because the spec is not of use right now. Obviously I do not see the connection

dev.4openID13:06:05

So that is the purpose of req-un then? Does that make the :keyword global?

frankitox13:06:56

Maybe you are mixing concepts. Keywords are just keywords, they can be qualified (`:test/timestamp`) or unqualified (`:timestamp`). :: It's a shortcut to write a qualified keyword with the same namespace as the current file.

aisamu13:06:30

Keywords can be qualified or unqualified. Unqualified can be thought of as "global", but it's better to stick with "lacking a namespace". When you type in ::timestamp, Clojure uses the current namespace and creates a qualified keyword. req-un means that the spec will match unqualified keys to the qualified keys, comparing just the name (i.e. discarding the namespace)

dev.4openID13:06:47

So in a more complex map structure, where I have several collections do I have to req-un for all definitions or only at the top level and it is inferred?

dev.4openID13:06:17

'Cos I req-un all my defs and now it does not work

aisamu13:06:05

Yes, you'd have to req-un all of them. Try breaking it into smaller pieces to find where it's failing

dev.4openID13:06:37

It is detailled

aisamu13:06:12

The spec for ::loca seems a bit off

dev.4openID13:06:08

seems to work.... what do you mean?

dev.4openID13:06:08

If it is all :req for keywords exampl2 works fine and conforms. So I am confused here?

aisamu13:06:10

Unless this is spec2, I don't think this is a valid definition: (s/def ::loca (s/keys :req-un {::addr [::country]}))

aisamu13:06:53

IIRC, req & req-un take a vector of keys, not a map

aisamu13:06:42

Might have worked by accident, or because you had a correct definition earlier that was stored in the registry

dev.4openID13:06:00

would (s/def ::loca :req {::addr [::country]}) do it? no it does not

dev.4openID13:06:55

Surely the country spec forces a s/keys?

aisamu13:06:06

I've never seen that syntax (feeding a map to :req). Does it work on a simple case?

dev.4openID13:06:40

2 rows above and it works -> (s/def ::addr (s/keys :req [::country])) (s/conform ::addr {::country "USA"}) ;; => #:myproj.trial.spec{:country "USA"}

dev.4openID13:06:59

It is effectively a map of map, no?

dev.4openID13:06:33

and :loca is the map of the above. But if I do not have s/keys it seems now not to work

aisamu13:06:29

These two are not the same:

(s/def ::addr (s/keys :req-un [::country]))
(s/def ::loca (s/keys :req-un {::addr [::country]}))
You're feeding a vector to the first req, but a map to the second

dev.4openID13:06:42

Agreed but I tried ::loca {:: addr {::country and it will not accept it

dev.4openID14:06:02

@U1UQEM078 BTW what do you mean as spec2? I am using clojure.spec.alpha latest version

dev.4openID15:06:45

@U3UFFB420 I agree with you, how does one relate the one to the other? I changed my spec to have :reg-un at the highest level and it fails spec is at <script src="https://gist.github.com/dev4openid/1becc5be61b764e330edc82d16f99eb6.js"></script>

frankitox15:06:46

Try first working with simpler examples, try to create a spec for a map with unqualified keys

dev.4openID15:06:35

If you look at the gist that Is what I am doing? No?

dev.4openID15:06:11

When all the keys are NOT reg-un then all my test cases pass

dev.4openID15:06:41

and so my spec seems OK ??

dev.4openID15:06:24

OK how? I know it is indicating a map but if I do not use s/keys it fails

dev.4openID15:06:55

https://app.slack.com/team/UEGT2541J  [2:57 PM] Agreed but I tried ::loca {:: addr {::country   and it will not accept

dev.4openID15:06:18

Even though I initially figured it wasa map of map of map

dev.4openID15:06:51

Oh oh!!!! I see it now!

frankitox15:06:25

I see several places you use :req-un instead of :req, which is the correct way for conforming/validating these examples

frankitox15:06:43

As I said, try creating a simpler spec for a non nested map and then try validating maps

dev.4openID15:06:52

Ok. But how do I use the spec against the data map (ex JSON) which is :keyword and not ::timeline (IYKWIM)

frankitox15:06:26

Here's an example for a qualified map

(ns myproj.trial.spec)

(s/def ::country string?)
(s/def ::street string?)

(s/def ::addr (s/keys :req [::country ::street]))

(s/valid? ::addr {::country "USA"
                  ::street "A street"})
;; => true

(s/valid? ::addr {:country "USA"
                  :street "A street"})
;; => false

(s/valid? ::addr {:myproj.trial.spec/country "USA"
                  :myproj.trial.spec/street "A street"})
;; => true

dev.4openID15:06:30

yes, without changing the above, how do you apply it to a map ex JSON) {:addr {:country "USA :street "A street"}}

frankitox15:06:42

As aisamu said before, if you are working with JSON you loose the namespace of the keywords, so you probably want to use :req-un in your specs.

(ns myproj.trial.spec)

(s/def ::country string?)
(s/def ::street string?)

(s/def ::addr (s/keys :req-un [::country ::street]))

(s/valid? ::addr {:country "USA"
                  :street "A street"})
;; => true

dev.4openID15:06:02

I was advise to use :req-un to allow for unqualif namesapces

frankitox15:06:18

Right, but then when you use valid? or conform you also have to use unqualified keywords for the map https://gist.github.com/dev4openid/1becc5be61b764e330edc82d16f99eb6#file-myproj-trial-spc-clj-L69-L81

dev.4openID15:06:08

(s/def ::country string?) (s/def ::addr (s/keys :req-un [::country])) (s/def ::loca (s/keys :req-un [::addr ::country ])) (s/conform ::country "USA") ;; => "USA" (s/conform ::addr {:country "USA"}) ;; => {:country "USA"} (s/conform ::loca {:addr {:country "USA"}}) ;; => #:myproj.trial.spec{:addr #:myproj.trial.spec{:country "USA"}} (s/explain-str ::loca {:addr {:country "USA"}}) ;; => "{:addr {:country \"USA\"}} - failed: (contains? % trial.spec/addr) spec: :trial.spec/loca\n{:addr {:country \"USA\"}} - failed: (contains? % trail.spec/country) spec: :parceltrax.tracking.spec/loca\n" ??why this??

dev.4openID15:06:32

I updated the gist <script src="https://gist.github.com/dev4openid/1becc5be61b764e330edc82d16f99eb6.js"></script> If you look at line 16 When I look sat it it is a map of map

frankitox15:06:53

Try conforming this

(s/conform ::loca {:addr {:country "USA"}
                   :country "OTHER"})

dev.4openID15:06:52

Yes, in theory that will work, however the data looks like this :pieceID ["ABC1" "DEF2" "GHI3"]}]}) (def exampl2 {::abc [{::loca {::addr {::country "USA"}} ::dets {::some{::timestamp "2020-04-09T15:27:00" ::Url "https://mywebpage.com/answer"

dev.4openID15:06:02

Look at line 1 of the gist

dev.4openID15:06:29

so loca is a map of map of map

dev.4openID15:06:54

The trouble I a,mm having is the compounded map that does not seems to play nicely

dev.4openID16:06:37

OK - it does conform to the definition Thx.  However it is not as per line 1 - which is how the data is meant to be.   So extracting from line 1:  :loca {:addr {:country "USA"}}

frankitox16:06:19

Yes, you need to modify ::loca so it looks like this (s/def ::loca (s/keys :req-un [::addr]))

dev.4openID16:06:32

@U3UFFB420 OK !!!! I think I have it ! I must take out the ::country in the s/def as the ::country is already defined (s/def ::loca (s/keys :req-un [::addr]))

dev.4openID16:06:54

I think that is what you were trying to get me to see

frankitox16:06:19

Yes, exactly! 🙂

dev.4openID16:06:50

@U3UFFB420 Thanks for your patience and help. I got it now!! my complete script now works! Kudos for you 🙂

🎉 3
frankitox13:06:38

The difference is that :: expands to the current namespace. So if you're working in a ns called my.test then ::timestamp expands to :my.test/timestamp. The spec fails with :timestamp because it's also expecting the namespace.

dev-hartmann19:06:09

Hi fellow clojurians, is anyone aware of a way to parse a swagger json to valid specs that can be used to generate test data?

elimisteve10:06:33

That's a great idea and something I'm interested in, too

dev.4openID11:06:01

This is what I did to learn spec. From this point on one can gen data. What you see is a map converted from a JSON structure. Now this is not a "take a swagger spec and convert" but is the basis of how to clojure spec based on a swagger that I did.

(def exampl {:abc [{:loca {:addr {:country "USA"}}
                    :dets {:some{:timestamp "2020-04-09T15:27:00"
                                 :Url ""
                                 :prod {:name "this product"}}}
                    :totalNumber 399
                    :pieceID ["ABC1" "DEF2" "GHI3"]}]})


(s/def ::country string?)
(s/def ::addr (s/keys :req-un [::country]))
(s/def ::loca (s/keys :req-un [::addr]))

(s/conform ::country "USA") ;; => "USA"
(s/conform ::addr {:country "USA"}) ;; => {:country "USA"}
(s/conform ::loca  {:addr {:country "USA"}}) ;; => {:addr {:country "USA"}}
(s/valid? ::loca  {:addr {:country "USA"}})  ;; => true
(s/check-asserts?)
(s/check-asserts true)                       ;; by default false!!
(s/assert ::loca  {:addr {:country "USA"}})  ;; => {:addr {:country "USA"}}
(s/assert ::loca  {:addr {:countryx "SA"}})  ;; note the exception error BUT look at the repl

(s/def ::timestamp (and not nil?
                        string?))     ;; this needs work for T-time issue
(s/def ::Url string?)                 ;; need to validate URLs - technique
(s/def ::name string?)
(s/def ::prod (s/keys :req-un [::name]))
(s/def ::some (s/keys :req-un [::timestamp ::Url ::prod]))
(s/def ::dets (s/keys :req-un [::some]))

(s/conform ::timestamp "2020-04-09T15:27:00")
(s/conform ::Url "")
(s/conform ::name "this product")
(s/conform ::prod {:name "this product"})
(s/conform ::some {:timestamp "2020-04-09T15:27:00"
                   :Url ""
                   :prod {:name "this product"}})
(s/conform ::dets {:some {:timestamp "2020-04-09T15:27:00"
                           :Url ""
                           :prod {:name "this product"}}})

(s/def ::totalNumber (and pos? int?))
(s/def ::pieceID (and (s/coll-of string?)
                      vector?))
(s/conform ::totalNumber 399) ;; => 399
(s/conform ::pieceID ["ABC1" "DEF2"]) ;; => ["ABC1" "DEF2"]

(s/def ::abc (s/coll-of (s/keys :req-un [::loca ::dets ::totalNumber ::pieceID])))
(s/conform ::abc [{:loca {:addr {:country "USA"}}
                   :dets {:some{:timestamp "2020-04-09T15:27:00"
                                :Url ""
                                :prod {:name "this product"}}}
                   :totalNumber 399
                   :pieceID ["ABC1" "DEF2" "GHI3"]}])

;; now ex is an map of a single instance)
ex
;; => {:abc [{:loca {:addr {:country "USA"}}, :dets {:some {:timestamp "2020-04-09T15:27:00", :Url "", :prod {:name "this product"}}}, :totalNumber 399, :pieceID ["ABC1" "DEF2" "GHI3"]}]}

(s/def ::an_instance (s/keys :req-un [::abc]))
(s/conform ::an_instance {:abc [{:loca {:addr {:country "USA"}}
                                 :dets {:some{:timestamp "2020-04-09T15:27:00"
                                              :Url ""
                                              :prod {:name "this product"}}}
                                 :totalNumber 399
                                 :pieceID ["ABC1" "DEF2" "GHI3"]}]})

(s/conform ::an_instance exampl)
(s/valid? ::an_instance exampl)
(s/explain-str ::an_instance exampl)
Also look at https://www.youtube.com/watch?v=f2hNQdS2VxQ to watch a conversion - the nearest I have found. Good luck with this 😉

dev-hartmann14:06:52

hey @UEGT2541J thanks for joining in. actually the challenge is not to create a clojure map from the json and then writing the spec. but parsing the json into specs . the difference being that I don't want to write them manually. but you are absolutely right, the chain is: json -> clojure-map -> transform clojure map entries to spec entries and compose them to upper level spec. then we can use the gen functions to create test data

dev-hartmann14:06:53

if we have a working parser for that, we can just call the endpoint on webservices, slurp the swagger json and create specs for data models

dev-hartmann14:06:11

which is a handy little tool 😄

dev-hartmann14:06:24

I'm not an expert for specs, but searching a swagger yaml/ json for the definitions

dev-hartmann14:06:17

and then using the type information there and maping those and the field names to specs sounds doable 😄

dev-hartmann14:06:21

definitions:
  Order:
    type: "object"
    properties:
      id:
        type: "integer"
        format: "int64"
      petId:
        type: "integer"
        format: "int64"
      quantity:
        type: "integer"
        format: "int32"
      shipDate:
        type: "string"
        format: "date-time"
      status:
        type: "string"
        description: "Order Status"
        enum:
        - "placed"
        - "approved"
        - "delivered"
      complete:
        type: "boolean"
        default: false
    xml:
      name: "Order"

dev-hartmann14:06:51

i like structured data 😄

dev-hartmann14:06:40

I'll work on that a bit an will put a link to a gist here if I can create something useful

flyboarder17:06:15

@U0J8XN37Y im really interested in this, we just rolled out swagger to our API’s both internally and externally, I would love to be able to have our apps handle specs for us

dev-hartmann17:06:50

will keep you posted too.

dev-hartmann17:06:03

I think it's easier than I thought.

dev-hartmann17:06:17

.. I mean until I hit the brick wall 😄

dev-hartmann17:06:18

at least the basics. translating the conditions on properties can take some time 😄

dev.4openID20:06:34

OK, similar to what you are stsing then: look st the library serene - It autoigenerates the specs for REST and graphql APIs.

dev.4openID20:06:43

I'll shut up now 🙂

dev-hartmann22:06:27

no, please don't. never heard of that library and it looks like they are doing the same thing, but for a different format. :thumbsup: