Fork me on GitHub
#malli
<
2023-03-16
>
ikitommi07:03:14

another thread about :map + :map-of parsing 🧵

ikitommi07:03:34

Given:

(def schema 
  [:map {:registry {'int [:orn ['int :int]]
                    'str [:orn ['str :string]]}}
   [:id 'int]
   ["name" 'str]
   [::m/default [:map-of 'str 'str]]])

(def valid 
  {:id 1, "name" "tommi", "kikka" "kukka", "abba" "jabba"})
which would be better: • 1️⃣ - merge the parse results into a single map
(m/parse schema valid)
;{:id [int 1]
; "name" [str "tommi"]
; [str "kikka"] [str "kukka"]
; [str "abba"] [str "jabba"]}
2️⃣ - keep the ::m/default parse results separately
(m/parse schema valid)
;{:id [int 1]
; "name" [str "tommi"]
; :malli.core/default {[str "kikka"] [str "kukka"]
;                      [str "abba"] [str "jabba"]}}

ikitommi07:03:59

defaulting to 2️⃣ here, without that, one has to reparse the parse-results to find out which entries are from the default. As a side-effect, this disallows :malli.core/default key in the map-instances, but that’s ok IMO.

ikitommi08:03:26

yeah, this is good, thanks duckie:

(def schema
  [:map
   [:id :int]
   ["age" :int]
   [::m/default [:map-of :string :string]]])

(m/parse schema {:id 1, "age" 13, "kikka" "kukka"})
;{:id 1, "age" 13
; :malli.core/default {"kikka" "kukka"}}

opqdonut08:03:53

I'd have expected 1️⃣

opqdonut08:03:18

I think mostly your processing code would "know" which keys are the defaults

opqdonut08:03:09

could you get something like 2️⃣ using :and if you really want it?

ikitommi08:03:09

:thinking_face:

ikitommi07:03:19

@US1LTFF6D - 1️⃣ has issues with key collisions, so we have to use 2️⃣. Given:

(def schema
  [:map {:registry {'str [:orn ['str :string]]}}
   [['str "kikka"] :any]
   [::m/default [:map-of 'str 'str]]])

(def value {['str "kikka"] true, "kikka" "kukka"})
with 1️⃣ :
(m/parse schema value)
; => {[str "kikka"] [str "kukka"]}

(m/unparse schema *1)
; => ::m/invalid
with 2️⃣ :
(m/parse schema value)
; => {[str "kikka"] true, :malli.core/default {[str "kikka"] [str "kukka"]}}

(m/unparse schema *1)
; => {[str "kikka"] true, "kikka" "kukka"}
… root cause is that the parse result tuples (returned via miu/-tagged) are implemented as MapEntries, which have equality to a vector of size two and thus can collide with user defined vector keys. Not common, but possible.

opqdonut07:03:52

Aren't collisions like that a problem in other places as well?

user=> (m/parse [:or [:orn [:s :string]] [:vector any?]] "s")
[:s "s"]
user=> (m/parse [:or [:orn [:s :string]] [:vector any?]] [:s "s"])
[:s "s"]

opqdonut07:03:47

user=> (m/parse [:map-of [:or [:orn [:s :string]] [:vector any?]] :int] {[:s "s"] 1 "s" 2})
{[:s "s"] 2}

ikitommi07:03:16

oh, true that. the first one works thou:

(def schema [:or [:orn [:s :string]] [:vector any?]])

(m/parse schema "s") ; => [:s "s"]
(m/unparse schema *1) ; => "s"

(m/parse schema [:s "s"]) ; => [:s "s"]
(m/unparse schema *1) ; => [:s "s"]

ikitommi07:03:23

.. but the second doesn’t, because they are pushed as keys into the map:

(= (miu/-tagged :s "s") [:s "s"]) ; => true

ikitommi07:03:26

so, the guideline should be “if you plan to use vector keys in maps, use tag names that don’t clash with your domain data”, e.g.

(m/parse
 [:map-of [:or [:orn [`s :string]] [:vector any?]] :int]
 {[:s "s"] 1 "s" 2})
; => {[:s "s"] 1, [malli.core-test/s "s"] 2}

opqdonut07:03:00

I'm reviewing the PR now

👍 2
ikitommi14:03:05

::m/default fallback works recursively:

(def schema
  [:map
   [:x :int]
   [::m/default [:map
                 [:y :int]
                 [::m/default [:map
                               [:z :int]
                               [::m/default [:map-of {:gen/max 4} :int :int]]]]]]])

(json-schema/transform schema)
;{:type "object",
; :additionalProperties {:type "integer"},
; :properties {:x {:type "integer"}
;              :y {:type "integer"}
;              :z {:type "integer"}},
; :required [:x :y :z]}

(m/explain schema {:x 1, :y "2", :z 3, 42 false})
;{:schema ...,
; :value {:x 1, :y "2", :z 3, 42 false},
; :errors ({:path [:malli.core/default :y]
;           :in [:y]
;           :schema :int
;           :value "2"}
;          {:path [:malli.core/default :malli.core/default :malli.core/default 1]
;           :in [42]
;           :schema :int
;           :value false})}

(mg/generate schema)
; => {:x -1, :y 423, :z -1130868, 41388000 -28671, -3869 41788, 38 -20324}

🤯 12
awesome 4
🚀 2
👏 2
cap10morgan18:03:51

it appears that you can't attach decode fns to map keys in a :map schema. is that correct? if so, is there another way to do that? I have been doing [:and [:map-of ::key-schema-with-decode :any] [:map [:specific-key ::val-schema]]] but it's kind of clunky and I'm hoping there's a more idiomatic way to do it.

ikitommi19:03:08

You can attach decode fns to map keys

cap10morgan19:03:31

oh great! I must have been holding it wrong then 🙂

cap10morgan19:03:02

I'm testing against your latest commit in https://github.com/metosin/malli/pull/871 and it is working great for my use case!

ikitommi19:03:53

good to hear, should be ready to review tomorrow.

👍 2
cap10morgan20:03:53

hmm... I can't get a decode fn on a :map key to work. it needs to decode the key not the value. is that supported?

cap10morgan20:03:38

I have this: [:map ["@context" {:optional true, :decode/edn my/decode-fn} ::context]]

cap10morgan20:03:11

my/decode-fn just checks for :context and returns "@context" instead

cap10morgan20:03:32

so I'm trying to get maps like {:context ...} to be decoded into {"@context" ...} by that schema w/ a transformer named :edn

cap10morgan20:03:20

I attached the decoder at the higher [:map {:decode/edn ...} ...] and checked for :context in the map itself instead and that works. so I'm good 👍

opqdonut06:03:08

Yeah I think the key-munging happens on the :map level. Let me check some old code of mine.

opqdonut06:03:47

(defn key-renamer
  "A transformer that renames keys of input maps based on the `column`
  properties in the schema. If no `column` property is found,
  `default-key-fn` is used (defaults to `identity`)."
  ([column]
   (key-renamer column identity))
  ([column default-key-fn]
   (mt/transformer {:name :key-renamer
                    :decoders {:map {:compile (fn [schema _]
                                                (let [mapping (key-mapping schema column)]
                                                  (mt/-transform-map-keys #(get mapping % (default-key-fn %)))))}}})))

opqdonut06:03:59

That's what I did for a similar case

opqdonut06:03:51

(deftest test-key-renamer
  (let [schema (m/schema [:map
                          [:our-a {:column "a"} :int]
                          [:our-b {:column "b"} :int]
                          ["c" :int]
                          [:our-nested {:column "nested"}
                           [:map
                            [:our-a2 {:column "a"} :int]
                            [:our-b {:column "b2"} :int]]]])]

    (is (= {:our-a 1
            :our-b 2
            :c 3
            :unknown 4
            :our-nested {:our-a2 5
                         :our-b 6}}
           (m/decode schema
                     {"a" 1
                      "b" 2
                      "c" 3
                      "unknown" 4
                      "nested" {"a" 5
                                "b2" 6}}
                     (domain/key-renamer :column keyword))))))