Fork me on GitHub
#malli
<
2023-03-15
>
ikitommi08:03:54

Could take a stab at merging :map and :map-of (https://github.com/metosin/malli/issues/43), best options so far: Given data:

{:id #uuid"839a35ba-1be1-43d1-af31-0c8cff06690b"
 123 "kikka"
 456 "kukka"} 
Schema to describe it could be: 1️⃣ ::m/default would validate just the extra keys as a map
[:map
 [:id :uuid]
 [::m/default [:map-of :int :string]]]
2️⃣ ::m/default would validate all extra key-value pairs via :tuple
[:map
 [:id :uuid]
 [::m/default [:tuple :int :string]]]
thoughs?

1️⃣ 8
dharrigan09:03:19

I think option #1 is clearer to read.

juhoteperi10:03:47

What about just [::m/default [:int :string]]? Just "special case" like other :map values. But OK, I see the value that it is a real Malli schema.

dharrigan10:03:23

I think it would be nice to make it explicit that it's either a tuple of values or a map-of values, rather than relying upon special "knowledge" that [:int :string] means a map. However, I'm have not that strong a feeling, and thus I can see the value in just [:int :string] too 🙂

juhoteperi10:03:18

I think option 2 is strange. Because the extra values are some number of those tuples, which :map-of describes. If we want to match regular :map keys drop :tuple.

Ben Sless10:03:54

1 is clearer

ikitommi19:03:33

About :map + :map-of + mt/strip-extra-keys-transformer🧵

ikitommi19:03:53

by default stripping works like this:

(m/decode
 [:map
  [:x :boolean]
  [:y :int]]
 {:x 1, "kikka" "kukka", 3 4}
 (mt/strip-extra-keys-transformer))
; => {:x 1}

ikitommi19:03:37

disabled for explixitely open maps:

(m/decode
 [:map {:closed false}
  [:x :boolean]
  [:y :int]]
 {:x 1, "kikka" "kukka", 3 4}
 (mt/strip-extra-keys-transformer))
; => {:x 1, "kikka" "kukka", 3 4}

ikitommi19:03:01

whatbout :map + ::m/default?

(m/decode
 [:map
  [:x :boolean]
  [:y :int]
  [::m/default [:map-of :int :int]]]
 {:x 1, "kikka" "kukka", 3 4}
 (mt/strip-extra-keys-transformer))
; 1) => {:x 1, "kikka" "kukka", 3 4}
; 2) => {:x 1, 3 4}

ikitommi20:03:07

I think: • 1️⃣ is more consistent with current behavior (and easier to implement) • 2️⃣ sounds what I would expect

escherize20:03:34

"kikka" "kukka" is an extra entry, so I think it should be stripped. so 2️⃣ seems more right

opqdonut06:03:35

if I had an API that had ::m/defaults, I'd be pretty peeved if they got stripped

opqdonut06:03:54

perhaps we need a separate strip-extra-keys-transformer and strip-unknown-keys-transformer

opqdonut06:03:17

I've always kinda understood strip-extra-keys as meaning strip-unknown-keys

ikitommi07:03:52

One can run apply the mt/strip-extra-keys-transformer in two phases: 1. before the ::m/default values have been transformed, e.g. we don’t know the validity of extra keys/values. With this, I would like to keep all extra keys 2. after ::m/default values have been transformed -> we could validate all (non-explicitly defined key-values) and strip away everything that doesn’t match the schema

ikitommi07:03:27

same numbers (1 & 2) as before, just with more context. For example, in current reitit, the mt/strip-extra-keys-transformer is applied BEFORE the normal transformers, so 2 would not work => the childs are not transformed and thus, all the entries could be invalid.

ikitommi07:03:02

I think this could be handled in mt/strip-extra-keys-transformer: 1. check if the schema has ::m/default key 2. If it has, mount a :leave transformer that validates all the extra key-values and strips away the invalid ones.

ikitommi07:03:52

boils down to what does “unknown” keys mean, ping @US1LTFF6D.

opqdonut08:03:58

yeah exactly

opqdonut08:03:47

> after ::m/default values have been transformed -> we could validate all (non-explicitly defined key-values) and strip away everything that doesn’t match the schema I think this is the right way to go

👍 2
ikitommi10:03:32

that is non-trivial to solve. Given schema:

[:map
 [:x :int]
 [::m/default [:map-of :int :int]]]
For value:
{:x 1, 1 1, 2 2, "3" "3", "4" "4"}
… we can trip the explicit keys easily so it’s:
{1 1, 2 2, "3" "3", "4" "4"}
… but how do we know which entries are valid? 1. validate them 1-by-1 {1 1}, {2 2}, … and merge all valid submaps together. This kinda works but as one can provide any schema as ::m/default, this might not always correct, e.g. given [::m/default [:map-of {:max 2} :int :int]] 2. validate everything once => doesn’t work 3. try all combinations => bad idea ideas welcome, @US1LTFF6D.

opqdonut10:03:40

dissoc the "known" keys, validate the rest using the default schema

opqdonut10:03:47

so it would fail with {:max 2}

opqdonut11:03:57

I'm not sure I see the problem. What were you thinking of?

ikitommi11:03:35

for extra keys:

{1 1, 2 2, "3" "3", "4" "4"}
correct answer after stripping the invalid keys would be:
{1 1, 2 2}
only way to do that I can think of is the 1) - validate key-values 1-by-1 and merge all valid ones together. This should work for many/most of the cases. but not for all.

ikitommi11:03:15

maybe this is just something to document, “best effort for ::m/default keys”

ikitommi11:03:32

and a option to disable it.

opqdonut11:03:37

I would've expected

(m/decode [:map-of :int :int] {1 1, 2 2, "three" "3"} (mt/strip-extra-keys-transformer))
to be {1 1, 2 2}

opqdonut11:03:48

but it seems strip-extra-keys-transformer doesn't do anything for map-of?

ikitommi11:03:45

currently, it does not, noticed the same, I’ll make it work too.

opqdonut11:03:19

yeah, well after that works, can't you just reuse that behaviour for ::m/default ?

opqdonut11:03:01

so (m/decode [:map [:x :int] [::m/default def]] val) would delegate to (m/decode def (dissoc v :x))

👍 2
ikitommi07:03:11

the -transformer of :map already covered the delegation, so just needed to add the :map-of stripping and it works. I’m surprised how well this played out!

(m/decode
 [:map
  [:x :int]
  [::m/default [:map
                [:y :int]
                [::m/default [:map-of :int :int]]]]]
 {:x 1, :y 2, :z 3, 1 1, "2" 2, 3 "3", "4" "4"}
 (mt/strip-extra-keys-transformer))
; => {:x 1, :y 2, 1 1}
https://github.com/metosin/malli/pull/871/commits/a1bb82f03e6e30a2d4b05f0404abdac8e2f58e90

ikitommi07:03:05

… and contructed a :tuple schema of the :map-of so we can iterate over key-values without stuffing those into a map. less garbage and don’t have to worry about :map-of level constraints like :min and :max properties.

opqdonut07:03:36

yeah it's pretty nice