Fork me on GitHub
#malli
<
2024-01-15
>
Tommi Martin13:01:19

Hello everyone, I'm trying to learn Malli and I was wondering. There are logical operators for the schema syntax such as [:or [:and [:=> Are there any detailed documentation available as to how they operate? I'm trying to write a schema for a messy data where i have a map such as

{:notification1 "stringValue"
 ;; several fixed keys from 1 to 24
 :notification24 "stringValue"
 ;; Following key is user defined in an external source.
 :<user-input> {;;homogenuous map of nodes
                node-1-<user-input> {...}
                node-2-<user-input> {...}}
}
Unfortunately I cannot modify the data structure at the source, only within my own application. Hence me trying to use Malli to transform it into a better format for my use. However I haven't been able to write a schema that successfully validates the random user input and the notification keys accurately. Does anyone have pointers where I could look to learn how to write that schema successfully on http://malli.io for example?

ikitommi14:01:13

Hi, something like this?

(require '[malli.generator :as mg])

(mg/sample
 [:map
  [:notification :string]
  [:notification24 :string]
  [::m/default [:map-of {:gen/max 2} :string [:map-of {:gen/max 4} :string :map]]]])
;({:notification "", :notification24 ""}
; {:notification "", :notification24 "2"}
; {:notification "8", :notification24 "5y", "" {}}
; {:notification "Q", :notification24 "DD", "" {"S8K" {}}}
; {:notification "5O38", :notification24 ""}
; {:notification "v4W2",
;  :notification24 "T",
;  "9J" {"8" {}, "4rLwM" {}, "" {}, "KoQZ" {}},
;  "0" {"kOro" {}, "Z" {}, "" {}}}
; {:notification "q6", :notification24 "", "2hxIz" {"e" {}}}
; {:notification "", :notification24 "", "0WVu" {}}
; {:notification "", :notification24 "egwdu"}
; {:notification "FBee", :notification24 "6UVm", "MjQ" {"M1Ig3" {}, "3O" {}, "a1t50QoD" {}}})

Tommi Martin13:01:04

This was precisely what I was looking for. I did manage to validate the incomind data using a modified version of your example. I'm a bit embarrased that I managed to miss / not understand how the m/default and map-of syntax worked from the readme.md. Thank you kindly for the answer

👍 2
Etzwane13:01:44

Hello This is my first post in this channel I am just starting out with Malli (in my first clojure project at work), and run into a simple problem I can not solve. I want to validate a map that kan have either one key or the other. I try to do it like this, but it does not seem to work:

(def registry (merge (m/default-schemas) (mu/schemas)))

Etzwane13:01:57

(m/schema
                                     "data"
                                     {:registry (merge registry {"dependency" [:map
                                                                               [:data map?]
                                                                               [:dependencies {:optional true} [:set [:ref "dependency"]]]
                                                                               [:type string?]
                                                                               [:update-path [:vector string?]]]

                                                                 "data"       [:union
                                                                               [:or
                                                                                [:map [:data map?]]
                                                                                [:map [:dependencies {:optional true} [:set [:ref "dependency"]]]]]
                                                                               [:map [:type {:optional true} string?]]]})})
So either :data or :dependencies should be there. I would expect something like:
[:map [:or [:foo string?] [:bar string?]]]
But that does not work either. The whole semantics of :union, :or & :merge are a bit unclear to me. Can someone give me a pointer?

Etzwane13:01:07

so this works:

(m/schema
             [:and
              [:map
               [:foo {:optional true} number?]
               [:bar {:optional true} number?]]
              [:fn
               {:error/message "either use :foo or :bar"}
               (fn [m] (or (some? (get m  :foo)) (some? (get m :bar))))]]

             {:registry registry})
but I doubt this is the idiomatic way to do this...

ikitommi15:01:40

currently, there isn’t more idiomatic way to do that. See https://github.com/metosin/malli/issues/474

ikitommi15:01:24

there is :multi that can be used, depending on the requirements.

ikitommi15:01:24

if someone can cook a good and simple syntax for defining the key requirements/dependencies, I'm all ears

Etzwane15:01:48

thanks for the reply. With my limited understanding so far, you can use :or either on the :map level

[:or 
  [:map [:foo number?]
  [:map [:bar number?]]
or on the value level
[:map [:foo [:or number? string?]]
but not on the property level:
[:map
 [:or 
  [:map [:foo number?]
  [:map [:bar number?]]
 [:baz number?]
That would seem pretty straightforward, but as I mentioned, I'm not sure I fully understand :or yet

ikitommi15:01:09

The entry-parser doesn't understand the :or currently. How would you differentiate it from expected map key :or?

ikitommi15:01:49

it could be ::m/or not to mix up with normal keys.

ikitommi15:01:16

[:map
 [:or [:map [:x :int]]]

ikitommi15:01:21

is that map with :or key with map as value, or :map key with :int value

ikitommi15:01:19

[:map
 [::m/or [:map [:x :int]]]
would work and be good enough?

Etzwane07:01:39

I see your point. from that perspective it would perhaps have been nice to namespace all Malli keywords... What makes me wonder, in your last example

[:map
 [::m/or [:map [:x :int]]]
Why the second :map? I would think that if you :or on this level, it would express nested maps, which is not what's happening and so a little surprising. What I would expect is semantics like this: suppose we have an address that can either be a zip code and a house number, or an city, a street name and a house number, I would not be surprised to see it expressed like this:
[:map
  [:house-number number?]
  [::m/or 
    [:zip-code string?]
    [::m/and
      [:city string?]
      [:street-name string?]]]]

ikitommi07:01:35

Exactly. Didn’t notice that my schema had a bug in it. Correct pair would be:

[:map
 [:or [:map [:tuple :int]]]
vs.
[:map
 [::m/or [:map [:tuple :int]]]
, with would mean (despite being silly):
"a map with EITHER a key :map key with :tuple of one :int OR nothing, e.g {:map [1]}" 
, so exactly what you described in:
[:map
  [:house-number number?]
  [::m/or 
    [:zip-code string?]
    [::m/and
      [:city string?]
      [:street-name string?]]]]

ikitommi07:01:39

implementation of this would require a change in the map parser + internal data structure, so some amount of work.

ikitommi07:01:09

could you your example (with reasoning) to the issue?

Etzwane09:01:43

Sure! Glad to be of help. Tomorrow I will have time

Etzwane07:01:02

Hi Tommi I added what I hope was the gist of our discussion to the issue.

Leo Poulson15:01:24

Hi all, happy monday I have been enjoying using malli.experimental for instrumenting expected input / output shape. I have a question about syntax. I have a lot of functions which look like this

(mx/defn my-function
  [{:keys [file-type] :as env} :- [:map [:file-type :string]]]
  ...)
I would like to avoid having to duplicate the name of the key when writing the expected schema, and instead write something like
(mx/defn my-function
  [{:keys [file-type :- :string] :as env}]
  (+ 1 2 3))
however this doesn’t work, because clojure reads :- and :string as binding variables. My questions are • is this currently possible (i.e. labelling expected schema of a variable inside a destructuring expression)? • if not, is this planned / is there a github issue I can track? hope this makes sense, thanks

ikitommi16:01:20

Hear you, just needs to implemented. Not sure if that would be even a big thing to do, but things like precedence should be thought.

ikitommi16:01:15

Here are the schematized parse tests: https://github.com/metosin/malli/blob/master/test/malli/destructure_test.cljc#L171-L287 Would you be interested in writing a minimalistic but complete expectation and attach that into an new issue of this?

ikitommi16:01:17

revisited te parser code, kinda dense, but there is not much of it as it's using malli for parsing 😉 https://github.com/metosin/malli/blob/master/src/malli/destructure.cljc

m s21:01:02

Hi all, good evening! I'm having trouble with the strip-extra-keys-transformer in conjunction with :and schemas:

(m/decode
 [:and [:map [:y :int]] [:map [:x :int]]]
 {:x 1 :y 2 :z 3}
 (mt/strip-extra-keys-transformer))
strips all keys, instead of stripping only :z and keeping :x and :y. I think this is a known issue, but the regular workaround of using :merge doesn't really work for my use case: I'm relying on :and to compose :map schemas together, which works great for documentation as I can have options on the composed schemas and use refs (and a registry) for the components (e.g. [:and {:title "composed schema"} ::component1 [:map {:title "component 2"} [:a :int]]]). :merge's options are overridden by the last component, and I also lose refs when it's converted to json-schema. Another issue is that I want to use strip-extra-keys-transformer to implement select (a la clojure.spec), and for the same reason, m/coerce also fails if I use it with strip-extra-keys-transformer. I tried extending strip-extra-keys-transformer with a special case for :and, and I was able to generate the complete keyset, but when the transformer descends into the children, it eventually drops all the keys (as long as all the children have a disjunct set of keys, which they almost always do in my case). Could someone please help me with implementing a transformer that strips extra keys correctly for [:and [:map ..] [:map ..]] schemas?

ikitommi20:01:47

Hmm. Interesting use case for :and! It works if as long as you don't close the maps.

ikitommi20:01:42

Can't recall the code, but what happens if you set the maps explicitly open with :closed false property. Would that work?

ikitommi20:01:16

How are you using "options for the composed schema"?

ikitommi20:01:35

losing refs with Json schema, could you show an example?

m s12:01:35

The issue with :closed false is that I would like to use the strip-extra-keys-transformer to implement select-schema (like metosin.schema-tools/select-schema), and that wouldn't work if I leave the maps open. By "options on the composed schema" I mean documentation keys like :title, :description :json-schema/format etc. I'll write a better example in a bit. Update: I realize I've been using the wrong term options while I actually meant properties. Sorry for the confusion.

m s12:01:48

adding as snippet as well for syntax highlight

m s13:01:20

To repeat, the two issues are: 1. When using mallo.json-schema/transofrm, :merge also merges the options properties of the schemas that are being merged and doesn't retain the options properties for the root (e.g. with [:merge props a b c d], only the options properties of d will remain). 2. refs also get resolved in the process But even if both of these behaviours would be changed in :merge, I would still want to keep using :and, because in some cases it describes the intention and the purpose of the schema more accurately than :merge.