Fork me on GitHub
#malli
<
2021-05-03
>
ikitommi05:05:38

@joel380 not atm. The new malli schemas of malli schemas feature allows us to describe the available properties formally as schemas, but it’s just the mechanism atm, no content shipped. Once all schmas are described (both properties and children), it will be THE documentation. For now, you need to read the sources. Documentation PRs always welcome.

ikitommi05:05:20

e.g.

;; currently
(m/properties-schema :string)
; => nil

;; after
(m/properties-schema :string)
; [:map
;  [:min {:description "length must be at least"} :int]
;  [:max {:description "length must be at most"} :int]]
… different schema applications (e.g. generation) will have their own overlays, which can be merged to get the full set of available keys. Should be easy to generate proper docs out of those.

👍 2
ikitommi05:05:20

after the children & property schemas are done, one can also generate valid schema-hiccup out of a given registry 🙂

Yehonathan Sharvit12:05:31

Is there a way to forbid some combination of keys in a map? For instance a map with either :a and :b or :c and :d

borkdude12:05:17

probably using another predicate?

borkdude12:05:38

or using two different maps

Yehonathan Sharvit12:05:41

What do you mean by “using another predicate”?

borkdude12:05:52

[:and [:map ...] [:fn (fn [...] ...)]]

borkdude12:05:21

Same as with clojure.spec

borkdude12:05:50

(def my-schema
  [:and
   [:map
    [:x int?]
    [:y int?]]
   [:fn (fn [{:keys [x y]}] (> x y))]])

Ben Sless12:05:57

I was just going to ask if there's a way to specify constraints on relations between keys which does not involve defining an ad-hoc function

borkdude12:05:22

Maybe using two separate closed maps also works. I think it depends on your domain perhaps?

Ben Sless12:05:14

Likely, although I was just thinking of a case like the one you illustrated, a constraint where the value at one key is greater than another

Ben Sless12:05:19

This pattern repeats so often, should it be a schema?

Ben Sless13:05:21

Something like:

(def preds {:> >, :>= >=, :< <, :<= <=, := =, :not= not=})

(defn -rel
  []
  (m/-simple-schema
   (fn [_ [pred a b]]
     {:type :rel,
      :pred (m/-safe-pred #((preds pred) (get % a) (get % b))),
      :min 3,
      :max 3})))

(m/validate
 (m/schema
  [:and
   [:map
    [:x :int]
    [:y :int]]
   [:rel :< :x :y]]
  {:registry
   (mr/composite-registry
    m/default-registry
    {:rel (-rel)})})
 #_{:x 1 :y 2}
 {:x 1 :y 1})

awesome 5
👍 2
borkdude13:05:18

reminds me of jet :)

$ echo  '{:a 1 :b 2}' | jet --query '(< :a :b)'
true
$ echo  '{:a 1 :b 2}' | jet --query '(> :a :b)'
false

👍 2
2
ikitommi13:05:47

I think it’s good idea to experiment new relational schemas in the user space, both -simple-schema & -collection-schma are good ways to make these, like @ben.sless you demoed. The more stuff pushed into these data-languages, the more it looks like a simple sci.

Ben Sless13:05:32

A big plus it could have over sci is getting the benefit of the JIT while still keeping the schema serializable

ikitommi13:05:45

only 👍

😄 2
ikitommi13:05:16

true, sci introduces a lot of overhead. and is big on cljs.

borkdude13:05:54

much smaller than self-hosted CLJS though

borkdude13:05:09

but how many people are really using schema serialization?

borkdude13:05:15

I think most people are not

ikitommi13:05:23

i think so too.

Ben Sless13:05:38

Does select-keys translate to mu/get of all the keys then closing the schema?

ikitommi13:05:55

but a lightweight map key dependency utility might be a good fit for many things. just (somone) needs (to invent) more syntax.

Ben Sless13:05:56

The schema serialization isn't my first (or third) priority here tbh, it's more about having more expressive schemas for the common 90% of cases and not having to define ad-hoc functions and errors for them, not to mention generators.

👍 2
borkdude13:05:35

yeah, I think that's nice. I think clojure spec also doesn't have a good answer to this yet and it's pretty common

Yehonathan Sharvit13:05:40

Is there an answer to this in TypeScript?

ikitommi13:05:13

I think you have to use union types with ts

Yehonathan Sharvit14:05:44

Does it help to define constraints on map keys

Joel17:05:29

I'm surprised that TypeScript is sophisticated enough to be used as inspiration, I guess I need to learn more. I've presumed Elm/Haskell are more sophisticated, eg.: https://www.youtube.com/watch?v=IcgmSRJHu_8

mynomoto02:05:33

Typescript is way more pragmatic than elm/Haskell. Structural typing almost looks like malli schemas enforced at compile time.

ikitommi13:05:45

would meander be great at describing the relations, as data?

borkdude13:05:44

malleander!

🤯 4
ikitommi13:05:43

using modified ben’s syntax:

[:and
 [:map
  [:a :any]
  [:b :any]
  [:c :any]
  [:d :any]]
 [:rel
  [:or
   [:and :a :b]
   [:and :c :d]]]]

Ben Sless13:05:00

The only objection I can come up with is that meander is too powerful, (i.e. the result will be hard to work with)

Ben Sless13:05:24

"Here, have this fighter-jet fueled by liquid plutonium" Scary

😂 3
borkdude13:05:22

I think it can be confusing to overload the meaning of :or within :rel

☝️ 2
borkdude13:05:34

Maybe just [:and [:map ....] [:or [:required-keys :a :b] [:required-keys :c :d]] [:gtk :a :b]]

borkdude13:05:42

(gtk = greather than for keys, or something)

Ben Sless13:05:08

The rel syntax needs to be properly thought out, I just invented something ad-hoc to see if it could work

borkdude13:05:14

[:keys/> :a :b]

Ben Sless13:05:37

How about :constraints?

borkdude13:05:43

[:keys/require :a :b]

Ben Sless13:05:16

Then constraints unify by default (like datalog) or disjoin when specified (via :or)

borkdude13:05:17

[:k> :a :b], [:kreq :a :b]

ikitommi13:05:01

this works already:

(m/validate
  [:schema {:registry {::a :int
                       ::b :int
                       ::c :int
                       ::d :int}}
   [:or
    [:map ::a ::b]
    [:map ::c ::d]]] 
  {::a 1, ::b 2})
; => true

Yehonathan Sharvit13:05:29

But it doesn’t scale. When you have other set of constraints you’d need to write down all the combinations of valid keys.

borkdude13:05:25

yeah, that's what I said in the start of the conversation: two disjunct map defs

ikitommi13:05:33

just not with non-qualified keys (for no good reason)

Ben Sless13:05:42

[:constraints
 [:or
  [:and
   [:requires :a :b]
   [:> :a :b]]
  [:and
   [:requires :c :d]
   [:< :c :d]]]]

borkdude13:05:54

but for rels between keys you could just have specialized ops like [:k> :a :b]: the :a key must be greater than the :b key, without introducing some new concept

2
ikitommi13:05:46

i kinda like it.

Ben Sless13:05:18

special ops rub me the wrong way, for some reason. What's wrong with just :>?

ikitommi13:05:34

but question is: from whom the declaration are for? schema writer? user? external docs?

ikitommi13:05:13

(m/validate [:> 6] 4)
; => false

ikitommi13:05:19

it already exists

Ben Sless13:05:23

hm, yes, but as a schema not an argument (if we go back to the [:rel x y z] or [:constraint ,,,] suggestion

borkdude13:05:24

> What's wrong with just :> That it means different things in different contexts, this can be confusing imo.

borkdude13:05:49

What if you want to really do numeric comparison like [:> :a 5] and 5 is also a key in a map?

borkdude13:05:03

In jet I chose [:> :a #jet/lit 5]

borkdude13:05:13

but quickly this became tedious so I wrote a clojure interpreter

Ben Sless13:05:37

The most shorthand constraint syntax I can think of would be :! An alternative which conforms to your suggestion would be :!/>

Ben Sless13:05:51

Downside - it looks like Perl Upside - concise and unique syntax

🚀 2
borkdude13:05:37

you should always make a good trade-off between adding extra syntax (= complexity) and how much people are really going to use this

☝️ 4
ikitommi13:05:14

[:and
 [:map
  [:a :any]
  [:b :any]
  [:c :any]
  [:d :any]]
 [:keys/or
  [:keys/and :a :b]
  [:keys/and :c :d]]]

borkdude13:05:26

yeah, I like that a lot better

borkdude13:05:05

why even :keys/or, you can just use the normal :or here?

Yehonathan Sharvit13:05:28

I was about to write the same thing

Ben Sless13:05:37

Then adding a predicate function on the keys would look like

[:and
 [:map
  [:a :any]
  [:b :any]
  [:c :any]
  [:d :any]]
 [:or
  [:and
   [:keys/and :a :b]
   [:keys/< :a :b]]
  [:keys/and :c :d]]]

Yehonathan Sharvit13:05:41

The disjunction must not be about the keys

Ben Sless13:05:32

:keys/or should mean "at least k1 or k2 should exist"

borkdude13:05:59

what does :keys/and mean: both keys must be present? what does [:keys/and :a] mean then? Maybe [:keys/req :a] is a better name?

2
Ben Sless13:05:47

Maybe even without a shorthand, keys/require

Yehonathan Sharvit13:05:04

:keys/req doesn’t convey the fact that :a and :b are related

borkdude13:05:35

but what exactly does [:keys/and :a :b] mean then, how are these things related?

Yehonathan Sharvit13:05:48

I didn’t say I liked :keys/and

Yehonathan Sharvit13:05:16

Maybe [:keys/present :a :b]?

borkdude14:05:12

but you also didn't say what the semantic relationship between those keys is according to that op. > :keys/req doesn’t convey the fact that :a and :b are related :present also doesn't communicate a relationship, just as :required doesn't, it just means all those keys must be required/present

Yehonathan Sharvit14:05:23

I was wrong. There is no relationship between the keys. But somehow I find it weird to say “required” inside an or condition

Yehonathan Sharvit13:05:12

And now, where would we put a custom predicate [:fn …]?

borkdude13:05:02

or :keys/with and :keys/without ;P

👀 2
ikitommi13:05:06

off to cook some food. good discussion, as always 👍

Ben Sless13:05:18

Thank you, feel like we hit on something useful and needed Bon appetit!

Yehonathan Sharvit14:05:11

By the way, my use case is real for a project at work.

Yehonathan Sharvit14:05:29

We are writing a Hbase driver

borkdude14:05:58

I guess you can already express required keys like this as well? [:and [:map ...] :a :b :c] or do keywords not behave like predicates in malli?

Ben Sless14:05:45

(m/schema [:and :a :b])
exception

borkdude14:05:13

ok, [:and [:fn :a] [:fn :b]] ? :thinking_face:

Ben Sless14:05:14

And non informative, and useless for generation

(me/humanize (m/explain (m/schema [:and [:fn :a]]) {:b 1}))
;; => #:malli{:error ["unknown error"]}

Yehonathan Sharvit14:05:30

The scan function receives a map that could contain either :from and :to or :starts-with

borkdude14:05:46

It looks like malli is becoming really popular in Israel?

Yehonathan Sharvit14:05:40

There are not so many Clojure shops i Israel. Therefore, we could already say that 10% of Israeli Clojure shops use malli 😁

Yehonathan Sharvit14:05:16

I’m struggling to write the schema of the map received by scan

[:map    
   [:starts-with :string]
   [:from :string]
   [:to :string]
 [:xor 
   [:keys/req :starts-with]
   [:keys/req :from :to]
   [:keys/req :from]
   [:keys/req :to]
 ]
]

Yehonathan Sharvit14:05:44

:or is not good in my case. It should be exclusive or

borkdude14:05:12

if it is xor, then why not use disjunct maps?

Yehonathan Sharvit14:05:26

Because there are other keys

borkdude14:05:42

maybe add a :type field or something?

Yehonathan Sharvit14:05:59

:time-range , :limit

Yehonathan Sharvit14:05:45

[:map    
   [:starts-with :string]
   [:from :string]
   [:to :string]
   [:limit :number]
   [:time-range [:map [:from-ms :int] [:to-ms :int]]
 [:xor 
   [:keys/req :starts-with]
   [:keys/req :from :to]
   [:keys/req :from]
   [:keys/req :to]
 ]
]

Yehonathan Sharvit14:05:59

What is a :type field?

borkdude14:05:57

I think you could have different types of ranges

borkdude14:05:03

{:range-type :closed}, {:range-type :open}

Ben Sless14:05:09

In this case I'd start with some base schema, merge the different options with it and or between them

Yehonathan Sharvit14:05:45

I don’t get what both of you suggested

borkdude14:05:19

@viebel I think this requirement is a bit weird:

:xor
[:keys/req :from :to]
[:keys/req :from]
if from is present, and to is not, then the second is true. but if to is present, then the first one is true. so to is optional.

borkdude14:05:50

I did not look at the other requirements, but I think your logic can be "refactored"

borkdude14:05:56

by modeling it differently

Yehonathan Sharvit15:05:28

I don’t see how to refactor the logic. All the followings are valid • :from 10 :to 20 • :form 10 • :to 20 • :starts-with aaa

Yehonathan Sharvit15:05:54

The followings are invalid

Yehonathan Sharvit15:05:29

• :from 10 :starts-with aaa • :to 20 :starts-with aaa • “from 10 :to 20 starts-with aaa

emccue14:05:36

hmm, this is a tricky one

emccue14:05:56

[:or [:map {:closed true}
       [:from ...]
       [:to   ...]]
     [:map {:closed true}
       [:starts-with ...]]]

emccue14:05:19

OR is an exclusive or if the types don't intersect

emccue14:05:36

[:map 
  [:from ...]
  [:to   ...]]

emccue14:05:51

but if you have it closed you can't have anything other than from and to

emccue14:05:29

and if you have it open you leave open the possibility that :starts-with is in the map as well

emccue14:05:41

what you need is basically

emccue14:05:08

[:or [:and [:map 
             [:from ...]
             [:to ...]]
           [:not [:map 
                   [:starts-with ...]]]]
     [:and [:map 
             [:starts-with ...]]
           [:not [:map 
                   [:from ...]
                   [:to   ...]]]]

emccue14:05:50

[:or [:and [:map 
             [:from ...]
             [:to ...]]
           [:not [:map 
                   [:starts-with ...]]]]
     [:and [:map 
             [:starts-with ...]]
           [:not [:map [:from ...]]
           [:not [:map [:to ...]]]]]

emccue14:05:27

not that not exists, but conceptually thats what you would need to represent the constraint

emccue14:05:54

since you only want to be closed against specific keys

emccue14:05:23

if you are willing to explicitly enumerate all the keys you can just do the version with {:closed

Ben Sless14:05:52

I meant something like

(def Base
  [:map
   [:limit :number]
   [:time-range [:map [:from-ms :int] [:to-ms :int]]]])

(def S1 (mu/merge Base [:map [:starts-with :string]]))
(def S2 (mu/merge Base [:map [:from :string] [:to :string]]))

Ben Sless15:05:30

Did some generalizing work, still not set on the syntax

(defn comparator-relation
  [sym msg]
  (let [f @(resolve sym)
        type (keyword "!" (name sym))]
    [type
     (m/-simple-schema
      (fn [_ [a b]]
        (let [fa #(get % a)
              fb #(get % b)]
          {:type type
           :pred (m/-safe-pred #(f (fa %) (fb %))),
           :type-properties
           {:error/fn
            {:en (fn [{:keys [schema value]} _]
                   (str
                    "value at key "
                    a ", "
                    (fa value)
                    ", should be "
                    msg
                    " value at key "
                    b
                    ", "
                    (fb value)))}}
           :min 2,
           :max 2})))]))

(defn -comparator-relation-schemas
  []
  (into
   {}
   (map (fn [[sym msg]] (comparator-relation sym msg)))
   [['>  "greater than"]
    ['>= "greater than or equal to"]
    ['= "equal to"]
    ['== "equal to"]
    ['<= "lesser than or equal to"]
    ['< "lesser than"]]))


(me/humanize
 (m/explain
  (m/schema
   [:and
    [:map
     [:x :int]
     [:y :int]]
    [:!/> :x :y]]
   {:registry
    (mr/composite-registry
     m/default-registry
     (-comparator-relation-schemas))})
  {:x 1 :y 1}))