Fork me on GitHub
Slack Archive

This page is not created by, affiliated with, or supported by Slack Technologies, Inc.

2021-07-20

Channels

#malli
<
2021-07-20
>
escherize03:07:35

Is there a malli schema for the clojure.core/defn form?

escherize05:07:03

thanks for your response @mike1452. What I am looking for is a malli schema that will say “that clojure.core/defn form you have is good and not malformed” or not.

escherize05:07:42

I think the schema should match these, but there may be more: 

['(defn f [a] 1)
 '(defn g [a] (+ a a))
 '(defn ^:ok g "hi" [a] (+ a a))
 '(defn ^:ok g "hi" ([a] (+ a a)))
 '(defn ^:ok g "hi" ([a] 1) ([a b] 2))]

escherize06:07:36

I thought I was getting close, but can’t quite get it for all cases. Maybe someone here can figure it out 

(def DefnSchema
  [:cat
   [:enum 'defn '-defn]
   symbol?
   [:? string?]
   [:alt
    [:cat [:vector symbol?] [:* any?]]
    [:* [:sequential [:cat [:vector symbol?] [:* any?]]]]]])

(mapv #(m/explain DefnSchema %)
      ['(defn f [a] 1) ;;<- good
       '(defn g [b] (+ b b))  ;;<- good
       '(defn ^:ok g "hi" [c] (+ c c))  ;;<- good
       '(defn ^:ok g "hi" ([d] (+ d d))) ;;<- fail
       '(defn ^:ok g "hi" ([e] 1) ([e e] 2))]);;<- fail

ikitommi11:07:38

m/explain might hint why it's not right

ikitommi11:07:59

oh, you had that.

ikitommi11:07:37

any hint about why they failed? (not near computer myself)

danielneal13:07:11

Is there already a transformer that will work with query parameters - specifically to convert single values into vectors where the schema specifies a vector. i.e. 

(malli.core/decode [:map
                    [:a [:vector int?]]]
                   {:a "1"}
                   some-transformer) => {:a [1]} (as opposed to {:a "1"})

(malli.core/decode [:map
                    [:a [:vector int?]]]
                   {:a ["1" "2"]}
                   some-transformer) => {:a [1 2]}
This seems to work - but is it right? 
(defn collection-transformer []
  (malli.transform/transformer
    {:decoders
     {:vector
      {:compile (fn [schema _]
                  (fn [x]
                    (if (vector? x) x [x])))}}}))

👍 2
ikitommi13:07:19

@escherize it should be: 

(def DefnSchema
  [:cat
   [:enum 'defn '-defn]
   symbol?
   [:? string?]
   [:alt
    [:cat [:vector symbol?] [:* any?]]
    [:+ [:schema [:cat [:vector symbol?] [:* any?]]]]]])

🔥 2
ikitommi13:07:30

e.g. not sequence :+ of sequence :sequence of sequences :cat, just sequence (`:+`) of sequences :cat.

😄 2
ikitommi13:07:02

@danieleneal maybe something like: 

(m/decode
  [:map
   [:a [:vector int?]]]
  {:a "1"}
  (mt/transformer
    (mt/transformer
      {:decoders {:vector (fn [x] (if (string? x) [x] x))}})
    (mt/string-transformer)))
; => {:a [1]}

danielneal14:07:55

ah cool, thanks, looks like I’m on the right lines 🙂

ikitommi14:07:58

you can use :compile if you want to access the schema ahead of time, like reading the separator per schma: 

(def decode
  (m/decoder
    [:map
     [:a [:vector {:separator ";"} int?]]]
    (mt/transformer
      (mt/transformer
        {:decoders
         {:vector
          {:compile (fn [schema _]
                      (let [separator (-> schema m/properties :separator (or ","))]
                        (fn [x]
                          (cond
                            (not (string? x)) x
                            (str/includes? x separator) (into [] (.split ^String x ^String separator))
                            :else [x]))))}}})
      (mt/string-transformer))))

(decode {:a "2"})
; => {:a [2]}

(decode {:a "1;2"})
; => {:a [1 2]}

danielneal14:07:25

nice!! Thanks :))))

kenny14:07:40

I'm curious if Malli has considered allowing the default registry to be set in a less invasive way (i.e., not through jvm props)? The requirement to set a jvm prop touches many places in a large application and, I imagine, will cause future developer confusion by requiring every repl to be launched with that prop.

➕ 7
ikitommi14:07:39

totally agree. the current way to enable custom registry is too much work and one can always define an immutable registry for the cases (multi-tenant env) when it matters. Just not sure what would be a best possible compromise between simple & easy here. Imperative programming with global state is not good either. Ideas?

ikitommi14:07:45

there was a discussion somewhere some time ago..

ikitommi14:07:25

1. immutable by default, swapping needs a custom jvm/compiler option 2. mutable registry by default, spec-like 3. immutable by default, but mr/set-default-registry! available without jvm options 4. something else

kenny15:07:42

We are very early in our usage of Malli, coming from a large use of Spec, so take anything I say with that grain of salt. I quite like the default mutable registry. Over the years, we have built up a large library of domain specs that are used all over the place. It's handy to be able to simply reference these specs by their keyword name. Of course, Malli may encourage different conventions (e.g., just write functions returning custom schema). I started down the path of creating our own immutable registry, but started to feel pain when I needed to pass my registry to every single Malli api call.

mike_ananev15:07:59

@ikitommi I found breaking changes between 0.5.1 and 0.6.0-snapshot in a function m/validate In 0.5.1, function m/validate returns true if data corresponds to spec. In 0.6.0-snapshot, function m/validate returns data, not boolean value true. In 0.5.1, If data is not corresponds to spec, then function m/validate returns false. In 0.6.0-snapshot, function m/validate returns in some cases value false, in some cases returns value nil (for strings for example).

ikitommi07:07:53

oh, that’s not good @mike1452. These might be related: • https://github.com/metosin/malli/commit/ae12531aede1ad936af7d7bde80543572cc907ae • https://github.com/metosin/malli/pull/479 … could you retest with the new SNAPSHOT (`metosin/malli-0.6.0-20210721.071739-2`) and if the problem persists, please write an issue.

mike_ananev08:07:05

@ikitommi thank you for quick reply! I'll check my tests and return soon. You can use malli spec to spec your functions to catch errors, while you are developing malli.spec. 😂

mike_ananev10:07:56

@ikitommi I found new strange behaviour during this cycle: edn1 -> encode-value -> json -> decode-value -> edn2 In v 0.5.1, edn1 = edn2. All uuids and zoned-date-time encoded/decoded correctly, they are data of the same type. In v 0.6.0-20210721.071739-2, edn1 != edn2 broken data for uuid and zoned-date-time - they are strings after decoding. Can't reproduce this error on a simple schema. I've more complicated schemas, but cannot publish them here. But in my code I do something like that.

mike_ananev10:07:56

(do (def custom-json-transformer (mt/transformer mt/string-transformer mt/json-transformer))
    (def my-spec1 [:map
                   [:a :zoned-date-time]
                   [:b :string]
                   [:c :uuid]
                   [:d :int]])
    (def my-spec [:map [:s1 my-spec1] [:s2 my-spec1]])
    (def edn-value (mg/generate my-spec))
    (def encoded-value (m/encode my-spec edn-value custom-json-transformer))
    (def json-string-value (json/write-value-as-string encoded-value json/keyword-keys-object-mapper))
    (def encoded-value2 (json/read-value json-string-value json/keyword-keys-object-mapper))
    (def edn-value2 (m/decode my-spec encoded-value2 custom-json-transformer))
    (is (m/validate my-spec edn-value2)))

Ben Sless12:07:44

Is this example sufficient for reproduction or is it just illustrative?

Ben Sless13:07:23

Some schemas are missing from the example, too

Ben Sless16:07:13

I MANAGED TO RECREATE IT 

(do
  (def custom-json-transformer (mt/transformer mt/string-transformer mt/json-transformer))
  (def my-spec1 [:map
                 [:a/c1  {:optional true} :uuid]
                 [:a/c2  :uuid]
                 [:a/c3  :uuid]
                 [:a/c4  :uuid]
                 [:a/c5  :uuid]
                 [:a/c6  :uuid]
                 [:a/c7  :uuid]
                 [:a/c8  :uuid]
                 [:a/c9  :uuid]
                 [:a/c10 :uuid]
                 [:a/c11 :uuid]
                 [:a/c12 :uuid]
                 [:a/c13 :uuid]
                 [:a/c14 :uuid]
                 [:a/c15 :uuid]
                 [:a/c16 :uuid]
                 [:a/c17 :uuid]
                 [:a/c18 :uuid]
                 [:a/c19 :uuid]
                 [:a/c20 :uuid]
                 [:a/c21 :uuid]
                 [:a/c22 :uuid]
                 [:d :int]])
  (def my-spec [:map [:x/y [:map [:b/foo [:map  [:a/s1 my-spec1]]]]]])
  (def edn-value (mg/generate my-spec))
  (def encoded-value (m/encode my-spec edn-value custom-json-transformer))
  (def json-string-value (json/write-value-as-string encoded-value json/keyword-keys-object-mapper))
  (def encoded-value2 (json/read-value json-string-value json/keyword-keys-object-mapper))
  (def edn-value2 (m/decode my-spec encoded-value2 custom-json-transformer))
  (m/validate my-spec edn-value2))

mike_ananev16:07:22

@UK0810AQ2 yeah! This code reproduces the bug. @ikitommi do you have any suggestions?

Ben Sless16:07:30

Now I can debug and investigate

mike_ananev16:07:56

The bug is when we have optional attribute in a map then process edn -> encode -> json -> decode ->edn2 is broken, edn != edn2 . This behavior is in version 0.6.0-20210721.071739-2. In 0.5.1 version all is working as expected.

Ben Sless17:07:33

Managed to narrow the example down a bit: 

(do
  (def custom-json-transformer (mt/transformer mt/string-transformer mt/json-transformer))
  (def my-spec1 [:map
                 [:a/c1  {:optional true} :uuid]
                 [:a/c2  :uuid]])
  (def s (m/schema my-spec1))
  (def my-spec (m/schema [:map [:x s]]))
  (def g (mg/generator my-spec))
  (def edn-value (mg/generate g {:seed 9999}))
  (def e (m/encoder my-spec custom-json-transformer))
  (def encoded-value (e edn-value))
  (def json-string-value (json/write-value-as-string encoded-value json/keyword-keys-object-mapper))
  (def encoded-value2 (json/read-value json-string-value json/keyword-keys-object-mapper))
  (def d (m/decoder my-spec custom-json-transformer))
  (def edn-value2 (d encoded-value2))
  (m/validate my-spec edn-value2))

Ben Sless17:07:50

This returns false consistently

Ben Sless18:07:16

I managed to find the commit which caused the change: 9dc8da7a0649ed93554c1fe16ea48c2bebd724ad fast collection transformers

Ben Sless18:07:07

I am not sure where the bug is, but changing the implementation of -map-transformer to 

(reduce
      -comp
      (map
       (fn [[k t]]
         (fn [x]
           (if-let [e (.entryAt x k)]
             (.assoc x k (t (.val e)))
             x)))
       ts))
Fixes it

ikitommi08:07:56

thanks @UK0810AQ2. I hate it when there is test to cover this. My bad. The minimal repro: 

(deftest regression-480-test
  (let [value {:b #uuid"f5a54a8f-7d78-4495-9138-e810885d1cdb"}
        schema [:map [:a :int] [:b :uuid]]]
    (is (= value
           (as-> value $
                 (m/encode schema $ mt/string-transformer)
                 (m/decode schema $ mt/string-transformer))))))

👀 2
ikitommi09:07:24

updated clojars; 

➜  ~ clj -Sforce -Sdeps '{:deps {metosin/malli {:mvn/version "0.6.0-SNAPSHOT"}}}'
Downloading: metosin/malli/0.6.0-SNAPSHOT/malli-0.6.0-20210722.085801-3.pom from clojars

ikitommi09:07:11

bare minmial repro btw was: 

(m/decode
  [:map [:a :int] [:b :int]]
  {:b "1"}
  mt/string-transformer)
; => {:b "1"}

mike_ananev10:07:24

@ikitommi @UK0810AQ2 thank you! Now, all test are passed. We will try to take version 0.6.0-20210722.085801-3 in our production.

mike_ananev15:07:21

This new behaviour of m/validate breaks some tests in my code.

mike_ananev15:07:15

What behaviour is expected in 0.6.0 release? Should we adapt our tests for this new behaviour of m/validate ?

greg16:07:30

I've just found out that mu/update-in can take a path that includes also catn options. It's so easy and concise to alter sequential schema with Malli! :grinning_face_with_star_eyes:

💯 3