Fork me on GitHub
#malli
<
2022-03-09
>
Ryan Tate03:03:15

Hello, Malli looks very cool. Is it possible in Malli to retain values generated during validation like spec/conform and spec/conformer? Or if not maybe you can think of some Malli-ish way to solve this: I have some expensive validations, for example I accept an xpath as input and validate by compiling it. I don't want to throw the resulting value away and have to re-generate it later. So I transform (with s/conformer) the xpath string to a map with a compiled version (object) of it during validation and this map is the result of validation (with s/conform):

(require '[clj-xpath.core :as xpath])

(defn expr-or-nil
  [maybe-expr]
  (try
    (xpath/xp:compile maybe-expr)
    (catch org.xml.sax.SAXException e)
    (catch javax.xml.xpath.XPathException e)))

(s/def ::xpath (s/and string?           
                      (s/conformer (fn [value] {:value value
                                                :expr (expr-or-nil value)})
                                    :value)
                       #(:expr %))) 

(s/conform ::xpath "//a/@href")
;; {:value "//a/@href",
;;  :expr
;;  #object[org.apache.xpath.jaxp.XPathExpressionImpl 0x1aa56eab "org.apache.xpath.jaxp.XPathExpressionImpl@1aa56eab"]}     <---DONT HAVE TO RE-PARSE LATER, YAY! :-)

(s/conform ::xpath "%#^@")
;; :clojure.spec.alpha/invalid
I do similar things when I accept regex strings as input - I validate by compiling them with re-pattern in a conformer then I save the output from s/conform for use later. So is it possible to do something like this with Malli? The :decode/math examples in the Value Transformation section of the README.md (https://github.com/metosin/malli#value-transformation) seem possibly close? But it's not clear to me if decode validates. Maybe Malli is trying to keep validation in a different stream from transformation? (I admit I don't understand the transformation section of the docs very well, particularly when it comes to doing custom functions as opposed to the pre-canned transformers for json and so forth.) Kiitos/thanks for any help

ikitommi06:03:17

@ryantate you have two models there: a string and a parsed map. With malli, you should describe either the string (source) or the target (a map). If the map presentation is what you expect, describe that and add a decoder from string->it. A naive impl:

(defn expr-or-nil [maybe-expr]
  (when (= maybe-expr "//a/@href") identity))

(defn decoder [value]
  {:value value
   :expr (expr-or-nil value)})

(def Xpath
  (m/schema
   [:map {:decode/string decoder}
    [:value :string]
    [:expr fn?]]))

(defn coerce [value]
  (let [decoded (m/decode Xpath value (mt/string-transformer))]
    (if-not (m/validate Xpath decoded)
      ::invalid
      decoded)))

(coerce "//a/@href")
; => {:value "//a/@href", :expr #object[clojure.core$identity 0x2c879e65 "clojure.core$identity@2c879e65"]}

(coerce "invalid")
;; => :user/invalid

๐Ÿ™ 1
ikitommi06:03:24

but yes, the transform and validation are separated. There might be use cases where this is not good, but for most cases, itโ€™s easy to compose the two using public apis. If/when someone finds out a need for one-pass-do-it-all &/ stateful transformers, would like to hear those ๐Ÿ˜Ž

Ryan Tate13:03:18

@ikitommi Ah thanks for the comprehensive reply!! ๐Ÿ‘ string -> compiled-xpath is the goal, the map is just there to allow s/unform which is not essential. In my case I would probably just accept the cost of two passes because I need the xpath compilation in order to validate the string) and also the xpath string needs to be composed into a more complicated validation so I can just do one m/validate call on the larger structure (which can have many xpath strings). So I think the coerce function would be tricky to use. I could just validate all xpath as string in the larger value and then m/validate and m/parse the larger value and then walk it and call coerce on each xpath item but I would have to do that also for regex and anything else that gets compiled - becomes basically building a new validation layer which defeats the point a bit. I did a benchmark and compiling xpath is only about 2 ฮผs and regex is even less so I will just put those checks in the schema so it is part of a compostable validation and then do it again in m/decode. It is a performance hit but a small one, hopefully not too bad ๐Ÿ˜…

ikitommi13:03:18

oh, if you want to validate the big thing, you could do schema for string, and then m/encode it to the parsed format. there is no easy way to customize m/parse per schema atm.

ikitommi13:03:05

for performance, you should use m/validator & m/decoder / m/encoder for MUCH better perf.

ikitommi13:03:15

;; pure functions, can be cached
(def validate (m/validator Xpath))
(def decode (m/decoder Xpath (mt/string-transformer)))

(defn coercer [value]
  (let [decoded (decode value)]
    (if-not (validate decoded)
      ::invalid
      decoded)))

Ryan Tate18:03:48

Issue with :orn schema inside decode. Any idea why this works:

(m/decode  [:string {:decode/string (fn [s] (try (re-pattern s) (catch java.util.regex.PatternSyntaxException e)))}] "foo" mt/string-transformer)
;;#"foo"
But this does not?:
(m/decode [:orn [:v vector?] [:re-string [:string {:decode/string (fn [s] (try (re-pattern s) (catch java.util.regex.PatternSyntaxException e)))}]]] "foo" mt/string-transformer)
;;"foo"
Is decode meant to work with :orn, :catn, :or, :and etc?

Ryan Tate18:03:18

Actually I think I have mixed up encode and decode. I found this which has helpful definition of differences (maybe later I can make a PR to merge this table into README.md?): https://cljdoc.org/d/metosin/malli/0.8.4/doc/value-transformation decode is for making values valid, encode is to transform valid values into something else. decode will revert value if it does not match the schema per https://clojurians.slack.com/archives/CLDK6MFMK/p1615225531254000 I rewrote this way which works:

(m/encode [:orn [:v vector?] [:re-string [:string {:encode/string (fn [s] (try (re-pattern s) (catch java.util.regex.PatternSyntaxException e)))}]]] "foo" mt/string-transformer)
;;#"foo"

ikitommi06:03:26

Doc PR most welcome

๐Ÿ‘ 1
ikitommi06:03:20

Also, a transformation debugger would be nice (and not hard to implement?): would emit which steps are run, in which order and how they change / not the value.

Ryan Tate19:03:15

Can I combine parse and encode functionality using a single schema? I can parse:

(def re-schema [:string {:encode/string (fn [s] (try (re-pattern s) (catch java.util.regex.PatternSyntaxException e)))}])
(def re-w-opts-schema [:and vector? [:catn [:pattern re-schema] [:opts [:* [:enum :i :s :u]]]]])
(def re-maybe-w-opts [:orn [:pattern re-schema] [:pattern-w-opts re-w-opts-schema]])

(m/parse re-maybe-w-opts ["foo" :i])
;;[:pattern-w-opts {:pattern "foo", :opts [:i]}]
I can encode:
(m/encode re-maybe-w-opts ["foo" :i] mt/string-transformer)
;;[#"foo" :i]
How to do both and get
[:pattern-w-opts {:pattern #"foo", :opts [:i]}]
?

ikitommi07:03:48

currently, you canโ€™t combine those and there is no schema property-based extension for parsers, so you could plug in custom logic to parsing. I think you could cal m/parse in a custom encoder :leaveyourself?

๐Ÿ‘ 1
๐Ÿ™Œ 1
ikitommi07:03:21

oh, the branching happens at parse, would not work that way. should add hook to parsing to allow custom steps or somehow mix the two (encode & parse). ideas welcome

๐Ÿ‘ 1
Ryan Tate15:03:59

Thanks for the reply, going to work on this for a bit. Parsing is more important for my use case than other transforms. Will come back if I have API extension ideas. Kiitos ๐Ÿ‘

Phil Jackson21:03:33

I came to ask exactly the same thing. @ryantate can you let me know if you come up with something?