Fork me on GitHub
Ryan Tate03:03:15

Hello, Malli looks very cool. Is it possible in Malli to retain values generated during validation like spec/conform and spec/conformer? Or if not maybe you can think of some Malli-ish way to solve this: I have some expensive validations, for example I accept an xpath as input and validate by compiling it. I don't want to throw the resulting value away and have to re-generate it later. So I transform (with s/conformer) the xpath string to a map with a compiled version (object) of it during validation and this map is the result of validation (with s/conform):

(require '[clj-xpath.core :as xpath])

(defn expr-or-nil
    (xpath/xp:compile maybe-expr)
    (catch org.xml.sax.SAXException e)
    (catch javax.xml.xpath.XPathException e)))

(s/def ::xpath (s/and string?           
                      (s/conformer (fn [value] {:value value
                                                :expr (expr-or-nil value)})
                       #(:expr %))) 

(s/conform ::xpath "//a/@href")
;; {:value "//a/@href",
;;  :expr
;;  #object[org.apache.xpath.jaxp.XPathExpressionImpl 0x1aa56eab "org.apache.xpath.jaxp.XPathExpressionImpl@1aa56eab"]}     <---DONT HAVE TO RE-PARSE LATER, YAY! :-)

(s/conform ::xpath "%#^@")
;; :clojure.spec.alpha/invalid
I do similar things when I accept regex strings as input - I validate by compiling them with re-pattern in a conformer then I save the output from s/conform for use later. So is it possible to do something like this with Malli? The :decode/math examples in the Value Transformation section of the ( seem possibly close? But it's not clear to me if decode validates. Maybe Malli is trying to keep validation in a different stream from transformation? (I admit I don't understand the transformation section of the docs very well, particularly when it comes to doing custom functions as opposed to the pre-canned transformers for json and so forth.) Kiitos/thanks for any help


@ryantate you have two models there: a string and a parsed map. With malli, you should describe either the string (source) or the target (a map). If the map presentation is what you expect, describe that and add a decoder from string->it. A naive impl:

(defn expr-or-nil [maybe-expr]
  (when (= maybe-expr "//a/@href") identity))

(defn decoder [value]
  {:value value
   :expr (expr-or-nil value)})

(def Xpath
   [:map {:decode/string decoder}
    [:value :string]
    [:expr fn?]]))

(defn coerce [value]
  (let [decoded (m/decode Xpath value (mt/string-transformer))]
    (if-not (m/validate Xpath decoded)

(coerce "//a/@href")
; => {:value "//a/@href", :expr #object[clojure.core$identity 0x2c879e65 "clojure.core$identity@2c879e65"]}

(coerce "invalid")
;; => :user/invalid

๐Ÿ™ 1

but yes, the transform and validation are separated. There might be use cases where this is not good, but for most cases, itโ€™s easy to compose the two using public apis. If/when someone finds out a need for one-pass-do-it-all &/ stateful transformers, would like to hear those ๐Ÿ˜Ž

Ryan Tate13:03:18

@ikitommi Ah thanks for the comprehensive reply!! ๐Ÿ‘ string -> compiled-xpath is the goal, the map is just there to allow s/unform which is not essential. In my case I would probably just accept the cost of two passes because I need the xpath compilation in order to validate the string) and also the xpath string needs to be composed into a more complicated validation so I can just do one m/validate call on the larger structure (which can have many xpath strings). So I think the coerce function would be tricky to use. I could just validate all xpath as string in the larger value and then m/validate and m/parse the larger value and then walk it and call coerce on each xpath item but I would have to do that also for regex and anything else that gets compiled - becomes basically building a new validation layer which defeats the point a bit. I did a benchmark and compiling xpath is only about 2 ฮผs and regex is even less so I will just put those checks in the schema so it is part of a compostable validation and then do it again in m/decode. It is a performance hit but a small one, hopefully not too bad ๐Ÿ˜…


oh, if you want to validate the big thing, you could do schema for string, and then m/encode it to the parsed format. there is no easy way to customize m/parse per schema atm.


for performance, you should use m/validator & m/decoder / m/encoder for MUCH better perf.


;; pure functions, can be cached
(def validate (m/validator Xpath))
(def decode (m/decoder Xpath (mt/string-transformer)))

(defn coercer [value]
  (let [decoded (decode value)]
    (if-not (validate decoded)

Ryan Tate18:03:48

Issue with :orn schema inside decode. Any idea why this works:

(m/decode  [:string {:decode/string (fn [s] (try (re-pattern s) (catch java.util.regex.PatternSyntaxException e)))}] "foo" mt/string-transformer)
But this does not?:
(m/decode [:orn [:v vector?] [:re-string [:string {:decode/string (fn [s] (try (re-pattern s) (catch java.util.regex.PatternSyntaxException e)))}]]] "foo" mt/string-transformer)
Is decode meant to work with :orn, :catn, :or, :and etc?

Ryan Tate18:03:18

Actually I think I have mixed up encode and decode. I found this which has helpful definition of differences (maybe later I can make a PR to merge this table into decode is for making values valid, encode is to transform valid values into something else. decode will revert value if it does not match the schema per I rewrote this way which works:

(m/encode [:orn [:v vector?] [:re-string [:string {:encode/string (fn [s] (try (re-pattern s) (catch java.util.regex.PatternSyntaxException e)))}]]] "foo" mt/string-transformer)


Doc PR most welcome

๐Ÿ‘ 1

Also, a transformation debugger would be nice (and not hard to implement?): would emit which steps are run, in which order and how they change / not the value.

Ryan Tate19:03:15

Can I combine parse and encode functionality using a single schema? I can parse:

(def re-schema [:string {:encode/string (fn [s] (try (re-pattern s) (catch java.util.regex.PatternSyntaxException e)))}])
(def re-w-opts-schema [:and vector? [:catn [:pattern re-schema] [:opts [:* [:enum :i :s :u]]]]])
(def re-maybe-w-opts [:orn [:pattern re-schema] [:pattern-w-opts re-w-opts-schema]])

(m/parse re-maybe-w-opts ["foo" :i])
;;[:pattern-w-opts {:pattern "foo", :opts [:i]}]
I can encode:
(m/encode re-maybe-w-opts ["foo" :i] mt/string-transformer)
;;[#"foo" :i]
How to do both and get
[:pattern-w-opts {:pattern #"foo", :opts [:i]}]


currently, you canโ€™t combine those and there is no schema property-based extension for parsers, so you could plug in custom logic to parsing. I think you could cal m/parse in a custom encoder :leaveyourself?

๐Ÿ‘ 1
๐Ÿ™Œ 1

oh, the branching happens at parse, would not work that way. should add hook to parsing to allow custom steps or somehow mix the two (encode & parse). ideas welcome

๐Ÿ‘ 1
Ryan Tate15:03:59

Thanks for the reply, going to work on this for a bit. Parsing is more important for my use case than other transforms. Will come back if I have API extension ideas. Kiitos ๐Ÿ‘

Phil Jackson21:03:33

I came to ask exactly the same thing. @ryantate can you let me know if you come up with something?