Fork me on GitHub
#malli
<
2020-08-07
>
borkdude08:08:51

Hoe does one combine validate and transform? E.g. this doesn't crash:

(prn (m/decode int? :foo mt/string-transformer))
I'm not saying it should, just wondering how to do it. Not clear from the README

borkdude08:08:10

Should I first call m/valid, if not valid, then m/explain and else m/decode, effectively traversing the structure twice?

ikitommi09:08:51

If you need decoding, the flow should be: 1. decode 2. validate 3. explain on error

ikitommi09:08:05

there is 2-3 walks in the error case.

ikitommi09:08:23

for happy case, 1-2

ikitommi09:08:20

having a seoarate optimized validate makes the happy case fast

ikitommi09:08:51

the docs could have examples on this...

ikitommi09:08:51

the m/decoder doesn't have to walk the structure, it returns an function to transform just the parts that need to be decoded. In case there is nothing to do, it returns identity

borkdude09:08:08

@ikitommi The concrete example I was going to try:

$ cat deps.edn
{:deps {metosin/malli {:git/url "" :sha "2bd749f7148e28a379f1e628a32188e7f6cf0bc4"}
        borkdude/edamame {:git/url "" :sha "64c7eb43950eb500ba7429dded48257cd15355ae"}}}
$ cat src/edamalli/core.clj
(ns edamalli.core
  (:require [edamame.core :as e]
            [malli.core :as m]
            [malli.transform  :as mt]))

(defrecord WrappedNum [obj loc])

(defn postprocess [{:keys [:obj :loc]}]
  (if (number? obj) (->WrappedNum obj loc) obj))

(defn -main [& args]
  (prn (e/parse-string "[:foo 42]" {:postprocess postprocess}))
  ;; TODO:
  ;; - validate that the WrappedNum contains value < 42
  ;; - then transform it to only that number
  ;; - else raise error, printing the location metadata of that number
  )
$ clojure -m edamalli.core
[:foo #edamalli.core.WrappedNum{:obj 42, :loc {:row 1, :col 7, :end-row 1, :end-col 9}}]

zclj11:08:30

@ikitommi I am parsing a schema into a malli-schema. The original might contain recursive references to other "entities" in the schema. I do not know this up front. Are there any trade-offs in putting all my potential recursive entity references in a [:ref ], even if they turn out not to be?

ikitommi11:08:50

@borkdude would [:foo 42] be transformed to [:foo 42] , as would [:foo :bar #{42}] to itself and {:a 41} would fail on the fact that there was a number that was not 42?

ikitommi11:08:18

happy to help, sample inputs -> outputs would help.

borkdude11:08:54

@ikitommi No, [:foo (WrappedNum. x y)] would be transformed to [:foo x] only if x < 42, else error with explain using y

ikitommi11:08:46

could that validation happen already in the :postprocess?

borkdude11:08:45

I just want to feed this data to malli and not intertwine parsing data from text to sexprs with malli validation

ikitommi11:08:23

ok. can the input be anything, e.g [:foo {:bar (WrappedNum. x y)}]?

borkdude11:08:23

[:foo y] could also be {:foo y} if that's easier for you

borkdude11:08:00

No, it's more like person: {:name "foo" :age 42}, let's say

borkdude11:08:06

so attribute+value

borkdude11:08:01

so unwrapping would just be (:obj wrapped), that would be the transform step

borkdude11:08:46

The use case for this is: normally edamame doesn't let you have location metadata for numbers and strings, but using a wrapped value you can have that

borkdude11:08:15

so I want to use malli like normally, but just use the location metadata in the wrapped value for reporting errors and discard it if the value is valid

borkdude11:08:35

I believe in spec you would write a conformer for this

ikitommi12:08:42

yes, this is kinda tricky with current malli, as there is not yet a parsing api, like conform.

ikitommi12:08:42

also, there is no custom overridable validator, so one needs to describe the given data structure (here, a tuple with keyword and a record).

borkdude12:08:19

ok, so one would write a schema using the records and if everything's ok, then postwalk yourself, unwrapping them?

ikitommi12:08:19

but, something like this:

;; schemas
(def <42 [:and int? [:< 42]])
(def Schema [:tuple keyword? [:map {:encode/success :obj, :encode/failure :loc} [:obj <42]]])

;; validator and encoders for both success & failure
(def valid? (m/validator Schema))
(def success (m/encoder Schema (mt/transformer {:name :success})))
(def failure (m/encoder Schema (mt/transformer {:name :failure})))

;; in action
(defn parse-validate-and-transform [s]
  (let [x (e/parse-string s {:postprocess postprocess})]
    (if (valid? x) (success x) (failure x))))
=>
(parse-validate-and-transform "[:foo 41]")
; => [:foo 41]

(parse-validate-and-transform "[:foo 42]")
; => [:foo {:row 1, :col 7, :end-row 1, :end-col 9}]

ikitommi12:08:03

yes, postwalk would do. or a recursive schema definition. if the wrapped records can be anywhere

borkdude12:08:12

let me try your snippet

borkdude12:08:52

$ clojure -m edamalli.core
Execution error (ExceptionInfo) at malli.core/-fail! (core.cljc:76).
:sci-not-available {:code ":obj"}
That was unexpected, I don't need sci in this example?

ikitommi12:08:24

oh, should not.

ikitommi12:08:22

(def Schema [:tuple keyword? [:map {:encode/success (fn [x] (:obj x)), :encode/failure (fn [x] (:loc x)) [:obj <42]}]])

ikitommi12:08:29

pushed e19872273c3660fbc482dcff4c2d8439dbcbb2a6, which should allow naked keywords as functions.

ikitommi12:08:53

(def Schema [:tuple keyword? [:map {:encode/success :obj, :encode/failure :loc} [:obj <42]]])

borkdude12:08:42

I'm still getting the sci-not-available error

borkdude12:08:56

When I do include sci, I get:

[:foo {:row 1, :col 7, :end-row 1, :end-col 9}]

borkdude12:08:24

I guess I should throw my own error in :encode/failure?

ikitommi12:08:21

yes, that’s one place to do it. will check why sci is needed. just a sec.

borkdude12:08:59

(defn parse-validate-and-transform [s]
  (let [x (e/parse-string s {:postprocess postprocess})]
    (if (valid? x)
      (prn "SUCCESS" (success x))
      (prn "ERROR" (failure x)))))
$ clojure -m edamalli.core
"ERROR" [:foo {:row 1, :col 7, :end-row 1, :end-col 9}]

borkdude12:08:14

haha, when I do this:

:encode/failure {:message "should be lower than 42"}
I get:
$ clojure -m edamalli.core
Execution error (ExceptionInfo) at sci.impl.utils/throw-error-with-location (utils.cljc:54).
Could not resolve symbol: should [at line 1, column 1]

borkdude12:08:38

I have no idea what I'm doing, since I don't know these APIs well. I'll take a look after work again some time

ikitommi12:08:40

8e067b3d004d1692cbfc695bc73d7e032ecb6e7f

ikitommi12:08:14

the code used sci for all non fn?s, no to all non ifn?s. need to add tests.

borkdude12:08:45

@ikitommi I now have this:

(def Schema [:tuple keyword? [:map {:encode/success :obj, :encode/failure {:message "should be lower than 42"}} [:obj <42]]])
Output:
"eval!" "should be lower than 42"
Execution error (ExceptionInfo) at malli.core/-fail! (core.cljc:76).
:sci-not-available {:code "should be lower than 42"}

borkdude12:08:14

I might be doing something wrong, but it seems there's a debug println in there?

ikitommi12:08:05

picard-facepalm my bad. but this kinda works (but is bad, should be better when we have the parsing api):

ikitommi12:08:31

➜  ~ clojure -Sdeps '{:deps {metosin/malli {:sha "230b1767729aad3e02568f1320855e2b45d2d9b5", :git/url ""}, borkdude/edamame {:sha "64c7eb43950eb500ba7429dded48257cd15355ae", :git/url ""}}}'
Checking out:  at 230b1767729aad3e02568f1320855e2b45d2d9b5

Clojure 1.10.1
user=> (ns edamalli.core
  (:require [edamame.core :as e]
            [malli.core :as m]
            [malli.transform :as mt]))

(defrecord WrappedNum [obj loc])

(defn postprocess [{:keys [:obj :loc]}]
  (if (number? obj) (->WrappedNum obj loc) obj))

(defn fail! [{:keys [:obj :loc]}]
  (throw (ex-info (str "so bad " obj "/" loc) {})))

(def <42 [:and int? [:< 42]])
(def Schema [:tuple keyword? [:map {:encode/success :obj,
                                    :encode/failure fail!} [:obj <42]]])

(def valid? (m/validator Schema))
(def success (m/encoder Schema (mt/transformer {:name :success})))
(def failure (m/encoder Schema (mt/transformer {:name :failure})))

(defn parse-validate-and-transform [s]
  (let [x (e/parse-string s {:postprocess postprocess})]
    (if (valid? x) (success x) (failure x))))

edamalli.core=> (parse-validate-and-transform "[:foo 41]")
[:foo 41]

edamalli.core=> (parse-validate-and-transform "[:foo 42]")
Execution error (ExceptionInfo) at edamalli.core/fail! (REPL:2).
so bad 42/{:row 1, :col 7, :end-row 1, :end-col 9}

ikitommi12:08:48

@zclj there is a small (have not measured) penalty for using ref-schemas, one function hop basically as the values are memoized.

zclj13:08:56

ok, that's fine since I have to do something to solve it anyway, by post-walking or such. Doing it up-front with malli ref considerable make the design simpler. Thanks for the info!

ikitommi17:08:38

rollback on the perf info @zclj . Validation & explain perf is about the same but transforming values is potentially much slower. Why? Malli can't optimize over :refs. Transformation behind :ref could be no-op, but it won't be removed as it's wrapped in a function. Still, orders of magnitude faster than with spec(-tools)

zclj09:08:41

thanks for the update! In my use-case I will also do generation from the schema, where I will blow the stack if I don't use :ref for recursive references. Are there any implications for using :ref for potentially non-recursive entities in that case?

borkdude12:08:53

works, thanks

👍 3
ikitommi17:08:38

rollback on the perf info @zclj . Validation & explain perf is about the same but transforming values is potentially much slower. Why? Malli can't optimize over :refs. Transformation behind :ref could be no-op, but it won't be removed as it's wrapped in a function. Still, orders of magnitude faster than with spec(-tools)