Fork me on GitHub
#malli
<
2021-04-13
>
Ben Sless08:04:33

Is it by design that schema transformations like mu/assoc don't play well with an unregistered schema?

ikitommi09:04:56

not by design, could you repro if there is something that doesn’t work?

Ben Sless09:04:27

(def Foo
  [:map
   [:a int?]])

(mu/assoc Foo :b ::bar)
Minimal example

ikitommi09:04:25

you should declare the ::bar so that is is visible, either: 1. override the default registry 2. pass the registry into Foo when creting it (it closes over the creation time registry) 3. pass the registry into mu/assoc

(def registry
  (merge (m/default-schemas) {::bar int?}))

(def Foo
  (m/schema
    [:map
     [:a int?]]
    {:registry registry}))

(mu/assoc Foo :b ::bar)
;[:map 
; [:a int?] 
; [:b :user/bar]]

(mu/assoc [:map [:a int?]] :b ::bar {:registry registry})
;[:map 
; [:a int?] 
; [:b :user/bar]]

Ben Sless09:04:57

I managed to get myself into this corner like so: • wanted content dependent schema • wanted to parametrize the schema (makes it extensible) • figured out I'd do it by delaying registry building and schema compilation to run-time. • With registry I need ::my-schema • Can't transform anything with ::my-schema at compile time I can create a placeholder registry for it but it seems like it would lead to errors down the line

Ben Sless09:04:40

I'd be happy to adopt a better idea

ikitommi10:04:34

me too 🙂 spec partially checks the references eagerly, partially lazily (e.g. s/keys), malli is currently eager.

ikitommi10:04:34

there is an internal escape hatch: :ref doesn’t check the reference if :malli.core/allow-invalid-refs option is truthy.

ikitommi10:04:44

it is used with local registries, which can have… holes.

ikitommi10:04:39

(m/validate
  [:schema {:registry {::foo [:ref ::bar]}} ;; incomplete registry
   [:tuple {:registry {::bar int?}}
    ::bar ::foo]]
  [1 2])
; => true

ikitommi10:04:33

ideas welcome how to make this good.

Ben Sless10:04:25

I don't know if it's good, but perhaps a :delay or :defer schema, which delays registry lookup to validation, with ample warning, care, etc.

Ben Sless10:04:24

Perhaps even wrap it in a function which will always emit warnings when it's called, i.e.

(mu/assoc Foo :b (m/schema (m/defer ::bar)))
STDERR: Deferred Warning *at* - instances of deferred schema must be provided with a registry at run time!
You can also throw when instantiating an explainer, transformer, or validator from it, which is when you actually need the registry

ikitommi10:04:14

could it be just [:ref {:lazy true} ::bar]?

Ben Sless10:04:40

Ah, laziness has to be explicit

ikitommi10:04:49

oh, ref’s are lazy already :thinking_face:

Ben Sless10:04:15

yeah, this didn't work 🙂

Ben Sless10:04:20

we need lazier laziness

ikitommi10:04:42

try (m/-lazy ::bar options)

ikitommi10:04:05

refs resolve eager by default, but one can create lazy refs with that.

ikitommi10:04:36

(let [-ref (or (and lazy (-memoize (fn [] (schema (mr/-schema (-registry options) ref) options))))
                      (if-let [s (mr/-schema (-registry options) ref)] (-memoize (fn [] (schema s options))))
                      (when-not allow-invalid-refs
                        (miu/-fail! ::invalid-ref {:type :ref, :ref ref})))

Ben Sless10:04:40

Cool, it worked 🙂

Ben Sless10:04:49

Always good to know some black magic

ikitommi10:04:06

could make a version of that which doesn’t require the options.

Ben Sless10:04:35

It makes me wonder why [:ref {:lazy true} ::bar] didn't work

ikitommi10:04:00

it’s a property of the IntoSchema, not Schema instance.

ikitommi10:04:05

by design, all the IntoSchemas are crated using a function, which can take properties how the IntoSchema works. Easy to extend the system that way and DCE drops all the unneeded schemas.

ikitommi10:04:53

for example, it’s reletively easy to create custom collection schema types:

(defn -collection-schema [{type :type fpred :pred, fempty :empty, fin :in :or {fin (fn [i _] i)} :as opts}] ...)

ikitommi10:04:18

:ref has:

(defn -ref-schema
  ([]
   (-ref-schema nil))
  ([{:keys [lazy type-properties] :as opts}] ...))

ikitommi10:04:56

but, could lift the lazy into a :ref schema property too. so one can say [:ref {:lazy true} ::bar] as data.

ikitommi10:04:09

if you need that, please write an issue.

Ben Sless10:04:10

I wonder if I should settle for m/-lazy

Ben Sless10:04:36

If I should consider functions prefixed with - as implementation detail, then I'd say that I shouldn't and open that issue

ikitommi10:04:33

things starging with - are ok to use: https://github.com/metosin/malli#alpha

Ben Sless10:04:03

> might evolve during the alpha That's a risk I'm willing to take. I think if m/-lazy develops in any direction it won't be one which will have friction with what I'm trying to do, on the contrary. Thanks again for the help and guidance, you rock

Ben Sless09:04:09

Another issue I managed to stumble on, I defined a dependent schema like https://github.com/metosin/malli#content-dependent-simple-schema It works well but throws when I pass it to reitit routes when the coercion is compiled

Ben Sless09:04:20

Great, now I'm unable to reproduce it 😞

ikitommi09:04:25

you should declare the ::bar so that is is visible, either: 1. override the default registry 2. pass the registry into Foo when creting it (it closes over the creation time registry) 3. pass the registry into mu/assoc

(def registry
  (merge (m/default-schemas) {::bar int?}))

(def Foo
  (m/schema
    [:map
     [:a int?]]
    {:registry registry}))

(mu/assoc Foo :b ::bar)
;[:map 
; [:a int?] 
; [:b :user/bar]]

(mu/assoc [:map [:a int?]] :b ::bar {:registry registry})
;[:map 
; [:a int?] 
; [:b :user/bar]]

yuhan10:04:07

Are there built-in functions to throw errors on invalid input? The plans for instrumentation in the above issue are nice, but I'm looking for something simple like spec/assert

nilern10:04:23

You can always (assert (thingy-validator dada)) but the error is not so useful

yuhan10:04:26

Yeah, I wrote my own for now:

(defn malli-assert
  ([schema value]
   (malli-assert schema value ""))
  ([schema value msg]
   (when-not (malli/validate schema value)
     (throw (ex-info (clojure.string/join "/n"
                       (cons msg
                         (flatten
                           (malli.error/humanize
                             (malli/explain schema value)))))
              {:value value})))))

nilern10:04:55

Make a PR?

yuhan10:04:57

It's not ideal because humanize returns nested messages according to the path of the error, which I just flatten into a single string

yuhan10:04:54

Ok I'll submit an issue, just wanted to check if it was a design decision not to have an assert

nilern10:04:31

Maybe AssertionError would be more appropriate :thinking_face:

nilern10:04:23

And maybe use *assert* and make it a macro

nilern10:04:59

Spec assert seems to use ex-info and a separate *assert* equivalent var

jcf10:04:42

I have a follow up question from https://clojurians.slack.com/archives/CLDK6MFMK/p1618257034389400 regarding emitting configuration for clj-kondo to pick up (which is an awesome feature by the way!). I have schematised the following code:

(m/=> hash-map-by
  [:=> [:catn [:f [:fn ifn?]] [:coll coll?]] map?])

(defn hash-map-by
  "Returns a map of the items in `coll` keyed by `f`."
  [f coll]
  (into {} (map (juxt f identity)) coll))
The function takes an arbitrary function, f, and a collection that will be converted into a map by applying f and identity to each item in the collection. Pretty standard stuff. 🙂 When I emit clj-kondo config with (mc/emit!), I get the following EDN:
{:lint-as #:malli.schema{defn schema.core/defn},
 :linters {:type-mismatch {:namespaces {example.hash-map {hash-map-by {:arities {2 {:args [:fn :coll], :ret :map}}}}}}}}
Please note, the 2-arity args say :fn and :coll returning a :map which means I get linting issues with something like (hash-map-by :user/id [{:user/id 1} {:user/id 2}]). Is this a bug worthy of a pull request or am I once again demonstrating my naivety? 🙈

ikitommi10:04:17

currently there is no way to override per schema instance how the clj-kondo works, but would be easy to add. also, having an ifn? schema built-in, it could have the correct clj-kondo type. interested in a PR?

ikitommi10:04:27

for the latter that is.

ikitommi10:04:38

for the first, for the second, something like:

ikitommi10:04:12

[:fn {:clj-kondo/type :ifn} ifn?]

jcf10:04:04

I'm very interested in implementing this as we'd need it to complete the replacement of clojure.spec with Malli in our codebase, I think.

jcf10:04:51

I can create a PR for sure. 💯

jcf11:04:00

@U055NJ5CC can I just clarify what you're thinking in terms of a PR, please? I can add #'ifn? to the predicate-schemas and then these tests pass:

(testing "ifn schemas"
    (let [schema (m/schema ifn?)]
      (is (true? (m/validate schema (fn []))))
      (is (true? (m/validate schema (constantly 1))))
      (is (true? (m/validate schema :keyword)))
      (is (true? (m/validate schema (reify clojure.lang.IFn
                                      (invoke [_] "Invoked!")))))))
Is that what you had in mind when you mentioned having an ifn? schema built in?

ikitommi11:04:54

yes, but also mappings for transformers, generators, json-schema, humanized errors and clj-kondo.

jcf11:04:44

I'll take a look at implementing the full feature set for the ifn? domain. 👍

jcf11:04:26

I'll not implement proper generation of interesting functions. Don't want to put us all out of a job. 😉

jcf11:04:48

Oh, I think I see what you mean. You want ifn? to be parameterised so you can schematize the args and return values…?

jcf11:04:48

So in the schema generator you can do something more than this:

(defmethod -schema-generator 'ifn? [_ _] (gen/return ::ifn))

jcf11:04:30

I'd need to generate a function that returns valid data given valid arguments.

ikitommi11:04:45

really, why?

ikitommi11:04:04

there is already :=> and :function which have proper input & output generators

jcf11:04:09

Because I thought that was what you wanted. 🙂

jcf11:04:13

I think I misunderstood.

ikitommi11:04:33

no, just the simple thing, lke fn? but bit different 🙂

jcf11:04:36

If ifn? can remain a simple predicate, I should be able to have a first pass at a PR before the lunchtime walk. 🙂

jcf11:04:27

One pull request with my stab at adding ifn? to Malli's list of supported predicates. Gotta go for a lunchtime stroll with the pup; I'll check in when I'm back in about an hour. 🙇 https://github.com/metosin/malli/pull/416

yuhan18:04:45

Is it possible for schemas to self-reference? Naively trying to make a recursive schema causes a stack overflow:

(malli/schema
  ::tree
  {:registry (merge (malli/default-schemas)
               {::node :int
                ::tree [:or
                        ::node
                        [:tuple ::tree ::tree]]})})
 

ikitommi19:04:57

@qythium use the :ref, luke:

(mg/generate
  (m/schema
    ::tree
    {:registry (merge (m/default-schemas)
                      {::node :int
                       ::tree [:or
                               ::node
                               [:tuple [:ref ::tree] [:ref ::tree]]]})})
  {:seed 3})
; => [[-26764 [[1 73136] [13307055 -1]]] [-381 [[-3 587742] -243724556]]]

yuhan19:04:38

awesome, thanks!

yuhan19:04:24

So schema references only in :ref and :map have to be qualified keywords? Strings appear to work too, but not plain keywords

yuhan19:04:00

Seems like it, I found malli.core/-reference? in the source code which checks for qualified-keyword? or string?, but this doesn't seem to be documented

yuhan19:04:10

Also it seems that the ::foo shorthand syntax doesn't work on the http://malli.io playground

ikitommi19:04:00

yes, references should be qualified keywords or strings. http://malli.io … must be a sci-thing.

yuhan19:04:07

Ok, filed an issue on the http://malli.io github

yuhan19:04:38

Trying to use unqualified keys as references tripped me up quite a bit at the beginning, since this requirement isn't documented and the error message just says :malli.core/invalid-schema

ikitommi19:04:48

doc enhancement PRs are most welcome.

ikitommi19:04:42

(the error keyword could be better here)

yuhan19:04:44

I'll just file issues for now if that's ok - still in the early stages of experimenting with the library and not confident of writing docs

yuhan19:04:58

Another strange thing I encountered:

;; This works as a schema
[:map {:registry {::foo :int}}
 ::foo]

;; so does wrapping the keyword in a vector
[:map {:registry {::foo :int}}
 [::foo]]
;; to pass it options
[:map {:registry {::foo :int}}
 [::foo {:optional true}]]


;; These are ok too
[:map {:registry {::foo [:tuple :int :int]}}
 ::foo]
[:map {:registry {::foo [:tuple :int :int]}}
 [::foo {:optional true}]]


;; But not this??
(malli/schema
  [:map {:registry {::foo [:tuple :int :int]}}
   [::foo]])
;; => Execution error (ExceptionInfo) at malli.impl.util/-fail! (util.cljc:16).
;;    :malli.core/invalid-schema {:schema [:tuple :int :int]}