Clojurians Log v2

calling schema will turn the datastructure into a schema. So one difference is if you pass the top one an invalid schema, it will fail before you def it.

you get fast failing, as you mentioned, you also get the caching of a schema object vs being converted to a schema with every invocation

Thanks!

that call to m/schema will resolve the entire schema at that point based on the current global environment. if the global environment is the same when you use the schema, then there is no real difference. however, if you use custom options or start defining schemas to the global environment in a different order, you might want to delay m/schema calls to later.

I think it's generally ok to use either if you use top-level def over say local registries.

I am using this approach so that I have qualified schemas. https://github.com/metosin/malli/blob/master/docs/reusable-schemas.md#schemas-via-global-registry

that's a good approach

you can maybe get some inspiration from the metabase malli registry util namespace, too https://github.com/metabase/metabase/blob/master/src/metabase/util/malli/registry.cljc#L119

you shouldn't need to call m/schema manually then

what is mr/def?

I don't understand why not doing this (since it "fails fast"):

(defn register! [type ?schema]
  (swap! *registry assoc type (m/schema ?schema)))

It's because you can have mutually recursive schemas. you need to delay the creation of a schema until both are registered.

Unlike spec, malli does not allow a ref to be unbound when the schema is created.

Hmm... if I don't have cycles than it is ok?

probably. I'm thinking through the case where you have two specs: foo that uses bar. What happens if you change bar, does foo get updated? I think no, refs are cached in malli. I think spec has more dynamism here.

Ignore the mr/def... I read your link too fast and thought it was from the malli repo... and confused everything, lol.

Thinking through this makes me reconsider the open :multi approach here https://github.com/metosin/malli/pull/1162 If we bind too early, then we won't see future extensions. Perhaps we need another kind of registry, indirection-registry, that wraps top-level schemas in way that allows dependees to see future changes.

Maybe that's how var schemas work?

Talking about recursion... clicked your link... went to github... there was a link to discussion... now I am back here at slack, lol.

yes sorry, see the README.md changes. you might find it relevant.

I am wondering if using simple defs instead of a registry would be better.

i'll say that at my last job, we exclusively used (def Foo (m/schema ...)) style across like 500 schemas and never ran into issues

but we never messed with mutual recursion

@nbtheduke did you have any issues with nested schemas? there would be no intermediate names to break up the schemas right?

It feels so weird to use CamelCase in clojure... but it makes the schemas stand out, I guess.

i don't have access to the code anymore, so i'm recreating this from memory:

(def User (m/schema [:map [:username :string] ...]))

(def Users (m/schema [:vector User]))

(def UsersEndpointResponse
  (m/schema [:map [:users Users] ...]))

Yeah, I guess if you generated a swagger page it would have a large schema as the route's response, instead of a named one pointing to smaller named ones.

we didn't use swagger so we never thought about that

i will say that this occasionally annoyed us because in repl-driven dev, you'd change User and evaluate it and nothing would change, because they're all def references, so we just got into the habit of evaluating the whole file every time

I have a feeling that will also happen if you used a global registry, or even drop the m/schema

(our schemas would live in the.company.user.schema namespaces)

that extra dynamism is missing in malli because of how refs are resolved. but I wonder if using the #'var syntax to refer to other schemas fixes that.

I think you only get that dynamism with a local registry but that's cheating since you're defining the world effectively.

It would be nice to have a dev mode where extra indirections are added to pick up the latest schemas.

then they're "directly linked" in prod

about the swagger example, I was more talking about some of the implications of having the abstractions set up by using def + var derefs directly. printing the schema doesn't tell you what the name is. error messages are larger, and will be missing names.

I was curious if any of those came up.

did you ever have a problem where you couldn't figure out where a schema was defined that was printed in an error message, for example.

I guess preserving the provenance of schemas is a big advantage of using the global registry over defs.

not that i remember, but we were also moving from an existing json-schema validation system to malli and i was being fairly "one to one" in my translation, so a lot of those issues were the norm

did you translate json schema refs to malli def's?

we didn't use those

we'd have something like (def User (common/object {:username common/string ...})) and then (def Users (common/list User)) which would produce plain clojure maps that would be passed to a https://github.com/luposlip/json-schema at run-time (no caching)

where did the names like User and Users come from? did they exist in any form in the original json schema?

lol maybe we should move this to a new thread, sorry hoynk

No problem at all! Please talk as much as you like 🙂 Might teach me something.

@ambrosebs did you ever play with the dev mode with dynamism idea? I've been pondering something similar. Maybe a defschema macro that at dev time rewrites things like User to #'User (is that possible?)

I can't remember. Maybe this is related? https://github.com/frenchy64/malli/pull/28/files

there's some interesting ideas there. IIRC an immutable global registry that accepts hooks that recompile the registry if it detects changes.

it's as i described:

(ns common)

(defn object [args]
  {:type "object"
   :properties args})

(defn array [t]
  {:type "array"
   :items {:type t}})

(ns other)

(def User (common/object {:username common/string ...}))

(def Users (common/array User))

there's no existing json file that held these, they were built in clojure by hand using the handrolled system, and the resulting objects were passed to the lupolisp/json-schema library's validate function

so the basic conversion to malli was pretty easy

just go piece by piece, convert the resulting "json schema" declarations to malli schemas

ok, that's what you mean by it not being any worse than before. It was always just one huge schema blob.

does not sound fun with 500 schemas

hah yeah it was a ton of work, but we gained back a lot of performance/speed cuz malli is fast and our way of doing json schema was not

looping back to the original topic, I think it could have been just as fast to use a global registry but you'd also surface the useful abstractions you created with the def names in the schemas themselves.

(thinking through the pros and cons of all these similar approaches)

right

yeah, i briefly experimented with setting up the global registry as described in https://github.com/metosin/malli?tab=readme-ov-file#composite-registry section of the readme, that worked pretty well, but i never got around to moving our existing stuff over before i was laid off

I think this is one weakness of not being as prescriptive as spec (`s/def`) or schema (`s/defschema`) in this area. these issues are not obvious.

maybe even a malli.def namespace containing the suggested idiom in the README.md

that could help put this issue to bed for newbies

yeah that'd be sick

I'm going to try this "dynamic dev mode" thing, then I'll play with that idea.

this is what i came up with for lazytest (before i removed malli entirely due to lack of usage):

(ns lazytest.malli
  (:require
   [malli.core :as m]
   [malli.registry :as mr]))

(defonce ^:private registry* (atom {}))

(mr/set-default-registry!
  (mr/composite-registry
    ;; Built-in schemas
    (m/default-schemas)
    ;; Var registry
    (mr/var-registry)
    ;; Custom schemas
    (mr/mutable-registry registry*)))

(defn register-schema
  [k ?schema]
  (swap! registry* assoc k ?schema))

(defmacro register!
  "Borrowed from malli documentation."
  [k ?schema]
  (assert (qualified-keyword? k) "Must provide a qualified keyword")
  `(do (register-schema ~k ~?schema)
       nil))

having the default schemas, var schemas, and then a mutable registry at the end was really nice

Yeah that is nice. It could support dev-time metadata too, including line numbers.

That could even propagate to external tooling like clj-kondo or Typed Clojure. They now know where to point the user if they need to know where the original malli schema is.

nice way to compose with defn