Fork me on GitHub
#malli
<
2022-01-05
>
Ben Sless07:01:37

Filed under "thanks I hate it" https://github.com/bsless/malli-jackson

🎉 3
eskos09:01:42

Can/could malli be used to extract/subselect a structure from a larger data structure? What I’m trying to achieve is that my CLJS app has one huge app-db ratom and from that I’d like to subselect certain parts which gets stored persistently with as little pain and whacking as possible, and I sort of had a thought that hey, it’d be nice to define the subselect as malli schema so that I could validate that the subselect is even persistable, plus malli schema is way easier to document than whatever else I’ve been thinking as solution to this so far. From system integrity’s point of view this could also work quite nicely, as marking specific branches/leaves of the data structure as persistable would mean writing a proper validator for it -> I wouldn’t have to worry someone breaking the, uh, persistability. And of course this is pretty straightforward thing to document to team members, both current and those who’ll join the project later that hey, just update the persistent malli schema if your data needs to be persisted.

ikitommi10:01:32

There is mt/strip-extra-keys-transformer to "select" a sub-schema.

eskos13:01:38

Hmm, interesting. :thinking_face:

lauri12:01:54

The following code snippet works with malli version 0.6.2:

(def instant
  (m/-simple-schema
   {:type            'instant
    :pred            (partial instance? Instant)
    :type-properties {:error/message "should be instant"}}))

(def Foo
  [:schema
   {:registry
    {::xyz
     [:map [:baz instant]]}}
   ::xyz])

(mu/merge [:map [:foo string?]]
          [:map [:bar Bar]])
=>
[:map
 [:foo
  [:schema
   {:registry #:test-namespace{:xyz [:map [:baz instant]]}}
   :test-namespace/xyz]]
 [:bar string?]]
but fails since version 0.7.0 with:
Execution error (ExceptionInfo) at malli.core/-fail! (core.cljc:137).
:malli.core/invalid-schema {:schema instant}
How can i execute it with the newest version on malli?

lauri13:01:10

when the type of the :baz field is changed to string? then it also works with the latest version but i’d like to be able to use a custom type..

lauri13:01:06

seems that the malli.core/schema function fails in the newest version:

(m/schema Foo)
Error printing return value (ExceptionInfo) at malli.core/-fail! (core.cljc:137).
:malli.core/invalid-schema {:schema instant}

ikitommi10:01:14

could you retest with latest code in master?

ikitommi10:01:20

e.g. property registries were written as forms, which was a bad idea as there is no way to re-construct non-registered schemas using just the m/type information.

lauri12:01:29

just checked with master and it works well now. Thanks a lot!

annapawlicka20:01:44

Hey folks, I’m trying to figure out if there’s a way to merge fields inside of registry. I’ve tried utility functions, tried declarative :merge , they don’t work. Here’s a small example of what I’m trying to do:

(def schema-registry-sample
  {::payment-authorization [:map
                            [:id :string]
                            [:approvalNumber :string]
                            [:authResultDescription :string]
                            [:authorizationDate :string]]

   ::sales-order-line-item [:map
                            [:id :string]
                            [:authorization [:maybe
                                             [:merge
                                              ::payment-authorization ;; <<<< I want to merge this with the fields below
                                              [:map
                                               [:type :string]
                                               [:activationCode :string]
                                               [:productCode :string]]]]]
                            [:offer {:optional true} [:multi {:dispatch :offerProgram}
                                                      ["INSTANTSAVINGS" [:merge
                                                                         ::offer ;; <<<< I want to add required field based on a certain condition
                                                                         [:map [:gtin :string]]]]
                                                      [::m/default ::offer]]]]

   :sales-order [:map
                 [:paymentAuthorization ::payment-authorization]
                 [:salesOrderLineItem [:vector ::sales-order-line-item]]]})

(def sales-order-explainer-sample
  (m/explainer [:schema {:registry schema-registry-sample}
                ::sales-order]))
The example here doesn’t work
{:type clojure.lang.ExceptionInfo
   :message ":malli.core/invalid-schema {:schema :merge}"
   :data {:type :malli.core/invalid-schema, :message :malli.core/invalid-schema, :data {:schema :merge}}
   :at [malli.core$_fail_BANG_ invokeStatic "core.cljc" 137]}
Other merge approaches fail as well. Any ideas? Is this just not doable right now?

ikitommi16:01:45

hi @U05087MJH! the :merge is not enabled by default. You can register it globally with:

(mr/set-default-registry!
 (mr/composite-registry
  (m/default-schemas) ;; malli defaults
  (mu/schemas))) ;; plus the utility schemas

ikitommi16:01:16

so, this works:

(ns demo
  (:require [malli.registry :as mr]
            [malli.core :as m]
            [malli.util :as mu]))

(mr/set-default-registry!
 (mr/composite-registry
  (m/default-schemas) ;; malli defaults
  (mu/schemas))) ;; plus the utility schemas

(def schema-registry-sample
  {::payment-authorization [:map
                            [:id :string]
                            [:approvalNumber :string]
                            [:authResultDescription :string]
                            [:authorizationDate :string]]

   ::sales-order-line-item [:map
                            [:id :string]
                            [:authorization [:maybe
                                             [:merge
                                              ::payment-authorization ;; <<<< I want to merge this with the fields below
                                              [:map
                                               [:type :string]
                                               [:activationCode :string]
                                               [:productCode :string]]]]]
                            [:offer {:optional true} [:multi {:dispatch :offerProgram}
                                                      ["INSTANTSAVINGS" [:merge
                                                                         ::offer                                                                  [:map [:gtin :string]]]]
                                                      [::m/default ::offer]]]]
   ::offer [:map]
   ::sales-order [:map
                  [:paymentAuthorization ::payment-authorization]
                  [:salesOrderLineItem [:vector ::sales-order-line-item]]]})

(def sales-order-explainer-sample
  (m/explainer [:schema {:registry schema-registry-sample}
                ::sales-order]))

ikitommi16:01:40

If you want spec-like custom mutable registry, here’s the thing:

(ns demo
  (:require [malli.registry :as mr]
            [malli.core :as m]
            [malli.util :as mu]))

(def registry* (atom {}))

(defn register! [type ?schema]
  (swap! registry* assoc type ?schema))

(mr/set-default-registry!
 (mr/composite-registry
  (m/default-schemas)
  (mu/schemas)
  (mr/mutable-registry registry*)))

(register!
 ::payment-authorization
 [:map
  [:id :string]
  [:approvalNumber :string]
  [:authResultDescription :string]
  [:authorizationDate :string]])

(register!
 ::sales-order-line-item
 [:map
  [:id :string]
  [:authorization [:maybe
                   [:merge
                    ::payment-authorization
                    [:map
                     [:type :string]
                     [:activationCode :string]
                     [:productCode :string]]]]]
  [:offer {:optional true} [:multi {:dispatch :offerProgram}
                            ["INSTANTSAVINGS" [:merge
                                               ::offer
                                               [:map [:gtin :string]]]]
                            [::m/default ::offer]]]])

(register!
 ::offer
 [:map])

(register!
 ::sales-order
 [:map
  [:paymentAuthorization ::payment-authorization]
  [:salesOrderLineItem [:vector ::sales-order-line-item]]])

(def sales-order-explainer-sample
  (m/explainer ::sales-order))

annapawlicka21:01:30

Thank you @U055NJ5CC! I just realized I posted incomplete example, but you made it work anyway 🙂

annapawlicka20:01:36

@U055NJ5CC Any chance to merge the mu/schemasinto my schema? I’d like to make the registry in this ns immutable as we’ll be using a lot of different schemas and there’s always a risk some other ns will set the global registry to something else. Simple merging doesn’t seem to work.

ikitommi20:01:09

I think it’s possible. Need few minutest to test that one out.

ikitommi20:01:17

e.g. create a common registry for the base-schemas (immutable), pass it as argument to m/schema , can still add local schemas on top of it.

ikitommi20:01:32

most workers (validators, explainers, parsers, unparsers, generators) are cached to the instance, so first m/validate on the Schema instance will create the validator, next will use the cached one.

ikitommi20:01:44

happy to give a tech talk / discussion about malli for you team you are interested. There are lot’s of options of doing things. Also curious on where are you using it.

ikitommi08:01:22

two more options (added to gist):

;;
;; 3) registering utility schema via local registry (requires latest code from MASTER)
;;

(def schema-registry-sample
  {:merge (mu/-merge)
   ::payment-authorization [:map
                            [:id :string]
                            [:approvalNumber :string]
                            [:authResultDescription :string]
                            [:authorizationDate :string]]

   ::sales-order-line-item [:map
                            [:id :string]
                            [:authorization [:maybe
                                             [:merge
                                              ::payment-authorization ;; <<<< I want to merge this with the fields below
                                              [:map
                                               [:type :string]
                                               [:activationCode :string]
                                               [:productCode :string]]]]]
                            [:offer {:optional true} [:multi {:dispatch :offerProgram}
                                                      ["INSTANTSAVINGS" [:merge
                                                                         ::offer [:map [:gtin :string]]]]
                                                      [::m/default ::offer]]]]
   ::offer [:map]
   ::sales-order [:map
                  [:paymentAuthorization ::payment-authorization]
                  [:salesOrderLineItem [:vector ::sales-order-line-item]]]})

(def sales-order
  (m/schema
   [:schema {:registry schema-registry-sample}
    ::sales-order]))

ikitommi08:01:37

;;
;; 4) using local component directly
;;

(def Merge (mu/-merge))

(def schema-registry-sample
  {::payment-authorization [:map
                            [:id :string]
                            [:approvalNumber :string]
                            [:authResultDescription :string]
                            [:authorizationDate :string]]

   ::sales-order-line-item [:map
                            [:id :string]
                            [:authorization [:maybe
                                             [Merge
                                              ::payment-authorization ;; <<<< I want to merge this with the fields below
                                              [:map
                                               [:type :string]
                                               [:activationCode :string]
                                               [:productCode :string]]]]]
                            [:offer {:optional true} [:multi {:dispatch :offerProgram}
                                                      ["INSTANTSAVINGS" [Merge
                                                                         ::offer [:map [:gtin :string]]]]
                                                      [::m/default ::offer]]]]
   ::offer [:map]
   ::sales-order [:map
                  [:paymentAuthorization ::payment-authorization]
                  [:salesOrderLineItem [:vector ::sales-order-line-item]]]})

(def sales-order
  (m/schema
   [:schema {:registry schema-registry-sample}
    ::sales-order]))

annapawlicka20:01:15

Thank you! It looks like there’s quite a few options. We might take you up on your tech talk offer but we need to gather a list of questions first 🙂 We’ve been slowly replacing Plumatic schema with malli. Plus we started adding malli to a recent project that had no Plumatic schema - we use it to validate payload from services we call, but also to validate data that we send in our requests to them. Generation of test data is there as well but I think validation is the biggest part. And since these are all different data sources that we validate, we’re trying to find the best way to organize it. We’ve just started with it in December IIRC, so it will possibly grow. It also helped us to identify how much of the payload we actually use (it’s a big project, different graphql resolvers using the same source of data in different ways), and helped us make a decision around choosing the right response version. @UEH4D93GS has been looking into it from a different project than I so he might have a different perspective.