Fork me on GitHub
#malli
<
2021-07-18
>
Vincent Cantin10:07:06

I have a schema that parses a value correctly but then the unparse operation returns :malli.core/invalid. Is it a known problem for some kind of schema or should I post a bug report?

Vincent Cantin11:07:22

To reproduce:

(let [my-schema (m/schema
                  [:and
                   list?
                   [:catn
                    [:wrapper [:= 'value]]
                    [:wrapped int?]]])]
  (->> (m/parse my-schema '(value 5))
       (m/unparse my-schema)))

Vincent Cantin11:07:30

m/unparse works if I remove list? from the schema, but then it's not the schema I need.

Vincent Cantin11:07:37

Is there an option in catn to specify the list of container of the sequence?

ikitommi12:07:11

currently no, but definetely it should.

ikitommi12:07:59

maybe something like?

[:cat {:type :vector} :int :int]

Vincent Cantin12:07:23

yes, but the name "type" is not descriptive enough as it relates to the container's type and not the sequence itself.

Vincent Cantin13:07:09

Right now, I am trying to port my model from minimallist to malli in the Vrac project, so I have less things to maintain and it's better for the users if they face a mainstream library like Malli.

👍 3
ikitommi13:07:21

have you checked the bundle size? malli starts from ~2kb (just the minimal working set of schemas), but is quite big if all schemas are used (40kb?). with all the bells & whistles, it can be >100kb.

ikitommi13:07:21

I’m not sure how :type and :coll-type are that different, but something like that would be good.

👍 3
Vincent Cantin13:07:23

:type will be fine

Vincent Cantin13:07:50

the bundle size won't be a problem, I can parse things offline.

Vincent Cantin18:07:21

:tuple would also need to have the {:type :list} option.

Vincent Cantin10:07:25

While chaining parse and unparse, I realized that the API would be more friendly to -> if the parameter order of ?schema and value were exchanged.

3
ikitommi13:07:11

did a quick run on improving performance of collection transformations, got easy x5 for CLJ in simple test:

(let [schema [:map
              [:id :string]
              [:type :keyword]
              [:address
               [:map
                [:street :string]
                [:lonlat [:tuple :double :double]]]]]
      decode (m/decoder schema (mt/json-transformer))
      json {:id "pulla"
            :type "herkku"
            :address {:street "hämeenkatu 14"
                      :lonlat [61 23.7644223]}}]
  ;; 920ns => 160ns
  (cc/quick-bench
    (decode json)))

ikitommi13:07:04

also, coercion (transform + validate) is 5x faster with that data compared to Plumatic Schema. Which is kinda nice as have considered (the awesome) Plumatic as a performance reference.

ikitommi13:07:25

merged in master.

Ben Sless13:07:18

I think you can close over the iteration completely

Ben Sless13:07:29

@U055NJ5CC you can gain at least 10% and probably gain some mechanical sympathy by closing over the transforms:

(map
 (fn [[k v]]
   (fn [^Associative x]
     (if-let [xe (.entryAt x k)]
       (.assoc x k (v (.val xe)))
       x)))
 ts)
then reduce over them with -comp

Ben Sless13:07:17

I can PR this if you'd like

ikitommi18:07:43

looks good. But, I thing the varargs version of -comp is actually slow as it uses the sequence abstraction, could be rewritten to use iterator? PR (and perf test before & after) most welcome.

Ben Sless19:07:47

I unrolled it to 16 args

ikitommi19:07:39

already this is much faster that the current:

(defn -comp
  ([] identity)
  ([f] f)
  ([f g] (fn [x] (f (g x))))
  ([f g h] (fn [x] (f (g (h x)))))
  ([f1 f2 f3 & fs]
   (let [fs (into [f1 f2 f3] fs)]
     (fn [x] (let [i (.iterator ^Iterable fs)]
               (loop [x x] (if (.hasNext i) (recur ((.next i) x)) x)))))))

Ben Sless19:07:32

I'll send an organized PR tomorrow, today is getting late

Ben Sless07:07:17

I wonder if there's an optimal maximum arity

ikitommi13:07:32

My initial thought was that having transform and validation as separate steps can actually make it faster - as the created transformation chain is usually small enough to fit into the JVM inlining budget - while validation is always “complete” and generated more code. Having those in one sweep means more code. Haven’t looked at the perf profile, so just quessing.

Ben Sless13:07:16

I was just thinking about this recently - would interleaving transform and validation be faster? Since it involves lots of iteration and allocation, we can't necessarily assume JIT friendly code would be faster than a single pass over two passes

Ben Sless13:07:12

> entryAt 😄

awesome 3
ikitommi13:07:59

the initial code was using reduce for all, which does a lot of things.

ikitommi13:07:47

stack size from 36 -> 6 (on malli side).

ikitommi13:07:54

but thanks @ben.sless for the validation perf, mostly same optimizations for transformers 🙇

Ben Sless13:07:37

I thought it looked familiar :thinking_face:

ikitommi13:07:06

reduce-kv should be fast, but for some reason, it’s not always used instead of reduce. Recall there was a related bug in clojure for this. writing by hand is always fast 🙂

Ben Sless13:07:23

Have you had a chance to look at my proposal on https://github.com/metosin/malli/issues/474?

Ben Sless13:07:14

I have no idea how to compile my proposal to something efficient, though I assume it is possible and someone clever like nilern could do it

Vincent Cantin20:07:00

It is possible in Malli to write a schema which represent Clojure's destructurations of a map? For example, something which would parse this kind of data:

{a :a
 b :b
 :keys [c d]
 :e/keys [f g]
 :as h}

ikitommi07:07:06

I don't think there is. Does spec have something for this?

ikitommi07:07:33

and what would be the expected parse result?

Vincent Cantin07:07:59

I don't know if spec can do something like that. In minimallist, I approach this problem by having map-of taking a schema of a pair [key value], so I am able to use :alt on the pair to valid multiple different cases where the key and the value are correlated.

ikitommi07:07:15

maybe m/parse could have a custom user-defined property to allow users to give the parse & unparse functions for the given schema.

Vincent Cantin07:07:25

An escape hatch, yes it would work, but the hardship would be on the user.

☝️ 4
ikitommi07:07:31

could you write a minimallist sample for that case? Sounds interesting

Vincent Cantin07:07:28

The escape hatch may not work well for the user in the case the model is using recursion, like in this example. That would become Malli -> fn -> Malli -> fn ...

respatialized20:07:04

Hi all! I've been working for a while on fabricate, a static website generator that takes advantage of malli in order to both provide a model for semantically valid HTML/Hiccup forms and to define its own order of operations. I wrote a post about my experience using malli schemas to define a finite state machine that dispatches functions on the basis of the schemas matched by the data passed in to them, which I have taken to calling "finite schema machines." https://fabricate-site.github.io/fabricate/finite-schema-machines.html I'd be really interested in what other users of malli think about this concept. It's highly experimental, but I've already found it quite interesting and useful.

👍 16
Ben Sless09:07:57

This looks very interesting and I've had FSMs on my mind recently. Will need to give a deeper look

Ben Sless10:07:19

Wonder how this relates to dependent types

respatialized12:07:58

I've never worked with dependent types, but my impression of malli is that you get a lot of the benefits of dependent types without a static type system. that said, when I think about how to validate/check these FSM definitions for cycles or non-termination at definition/compile time, I wonder how close I'm actually getting to reinventing a type system.

6