This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2020-02-24
Channels
- # announcements (5)
- # aws (24)
- # babashka (41)
- # beginners (130)
- # bristol-clojurians (2)
- # calva (39)
- # chlorine-clover (64)
- # cider (30)
- # clojure (202)
- # clojure-belgium (1)
- # clojure-dev (99)
- # clojure-europe (5)
- # clojure-hungary (4)
- # clojure-italy (10)
- # clojure-losangeles (8)
- # clojure-nl (11)
- # clojure-norway (6)
- # clojure-spec (7)
- # clojure-uk (12)
- # clojurescript (52)
- # core-typed (26)
- # cursive (19)
- # data-science (19)
- # datomic (19)
- # duct (10)
- # emacs (17)
- # fulcro (22)
- # graalvm (11)
- # jobs (3)
- # kaocha (28)
- # leiningen (6)
- # lumo (2)
- # malli (10)
- # nrepl (2)
- # off-topic (23)
- # pathom (2)
- # pedestal (7)
- # re-frame (3)
- # reagent (30)
- # reitit (2)
- # remote-jobs (2)
- # shadow-cljs (77)
- # sql (10)
- # test-check (22)
- # tools-deps (37)
- # vscode (1)
- # yada (3)
Hey everyone. Love using this tool.
Seems like validation works blazing fast, however mp/provide
is slow.
for example - creating a schema from a vector of 200 items will take more than 2 minutes. The processing growth is linear so 20,000 would take about 9 hours :(.
I think it has to do with the fact the schema is built using exceptions.
oh, that's horrible. The only really perf optimized trails are -validator
, -explainer
and -transformer
paths of schemas. Ideas welcome how to make the providers faster.
the algo is currently brute force, but I think if there were some kind of type-based narrower, it would not throw. e.g. Java Long
would only be checked against predicates that work with that.
off the top of my head, wouldn't "errors as values" be faster? i.e. instead of throwing, return the data in a map with an :error
key
Errors as values would be slower in the happy case / success - one would have to check always is the result an error or not. Try is basically zero cost in JVM, so the perf penalty (of throwing) only happens at the unhappy path.
I think the exceptions originate from the fact that the inferrer tries to make all registered schemas from it and they throw on invalid childs. E.g. for value 1
, it tries to make a map with [:map 1]
, which throws on creation.
narrowing possible schemas to test would make it orders of magnitude faster, my guess on 2 (orders)
There’s IIRC some newer optimizations in post-JDK9 versions for making stack&exception handling way faster, but I haven’t really looked into those beyond StackWalker…at least one of them was a throw-without-resolving-stack kind of operation; the stack unrolling is what generally is the slow part when exception gets thrown.