Fork me on GitHub
#malli
<
2020-02-24
>
Scar14:02:29

Hey everyone. Love using this tool. Seems like validation works blazing fast, however mp/provide is slow. for example - creating a schema from a vector of 200 items will take more than 2 minutes. The processing growth is linear so 20,000 would take about 9 hours :(. I think it has to do with the fact the schema is built using exceptions.

đź‘€ 4
ikitommi17:02:52

oh, that's horrible. The only really perf optimized trails are -validator, -explainer and -transformer paths of schemas. Ideas welcome how to make the providers faster.

ikitommi17:02:59

the algo is currently brute force, but I think if there were some kind of type-based narrower, it would not throw. e.g. Java Long would only be checked against predicates that work with that.

Ben Sless08:02:11

off the top of my head, wouldn't "errors as values" be faster? i.e. instead of throwing, return the data in a map with an :error key

ikitommi15:02:20

Errors as values would be slower in the happy case / success - one would have to check always is the result an error or not. Try is basically zero cost in JVM, so the perf penalty (of throwing) only happens at the unhappy path.

ikitommi15:02:19

I think the exceptions originate from the fact that the inferrer tries to make all registered schemas from it and they throw on invalid childs. E.g. for value 1, it tries to make a map with [:map 1], which throws on creation.

ikitommi15:02:42

narrowing possible schemas to test would make it orders of magnitude faster, my guess on 2 (orders)

eskos12:02:27

There’s IIRC some newer optimizations in post-JDK9 versions for making stack&exception handling way faster, but I haven’t really looked into those beyond StackWalker…at least one of them was a throw-without-resolving-stack kind of operation; the stack unrolling is what generally is the slow part when exception gets thrown.

eskos12:02:41

It is also maybe possible to memoize the exception itself, although I wonder if that would work at all with the ex-info…