Is there a simple flow to run generative testing on a function defined with mx/defn?
mi/check can only run it for all instrumented functions
Unless I am missing something, there is no way to get the schema of a function defined with mx/defn as the schema is registered as a side effect?
we have our own malli/defn at metabase that I wrote some code to do that with
it is all open source, let me find the gist with the generative testing stuff
metabase.util.malli.defn/defn: https://github.com/metabase/metabase/blob/master/src/metabase/util/malli/defn.clj#L60-L106
check-fn:
https://github.com/metabase/metabase/blob/master/test/metabase/permissions/util_test.clj#L208-L213
then calling it:
(mu/defn square :- :int
[x :- [:int {:min 0 :max 100}]]
(if (= x 3) "woops, 3 doesn't work" (* x x)))
(check-fn #'square)
returns
{:total-nodes-visited 2,
:depth 0,
:pass? false,
:result "Invalid output: [\"should be an integer, got: \\\"woops, 3 doesn't work\\\"\"]",
:time-shrinking-ms 0,
:smallest [(3)],
:malli.core/result #error {
:cause "Invalid output: [\"should be an integer, got: \\\"woops, 3 doesn't work\\\"\"]"
:data {:type :metabase.util.malli.fn/invalid-output, :error {:schema :int, :value "woops, 3 doesn't work", :errors ({:path [], :in [], :schema :int, :value "woops, 3 doesn't work"})}, :humanized ["should be an integer, got: \"woops, 3 doesn't work\""], :schema :int, :value "woops, 3 doesn't work", :fn-name square}
:via
[{:type clojure.lang.ExceptionInfo
:message "Invalid output: [\"should be an integer, got: \\\"woops, 3 doesn't work\\\"\"]"
:data {:type :metabase.util.malli.fn/invalid-output, :error {:schema :int, :value "woops, 3 doesn't work", :errors ({:path [], :in [], :schema :int, :value "woops, 3 doesn't work"})}, :humanized ["should be an integer, got: \"woops, 3 doesn't work\""], :schema :int, :value "woops, 3 doesn't work", :fn-name square}
:at [metabase.permissions.util_test$fn__186566$fn__186569 invoke "NO_SOURCE_FILE" 229]}]
:trace } Thanks for sharing! I wonder what the memory issue is with mx/defn, its a private slack convo? Maybe it has been fixed now? I ended up doing this:
;; using :malli/schema in metadata: instrumentation not reloaded when schema changes
;; using mx/defn: hard to access function schema
(defn get-schema-from-var [v]
(let [m (meta v)]
(get-in (m/function-schemas)
[(symbol (str (:ns m)))
(symbol (:name m))
:schema])))
(defmacro check
"Given a function, it will grab its schema from the registry of
instrumented functions and run generative testing on it"
[func]
`(let [func# (meta (var ~func))
ns# (:ns func#)
schema# (get-in (m/function-schemas)
[(symbol (str ns#))
(symbol (:name func#))
:schema])]
(assert schema# ~(str func " is not instrumented"))
(tests
~(str "Generative testing of " func)
(mg/check schema# ~func) := nil)))
so when I want a function to be tested with generative testing I do
(check my-function)I'm trying to write a Malli spec to parse IRC commands for fun How can I express optional sequence items when it involves the sequence ending? I've tried a lot of things and can't seem to figure out how to make the end of sequence nil value validate e.g.
["ADMIN"] => valid
["ADMIN" "mike"] => also valid
Thanks!
Looks like this:
[:cat [:= "ADMIN"] [:? [:= "mike"]]]
Or I misunderstood what do you want