Fork me on GitHub
#malli
<
2022-08-06
>
hjrnunes10:08:22

I need to validate a map-like type with metadata. Actually, values are collections - vector-like types - which also have metadata that needs validation. I've played around with the -map-schema and -collection-schema implementations and I can make them support my custom types easily, but I am unsure how I can express the metadata schema. I could use properties but I wonder if that makes sense. The metadata is as important as the rest. Actually it's where the most variation will be. Is there a better way than simply maintaining two schemas - one for the actual type and another for the metadata map - and running validation for each independently?

ikitommi06:08:54

currently, there isn’t an extension point for validation, like there is for most other schema applications (transforming, generating, …), so there isn’t an easy way to add a generic, properties-based declaration for meta-data, if there was, could be something like:

[:map {:metadata [:map [:closed :boolean]]} [:x :int]]

ikitommi06:08:49

if you can provide an example how and where you would like to define the metadata schemas, could suggest ways to build such thing. It’s definetely doable, many ways to do that.

ikitommi06:08:10

Please write an issue with some sample data?

hjrnunes07:08:21

Well, that's the thing. I'm not really sure how to best handle it. I can easily do it via properties like the snippet you shared, but it just felt odd to have schema in the properties. Maybe I'm just overthinking it.

ikitommi07:08:44

could also be a lookup, e.g. [:map {:domain/type ::object} …], so that the metadata-schemas are in a separate registry and looked out with a property-key. You can the create a helper that takes a schema and returns a metadata validator and use that when needed. keeps the validation of metadata and values as separate. should be easy’ish to do with schema walking.

Eugen21:08:11

hi, is there a guide to learn malli for dummies? I am new to clojure and I kind of struggle to get my head around malli. I would like to use it for data modeling - as an exploratory tool. I would like to define some entities and be able to customize the generated output. (have a list of names, companies, etc). I read through the section and I still feel like I am missing some context. For example, I would like to generate company names like: Acme + some random smallish number ?!: ACME 12 I have defined an ID like

(def id [:string {:min 1, :max 20}]) 
and I would like to limit what characters it should contain (it should have only printable characters for example)

hanDerPeder22:08:55

Malli, as spec, leverages the test.check library for generating values. I would start there. There are some very useful guides linked in the readme of that project.

hanDerPeder22:08:33

This one seems relevant

❤️ 1
Eugen22:08:36

I think my end goal is very similar to what @U07FP7QJ0 talked about here https://www.youtube.com/watch?v=ww9yR_rbgQs&amp;ab_channel=LambdaIsland . Thank you @U07FP7QJ0 for making the video ! I would love to have a more in-depth presenatation of that. If you have anything please share. BTW, can you share how the project evolved? It's been two years.

👍 1
pppaul13:08:54

(def id [:string {:gen/elements [list of names to use] :min 1, :max 20}]) 

👍 1
Eugen13:08:54

thanks for all the help, I am getting my head around it - but it's lots of new stuff to learn (test-check).

Eugen13:08:08

I am thinking of writing some generators that will leverage data sources like WikiData to get real world examples. This should be more to generate some lorem ipsum style of data.

pppaul13:08:10

if you aren't experienced with generators there is a bit of a learning curve, but it's not too hard. i find programmatically attaching the generators harder than making them.

Eugen13:08:50

thanks for the feedback. That is a question I have - how to use the generators in the app. Since it's a lot of new stuff I am bound to make mistakes, hence me asking.

pppaul13:08:42

if you have schema's that can generate nested things, you will want to control those generators a lot (any? is a very nasty one).

Eugen13:08:42

if I make it I will be bale to write on CV: generator tamer 🙂

pppaul13:08:56

also, the gen/ properties don't work together, so if you want to use elments->fmap you have to use gen/gen instead. if you are converting to json, you need to watch out for symbol/string/keyword keys causing dupe key issues

Eugen13:08:35

don't know what that means for now, but I'll try to keep an eye out for it.

pppaul13:08:48

maybe not a normal issue to run into, but it was an issue for me when i was making generators for schemas that i converted from json to malli (i didn't own the schema)

pppaul14:08:22

{'key :a :key :c "key" :d} that in json is a map where all keys are the same, and thus invalid

pppaul14:08:22

anyway, biggest issue with generators is the recursive stuff, so if you have any of that you have to take care of it right away and figure out how to limit the recursions, or your data is going to get extremely large

Eugen14:08:54

I'll try to stay away from recursion