Fork me on GitHub
#cljdoc
<
2021-08-24
>
lread18:08:55

Random cljdoc dev team code observation: we named the user config file cljdoc.edn and the API analysis results files cljdoc.edn. This can make things confusing when rooting through the code and figuring out which is which.

martinklepsch22:08:42

Yup 😅 I think the analysis one was the first one to exist and then for the “user facing api” in people’s repos any other name than cljdoc.edn felt wrong

lread22:08:55

Yeah, makes sense!

lread22:08:29

Huh, I don’t think we do any validation on the user’s cljdoc.edn. Well… other than reading it as edn. Probably not a great idea to add in strict validation now for existing stuff? (Or maybe we could but only apply on new builds and not rebuilds). I will likely validate :cljdoc/languages and fail a build if there are unexpected values for that.

martinklepsch22:08:23

Oh yeah, that sounds like a great idea!

lread22:08:20

Spec or malli… hmmm… probably spec because we are already using spec…

lread23:08:59

But I might start with malli cause I like it and know it better….

martinklepsch11:08:58

Yeah, I’ve also been using malli lately, seems nice 🙂

lread11:08:17

Yeah, it does! But currently scratching my head on a malli schema that will work for user cljdoc.edn! Probably just my newbieness.

martinklepsch12:08:19

Yeah, also still finding my way around. At some point I tried to merge schemas and that seemed non trivial. In the end I just ended up appending to the [:map ] vector using into — it’s just data 🙂

lread12:08:19

Interesting! My goal is to validate cljdoc.edn but also provide sensible error messages to the library author. If that is not easy with malli or spec I just might end up rolling my own wee validate fn.

martinklepsch12:08:34

yeah that makes sense

martinklepsch12:08:48

The error messages malli provides look really great though

martinklepsch12:08:04

What is it you’re stuck with right now?

lread12:08:19

I've tried humanize support and found it pretty nice!

lread12:08:32

Not sure how to express our doc tree structure. Seems like a recursive schema? But not sure yet. Playing around....

martinklepsch12:08:09

Ah yeah, that is recursive. This looks interesting https://github.com/metosin/malli/pull/209

lread12:08:24

Yeah, been looking at that and the recursive example in the readme but am not grokking yet!

lread17:08:17

It might not be that hard after all, after some playing around, here’s user cljdoc.edn malli schema so far:

[:schema
 {:registry {"docs" [:catn
                     [:title string?]
                     [:opts [:map {:closed true}
                             [:file {:optional true} string?]]]
                     [:children [:* [:schema [:ref "docs"]]]]]}}

 [:map {:closed true}
  [:cljdoc.doc/tree {:optional true}
   [:vector {:min 1} [:+ "docs"]]]
  [:cljdoc/include-namespaces-from-dependencies {:optional true}
   [:vector {:min 1} symbol?]]
  [:cljdoc/languages {:optional true}
   [:set {:min 1} [:enum "clj" "cljs"]]]]]

lread17:08:42

(with the new languages support)

lread20:08:07

I’m finding that adding reasonable constraints helps (especially for recursive bit when generating samples, I otherwise suffer stack overflow exceptions), so now have:

[:schema
 {:registry {"docs" [:catn
                     [:title [:string {:min 1 :error/path ["doc-title"]}]]
                     [:opts [:map {:closed true :error/path ["doc-opts"]}
                             [:file {:optional true} [:string {:min 1}]]]]
                     [:children [:repeat {:min 0 :max 20} [:schema [:ref "docs"]]]]]}}

 [:map {:closed true}
  [:cljdoc.doc/tree {:optional true}
   [:vector {:min 1} "docs"]]
  [:cljdoc/include-namespaces-from-dependencies {:optional true}
   [:vector {:min 1} [symbol? {:error/path ["namespace-name"]}]]]
  [:cljdoc/languages {:optional true}
   [:set {:min 1} [:enum {:error/path ["language"]} :clj :cljs]]]]]
Don’t know if I should be checking string content so am only checking for min length of 1 for now to help with typos. I can’t remember if there are any db limitations for max sizes of these strings when they get stored… I am remembering blobs… but will check. Could check cljdoc.edn file size itself to see if it seems reasonable-ish. Will continue to explore what error message reporting gives, but for our structure it seems to help to define some :error/paths. More tweaks to come as I learn…

👍 3
martinklepsch21:08:11

I’m also worked on some Malli stuff today, thanks for sharing this I think I might just steal some of it 🙂

lread21:08:08

Awesome! Don't forget I'm a newbie though (have a question over on #malli right now). I'll post progress here as I learn, which might just end up being me talking to myself, but it won't be the first time that has happened!

martinklepsch22:08:09

I want to do a twitch programming stream some time, I expect that to be a similar experience 😄

lread22:08:07

Ha! I always reassure myself that at least I am good company!

lread22:08:31

I won’t rename the physical filenames, but will likely clarify the code by renaming some vars/fns.

lread22:08:01

And not griping, just observing, could have been me who did this! simple_smile