Clojurians Log v2

Thanks this was really helpful!

Ok, I've published the repo here: https://github.com/Ramblurr/cljdocset It's a babashka cli tool and I think the docs it generates are on par with the auto-Dash docs, and even better in some ways. Would love some feedback if people take it for a spin. Ideas for improvments: 1. Add a web server and host Dash/Zeal feeds so no one needs to install this tool directly (then can use add the feed url to Dash/Zeal) 2. Explore how to integrate into http://cljdoc.org @lee If you want to give the code a gander and let me know if you see a way forward with this, I'll be happy to talk about what to do next.

Heya Casey! Are you planning on adding support to cljdoc? There's an issue for that: https://github.com/cljdoc/cljdoc/issues/646

I'm not a Dash user, but do have a license for cljdoc support reasons. If no other Dash enthusiast chimes in, I'll try to help you out.

I'm guessing you've already read https://github.com/cljdoc/cljdoc/blob/master/doc/cljdoc-developer-technical-guide.adoc#dash-integration

Yup I found both during my research. Currently I am implementing option 1 from that issue. I'm using hickory to parse the html and feed the vars into the SQLite database. It's working well! I wouldn't object to it being integrated into cljdoc itself, but to start I wanted something standalone just to prove it could work.

The Dash integration you mention is something that's only in the MacOS app and I suppose is doing something similar to what my code is doing. On Linux we have Zeal which can only consume prebaked docsets.

Let me know if you want/need anymore

Could you please share metosin/reitit and weavejester/hiccup?

@lee I've been using this heavily the past weeks and it is very useful! I think it would be great to implement this into cljdoc. We could also provide XML feeds so users get auto updates of the docs. How do you suggest we proceed? I am happy to help maintain/own the code in cljdoc.

That’s great Casey, I’ll take a peek at your code.

And I’ll try out your tool with Zeal.

If we integrate, would this be interesting to the author of Dash too?

FYI: The bb example from the README did not work for me, but I can run via clojure like so:

clojure -M -m cljdocset.cli build --output-dir ./out rewrite-clj/rewrite-clj

Ah interesting. You also download all images so docset is more truly offline.

FYI: medley is in the deps.edn but seems unused.

FYI: I installed Zeal via flatpak. The docset directory was different from what your README stated. I discovered the correct docset dir for my Zeal via Zeal->Edit->Preferences->Docset storage. For me it was: ~/.var/app/org.zealdocs.Zeal/data/Zeal/Zeal/docsets

I expect this is the same for Dash (only used ages ago to test) but I don't find the separation of the API into "Functions" "Macros" and "Variables" terribly useful for clojure libs.

Also probably same for Dash, but "Sections" section seems a bit weird, don't you think? It has headings from Guides with no context.

The zeal docs say I can search within a single docset via <docsetname>:<search string>. What determines <docsetname>? I can't seem to get it to work with rewrite-clj, for example.

Thinking about Dash again... Zeal and Dash seem to have a symbiotic relationship. Dash provided Zeal a bunch of docsets, and in return, Zeal promotes Dash as the solution on macOS. I don't think integrating this work into cljdoc would hurt Dash at all, but I'll probably ping the Dash author out of courtesy. I'm guessing he'd prefer your docsets because he would have to do no processing on them.

So, this looks all very interesting for integration. But what problems might we have? 1. Will the processing time and resources be an issue? We regenerate zip offline download for each request (no caching). This is pretty heavy but I haven't seen an issue with it yet. Probably due to light demand? Your docset processing is heavier. I wonder if that could cause an issue. 2. What about bundling the images? This is great in that the docset is now truly offline. But the download is bigger. And do we potentially get into any trouble for redistributing images? That might be copyrighted?

> Ah interesting. You also download all images so docset is more truly offline. > That's important to me, I like to be able to work offline Re medley: good catch thanks Re install dir: you must have installed it via a flatpak or something? Depends on your install method. But I can add the instructions for finding the dir. Good idea. Re Sections/Macros/Functions: I think the Sections isn't that useful, I added it after looking at the docsets from Dash that Anders shared. The only reason to have them is so that they are indexed and can thus be searched. I think the others are useful, not for browsing but when searching you get a nice little icon. Re Dash integration: I agree with you. Dash has a bunch of features that Zeal lacks and will probably never get. One of those is w/e feature in Dash that exists now to show cljdoc. I don't think he will be peeved, he may be inclined to use cljdoc's feed if you choose to adopt this. I doubt he will care one way or another if it stays in my little repo like it is now. Re rest: will return to this thread later this evening:pray:

Absolutely loving this btw. When my last macbook bites the dust the path for my return to linux has been paved.

("this evening" is now "next morning", but better late than never) > The zeal docs say I can search within a single docset via <docsetname>:<search string>. What determines <docsetname>? I can't seem to get it to work with rewrite-clj, for example. Good point! I will investigate > Will the processing time and resources be an issue? I can't really speak to that. I of course don't know the compute resources / budget of http://cljdoc.org. I think generating the docset should only be done when its first requested, but it should be cached, maybe with a TTL. Demand won't be that high I imagine, but that's always relative to http://cljdoc.org's capacity. > What about bundling the images? do we potentially get into any trouble for redistributing images that might be copyrighted? IANAL. But why would images be special case versus the text of the documentation itself? A cursory look at http://cljdoc.org and I can't find a TOS, so I think bundling for offline use seems reasonable and aligned with the spirit of documentation hosting. > This is great in that the docset is now truly offline. But the download is bigger. The download is bigger for containing images, but my experience says 99% of docsets do not contain images, and if they do its a badge or two in the README. I don't think it makes a meaningful difference. We could add some safeguards like max images / max size processed if we think it is a concern.

Cljdoc hosting is generously sponsored by Exocscale. We are currently running on a single compute instance with 8gb RAM and 4 CPUs. We make use of their Object Store for db backups. We out-source API analysis to CircleCI jobs, with their blessing (at least historically!). I'll ping the author of Dash, just to let him know what we are musing about, and to keep him in the loop.

> We out-source API analysis to CircleCI jobs, with their blessing (at least historically!). Interesting! We could do the same for the docset builds if CircleCI is still ok with it?

That's a possibility.

I think an initial, clear, and easy win would be to describe your new tool in the cljdoc docs. I'll go ahead and do that.

Sorry I don't follow, what do you mean by "describe the tool in the cljdocs"?

I think you misread maybe? I wrote: "describe your new tool in the cljdoc docs". I'll describe it in the https://github.com/cljdoc/cljdoc/blob/master/doc/userguide/for-users.adoc.

BTW, Dash author got back to me... no concerns at all.

cljdoc docs update: https://github.com/cljdoc/cljdoc/pull/1068

We could consider this done if we are happy with Option 1 of https://github.com/cljdoc/cljdoc/issues/646

I just re-read issue 646, nice work by that lread fellow! I'd like to meet him someday!

I was thinking more about how to support feeds. In Zeal you can add a feed url, a bit of xml described in 646. Each project in cljdoc would get its own feed url. Presumably a user could copy paste from somewhere (maybe the footer) a feed url. The issue is how and when cljdoc generates the docsets for the feeds. That could be tricky.

Well, we could be done for now, @ramblurr. My analysis from 646, suggested Option 1 (which you have kindly implemented!) if there are only a handful of people interested in this feature. I think that's the case here. Watcha think? Does that make sense for now?