Fork me on GitHub
Daw-Ran Liou04:10:04

Hi maintainers, I’m working on adding sitemap to cljdoc. What do y’all think should be in the sitemap? I’m think of adding all the /d/:group-id/:artifact-id routes. Do you think that’s sufficient? Frankly, I’ve never created a sitemap before so don’t hesitate to let me know if I misunderstood something. Thanks.


Hey 😊 I’m also not sure about this, some googling doesn’t give a clear answer either. Maybe let’s start with what you suggested and then we can see how it affects google indexing, adding more pages if it seems necessary.


Also maybe @U051KLSJF has more knowledge on this since he opened a few SEO related issue


Yeah I think just doing the current routes is enough


Ie the routes for the current pages


That would certainly be a good start, and get you a long way


The Googlebot can probably crawl from there


This presents a decent argument for having a /current


Because you get a limited crawl budget with Google, especially for a new site like cljdoc, it may take some time for pages to get re-indexed, so the search results for pages may be out of date


I noticed this today on a cljdoc search


@U7PQH43PS so what Daniel is saying is that instead of putting /d/group/artifact in the Sitemap we should put in /d/group/artifact/CURRENT so the indexing budget is spent on the most recent versions at first... Maybe we should make the link that gets users to the list of versions rel=nofollow so crawlers don’t get to those pages


Ah and @U051KLSJF, you mean /current as in a proper url segment not just for redirecting, right? :thinking_face:


Will explain more tomorrow

Daw-Ran Liou15:10:23

@U050TNB9F Sounds good to me but I don’t get the last part about rel=nofollow

Daw-Ran Liou15:10:00

I’d like to test my sql function in the repl. How do I create the db-spec for local testing?


(cljdoc.config/db (cljdoc.config/config))


It’s a bit long I usually just put it in a def

Daw-Ran Liou16:10:06

Thanks! Yeah that’s a good idea.

Daw-Ran Liou16:10:04

I created a new storage api all-distinct-docs to retrieve all the ingested documentations. However, I don’t quite understand how to use the API. When I called ( db-spec)

CompilerException java.lang.IllegalArgumentException: No implementation of method: :all-distinct-docs of protocol: #' found for class: clojure.lang.PersistentArrayMap, compiling:(/Users/dawran6/Developer/cljdoc/src/cljdoc/server/sitemap.clj:22:1)

Daw-Ran Liou17:10:55

It seems like I’m passing the wrong db-spec type into my API, right?

Daw-Ran Liou17:10:32

Oh I got this after typing out my question. I need to use ->SQLiteStorage to convert the type

Daw-Ran Liou18:10:15

I made a PR for adding the sitemap. Please feel free to give me some comments. Thanks!

Daw-Ran Liou18:10:08

Oops I broke something. CI build didn’t pass.


cljdoc is trying to analyze a file that isn't on the classpath. shouldn't it just ignore that?


this file is consumed by the project's test, but it's not meant for direct consumption


If it’s only relevant for tests maybe not including it in the jar would be an option as well?


good call 🙂