Fork me on GitHub
#cljdoc
<
2018-10-31
>
Daw-Ran Liou04:10:04

Hi maintainers, I’m working on adding sitemap to cljdoc. What do y’all think should be in the sitemap? I’m think of adding all the /d/:group-id/:artifact-id routes. Do you think that’s sufficient? Frankly, I’ve never created a sitemap before so don’t hesitate to let me know if I misunderstood something. Thanks.

martinklepsch08:10:52

Hey 😊 I’m also not sure about this, some googling doesn’t give a clear answer either. Maybe let’s start with what you suggested and then we can see how it affects google indexing, adding more pages if it seems necessary.

martinklepsch08:10:36

Also maybe @U051KLSJF has more knowledge on this since he opened a few SEO related issue

danielcompton09:10:18

Yeah I think just doing the current routes is enough

danielcompton09:10:35

Ie the routes for the current pages

danielcompton09:10:57

That would certainly be a good start, and get you a long way

danielcompton09:10:11

The Googlebot can probably crawl from there

danielcompton09:10:01

This presents a decent argument for having a /current

danielcompton09:10:29

Because you get a limited crawl budget with Google, especially for a new site like cljdoc, it may take some time for pages to get re-indexed, so the search results for pages may be out of date

danielcompton09:10:52

I noticed this today on a cljdoc search

martinklepsch09:10:15

@U7PQH43PS so what Daniel is saying is that instead of putting /d/group/artifact in the Sitemap we should put in /d/group/artifact/CURRENT so the indexing budget is spent on the most recent versions at first... Maybe we should make the link that gets users to the list of versions rel=nofollow so crawlers don’t get to those pages

martinklepsch09:10:39

Ah and @U051KLSJF, you mean /current as in a proper url segment not just for redirecting, right? :thinking_face:

danielcompton11:10:17

Will explain more tomorrow

Daw-Ran Liou15:10:23

@U050TNB9F Sounds good to me but I don’t get the last part about rel=nofollow

Daw-Ran Liou15:10:00

I’d like to test my sql function in the repl. How do I create the db-spec for local testing?

martinklepsch16:10:37

(cljdoc.config/db (cljdoc.config/config))

martinklepsch16:10:20

It’s a bit long I usually just put it in a def

Daw-Ran Liou16:10:06

Thanks! Yeah that’s a good idea.

Daw-Ran Liou16:10:04

I created a new storage api all-distinct-docs to retrieve all the ingested documentations. However, I don’t quite understand how to use the API. When I called (cljdoc.storage.api/all-distinct-docs db-spec)

CompilerException java.lang.IllegalArgumentException: No implementation of method: :all-distinct-docs of protocol: #'cljdoc.storage.api/IStorage found for class: clojure.lang.PersistentArrayMap, compiling:(/Users/dawran6/Developer/cljdoc/src/cljdoc/server/sitemap.clj:22:1)

Daw-Ran Liou17:10:55

It seems like I’m passing the wrong db-spec type into my API, right?

Daw-Ran Liou17:10:32

Oh I got this after typing out my question. I need to use ->SQLiteStorage to convert the type

Daw-Ran Liou18:10:15

I made a PR for adding the sitemap. Please feel free to give me some comments. Thanks! https://github.com/cljdoc/cljdoc/pull/191

Daw-Ran Liou18:10:08

Oops I broke something. CI build didn’t pass.

plexus22:10:33

cljdoc is trying to analyze a file that isn't on the classpath. shouldn't it just ignore that? https://circleci.com/gh/martinklepsch/cljdoc-builder/6450

plexus22:10:56

this file is consumed by the project's test, but it's not meant for direct consumption

martinklepsch00:11:02

If it’s only relevant for tests maybe not including it in the jar would be an option as well?

plexus14:11:23

good call 🙂