Fork me on GitHub
#cljdoc
<
2020-12-23
>
mccraigmccraig11:12:14

i'm seeing an analysis failure with no errors - https://cljdoc.org/builds/38385 - when i try running the analyzer locally

clojure -Sdeps '{:deps {cljdoc/cljdoc-analyzer {:git/url "" :sha "9ff24dd10772b5c95d33d1c93c866a9e38884c17"}}}' -m cljdoc-analyzer.main analyze \
  --project cc.qbits/alia-all --version "5.0.0-alpha2-SNAPSHOT" \
  --output-filename "cc-qbits-alia-all-5.0.0-alpha2-SNAPSHOT.edn" \
  --exclude-with :no-doc \
  --exclude-with :skip-wiki \
  --extra-repo "clojars " \
  --extra-repo "central "
i see a similar failure with nothing logged to STDOUT or STDERR anyone have any clues on how to debug this ?

lread14:12:23

I can take a peek @mccraigmccraig

lread14:12:12

And.. there might be something fishy with the modules too? If I try to run the analyzer on cc.qbits/alia-joda-time version 5.0.0-alpha1 it seems to be referencing version 5.0.0 of alia which is not up on clojars.

mccraigmccraig14:12:10

ah, yeah - there was an error in the deps ... it's fixed now, but clearly i need to do another release

mccraigmccraig14:12:52

ok, that's a couple of things to run with ... thanks for the pointers @lee

lread14:12:40

Also… don’t forget to make sure scm gets setup appropriately in your pom.xml so that cljdoc can fetch and display your docs.

lread14:12:49

After addressing those issues, and if you are still running into probs, come on back with more Qs!

lread14:12:35

Yeah, I recently reworked that doc, feedback is always appreciated!

mccraigmccraig16:12:17

should i be able to run an ingest locally on the multi-module project if all the deps are only installed in my local repo @lee? i've added :scm keys to all modules and a doc/cljdoc.edn and i'm getting a very similar failure locally to that i was seeing on cljdoc circleci : https://gist.githubusercontent.com/mccraigmccraig/642198b30a0c8a28e61dc35b1d1d66dd/raw/f526a995eba994154907c1251d7626332cbaea7c/failed%2520local%2520cljdoc%2520alia%2520ingest

lread16:12:29

I’m not sure I ever tried a module project locally. Lemme check…

mccraigmccraig16:12:26

i also tried it without the explicit jar/pom paths, and it retrieved them from the local repo ok... updated the gist to show that

lread16:12:00

I just tried

clojure -m cljdoc-analyzer.main analyze \
  --project metosin/reitit --version "0.5.10" \
  --output-filename "/tmp/erp.edn" \
  --exclude-with :no-doc \
  --exclude-with :skip-wiki 
And that worked fine… so that’s reassuring.

lread16:12:11

Lessee…

mccraigmccraig16:12:18

using ../cljdoc/script/cljdoc rather than docker

mccraigmccraig16:12:52

i'll try ingesting reitit too, as a test of my local setup

lread16:12:04

good idea

lread16:12:58

It’s kind of a horrible error message, that we deliver for this scenario! simple_smile Have you setup your doc/cljdoc.edn?

lread16:12:11

The :cljdoc/include-namespaces-from-dependencies would be important for alia

lread16:12:18

I’m comparing your cljdoc.edn to the one in reitit…

lread16:12:19

Reitit is listing what look like maven coords, and I think you might be listing namespaces?

mccraigmccraig16:12:00

:man-facepalming:

lread16:12:55

It’s not entirely clear from our docs what to list, we should certainly add a tip/note.

mccraigmccraig16:12:09

clearly i read that key :cljdoc/include-namespaces-from-dependencies and though "aha, this is a list of namespaces" , without looking carefully at what reitit had done

lread16:12:33

yeah, I can see why you (and others) might do that!

lread16:12:44

I don’t think there are a ton of projects on cljdoc that use modules, so while you might be going through a bit of pain here, we’ll end up at least improving our docs!

lread16:12:45

I sense extreme success for alia on cljdoc on the horizon. simple_smile

mccraigmccraig17:12:27

darn, nope. same error 😞

mccraigmccraig17:12:24

when i ran the analyzer directly on reitit, it is successful

mccraigmccraig17:12:04

but it still fails on alia, so it is some attribute of the alia project which is causing the issue

lread17:12:35

oh right… I tried the analyzer rather than ingest on reitit. Did you successfully ingest reitit?

mccraigmccraig17:12:07

i also tried ingest (on reitit-parent which is the multimodule project) - it seemed to succeed

lread17:12:27

So you’ve installed all your modules to your local maven repo?

lread17:12:41

Is everything currently pushed to alia? Maybe I can diagnose. I’ll need the command to install to local repo.

mccraigmccraig17:12:46

everything is currently pushed. to install, lein sub install ; lein install should do the trick

mccraigmccraig17:12:51

you could try lein test, but unless you have a cassandra instance on localhost:9042 that will fail... although it will fail after loading namespaces, so you should be able to see that all the submodules were installed correctly

lread17:12:35

cool, won’t run the tests, but will install and try to see what’s up

lread17:12:05

ok… good, I am seeing the same error as you.

lread17:12:19

still poking around… starting with one of your modules… which oddly does not seem to analyze either.

mccraigmccraig17:12:20

don't know if this is relevant - when i ran vanilla codox i had to do some :exclusions messing to get it to run - there were asm clashes between one of codox's deps and com.datastax.oss/java-driver-core-shaded

mccraigmccraig17:12:02

com.datastax.oss/java-driver-core-shaded is a dep of the "main" module cc.qbits/alia

mccraigmccraig17:12:08

once i excluded all the asm from com.datastax.oss/java-driver-core-shaded, and let codox have its own preferred version, it worked ok

lread17:12:05

that might be relevant, but first…

lread17:12:38

it seems that cljdoc does understand the use of dependencyManagement in a pom.xml. If I explicitly put versions in dependencies I can successfully locally ingest cc.qbits/alia-joda-time

mccraigmccraig17:12:11

doh! lol, i only put the dependencyManagement stuff in yday - it cleaned up all the project.cljs quite a lot

mccraigmccraig17:12:16

ok, so i'll have to get rid of the dependencyManagement for the moment then... should i file a ticket against https://github.com/cljdoc/cljdoc-analyzer ?

lread17:12:25

But… reitit uses :managed-dependencies in its project.clj

lread17:12:15

But always specifies a version on :dependencies ?

lread17:12:05

So maybe there is some partial advantage to use :managed-dependencies? Not a lein guy, dunno.

mccraigmccraig17:12:23

i'm looking in pom.xml for metosin/reitit and its not got any version against the <dependencies> , only against the <dependencyManagement>

lread17:12:31

Huh, right you are…

lread18:12:45

maybe only for his modules?

mccraigmccraig18:12:59

all the <dependencies> in the top level metosin/reitit-parent project do seem to have versions

mccraigmccraig18:12:57

ok, well i think i can add explicit versions for all the top-level deps easily enough

lread18:12:12

hmmm… I don’t think we’ve figure it out yet.

mccraigmccraig18:12:12

2020-12-23 18:05:58,931 INFO cljdoc-analyzer.runner - Analysis succeeded.

lread18:12:12

oh… if I remove the explicit dep on org.clojure/clojure in the pom, I have success for cc.qbits/alia-joda-time

lread18:12:15

So… I currently think I might have been wrong about dependencyManagement

mccraigmccraig18:12:47

i've added explicit versions to all the <dependencies> and now i'm passing analysis and ingest seems to be working

lread18:12:28

ok, I’m not sure of the root cause, but if you are happy, I am happy.

mccraigmccraig18:12:07

not 100% sure yet - how do i see the results on the local server ? when i search on localhost:8000 , it seems to hit the remote api ?

lread18:12:23

Did your docs import?

mccraigmccraig18:12:31

although i'm not seeing any namespaces 😞

mccraigmccraig18:12:43

just the README

lread18:12:53

goodness gracious

lread18:12:25

so the import of your modules did not work, I guess.

mccraigmccraig18:12:00

no, but it crashed less hard than previously

lread18:12:09

baby steps

mccraigmccraig18:12:19

i will now see what happens if i put explicit versions in all the module dependencies

lread18:12:46

from what I see on reitit, that should not be necessary… but…

mccraigmccraig18:12:53

it's worth a try, it's the only action which has shown any progress at all so far

lread18:12:51

Ok, FWIW, here’s an observation: If I add an explicit version only to org.clojure/clojure in your poms I am getting success.

lread18:12:02

At least on the two modules I tried it on.

mccraigmccraig18:12:54

analysis success, or you can see the API docs ?

lread18:12:10

ingest success

mccraigmccraig18:12:36

yeah, i'm getting ingest success, can't see any API docs though

lread18:12:36

still working on root cause of original failure… There’s something that does not like a versionless org.clojure/clojure dep.

lread18:12:11

ok, I’ll move on… have you pushed your changes?

lread18:12:35

Not to overwhelm you but you’ll probably want to be more specific with your scm . So that cljdoc can point back to the correct revision on github.

lread18:12:54

(Unrelated to your current issue.)

mccraigmccraig18:12:48

np, not overwhelmed yet 🙂

mccraigmccraig18:12:31

i just pushed changes to add explicit versions everywhere... i also added a scope to all the org.clojure/clojure deps

lread18:12:27

oh. I see. gotta ingest all modules when testing locally

mccraigmccraig18:12:49

oh... got to ingest them in dependency order ?

lread18:12:57

cljdoc is consulting its db for modules at render time.

lread18:12:28

order is unimportant. Modules that are not ingested won’t show up under namespaces.

lread18:12:35

In production this would just happen automatically after the first cljdoc load.

mccraigmccraig18:12:50

oh - w00t - i ingested a module, and re-ran the top-level, and now i see API docs 😃

mccraigmccraig18:12:38

and in all the modules too...

mccraigmccraig18:12:20

it was coming through to the pom.xml last i looked

lread18:12:48

hmm… maybe lein does something different for local install than for deploy?

lread18:12:25

If I look at a reitit pom I see

<scm>
    <url></url>
    <connection>scm:git:>
    <developerConnection>scm:git:>
    <tag>e27d5c02138fd129280bcd1d18c49ac2067f93fc</tag>
  </scm>

lread19:12:25

Oh no wait yours seems fine now, I must have been looking at an earlier version that only had url

mccraigmccraig19:12:04

cool - yeah, i added it after you first said way back when

lread19:12:23

ah! good stuff!

lread19:12:36

so, extreme success?

mccraigmccraig19:12:55

looking that way ... i'm just seeing how much duplication i can get rid of in the metadata, then i'll try a new release

mccraigmccraig19:12:13

thank you for all the help @lee - do you need an issue submitting on gh ?

mccraigmccraig19:12:25

ok, i note that metosin/reitit has no org.clojure/clojure deps at all in its modules - which also avoids the issue - so, i think i'll follow that pattern, rather than have a hard-coded version in each module

lread19:12:53

So… what did we uncover? 1. docs on :cljdoc/include-namespaces-from-dependencies could use more guidance 2. docs on running locally should mention that you’ll need to ingest all your modules 3. there is something up with having org.clojure/clojure under dependencyManagement and then versionless under dependencies Anything else?

lread19:12:55

I’ll hit 1 and 2 today. Not sure what to do about 3. Is it a Clojure thing? Or a cljdoc thing?

mccraigmccraig19:12:45

i thought versionless under dependencies is the point of dependencyManagement - so you can specify a single version once across multiple libs, and i presumed its a maven feature rather than anything specifically clojure

mccraigmccraig19:12:29

but that's just my impression, i haven't really looked at it beyond reading some docs which suggested it as a way of reducing duplication and inconsistency

lread19:12:18

Yeah, in maven/java world it works fine. Don’t know root cause. I might poke around a small bit.

mccraigmccraig19:12:06

it's interesting that the issue is only on the org.clojure/clojure dep - i've just removed the version from another dependency and it seems to analyse fine

lread19:12:00

weird huh?

mccraigmccraig19:12:31

the thing i think is a (maybe indirect) cljdoc issue is that there was no useful error message

lread19:12:59

oh right! ya, so when there were no resolvable deps cljdoc got very mysterious with us.

lread19:12:19

So if I am right about the indirect cause, an error message like “no dependencies found” might have been helpful.

lread19:12:10

I think it should be easy to setup a small reproducible project for that.

lread19:12:41

I can log that one.

mccraigmccraig20:12:40

yes, something which gives some basis for experimentation - if i had gotten any inkling that it was something to do with the org.clojure/clojure dep and version i would probably have tried removing it and figured out a path to success

mccraigmccraig20:12:02

yep, that covers it all i think @lee

mccraigmccraig20:12:24

thanks again for your help - i doubt i would have gotten a solution without it

lread20:12:28

My pleasure @mccraigmccraig, thanks for uncovering some interesting problems!

mccraigmccraig21:12:49

i think that covers it @lee

👍 3