Clojurians Log v2

update, I went back and set a v1.0.149 tag and now some stuff worked, but still not quite 100%: There was an error analysis-job-failed: 2025-09-13 02:24:50,237 INFO cljdoc-analyzer.runner - launching metagetta for: io.github.timothypratley/happyapi languages: :auto-detect 2025-09-13 02:24:52,712 INFO cljdoc-analyzer.runner - metagetta results: exit-code 1 stdout: Args: {:namespaces :all, :root-path "/tmp/cljdoc-io.github.timothypratley-happyapi-1.0.1514578196223357790175/src/main/clojure", :languages :auto-detect, :output-filename "/tmp/io.github.timothypratley-happyapi3156680318867407073.edn", :exclude-with [:no-doc :skip-wiki :mranderson/inlined]} Java version 24.0.2 Analyzing for clj Clojure version 1.11.3 stderr: {:clojure.main/message "Execution error (ClassNotFoundException) at http://java.net.URLClassLoader/findClass (URLClassLoader.java:349). and more stack trace than can fit here, i'll try to understand it more.

Hi @timothypratley ! I am done for the day, but will be happy to help tomorrow.

Thanks! I think I'll be able to figure it out from the stack from here 😄

👍

It's not urgent at all, I'm just updating what I have discovered: My optimism was hubris, and I'm not able to solve it. The analysis issue is while processing dep.clj: https://github.com/timothypratley/happyapi/blob/main/src/happyapi/deps.clj Which is a little bit strange because it is a dynamic dependency discovery namespace. Just to explain it; happyapi is just middleware that can work with several different implementations of json and http-servers. It intentionally leaves it up to the user which libraries they wish to rely on. So the issue (I think) is that for some reason the analysis is trying to find one of these libraries which is not a dependency. It occurs to me that there might be several strategies to work around this: 1. Maybe there is a way to nominate extra dependencies to use during analysis somehow. The user selectable dependencies are: clj-http/clj-http http-kit/http-kit cheshire/cheshire org.clojure/data.json metosin/jsonista (I don't know how to do this if it is possible, maybe adding devDependencies to pom.xml????) 2. Maybe there is a way to flag the happyapi.deps namespace to not generate documentation. That would be slightly sub-optimal as I think the docstrings are useful, but would get around the problem. I think I can try this by adding ^:no-doc metadata on the namespace or vars that are problematic.

I attempted (2) and now it fails in the same way on setup.clj: https://github.com/timothypratley/happyapi/blob/main/src/happyapi/setup.clj Possibly adding ^:no-doc there will solve it, but for now I think I'll hold off to avoid creating extra deployments and try to rethink a better approach.

I can reproduce the problem locally with clojure -X:codox

clojure -X:dev:codox works as expected because the optional dependencies are present. but I'm not sure that helps to know.

Unfortunatley ^:no-doc on a ns appears to not help when the ns is required by another ns, and this ns is transitively required in many places, so metadata annotation is a bit of a dead end for now.

That's right, if another ns includes a :no-doc ns it will be loaded during analysis.

I'll start at the top. Could not find revision v1.0.149 - let's take a look at the pom on clojars: https://repo.clojars.org/io/github/timothypratley/happyapi/1.0.151/happyapi-1.0.151.pom. Notice that the scm tag is incorrect and points to v1.0.149

<scm>
    <url></url>
    <connection>scm:git:</connection>
    <developerConnection>scm:git:</developerConnection>
    <tag>v1.0.149</tag>
  </scm>

Next up, the analysis failure. The current way that cljdoc handles optional dependencies is via provided deps. https://github.com/cljdoc/cljdoc/blob/master/doc/userguide/for-library-authors.adoc#getting-dependencies-right.

I think that covers your current stumbling blocks. Happy to chat more if you need a hand.

Thanks so much for the detailed help, I really appreciate it. Successfully built https://cljdoc.org/d/io.github.timothypratley/happyapi/1.0.157/ following your recommendations. As a record of what I changed: 1. I added a :provided alias to my deps.edn listing the optional dependencies as both :scope "provided" and :optional "true" 2. Creating the basis as (b/create-basis {:project "deps.edn" :aliases [:provided]}) 3. tools.build does not respect the :scope field, so did some hackery 🤮

(defn write-pom [x]
  (let [to-dep-orig @#'wp/to-dep]
    (with-redefs [wp/to-dep
                  (fn [[_lib {:keys [scope]} :as entry]]
                    (cond-> (to-dep-orig entry)
                            scope (conj [:xmlns.http%3A%2F%2Fmaven.apache.org%2FPOM%2F4.0.0/scope scope])))]
      (b/write-pom x)))) 

@daslu released 2-beta56, and everything looks great: https://cljdoc.org/d/org.scicloj/clay/2-beta56 🥳 Thanks Lee so much for guiding us to improve the release process

Heya @timothypratley, it is slow and honestly a bit painful, but if you ever get the itch to do a full local preview of your docs you can: https://github.com/cljdoc/cljdoc/blob/master/doc/running-cljdoc-locally-author.adoc If this interests you, lemme know, I've created bb task wrappers in various projects.

reading further I see the way to use curl to request a build:

curl -X POST -d project=org.scicloj/clay -d version=2-beta54 

exception-during-import

{:cause
   "Could not find revision v2-beta54 in repo "
 :data {:origin "" :rev "v2-beta54"}

Maybe :rev is meant to be a SHA rather than a tag or something along those lines or maybe something else 🤷

Maybe I was too impatient, I just refreshed and now there is a link to a build error:

WARNING: Restricted methods will be blocked in a future release unless native access is enabled

{:clojure.main/message
 "Execution error at libpython-clj2.python/initialize! (python.clj:129).\nFailed to find a valid python library!\n",
 ...
    "Could not generate Clojure documentation for scicloj.clay.v2.libpython-plotly",
    "Syntax error macroexpanding at (libpython_clj2/metadata.clj:17:1).",
    :message "Failed to find a valid python library!",
    :at
    [libpython_clj2.python$initialize_BANG_
     invokeStatic
     "python.clj"
     129]}],

Cljdoc should sort releases by Maven version numbers. I can double-check we are adhering to those rules.

Cljdoc analyzes articles separately from APIs. So articles can appear even when API analysis fails and vice versa.

So libpython requires python to be installed, eh? Yeah, that won't currently work for cljdoc API analysis.

I do still plan to add static API analysis to cljdoc (using clj-kondo). The pro of static analysis is that it doesn't dynamically load sources to discover the API and therefore does not care about any missing dependencies. Static analysis won't discover vars that are created at load time, but only a subset of libraries create their API vars at load time.

Yeah, I think we aren't strictly following maven version sorting rules. I'll raise an issue.

https://github.com/cljdoc/cljdoc/issues/1092

Thanks for pointing out that version sorting issue, fixed: https://github.com/cljdoc/cljdoc/pull/1093

Amazing! Thank you so much for all the detailed guidance, and adjusting the versioning.

My pleasure, happy to answer more questions when you have them.

@daslu https://github.com/scicloj/clay/pull/287 should help: I've replaced the libpythonclj interop with calling out to the shell, and tested locally this works to produce the PNG file (assuming you have python and libs available), and also does not cause codox any issues (because it can load the namespace without any python stuff). Also a tag step is added to the deploy step of build. It should be safe to merge whenever you feel comfortable, or let me know if you have any concerns. It's not urgent to make a new release, just updating you on the status so hopefully next release can be cljdocs compatible. One small remaining mystery, now when I run clojure -X:codox it never terminates (which is better than before when it would throw an exception), however we might see cljdocs also never complete. I'm not entirely sure the best way to figure out if this presents a problem, I guess we will find out.

(good idea to use the shell). great, thanks, i'll merge

thanks. Just a small update regarding running codox locally: 1. Using the latest codox 0.10.8, clay fails due to a clojurescript error 2. working around that, instrumenting codox to report it's progress, it completely finishes the task (`generate-docs`) but still hangs when run from clojure -X:codox but it's not clear if that's a real issue or just with the way I run it. Possibly we should exclude ClojureScript as a transitive dependency (I'll think more about this and submit a PR about it if needed).

Correction: Clay does not have a transitive dependency on ClojureScript, it's codox that is trying to pull in an old ClojureScript so I don't think that behavior is a Clay issue.

From reading more I'm convinced my approach of running codox locally is largely irrelevant. The prescribed way is to use Docker which makes sense. I don't have Docker installed and don't plan to run this test locally. TLDR I'm hoping that the libpython-clj removal will be enough and success is immenent.

@timothypratley, I was curious, so I tried cljdoc-analyzer (originally based on codox, but different now, and what cljdoc uses) on your clay branch. Like your experience on codox, I found API analysis did not exit. I've found that this is usually because a namespace starts some server or thread at namespace load time, which typically is not a great practice. I dumped threads and do see what looks like httpkit running? Does that make sense?

Ah, that's a great clue! Thank you 😄 Part of the purpose of Clay is to indeed launch a webserver. https://github.com/scicloj/clay/blob/main/src/scicloj/clay/v2/server.clj is the namespace where a httpkit server is started. Just looking at the code, it looks like it correctly does nothing unless the function core-http-server is called (which I don't think analyzer would trigger)... I think by instrumenting the Clay code in server I can learn more. Thanks for the extra motivation and valuable clue 👍

Found it! 🎉 https://github.com/scicloj/clay/blob/main/src/scicloj/clay/v2/util/portal.clj Clay does a little trick here which has the side-effect of running a server is not closed. I'll make a PR soon.

Great idea to dump the threads 💡

You've inspired me to create 2 new issues for cljdoc-analyzer. One is to exit explicitly when analysis is complete. If that works, it shouldn't matter to cljdoc-analyzer if clay starts a web server on ns load.

https://github.com/scicloj/clay/pull/289 removes the side-effect portal server

Not that it should matter to Clay anymore, but I've released my fix https://github.com/cljdoc/cljdoc-analyzer/issues/118

Awesome, thank you 😄 And thank you for pointing out that https://github.com/cljdoc/cljdoc-analyzer superseeds codox. I'm able to run it locally as deps.edn alias like so:

;; for testing cljdocs process `clojure -M:cljdocs`
           :cljdocs {:extra-deps {cljdoc/cljdoc-analyzer {:git/url ""
                                                          :git/sha "f83f0f6ff53c52b96a625790abbee3a69ccdfdad"}}
                     :main-opts ["-m" "cljdoc-analyzer.cljdoc-main"
                                 {:project "org.scicloj/clay"
                                  :version "2-beta54"
                                  :jarpath "target/org.scicloj/clay-2-beta54.jar"
                                  :pompath "target/classes/META-INF/maven/org.scicloj/clay/pom.xml"}]}