Fork me on GitHub
#cljdoc
<
2023-06-14
>
Joshua Smock12:06:39

Hi 👋 I’m trying to get the cljdoc-analysis build to pass for cljest, since at the moment it’s failing due to one of my functions erring during the analysis (https://app.circleci.com/pipelines/github/cljdoc/builder/42768/workflows/ad287d58-cfb7-4be9-9b80-56d649e72685/jobs/59143). I added ^:no-doc in an attempt to get it to work, but it didn’t seem to, and I couldn’t figure out otherwise how to make it skip attempting to run this function, and I also couldn’t figure out how to run cljdoc or the analyzer locally to play around myself. Could someone help me with this?

lread14:06:11

Ya sure @U02G8GJ0NMR, I can take a peek.

lread14:06:29

So, currently, cljdoc loads all of a library's namespaces to learn about its API. It will only skip directly loading namespaces (not vars) that have :no-doc metadata. In the case of cljest, during cljdoc API analysis, we see the exception thrown with msg:

Error: A cljest.edn should exist in the same directory as you started the cljest server. This file should be located next to your jest.config.js file (or equivalent).
If I search for what throws this, I find: https://github.com/pitch-io/cljest/blob/cafc7a3541ea61e36ff04351ef72d7c9d8a52fbb/cljest/src/cljest/compilation/config.clj#L87! It I look for what call this, I find: https://github.com/pitch-io/cljest/blob/cafc7a3541ea61e36ff04351ef72d7c9d8a52fbb/cljest/src/cljest/compilation/config.clj#L99! And I look for what calls this, I see: https://github.com/pitch-io/cljest/blob/cafc7a3541ea61e36ff04351ef72d7c9d8a52fbb/cljest/src/cljest/core.clj#L8-L10 which is a var (not a fn). So that's a problem. That var will always be loaded/evaluated when the cljest.core namespace is loaded, which will eventually call load-config! and trigger the exception. So I think cljest code is saying that it is not valid to load my namespaces without a cljest.edn but cljdoc is trying to do just that to learn about its API. Does that make sense?

Joshua Smock15:06:21

Ahhhh. Yeah, that makes perfect sense, and thanks for looking into it. I’m guessing there’s not a way to conditionally hide code from cljdoc-analyzer when it loads some namespace, right? Something like:

(when-not (System/getEnv "CLJDOC") ...)
Maybe this isn’t even the best approach regardless but just thinking if there’s a simple way to get this code to not be evaluated when loaded by the analyzer

Joshua Smock15:06:35

Another semi-related question too: I tried to get the Docker setup working locally but seemed to really struggle to, is there a simpler way that I could run the analyzer on my code w/o needing to publish a new version to Maven and then triggering a build?

lread15:06:57

Yeah the local preview story is pretty awkward/slow/complex at the moment. I do it for my projects, but not everybody has so much patience! If you just want check that API analysis will pass, without doing a full preview, you can try just running the https://github.com/cljdoc/cljdoc-analyzer on your jar. There is a https://github.com/cljdoc/cljdoc-check-action that does this too, but it is a tad out of date (I'm not the original author of that one, and people can get distracted and move on). I have it on my todo list to freshen it, but I have other distractions too.

👀 2
lread15:06:42

For your failure... static API analysis might help. That is something I am currently working on, but it is not available yet.

lread15:06:57

Without understanding cljest better, dunno. For general use, is it ok/desirable for cljest to throw when loading cljest.core if cljest.edn does not exist? Is that the intent?

Joshua Smock15:06:30

I’ll take a look at trying to run cljdoc-analyzer on a JAR and reach out if I have any issues 🙂 As far as if cljest.core should throw, yes and no. In theory no, it shouldn’t throw because the code should still be able to compile, but in practice since cljest.core and related nses is going to be compiled by cljest.compilation.server which won’t start w/o cljest.edn, it’s not really a concern. But I would like the API docs to compile because I spent time on nice docstrings, so I am keen to make it not throw 🙂

lread15:06:34

One thing folks sometimes do is defer evaluation until the var is used by a call from the user. Not sure if that makes sense for cljest. You'd get your failure on first usage rather than ns load.

Joshua Smock15:06:27

Yeah that’s what I was more or less thinking too, turning that variable into a function and then calling it when necessary, which sets a variable like *formatters-set* or something

Joshua Smock16:06:30

Okay so I think I got the analysis kind of working, but now I’m getting a different error:

Execution error (FileNotFoundException) at cljdoc-analyzer.runner/eval1689$loading (runner.clj:1).
Could not locate babashka/fs__init.class, babashka/fs.clj or babashka/fs.cljc on classpath.
Should I invoke the analysis differently than in the readme, e.g.
clojure -M -m cljdoc-analyzer.cljdoc-main \
  '{:project "com.pitch/cljest"
    :version "1.1.0"
    :jarpath "/path/to/cljest.jar"
    :pompath "/path/to/cljest-pom.xml"}'

lread17:06:57

Does cljest use babashka/fs? If you push your cljest changes, I can see if I can replicate.

lread17:06:05

Your syntax matches what cljdoc itself uses. For whatever reason, I usually install my jar to my local maven repo, then use https://github.com/cljdoc/cljdoc-analyzer#general-use.

Joshua Smock17:06:22

cljest doesn’t use babashka :thinking_face: I pushed up the commit that gets it to that point here https://github.com/pitch-io/cljest/tree/cljdoc-analysis And to generate the JAR and pom.xml file you can do this:

# Inside of `cljest/cljest` directory
clj -A:publish -M -m deps-library.release pom
clj -A:publish -M -m deps-library.release jar

lread17:06:23

rm -rf .cpcache?

lread17:06:59

Ok, will give it a shot from over here...

Joshua Smock17:06:08

Yeah, same thing when removing .cpcache 😕

lread17:06:28

Huh. I see it too. I'll poke around and figure out what is going on.

gratitude-thank-you 2
Joshua Smock18:06:43

Okay so I have some kind of an idea what could be going on. I was able to fix the babashka.fs issue by adding babashka/fs {:mvn/version "0.4.19"} to my deps, rebuilding the JAR and pom.xml. When I reran the analysis, I get a different error:

Execution error (FileNotFoundException) at cljdoc-analyzer.deps/eval2339$loading (deps.clj:1).
Could not locate clojure/tools/deps__init.class, clojure/tools/deps.clj or clojure/tools/deps.cljc on classpath.
This makes me think that somehow the deps from cljdoc-analyzer aren’t being pulled in when I execute clojure -M -m cljdoc-analyzer.cljdoc-main

lread18:06:32

Something is afoul with the cljest jar creation. Try an unzip -l target/project.jar. I'm seeing a bunch of the cljdoc sources in mine, are you?

Joshua Smock18:06:09

:thinking_face: :thinking_face:

lread18:06:00

Ah you have a ref to cljdoc-analyzer in your deps.edn default deps.

lread18:06:24

Sooo... it you turf that dep... clone cljdoc-analyzer and run from its root... that will get rid of that.

lread18:06:41

Or... if you want to use cljdoc-analyzer as a dep, put it under its own separate alias. But try the above first, it is typically how I run cljdoc-analyzer.

lread18:06:14

So after you try that ask me what else I'm wondering about.

Joshua Smock18:06:07

Okay so I tried putting it under an alias and then rebuilding the JAR and pom, and then re-ran the analysis and it got to the point of actually trying to analyze

Joshua Smock18:06:15

But it still fails with that original error :thinking_face:

Joshua Smock18:06:34

(after I set one of the test files to ^:no-doc ). Is that what you also see?

lread18:06:42

I didn't retry yet. So your jar contents look correct now?

Joshua Smock18:06:00

They look correct now

lread18:06:12

Coolio. So by original error you mean the exception with msg?

Error: A cljest.edn should exist in the same directory as you started the cljest server. This file should be located next to your jest.config.js file (or equivalent).

Joshua Smock18:06:09

Yeah, but now it makes sense why, I have some code in my tests that eventually does execute the same code path to load that function that eventually tries to load the config, so now I need to get the test namespace skipped from analysis, or somehow skip any files that end in a certain ns suffix

lread18:06:52

So cljdoc only analyzes what is in the jar. Typically test namespaces are not included in the jar.

Joshua Smock18:06:52

Or maybe only just analyze clj files, not cljs files

Joshua Smock18:06:05

Ah okay, let me see what’s going on there

Joshua Smock18:06:03

I guess it’s because I colocate my tests with my source files, so they’re just being picked up by default when I build the JAR. I’ll play around with getting them omitted and then run the analysis again, and if I still have some issues I can’t figure out I’ll reach out again

Joshua Smock18:06:38

So I think I’m good for now 🙂 But you did mention one other thing you were wondering about?

lread18:06:45

Yeah, should cljest be including shadow-cljs as a default dep? Isn't it a build tool?

lread18:06:12

Or maybe it makes sense for cljest, what do I know? simple_smile

Joshua Smock19:06:19

Ah 😄 So shadow is a build tool, but cljest kind of abstracts it, such that you don’t really need to know too much about shadow to use cljest (so you can call clj -X cljest.compilation/watch and it will work without any config file by default, for example). I did it both to make it a little nicer to work with as a user (Jest needs some specific build settings to make things like spying work and I don’t want to make users configure that), and the Jest test runner also takes care of automatically watching files, so I don’t want shadow to do the same or else weird stuff can happen

lread19:06:21

Ah ok, so all cool then. I don't know a ton about ClojureScript, and thought it looked a bit odd.

lread19:06:40

If you run into more mysteries, drop by with questions and we'll do our best to help you out. Even if you don't have anymore mysteries, and get cljest docs working on cljdoc, please let us know of your success!

Joshua Smock19:06:24

Will do 🙂 Stay tuned for some nice cljdoc docs for cljest!

Joshua Smock19:06:47

And thanks again for helping me debug this, I probably wouldn’t have figured this out myself and/or as quickly without it

lread19:06:35

Yeah, my pleasure @U02G8GJ0NMR.