Fork me on GitHub
#cljdoc
<
2019-05-24
>
borkdude09:05:02

nice work on the Docker image. I’ve been able to run it like this:

docker run --rm -p 8000:8000 cljdoc/cljdoc:0.0.1406-030763c
it’s probably still undocumented? it would be nice to have some pointers around: persisting state, importing local docs

borkdude09:05:39

the DB migration could probably happen at build time, so it would save time at startup

borkdude09:05:55

(nothing urgent, I was just playing with it)

martinklepsch09:05:12

@borkdude yeah, that could probably be documented better

martinklepsch10:05:15

I'm not entirely sure where it should go (maybe into running cljdoc locally) but if you put up the rough steps and instructions I'm happy to fill in any blanks that aren't clear

borkdude10:05:30

I think “running cljdoc locally” makes sense yeah. Docker would probably be my first goto instead of trying to build all these npm deps

borkdude10:05:59

I’ll first make an issue for this, so we can think about it a little bit

martinklepsch10:05:12

should be just npm i; npm package or something right?

borkdude10:05:57

yeah, but that requires npm, which not everyone uses (really, some people don’t do anything with front-end or JS). I think Docker is the path of least resistance without cluttering your machine with stuff

dominicm10:05:44

Of course, some people don't want their machine cluttered with docker

borkdude10:05:07

yeah, so more choices for everyone

martinklepsch10:05:53

We can also discuss the details in a PR, I think it would be fantastic to have a first cut that can be iterated upon 🙂

martinklepsch10:05:40

Also... two small "announcements": - cljdoc is now linked to from every Clojars artifact page: https://clojars.org/rum - If you want to support my work on cljdoc you can sponsor me on GitHub: https://github.com/users/martinklepsch/sponsorship) For the entire first year GitHub will match your sponsorship! ‼️ 🙂

🎉 8
borkdude10:05:16

(the branch allows edits from maintainers, so feel free to push to this PR)

borkdude15:05:36

I’ve got something going:

docker run --rm -p 8000:8000 -v $HOME/Dropbox/dev/clojure/clj-kondo:/clj-kondo --entrypoint "clojure" cljdoc/cljdoc:0.0.1407-32d00ce -A:cli ingest -p borkdude/clj-kondo -v 2019.05.20-alpha-SNAPSHOT -g /clj-kondo
but it complains about :cause Requested version cannot be found in configured repositories: [borkdude/clj-kondo 2019.05.20-alpha-SNAPSHOT] Not sure what this is.

borkdude15:05:56

it probably needs the jar to be in the .m2 directory? I’m not sure why since I’m ingesting a git repo?

borkdude15:05:59

this didn’t help:

docker run --rm -p 8000:8000 -it -v $HOME/Dropbox/dev/clojure/clj-kondo:/clj-kondo -v $HOME/.m2:/root/.m2 --entrypoint "clojure" cljdoc/cljdoc:0.0.1407-32d00ce -A:cli ingest -p borkdude/clj-kondo -v 2019.05.20-alpha-SNAPSHOT -g /clj-kondo

borkdude15:05:26

I’ll continue some other time

martinklepsch15:05:18

Jars are always used for analysis. Git is only used for articles. Sorry for being terse, afk

borkdude15:05:50

ah now I get it

martinklepsch15:05:52

Try mounting your m2 into docker as well

borkdude15:05:06

yes, that’s what I did, but I got my version wrong I think

borkdude15:05:41

cool, it worked

borkdude15:05:26

Cool, after ingesting and preserving the sqlite data outside the container, and then running cljdoc, it worked 🙂 https://www.dropbox.com/s/3u4dm1timpi84b3/Screenshot%202019-05-24%2017.43.53.png?dl=0

borkdude15:05:24

I’ll update the docs later, but it’s already described in the PR

borkdude16:05:51

am I right that cljdoc does not scan your local m2 directory for newer versions? if that were possible, then I could just ingest after I started the server, using the UI 🙂

martinklepsch16:05:32

You mean like actively scan in the background?

borkdude16:05:57

I mean scan when you visit e.g.:


Jacob O'Bryant16:05:01

I'm trying to follow the steps from https://cljdoc.org/d/cljdoc/cljdoc/0.0.1315-c9e9a7e/doc/running-cljdoc-locally. The muuntaja example works, but when I try to ingest my own lib, I get this during analysis:

Caused by: java.io.FileNotFoundException: Could not locate taoensso/encore__init.class or taoensso/encore.clj on classpath., compiling:(trident/util/core.cljc:1:1)
Earlier in the analysis output, I see:
Used dependencies for analysis:
org.clojure/clojure 1.9.0
  org.clojure/core.specs.alpha 0.1.24
  org.clojure/spec.alpha 0.1.143
trident/util /home/arch/.m2/repository/trident/util/0.1.1/util-0.1.1.jar
org.clojure/clojurescript 1.10.339
...
So my lib (`trident/util`) is included, but taoensso/encore (which my lib depends on) isn't. I'm assuming encore should be listed here, is that correct?

Jacob O'Bryant16:05:19

I would think the dependency on encore would've been picked up from the pom file, but apparently that's not happening. A difference I noticed between my lib and muuntaja is that the jar for my lib doesn't have anything under META_INF, so maybe cljdoc/codox depends on certain files being in there?

martinklepsch16:05:29

Dependencies are extracted from the Pom so if it’s in there it should be picked up

martinklepsch16:05:10

I’m away from the computer right now but happy to help you out later

Jacob O'Bryant16:05:48

k thanks. I'm putting together a minimal example now and I'll let you know if I make any progress

Jacob O'Bryant17:05:20

Got a minimum example up here: https://github.com/jacobobryant/super-duper-octo-barnacle That repo packages and installs the same way that I do for trident/util (i.e. with deps.edn, juxt/pack.alpha and deps-deploy), so I'm guessing it has something to do with that

dominicm18:05:06

Hello, will fix anything that needs fixing 👋

dominicm18:05:03

I guess we need to see the pom.xml for trident.util :)

dominicm18:05:39

If you run jar tf myjar.jar that output is useful

Jacob O'Bryant18:05:40

I actually just figured out the problem! if you include META-INF/pom.xml in the jar it works

Jacob O'Bryant18:05:01

not sure if this is expected behaviour or not (@U050TNB9F?)

dominicm18:05:07

That's exactly right!

dominicm18:05:20

You need the pom for cljdoc to read the dependencies

Jacob O'Bryant18:05:04

:thumbsup: Before I was assuming that the pom at (e.g.) ~/.m2/repository/io/dominic/super-duper-octo-barnacle/0.1.3/super-duper-octo-barnacle-0.1.3.pom would be used

martinklepsch18:05:36

thats interesting, i think there was a PR recently that changed this, focusing on the Pom in the jar and not so much on the other Pom

dominicm19:05:51

I see, interesting

dominicm19:05:42

I would have thought they were equally valid?

martinklepsch19:05:21

They are but tools deps supports resolving deps via a jar only too and so I think we switched to that. You can probably dig up the PR very easily

martinklepsch22:05:22

Please open an issue if the behavior discussed here should be considered a bug! :)

dominicm06:05:15

So the behaviour is this way because it allows building jars locally?

martinklepsch07:05:58

not sure I understand 😄

martinklepsch07:05:20

What we did before was just use :mvn/version and add that to the classpath. But when users passed a custom jar location (ie. something that would not be found when using :mvn/version) that caused problems. So now instead of depending on tools.deps to find the right jar we depend on the actual jar file the user provided: https://github.com/cljdoc/cljdoc/pull/276/files#diff-aee7953db795ae539bf8139b6e6b82a1

martinklepsch07:05:11

tools.deps supports resolving dependencies for jars like this (at least in the limited testing we did) and generally most libraries seem to be packaged in the way that is compatble with the way tools.deps handles this

martinklepsch07:05:43

Maybe it would make sense for tools.deps to surface an error or warning if it can't resolve transitive dependencies of a jar but not sure

dominicm11:05:48

Seems like there's multiple ways the pom can be found in different contexts

borkdude16:05:21

A general remark about the scripts in cljdoc: it would be a lot easier if you assume that people will call the scripts from the root dir I think? No need to worry about ../.. and doing things like cd $(git rev-parse --show-toplevel)

borkdude16:05:08

but maybe there are reasons that it doesn’t work like that

borkdude16:05:20

I updated the PR now, I think it’s pretty much ready 🙂

borkdude16:05:56

one assumption in the updated README is that there is a Docker image with a latest tag on Dockerhub. I also updated the config for that

lread16:05:29

cool to see docker option for testing locally! Is using docker locally a better option if I want to more closely match what happens in cljdoc production?

borkdude16:05:52

I think it’s pretty much the same thing?

martinklepsch16:05:16

Yeah, same thing

🤓 4
lread16:05:32

cool! thanks!