Clojurians Log v2

Hiya @lasse.olavi.maatta it should be good now. Fixed docs & the bug I introduced. See https://github.com/cljdoc/cljdoc/pull/959#issuecomment-2564490431 for cmds I tested with. Note that the above server launch is changed to include m2 repo volume:

docker run --rm \                                                                                                                                                   
  --publish 8000:8000 \
  --volume "$HOME/.m2:/root/.m2" \
  --volume ./.cljdoc-preview:/app/data \
  --platform linux/amd64 \
  cljdoc/cljdoc

Oh and also: thanks for raising the problem!

As per the instructions, I got the local cljdoc website running with docker within WSL

This is what happens when I try to ingest my project in a separate terminal:

lassemaatta@DesktopPc:~/personal/murmeli$ docker run --rm --volume $(pwd):/repo-to-import --volume "$HOME/.m2:/root/.m2" --volume ./.cljdoc-preview:/app/data --platform linux/amd64 --entrypoint clojure cljdoc/cljdoc -Sforce -M:cli ingest --project murmeli/murmeli --version 0.1.0-SNAPSHOT --git /repo-to-import --rev $(git rev-parse HEAD)
INFO [2024-12-23 10:45:59,798] main - cljdoc.server.system Starting :cljdoc/build-tracker
INFO [2024-12-23 10:45:59,799] main - cljdoc.server.system Starting :cljdoc/storage
INFO [2024-12-23 10:45:59,800] main - cljdoc.server.system Starting :cljdoc/analysis-service
INFO [2024-12-23 10:45:59,922] clojure-agent-send-off-pool-0 - cljdoc.analysis.git Cloning Git repo from /repo-to-import
INFO [2024-12-23 10:46:06,210] clojure-agent-send-off-pool-0 - cljdoc.analysis.git Analyzing git repo at revision: 70ebcefb0fc5791bd0ce8ab91c8a9e54202ccd54
INFO [2024-12-23 10:46:06,226] clojure-agent-send-off-pool-0 - cljdoc.server.ingest Importing Articles from /repo-to-import
INFO [2024-12-23 10:46:06,252] clojure-agent-send-off-pool-0 - cljdoc.analysis.service Starting local analysis for murmeli/murmeli 0.1.0-SNAPSHOT /root/.m2/repository/murmeli/murmeli/0.1.0-SNAPSHOT/murmeli-0.1.0-SNAPSHOT.jar
INFO [2024-12-23 10:46:14,518] clojure-agent-send-off-pool-0 - cljdoc.analysis.service Got file from Local AnalysisService /tmp/cljdoc/analysis-out/cljdoc-analysis-edn/murmeli/murmeli/0.1.0-SNAPSHOT/cljdoc-analysis.edn
INFO [2024-12-23 10:46:14,534] clojure-agent-send-off-pool-0 - cljdoc.server.ingest Verifying cljdoc analysis edn contents against spec
INFO [2024-12-23 10:46:14,534] clojure-agent-send-off-pool-0 - cljdoc.server.ingest Importing API into database murmeli 0.1.0-SNAPSHOT
INFO [2024-12-23 10:46:14,538] clojure-agent-send-off-pool-0 - cljdoc.storage.sqlite-impl version-id 1

It seems to be doing something but no idea if the above output is what it should look like or not. In any case, the cljdoc server has no idea about my project

The corresponding https://github.com/lassemaatta/murmeli/blob/1d70aae659dbe5ce7cc15a1d0833e65bc21975aa/project.clj#L1

It looks like it's working fine! How did you diagnose "the cljdoc server has no idea about my project"

are you using the search?

try visiting /d/murmeli

yeah, the search returns all sorts of libraries but not that murmeli project. Also, just returns a "404 - Library not found" error page

the cljdoc-preview directory contains

lassemaatta@DesktopPc:~/personal/murmeli/.cljdoc-preview$ ll
total 164
drwxr-xr-x  3 root        root          4096 Dec 23 12:46 ./
drwxr-xr-x 11 lassemaatta lassemaatta   4096 Dec 23 12:45 ../
-rw-r--r--  1 root        root        151552 Dec 23 12:46 cljdoc.db.sqlite
drwxr-xr-x  2 root        root          4096 Dec 23 12:22 heapdumps/

I'm sure the problem is some silly typo somewhere, it took me like a dozen tries to get the docker commands working 🙂

Does /d/murmeli also yield nothing?

fyi: I start the server with

lassemaatta@DesktopPc:~/personal/murmeli$ docker run --rm --publish 8000:8000 --volume ./.cljdoc-preview:/app/data --platform linux/amd64 cljdoc/cljdoc

> Does /d/murmeli also yield nothing? yeah, I get the same error

Can you try inspecting the sqlite db if it has contents? (you can use /schema to understand the table structure, the versions table should have your project)

the sqlite looks ok, using emacs to browse it. I can see a single build for the project, the namespaces are listed fine and the versions table lists a single version

Not sure if this is relevant, but when I run the ingest tool, there's no logging happening in the cljdoc webserver output. Should there be something logged when a library is ingested?

Interesting. I stopped the webserver, removed the .cljdoc-preview directory and started the server again. The server claimed to run the ragtime migrations, but the .cljdoc-preview contained only that heapdumps directory. Once I ran the ingest tool it also first reported that it ran the migrations before analyzing the code. Afterwards, the .cljdoc-preview contained the sqlite file (which listed the migrations within). Is it possible, that the server is reading a different sqlite file and it's ignoring the one with only my project inside it?

> there's no logging happening in the cljdoc webserver output that is correct, I think the ingest tool runs directly against the SQLite db if I remember correctly

should the sqlite db contain only my project or all the projects known to cljdoc?

only yours when you run it locally

ok, so I suppose there's another db within the webapp and there's some merging involved when looking at local projects

there's only one DB but the search runs against a lucene index that is built from remote APIs (clojars+maven) to support searching for projects that have no builds yet

ok. I wonder what that loggin related to db migrations was when the server was starting 🤔

it definitely seems like the web app and the ingest tool run against different databases, could you share both commands you use for starting the web app and the ingest command?

one thing to try is also finding a project via search and building it in the web app, when everything is done right the project data should show up in the DB

starting the server (after recreating an empty .cljdoc-preview directory):

lassemaatta@DesktopPc:~/personal/murmeli$ docker run --rm --publish 8000:8000 --volume ./.cljdoc-preview:/app/data --platform linux/amd64 cljdoc/cljdoc
Preparing heap dump dir
- will save heap dump on out of memory error to: /app/data/heapdumps/heapdump.hprof
Launching cljdoc server
VM settings:
    Min. Heap Size: 1.00G
    Max. Heap Size: 1.00G
    Using VM: OpenJDK 64-Bit Server VM

INFO [2024-12-23 11:00:50,628] main - cljdoc.server.system Starting :cljdoc/analysis-service
INFO [2024-12-23 11:00:50,721] main - cljdoc.server.system Migrating :up 001-create-builds-table
INFO [2024-12-23 11:00:50,743] main - cljdoc.server.system Migrating :up 002-create-releases-table
INFO [2024-12-23 11:00:50,762] main - cljdoc.server.system Migrating :up 003-add-api-git-columns-to-builds
INFO [2024-12-23 11:00:50,799] main - cljdoc.server.system Migrating :up 004-add-grimoire-style-tables
INFO [2024-12-23 11:00:50,838] main - cljdoc.server.system Migrating :up 005-add-analyzer-version-column-to-builds
INFO [2024-12-23 11:00:50,858] main - cljdoc.server.system Migrating :up 006-add-namespaces-count-column-to-builds
INFO [2024-12-23 11:00:50,886] main - cljdoc.server.system Migrating :up 007-builds-index
INFO [2024-12-23 11:00:50,906] main - cljdoc.server.system Migrating :up 008-create-stats-table
INFO [2024-12-23 11:00:50,925] main - cljdoc.server.system Migrating :up 009-add-error-info-column-to-builds
INFO [2024-12-23 11:00:50,944] main - cljdoc.server.system Migrating :up 010_convert_builds_error_info_from_exception_to_data
INFO [2024-12-23 11:00:50,948] main - migrations.010-convert-builds-error-info-from-exception-to-data rows to udpate: 0
INFO [2024-12-23 11:00:50,968] main - cljdoc.server.system Migrating :up 011-adjust-clojars-stats
INFO [2024-12-23 11:00:50,996] main - cljdoc.server.system Migrating :up 012-add-retry-count-to-releases
INFO [2024-12-23 11:00:51,016] main - cljdoc.server.system Migrating :up 013-remove-duplicate-queued-releases
INFO [2024-12-23 11:00:51,027] main - cljdoc.server.system Starting :cljdoc/build-tracker
INFO [2024-12-23 11:00:51,040] main - cljdoc.server.system Starting :cljdoc/tea-time
INFO [2024-12-23 11:00:51,041] main - cljdoc.server.clojars-stats Starting :cljdoc/clojars-stats
INFO [2024-12-23 11:00:51,045] main - cljdoc.server.db-backup Database backup disable, skipping  :cljdoc/db-backup
INFO [2024-12-23 11:00:51,045] main - cljdoc.server.metrics-logger Starting :cljdoc/metrics-logger
INFO [2024-12-23 11:00:51,046] main - cljdoc.server.search.api Starting :cljdoc/searcher
INFO [2024-12-23 11:00:51,047] Tea-Time 2 - cljdoc.server.metrics-logger mem-stats {:heap-used 163548416, :heap-committed 1073741824, :heap-max 1073741824, :non-heap-used 208384160, :non-heap-committed 253034496}

running the ingest tool

lassemaatta@DesktopPc:~/personal/murmeli$ docker run --rm --volume $(pwd):/repo-to-import --volume "$HOME/.m2:/root/.m2" --volume ./.cljdoc-preview:/app/data --platform linux/amd64 --entrypoint clojure cljdoc/cljdoc -Sforce -M:cli ingest --project murmeli/murmeli --version 0.1.0-SNAPSHOT --git /repo-to-import --rev $(git rev-parse HEAD)
INFO [2024-12-23 11:01:57,363] main - cljdoc.server.system Migrating :up 001-create-builds-table
INFO [2024-12-23 11:01:57,385] main - cljdoc.server.system Migrating :up 002-create-releases-table
INFO [2024-12-23 11:01:57,405] main - cljdoc.server.system Migrating :up 003-add-api-git-columns-to-builds
INFO [2024-12-23 11:01:57,456] main - cljdoc.server.system Migrating :up 004-add-grimoire-style-tables
INFO [2024-12-23 11:01:57,494] main - cljdoc.server.system Migrating :up 005-add-analyzer-version-column-to-builds
INFO [2024-12-23 11:01:57,513] main - cljdoc.server.system Migrating :up 006-add-namespaces-count-column-to-builds
INFO [2024-12-23 11:01:57,532] main - cljdoc.server.system Migrating :up 007-builds-index
INFO [2024-12-23 11:01:57,551] main - cljdoc.server.system Migrating :up 008-create-stats-table
INFO [2024-12-23 11:01:57,570] main - cljdoc.server.system Migrating :up 009-add-error-info-column-to-builds
INFO [2024-12-23 11:01:57,589] main - cljdoc.server.system Migrating :up 010_convert_builds_error_info_from_exception_to_data
INFO [2024-12-23 11:01:57,593] main - migrations.010-convert-builds-error-info-from-exception-to-data rows to udpate: 0
INFO [2024-12-23 11:01:57,612] main - cljdoc.server.system Migrating :up 011-adjust-clojars-stats
INFO [2024-12-23 11:01:57,651] main - cljdoc.server.system Migrating :up 012-add-retry-count-to-releases
INFO [2024-12-23 11:01:57,670] main - cljdoc.server.system Migrating :up 013-remove-duplicate-queued-releases
INFO [2024-12-23 11:01:57,681] main - cljdoc.server.system Starting :cljdoc/build-tracker
INFO [2024-12-23 11:01:57,682] main - cljdoc.server.system Starting :cljdoc/storage
INFO [2024-12-23 11:01:57,682] main - cljdoc.server.system Starting :cljdoc/analysis-service
INFO [2024-12-23 11:01:57,827] clojure-agent-send-off-pool-0 - cljdoc.analysis.git Cloning Git repo from /repo-to-import
INFO [2024-12-23 11:02:04,150] clojure-agent-send-off-pool-0 - cljdoc.analysis.git Analyzing git repo at revision: 70ebcefb0fc5791bd0ce8ab91c8a9e54202ccd54
INFO [2024-12-23 11:02:04,166] clojure-agent-send-off-pool-0 - cljdoc.server.ingest Importing Articles from /repo-to-import
INFO [2024-12-23 11:02:04,191] clojure-agent-send-off-pool-0 - cljdoc.analysis.service Starting local analysis for murmeli/murmeli 0.1.0-SNAPSHOT /root/.m2/repository/murmeli/murmeli/0.1.0-SNAPSHOT/murmeli-0.1.0-SNAPSHOT.jar
INFO [2024-12-23 11:02:13,932] clojure-agent-send-off-pool-0 - cljdoc.analysis.service Got file from Local AnalysisService /tmp/cljdoc/analysis-out/cljdoc-analysis-edn/murmeli/murmeli/0.1.0-SNAPSHOT/cljdoc-analysis.edn
INFO [2024-12-23 11:02:13,951] clojure-agent-send-off-pool-0 - cljdoc.server.ingest Verifying cljdoc analysis edn contents against spec
INFO [2024-12-23 11:02:13,951] clojure-agent-send-off-pool-0 - cljdoc.server.ingest Importing API into database murmeli 0.1.0-SNAPSHOT
INFO [2024-12-23 11:02:13,955] clojure-agent-send-off-pool-0 - cljdoc.storage.sqlite-impl version-id 1

I manually built core.async, it succeeded but the local sqlite db still only contains data about my project

If you don't run the ingest, delete the .cljdoc-preview directory and then run the server the DB doesn't show up at all right?

I looked within the running container and I suppose /var/cljdoc/cljdoc.db.sqlite is the other database

yeah, correct

interesting, what happens if you use /var/cljdoc over /app/data ?

I'm not sure the /app/data thing is right, I can't find any references of it in the codebase but can find some for /var/cljdoc

https://github.com/cljdoc/cljdoc-exerciser/blob/master/script/cljdoc_preview.clj refers to it

Give this a try and see

docker run --rm --publish 8000:8000 --volume ./.cljdoc-preview:/var/cljdoc --platform linux/amd64 cljdoc/cljdoc

It doesn't seem to help. It injects my project to the sqlite db under /var/cljdoc. It seems to also contain a lot of stuff in the clojars_stats table. But the end result is that the server still does not find my project

I think I'll need to return later to debug this, thanks a lot for the help 🙂

Sounds good, we'll get back to this some time :) To summarize: You ran the server with

docker run --rm 
           --publish 8000:8000 
           --volume ./.cljdoc-preview:/app/data 
           --platform linux/amd64 
           cljdoc/cljdoc

docker run --rm 
           --volume $(pwd):/repo-to-import 
           --volume "$HOME/.m2:/root/.m2" 
           --volume ./.cljdoc-preview:/app/data 
           --platform linux/amd64 
           --entrypoint clojure cljdoc/cljdoc -Sforce -M:cli ingest --project murmeli/murmeli --version 0.1.0-SNAPSHOT --git /repo-to-import --rev $(git rev-parse HEAD)

I'll read up on the above sometime soon and see what I can see. I've never tested this on WSL... only macOS and Linux. I can try to preview on Linux and see what happens.

No worries, I’ll happily volunteer as a test subject :)

I think I’ll test this tomorrow on my mac, just to see if I get different results

I have a bit of time now. I'll give it a go on Linux and report back.

Yeah... I think local preview is currently broken. I can take a look and fix sometime soon.

https://github.com/cljdoc/cljdoc/issues/957

Don't sweat it, enjoy the holidays 🙂

Happy holidays to both of you 🫶

Great, thanks for fixing it 👍