Next, I'll move all my .edn files away from my project directory to see if it has any effect. (will be AFK for few hours though)

I think clojure-lsp will initially allocate more memory when the cache is empty, and then cool it when that is done. Still 4.6 gig seems like a lot. I have 7 projects open and none of the clojure-lsp processes consume more than 500 meg.

OS is Linux camel 5.17.12-100.fc34.x86_64 #1 SMP PREEMPT Mon May 30 17:47:02 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

And 4.6 gig is not the largest memory footprint, at one time it consumed 14.2G...

@U0267SNCPEY Is it possible to make a repro repository so clojure-lsp maintainers can reproduce this locally?

We've seen this issue before but we haven't managed to reproduce this, which is necessary to be able to fix it

@U04V15CAJ will try when back at work 👍:skin-tone-2:

Hum, since latest release clojure-lsp now care about edn and quoted forms l analysis, maybe there is some corner cases we don't know, a repro would certainly help

Perhaps there is a loads of keyword + symbol analysis stored in that single transit file?

Probably, but we load it once per startup

but all of this analysis is in memory right

if you have a gazillion keywords/symbols in an EDN file, that's a huge bucket

Yes

I moved my project specific dev tmp file directory away and the problem disappears. I had several .edn-files in there totalling in 10 238 652 lines of data 🙈

Glad you fixed it, if the file was in the source-paths, clojure-lsp would consider it and analyze it indeed

The directory was in the project root, have to check my config because I don't remember adding it to the project path (polylith project). I did use the files from my comment code.

When you edit the files themselves, they're also analyzed right

Read in as edn yes

I mean, edit in your editor

and analyzed as in analyzed by clj-kondo

Then it wouldn't matter if they were moved away?

right

And reprocessed, since I do the calculation on data that I don't want to use internet to fetch evertyime. I usually do not open them in editor, since they are large

I mean that I dump data from a REST API locally for dev purposes

What you do with them in the REPL doesn't matter for clojure-lsp.

If they are not on the classpath and you're not opening them in your editor, I don't see how they would matter for clojure-lsp, unless clojure-lsp does too much

like the watcher functionality maybe? does that also watch files not on the classpath?

maybe, it depends on the editor

if there is a didChangeWatchedFiles for that URI, yes

Checked the classpath, my project root is not included nor the directory I used for storing the data dumps

Please, check <https://clojure-lsp.io/troubleshooting/#client-server-log%7Cclient&lt;-&gt;server> communication and look for mentions of that file

After few minutes, CPU usage dropped but the memory footprint remains

491112 jmo       20   0 8920504   8.3g  72000 S   2.6  26.7   8:21.93 clojure-lsp

So, I moved the problematic directory back to the project root, started Calva and I have the autostart configured for clojure-lsp so it immediately started to hog m emory

that is the server logs, we need the client<->server logs, the exact link I mentioned

ah, my bad!

It seems to browse through my temporary data dump files.

[Trace - 16:32:41] Received notification 'textDocument/publishDiagnostics'.
Params: {
    "uri": "file:///home/jmo/projects/carbonlink/invoice-51456979.edn",
    "diagnostics": []
}


[Trace - 16:32:41] Received notification 'textDocument/publishDiagnostics'.
Params: {
    "uri": "file:///home/jmo/projects/carbonlink/tmp-data/invoice-48713344.edn",
    "diagnostics": []

log has 14300 lines and it includes some of our code (dev-helpers, etc) is there a specific part that is of interest?

Hmm, maybe this is also relevant for the @U02CX2V8PJN issue

For me it only does this when I actually open a large file when it open in the main window in VSCode. I added the following setting to .clj-kondo/config.edn: :output {:exclude-files ["dev-resources/**" "../_docs-code/**" "..\\_docs-code\\**" "/resources/clients/**"]} This stops it from evaluating it before hand

@U0267SNCPEY textDocument/publishDiagnostics means server published those diagnostics after analyzing it, we need to know what is triggering the analysis: • if it's on classpath, it will be triggered on startup (probably this case, I see "/home/jmo/projects/carbonlink" in your server logs) • if the file is opened we lint and publish diagnostics • if the file is watched on client and client send to server via didChangeWatchedFiles

"workspaceFolders": [
        {
            "uri": "file:///home/jmo/projects/carbonlink",
            "name": "carbonlink"
        }
    ],

hum, I don't think that's the issue, clojure-lsp run a external command to retrieve the classpaths, for a polylith project should be something like: clj -A:dev:test -Spath , could you run that in your terminal and check if the project root is there?

Sure!

bases/api/test/resources
src
/home/jmo/projects/carbonlink/components/airlines/src
/home/jmo/projects/carbonlink/components/airlines/resources
/home/jmo/projects/carbonlink/components/airports/src
/home/jmo/projects/carbonlink/components/airports/resources
/home/jmo/projects/carbonlink/bases/api/src
/home/jmo/projects/carbonlink/bases/api/resources
/home/jmo/projects/carbonlink/components/categorization/src
/home/jmo/projects/carbonlink/components/categorization/resources
/home/jmo/projects/carbonlink/bases/cli/src
/home/jmo/projects/carbonlink/bases/cli/resources
/home/jmo/projects/carbonlink/components/co2e/src
/home/jmo/projects/carbonlink/components/co2e/resources
/home/jmo/projects/carbonlink/components/co2e-factor/src
/home/jmo/projects/carbonlink/components/co2e-factor/resources
/home/jmo/projects/carbonlink/components/config/src
/home/jmo/projects/carbonlink/components/config/resources
/home/jmo/projects/carbonlink/components/currency/src
/home/jmo/projects/carbonlink/components/currency/resources
/home/jmo/projects/carbonlink/components/db/src
/home/jmo/projects/carbonlink/components/db/resources
/home/jmo/projects/carbonlink/bases/dev-helper/src
/home/jmo/projects/carbonlink/bases/dev-helper/resources
/home/jmo/projects/carbonlink/components/yyy/src
/home/jmo/projects/carbonlink/components/yyy/resources
/home/jmo/projects/carbonlink/components/einvoice/src
/home/jmo/projects/carbonlink/components/einvoice/resources
/home/jmo/projects/carbonlink/components/emce/src
/home/jmo/projects/carbonlink/components/emce/resources
/home/jmo/projects/carbonlink/components/envimat/src
/home/jmo/projects/carbonlink/components/envimat/resources
/home/jmo/projects/carbonlink/components/espoo-opendata/src
/home/jmo/projects/carbonlink/components/espoo-opendata/resources
/home/jmo/projects/carbonlink/components/expense-csv/src
/home/jmo/projects/carbonlink/components/google-auth/src
/home/jmo/projects/carbonlink/components/google-auth/resources
/home/jmo/projects/carbonlink/components/helsinki-opendata/src
/home/jmo/projects/carbonlink/components/helsinki-opendata/resources
/home/jmo/projects/carbonlink/components/hus-opendata/src
/home/jmo/projects/carbonlink/components/hus-opendata/resources
/home/jmo/projects/carbonlink/components/microsoft/src
/home/jmo/projects/carbonlink/bases/mssql-digger/src
/home/jmo/projects/carbonlink/bases/mssql-digger/resources
/home/jmo/projects/carbonlink/components/zzz/src
/home/jmo/projects/carbonlink/components/zzz/resources
/home/jmo/projects/carbonlink/components/oidc/src
/home/jmo/projects/carbonlink/components/pathom/src
/home/jmo/projects/carbonlink/components/xxx/src
/home/jmo/projects/carbonlink/components/xxx/resources
/home/jmo/projects/carbonlink/components/railways/src
/home/jmo/projects/carbonlink/components/railways/resources
/home/jmo/projects/carbonlink/components/util/src
/home/jmo/projects/carbonlink/components/util/resources
/home/jmo/projects/carbonlink/bases/xtdb-inspector/src
/home/jmo/projects/carbonlink/bases/xtdb-inspector/resources
/home/jmo/.m2/repository/kaocha-noyoda/kaocha-noyoda/2019-06-03/kaocha-noyoda-2019-06-03.jar
/home/jmo/.gitlibs/libs/lambdaisland/kaocha/6bac06a47235930a84fc4167d475d2ed1f1ce7a2/src
/home/jmo/.gitlibs/libs/lambdaisland/kaocha/6bac06a47235930a84fc4167d475d2ed1f1ce7a2/resources
/home/jmo/.m2/repository/org/clojure/clojure/1.11.1/clojure-1.11.1.jar
...

Rest of the paths were pointing to .m2/, also redacted few customer specific names

so no project root right?

Not that I could find no

Did run (->> (java.lang.System/getProperty "java.class.path") (re-seq #"[^;:]+") (map println) dorun) in REPL too with the same result

could you: • rm -r .lsp/.cache • restart the server • paste server logs, especially the part that says the command it will run to retrieve the classpaths and the classpath found

INFO [clojure-lsp.classpath:103] - Finding classpath via `/usr/local/bin/clojure -A:test:dev -Spath

same result with that exact cmd

but did it mention the project root in the classpath as mentioned https://clojurians.slack.com/archives/CPABC1H61/p1675174738185739?thread_ts=1675155395.313099&amp;cid=CPABC1H61?

just a sec

Nope, now that it didn'n use the cached classpath the root dir is not there.

So, with a fresh start, .lsp/.cache removed it seems to have a sane classpath.

(gotta run now, will be back in few hours perhaps, tomorrow at latest)

yeah, weird how that folder was in the classpath, but that should be enough to not scan the file unnecessarily

There are plenty of instances where I've seen edn in the classpath, seems like if we started analyzing them we should have a config flag to not do so.

yeah, maybe, it should be rare to have huge edn files, but we should have a flag to not analyze indeed

@U0267SNCPEY feel free to create an issue about that

or maybe :exclude-files to never analyze those paths (with regex patterns, similar to in clj-kondo, matched via re-find

yes, looks more extensible

I just pushed a new setting :paths-ignore-regex doing that ☝️, so now it's possible to exclude from linting/analyzing specific paths by regex, available at #C032YH7P0R2

is :paths-ignore-regex a single regex or a vector of regex-patterns?

A vector, following source-paths-ignore-regex existing setting standard

ok

@UKFSJSM38 do you still need an issue or is everything ok for clojure-lsp development?

I will test this new setting aswel!

No issue needed anymore, would be nice if you could test it

Sure can!

:paths-ignore-regex <- this setting, do I put it in my workspace file, in a new config.edn in .lsp, in the config.edn in .kondo or somewhere else?

It's a clojure-lsp setting, so in .lsp/config.edn

Okay thanks

last nightly exe crashes for me, will try the jar file now.

[Error - 12:19:04] The Clojure Language Client server crashed 5 times in the last 3 minutes. The server will not be restarted. See the output for more information.
[Error - 12:19:04] Server initialization failed.
  Message: Pending response rejected since connection got disposed
  Code: -32097 

Tried with setting version to nightly without an lsp path aswell

.jar file starts without issues (using batch file to start it)

Running without batch file now, didnt get a connection to the process. Will report back how this works

Running Calva now with the nightly build (`.jar`), findings so far • "." is found on source-paths as well as on classpath - I have no idea why? • Without the new config option, the problem persists • With the new option set to {:paths-ignore-regex ["tmp.*"]} the startup takes some time as expected, but the symptoms are gone (my evil directory is tmp-data)

Good

I tried the following. Mind you, these are all folders containing either large files or files I don't want analyzed as they are not part of the project. Having lsp-version set to latest: .clj-kondo/config.edn: :output {:exclude-files ["dev-resources/**" "../_docs-code/**" "..\\_docs-code\\**" "/resources/clients/**"]} Memory usage on startup is 786MB~1GB and stays around that size. .clj-kondo/config.edn: :output {:exclude-files ["dev-resources/**" "../_docs-code/**" "..\\_docs-code\\**"]}If I remove one of the excluded folders from this list should increase the memory usage? But for some reason it doesn't and the startup memory is around 500MB~1GB but goes down to about 420MB after a minute or so. If I remove the output exclude-files line completely the startup memory is around 650MB~700MB and stays at that. Having lsp-path set to the latest nightly jar (exe file crashes): .clj-kondo/config.edn: :output {:exclude-files ["dev-resources/**" "../_docs-code/**" "..\\_docs-code\\**" "/resources/clients/**"]} .lsp/config.edn: {:paths-ignore-regex ["dev-resources/**" "../_docs-code/**" "..\\_docs-code\\**" "/resources/clients/**"]} This makes the lsp-server crash startup progress at different percentages, restarts, tries again and crashes again.

[Error - 13:05:02] Server initialization failed.
  Message: Internal error
  Code: -32603 
[object Object]
[Error - 13:05:02] Clojure Language Client client: couldn't create connection to server.
  Message: Internal error
  Code: -32603 
[object Object]

if you use the new clojure-lsp setting you don't need to change anything on kondo config as clojure-lsp won't even pass to kondo those paths.

I don't know why it's crashing for windows the nightly, needs more debugging and I don't have a windows machine nearby unfortunately

maybe there is something to improve on the new setting regex https://github.com/clojure-lsp/clojure-lsp/blob/master/lib/src/clojure_lsp/kondo.clj#L257? I can take a closer look later trying to repro with your paths that crash

There's probably something else wrong with my configuration, I'm not getting any warnings or errors in the PROBLEMS tab in VSCode, not even if I make intentional errors and reload vscode.

if those paths are ignored in the new setting, that's expected.

yes, but not even in project files

probably something with the regex

user=> (re-pattern ""..\\_docs-code\\**")
Syntax error reading source at (REPL:1:29).
Unsupported character: \\_docs-code
Syntax error reading source at (REPL:1:33).
Unsupported character: \\**

@UKFSJSM38 I think it might be nice if you use fs/unixify so you would only have to provide only one regex for all OSes

(this is a new function in babashka.fs)

I just noticed that too, this isn't logged anywhere tho. It only popped up for me when I (deliberately) activated the separate clj-kondo extension. I have now disabled that again and edited the regex and the linter is working again

@U02CX2V8PJN Try this:

user=> (re-pattern "..\\\\_docs-code\\\\.*")
#"..\\_docs-code\\.*"

I'll try to add log of what paths matched the regex when ignored

** isn't valid in a Java regex

yes thats my bad, I used .gitignore entries :face_palm:

Is there a setting that makes clojure-lsp not start automatic? I probably did something while testing and now I have to start it manually everytime

No

Apparently you can use this for literal strings:

"\\Qc:\\foo\\E"

But this doesn't work, never mind:

user=> (re-find (re-pattern (java.util.regex.Pattern/quote "c:\\foo")) "foo")
nil

oh of course:

user=> (re-find (re-pattern (java.util.regex.Pattern/quote "c:\\foo")) "c:\\foo\\bar")
"c:\\foo"

Have to click this and then click run otherwise it just wont start

oh a great, a path with both / and \ ?

well, calling fs/unixify on that should take care of it

Pushed a commit using fs/unixify, should be available soon on #C032YH7P0R2

I think you should use re-find here: re-matches

https://github.com/clojure-lsp/clojure-lsp/commit/de3c28b8ab0f38941179695f4466ad5720bd0953#diff-958d06f7efa75f91ba32de5cb3312161507f191c7394e31c77dfa7ac37896dccR231

and also I think you still need to match against the original path too in case Windows users write \\

only on Windows, which you can check with fs/windows?

@U04V15CAJ I did not add the leading / in the clojure-lsp prick a project thing. For some reason it's now seeing all those folders as project folders? Could this be the reason why clojure-lsp isnt automatically starting? It did before

I don't get it, this is set correctly and still it doesn't auto start.

that looks like the Calva regression introduced on recent release, check #CBE668G4R

Indeed

Going 1 version before the one released an hour ago makes it auto start again

I have a question about :paths-ignore-regex not ignoring the paths in the regex making lsp read and scan files it shouldn't causing this [Trace - 10:46:25] Received response 'initialize - (0)' in 601009ms. 10 minute startup and 10GB of memory usage :melting_face: I have this in my .lsp/config.edn : :paths-ignore-regex ["clients.*" "dev-resources.*" "docs-code.*"] but for some reason it still scans stuff in the folders within clients.* . I double checked this on http://regex101.com and with (re-find (re-pattern #"clients.*") "resources/clients/16/2023-02-15T13.00.16-update-sync-contextdata.clj") which is an actual path to one of those files and they check out. That file (and others) in the resources/clients/* folder and sub folder still get scanned. Am I doing something wrong @UKFSJSM38?

That setting doesn't exist, I think you meant source-paths-ignore-regex , even so, does those folder have clj or edn files you want to ignore? 600.000ms looks a lot indeed

https://clojure-lsp.io/settings/

ahh changes have been made, thanks for pointing that out!

Or do clojure-lsp monitor changes in .lsp/config.edn and update cp?

If the file sha changed, yes

Either way you can restart the lsp server with lsp-workspace-restart and then you will see the new classpath being used

Yeah, keep in mind that no changes are monitored if you change the deps file, you still need to restart aerver

got it thanks