Interesting, is your project huge? Could you try: .lsp/config.edn

{:java nil} 

but 10GB is a lot.. it's weird

Yes project is a big...

coud you show the ls -lh .lsp/.cache/db.transit.json

-rw-r--r-- 1 audrius staff 1.0G Oct 27 17:03 .lsp/.cache/db.transit.json

uow, 1GB is a lot of analysis...

is that a mono-repo or something?

it is not that big 😄 and lsp cache takes more memory than everything else 🙃

I just removed the cache. restarted everything and it is all better now - only takes 35 MB yet. Such relieving 😊

ah yeah... I was to recommend that, you are not the first one which the cache for some crasy reason caches a huge old analysis

it is now 10GB again 😞

hum, I have a guess it's the post startup analysis which are mostly java analysis, did you try to disable java?

well, I added this {:java nil}

hum, it's enough

is your project public by any chance?

no 😞 I just started to work on it and this.

That gave me the idea that we should have a command to tell how the project in terms of analysis behave, like how many namespaces and how many keywords, that may help debug these kind of issues and it's not hard to implement

If it is a #C013B7MQHJQ project, the poly tool can provide quite a lot of that, I think.

If the 10GB is too much for your machine, @U7S4LP2TD, you can tell Calva to not autostart clojure-lsp. You’ll not have all the functionality, but it is a competent Clojure environment still. Here’s hoping we will figure out how to calm down the memory consumption!

10gb for clojure-lsp is too much, I never saw a project use that even https://github.com/metabase/metabase which is the biggest clojure codebase I know, there is something weird here

I’m often running with 15+ projects open at the same time. Some pretty big. I have not seen clojure-lsp being the major memory hog on my machine. I don’t think it totals to 10GB even.

yeah, the graal image helps a lot with that indeed

fwiw I work on metabase (200,000 LOC) with calva on an M1 Mac too. clojure-lsp generally uses about 8GB-10GB of my memory.

I'll work on these analysis count be available on serverInfo command so we can understand better on projects what takes lot of mem, my biggest guess are keywords and/or java analysis

ls -lh .lsp/.cache/db.transit.json gives me 350MB

yeah, for metabase that makes sense, which is quite different from @U7S4LP2TD’s cache

I disabled in in the config with '{:java nil :clojure-lsp {:enabled false}}'

it still starts but takes only 35 MB and doesn't create cache file.

there is no :clojure-lsp {:enabled false} config

To stop Calva from starting the server, set Calva: Enable Clojure Lsp On Start to never.

@U7S4LP2TD just did a push to master adding analysis-summary field, that may contain the count of some analysis which may help understand how big a project is and what is taking that much mem, it should be available in some mins in #C032YH7P0R2, would be great if you could test that

ok I can test 🙂 How to pull your changes?

Download from that channel #C032YH7P0R2 the latest one for your pc, extract the zip, change calva to point to that path, restart vscode, and then call the comand Clojure LSP server info

Clojure LSP server info command worked with the old version, but after the upgrade IDK where the output goes, even when I switched to the old version. 😞 Where may the out go?

I think calva prints it right @U0ETXRFEW?

It’s printed in the Output panel Calva says (iirc)

it printed but with a huge delay, but. for the old version. now printing for the new version...

https://pastebin.com/f06FKHtJ takes 10 mins to output 😱

"analysis-summary": {
    "external": {
      "keyword-usages": 3340170,
      "java-class-definitions": 34412,
      "keyword-definitions": 207230,
      "symbols": 0,
      "java-member-definitions": 282904,
      "var-usages": 0,
      "namespace-definitions": 1582,
      "java-class-usages": 0,
      "var-definitions": 18033
    },
    "internal": {
      "keyword-usages": 343307,
      "java-class-definitions": 0,
      "keyword-definitions": 4870,
      "symbols": 904,
      "java-member-definitions": 0,
      "var-usages": 410351,
      "namespace-definitions": 1942,
      "java-class-usages": 3739,
      "var-definitions": 30509
    }
  },

not in your project, but in some dependency, clojure-lsp should have a way to disable keyword analysis and maybe we should discard keyword-usages from external deps :thinking_face: which would result in way less things to have in memory + cache. I'll work on that but should take some days

https://github.com/clojure-lsp/clojure-lsp/issues/1700

I'll try to include on next release (couple of days), but no guarantee

I have a shizload of extensions in VSCode. Do you think they could be responsible for all this? My suspects other JVM languages. Especially Groovy - it has some JVM tricks.

I don't think so

@U0123R34Q9Y would be great if you could get the server info as well for metabase

@U7S4LP2TD Good news: • I improved clojure-lsp external analysis to drop keyword-usages as we don't really use it, so your 3M keywords will be gone and use way less memory probably • I implemented ways to improve memory usage disabling specific features/analysis • there is a doc about troubleshooting memory usage, check https://clojure-lsp.io/troubleshooting/#high-memory-usagefor more details This is already available on #clojure-lsp-builds but will be available on next release (couple of days), I hope it helps!

this config.edn

{:analysis {:java {:class-definitions false :member-definitions false}}  :java nil}

"analysis-summary": {
    "internal": {
      "keyword-usages": 344105,
      "keyword-definitions": 4872,
      "instance-invocations": 1049,
      "protocol-impls": 990,
      "local-usages": 215625,
      "namespace-alias": 19801,
      "namespace-usages": 21144,
      "symbols": 907,
      "var-usages": 411432,
      "namespace-definitions": 1944,
      "java-class-usages": 3753,
      "locals": 134118,
      "var-definitions": 30603
    },
    "external": {
      "namespace-definitions": 1595,
      "namespace-usages": 3811,
      "namespace-alias": 2926,
      "var-definitions": 18178,
      "keyword-definitions": 207230,
      "protocol-impls": 3247
    }
  },

{:analysis {:keywords {:definitions false
                       :usages false}}}

besides that your proejct doesn't seem that big, I'd expect way less memory than 10GB after this 3M elements removal

"analysis-summary": {
    "internal": {
      "instance-invocations": 1049,
      "protocol-impls": 990,
      "local-usages": 215625,
      "namespace-alias": 19801,
      "namespace-usages": 21144,
      "symbols": 907,
      "var-usages": 411432,
      "namespace-definitions": 1944,
      "java-class-usages": 3753,
      "locals": 134118,
      "var-definitions": 30603
    },
    "external": {
      "java-class-definitions": 34443,
      "java-member-definitions": 283046,
      "namespace-definitions": 1582,
      "namespace-usages": 3799,
      "namespace-alias": 2919,
      "var-definitions": 18034,
      "protocol-impls": 3237
    }
  },

added that conf you offered. removed the cache

looks good, how about mem?

but memory usage keeps growing. 5 GB yet and countin. cache 150mb

that's really weird..

I think from now, we would need to do a mem profiling on the JVM binary

ok - how to do it? I remember using Visual VM. What do you use for it?

I used visual VM as well

you need the JVM version, not the native-image, there are 2 ways: • you can find it in #C032YH7P0R2 as well, but it's the clojure-lsp-standalone.jar, but you need to have a second executable file like clojure-lsp that will java -jar clojure-lsp-standalone.jar • you can clone clojure-lsp repo and run bb debug-cli which should generate a clojure-lsp JVM binary

after server starting just spawn visual vm or something so we can check mem

Will Calva not start another process of LSP this way?

no if you point the calva setting to that file

there is a calva setting called clojureLspPath or something

so it will run from JAR?

no, the clojure-lsp file you need to create that inside has a java -jar clojure-lsp-standalone.jar

ah yes 🙂

should this:

(defn ^:private server-info [{:keys [db*]}]
  (let [db @db*]
    {:project-root-uri (:project-root-uri db)
     :server-version (shared/clojure-lsp-version)
     :clj-kondo-version (lsp.kondo/clj-kondo-version)
     :log-path (:log-path db)
     :project-settings (:project-settings db)
     :classpath-settings (:classpath-settings db)
     :client-settings (:client-settings db)
     :final-settings (settings/all db)
     :classpath (:classpath db)
     :cljfmt-raw (binding [*print-meta* true]
                   (pr-str (f.format/resolve-user-cljfmt-config db)))
     :analysis-summary (q/analysis-summary db)
     :port (or (:port db)
               "NREPL only available on :debug profile (`bb debug-cli`)")}))

I never needed that for debugging, having >java11 is enough, but wouldn't harm have that as well when not a graal image

I will try to add that and test with JAR soon

I found out that developers on the same project also had the same problem some time ago. Downgrading to

~ clojure-lsp --version
clojure-lsp 2022.11.03-00.14.57
clj-kondo 2022.11.02

it's one year of features and bugfixes, we added keyword and java analysis since then, but with the new option to disable keyword and java analysis that should not be a problem, but it could be some other performance issue we didn't catch yet