memory usage

clojure-lsp --version
clojure-lsp 2022.01.03-19.46.10
clj-kondo 2021.12.20-SNAPSHOT

I’m running macOS Monterey 12.0.1 (21A559)

when I attempt to run lsp-doctor in Emacs in a state like this it hangs and I can only force quit it

this is the output of lsp-doctor after a Emacs restart

Checking for Native JSON support: OK
Check emacs supports `read-process-output-max': OK
Check `read-process-output-max' default has been changed from 4k: OK
Byte compiled against Native JSON (recompile lsp-mode if failing when Native JSON available): OK
`gc-cons-threshold' increased?: OK
Using `plist' for deserialized objects? (refer to ): OPTIONAL
Using emacs 28+ with native compilation?: OPTIONAL

That's odd, I know there was a memory leak on lsp-mode but it was fixed some months ago. Does restarting the server with lsp-workspace-restart help? If so, probably is some memory issue with closure-lsp otherwise lsp-mode

would that be per project?

does it restart when I restart Emacs?

this above is after I restarted emacs

and I have three processes running after killing the one with a ton of memory

guess I should check next time what files it has open using lsof -p

If you close emacs it should close running clojure-lsps

It seems like clojure-lsps process is freeze or something, by any chance does your project use Potemkin?

We had a kondo issue where there was a infinite loop on Potemkin macros lint

not directly but this could be in any dep I guess, correct? I'll check in a bit

I think so

yep, looks like I have a transitive dep on *pote*mkin/*pote*mkin 0.4.5

yeah, I'm releasing a new clojure-lsp version which contains the kondo fix

Do you use macos?

any reason why not installing it via brew?

there are plenty ways to install it as well: https://clojure-lsp.io/installation/

@U0DJ4T5U1 there is a bug on that script

I'll fix it

@U0DJ4T5U1 I fixed one of the bugs, but the reason to not finding the version is because I'm deploying a new version right now 😅

it should work in a few minutes/hours if CI helps 😅

thats awesome, ill try again later

fyi, via brew upgrade, @UKFSJSM38 - i guess you need a new version in brew too?

==> Upgrading 1 outdated package:
clojure-lsp/brew/clojure-lsp-native 2022.01.03-19.46.10 -> 2022.01.20-14.12.43
==> Downloading 
curl: (22) The requested URL returned error: 404

Error: clojure-lsp-native: Failed to download resource "clojure-lsp-native"
Download failed: 

just tried brew upgrade clojure-lsp-native and got the same 404 as well

yes! I'm releasing the fix that should fix brew as well, I let you know when it's done

thanks Eric!

yes, and possible

already faced that issue as well

makes sense, feel free to create a issue

np 🙂

Here okay?

I’m not able to reproduce on a very small project, FWIW

I'm completing a github issue, I'm sure others can chime in when I've posted it

Ditto. I’m using emacs. It finishes analyzing a small project, but stalls analyzing a large project (`clojure-lsp` itself actually)

Does anyone have a project that this happens?

I can't repro with the static binary on the clojure-lsp repo

I've posted an initial issue, others can add to it

Alright, managed to repro, it seems related with graalvm, it freezes when analyzing with clj-kondo :thinking_face:

sorry @UKFSJSM38 for not making a thread to begin with!

No problem!

I had this issue before, and I thought it was a corrupted jar in my m2, after removing it it solved my issue

but since more people and I managed to repro again, seems to be something else

I'll revert the release for now

@U04V15CAJ any chance this is related with the new clj-kondo version? It seems to be freezing on the call to clj-kondo, not easy to repro though

In an example project I have up on github , the issue occurs too (if I open up the main.clj)

So, if you're looking for another project on which this fails, that is another example.

appreciate the quick response

Will you be pushing out a new version? As a maintainer of the clojure-lsp version on arch, it's tricky to go back for a release...

(even if the new version is identical to the previously good version - it needs a higher version number)

I see, not easy to revert all changes though

I just removed the latest github entry so it's possible to get previous version from install script

I'll probably revert brew as well

ehm... can you reproduce with clj-kondo only?

I'll try it!

I can't repro with JVM clojure-lsp, so I'd need to test with clj-kondo native

I'm building a clojure-lsp static binary with previous clj-kondo version to make sure as well

For clojure-lsp project, it seems to stuck when analyzing /home/greg/.m2/repository/com/taoensso/timbre/5.1.2/timbre-5.1.2.jar , the same jar I faced the same issue before :thinking_face:

I can't repro with kondo native though

My startrek one doesn't use timbre

it seems to be a inner timbre deps

because it fails with taoensso/encore as well

you can't repro with kondo itself?

yes, I tried to repro with kondo static binary, but it linted correctly

quite weird

I ran clj-kondo v2022.01.15 on my startrek one, with no issues

can you repro with clojure-lsp JVM version?

are you using clj-kondo master in lsp? try with that one?

@U11EL3P9U I'building a new native clojure-lsp with more logging on what we are sending to kondo to lint, it should be avaialbe here soon to download https://github.com/clojure-lsp/clojure-lsp/actions/runs/1725530702

It doesn't happen with JVM clojure-lsp the issue

I'm using latest stable clj-kondo which doesn't include only latest clj-kondo commit, do you think that commit could be related with this issue?

no

I'm happy to hear you didn't include master since the latest commit contains a false positive with bb.edn ;)

timbre project does lint correctly to me with the native clojure-lsp

it's weird, seems to be related with that jar on my m2

that's why I thought it was a corrupted jar first time I saw that issue, let's see on what jar stuck for @U11EL3P9U’s project when the CI finish

removed the timbre jar and the issue keeps happening, so it's not a corrupted file or something like that

this native bin contains more logs regarding kondo file being analyzed: https://github.com/clojure-lsp/clojure-lsp/actions/runs/1725530702

@U04V15CAJ it seems that is freezing indeed on the kondo call, I know the best would be to repro with clj-kondo itself, I'll keep investigating

The issue doesn't happen to me with your https://github.com/dharrigan/startrek project though @U11EL3P9U, so it seems to not be consistent, harder to debug :(

I wonder if others have any leads too?

other projects to test against 🙂

yeah, that could help, could you try my zip above and check what file is stuck on the logs?

I can't unfortuantely, I need a linux native image

let me dm you

this one is a linux static one

okay, trying

okay, spinning it up

a moment

@U11EL3P9U project hangs on other jar in his machine, so it seems to have no pattern the issue

seems to be a hairy one 😔

I'll keep trying narrow down

I’m trying to prune out a private, single src namespace, single test namespace project that seems to exhibit the issue

(yes, it was hanging for me on kaocha library)

For me it freezes on jars: after I remove one of them from my m2 manually it stucks on the next

/home/greg/.m2/repository/com/taoensso/timbre/5.1.2/timbre-5.1.2.jar
/home/greg/.m2/repository/com/taoensso/encore/3.12.1/encore-3.12.1.jar
/home/greg/.m2/repository/com/taoensso/nippy/3.1.1/nippy-3.1.1.jar

I wonder what they have in common

tee hee, from your examples, I hope not all from taoensso 😄

haha yeah, but for you it's kaocha, so 🤷

Let me see what it is with my bigger project...

hanging with kaocha

any news? if there's anything I can do to help, I would

Thanks, I'm still debugging trying to compile native with previous commits to narrow down

for now I can't find any issues with kondo, but it's indeed freezing on the kondo call

and you can't repro with the JVM ?

exactly

https://github.com/dpassen/lsp-kondo-repro is a very silly example

which graalvm version are you using?

that exhibits the issue for me

(eglot and emacs)

I'll try your project @UPEKQ7589 @U04V15CAJ 21.3.0

thank eric

i hope it suffices

@UKFSJSM38 try 22.0.0 which came out yesterday? ;)

btw, also try linting with --parallel if you are reproducing with clj-kondo only

use the exact same arguments as lsp does

hum, good point, forgot about that parameter

@UPEKQ7589 your project indeed freeze with me, but linting the project source not the jar!

that's a perfect sample it seems, I'll take a closer look

great

kinda!

does it freeze for you in the project code source analysis as well or during classpath scan?

not sure eglot shows that though

[server-notification] Thu Jan 20 15:58:26 2022:
(:jsonrpc "2.0" :method "$/progress" :params
          (:token "clojure-lsp" :value
                  (:kind "report" :message "Analyzing project files" :percentage 95)))

@U04V15CAJ Using @UPEKQ7589’s project, I found it freezes clojure-lsp during the analysis of this simple code:

(ns lsp-kondo-repro.core-test
  (:require
   [com.gfredericks.test.chuck.clojure-test :refer [checking]]))

(checking "illegal charges" 100)

oh, if I start clojure-lsp with

;; (checking "illegal charges" 100)

nice simplification

only that code?

exactly 😱

and with clj-kondo itself?

Now I doubt all these projects are using test.chuck

no freezes with clj-kondo static binary 😔

I pushed up the simplification to the repo

have you tried with clojure-lsp from previous release?

perhaps you can git bisect

yes, clojure-lsp from previous release works perfectly

ok, then bisect

uh, I have a clue of what could be

the re-frame unusued-public-var code, that is used on the custom-lint-fn

that's why I can't repro with CLI clj-kondo, not sure there is a option to pass custom-lint-fn there

no, but you can on the JVM

will try to revert that code and compile a native one

on the JVM it works

oh, is not that, the bisect I tested and it worked already contains the re-frame changes, damm

take it easy, people can just use the previous release

and we’ve got a very small repro case now

I wish I could use a REPL in a graalvm native image 😂

I pushed up another narrowing commit. Dropped some dependencies

As I was afraid the issue started to happen after https://github.com/clojure-lsp/clojure-lsp/pull/696, which unfortunatelly was the refactor of splitting to submodules and had a lot of changes (just moved code)

the issue is that I already commited some other changes, so no easy to just revert that

Not sure this is helpful, but… I waited for a long time and it finally finished prepping clojure-lsp. Checking whether it stalls when I restart the lsp workspace and when I restart emacs…

Oh wait, I take that back. It’s still not working for clojure-lsp. But I had some ~/.emacs.d/workspace/.cache/rewrite_clj.* buffers open from using lsp-find-definition. And LSP is working in those buffers. Weird. Anyway, probably best to ignore what I said before.

Alright, issue fixed, I tried to explain the details on https://github.com/clojure-lsp/clojure-lsp/pull/710that fixed it. thank you very much everyone for the help especially @UPEKQ7589 and @U11EL3P9U for the project samples! I'm releasing the fix right now, it should be available in some minutes/hours

TBH I didn't understand the issue exactly, but I hope @U04V15CAJ will help understand this later:joy:

Thank you so much for the quick fix. Really appreciate all the hard work you do to keep this thing working

😄 Thank you @UEENNMX0T for the quick reply and the feedback!

Same Eric. Thank you. As someone mentioned earlier, LSP became part of my workflow so quickly

CI is having issues to build the native image for windows as always, will leave the release to tomorrow morning

@UKFSJSM38 Glad you solved it. Are you able to reproduce the issue on the JVM with your project that didn't explicitly include clj-kondo? I've seen a similar issue recently, it may be due to a different version of SCI being pulled in somewhere or so

so it would be good to look at the deps tree of the problematic project, I think you should be able to reproduce it on the JVM

Yeah, I couldn't repro with JVM, but what you say makes sense, clojure-lsp has a javac task that uses americano to compile the java sources to a classes folder that is included on the classpath, when compiling the uberjar, depstar put the compiled code into classes as well and I noticed that sci was not there when that deps was not on the cli submodule

perhaps move to tools.build?

depstar is deprecated

yeah, forgot about that, will do!

@U04V15CAJ you were right, after changing to tools.build I can repro the same issue with JVM, but I'll leave to debug tomorrow

First, it doesn’t work when you’re on whitespace or a comment, even though the tests prove that it can. That is, in the tests, if you start on a whitespace/comment node, you can successfully move up or down. But in the editor, if the cursor is on a whitespace/comment node, invoking move up / down doesn’t do anything. I have a guess of what’s happening, based on something I saw when writing tests. I think that the code path we’re using must translate a cursor position into a zloc by looking only at “real” nodes, i.e., not whitespace or comments. So, if you’re in a map and on whitespace or a comment, it chooses the map itself as the zloc. … Actually, yes, I can confirm this must be the case. If I put the map inside another sequence, and invoke up/down, it moves the whole map. It must be choosing the whole map as the zloc.

The second issue is related to the first. When moving up/down, the cursor follows the moved node. This is by design, so that you can repeatedly invoke the action to keep moving the node. However, sometimes the cursor ends up on a comment associated with the node, and then the above situation arises… and so movement stops

I’m going to dig around to figure out where the code chooses the zloc that it passes to the actions. But, I’m skeptical of changing that. I bet there’s other code that assumes that zloc will be a real (non whitespace/comment) node. But I’ll experiment anyway and report back what I find.

And a final thing, not so much a bug as a new feature… It’d be nice if it moved pairwise in case , cond and other similar statements.

> … Actually, yes, I can confirm this must be the case. If I put the map inside another sequence, and invoke up/down, it moves the whole map. It must be choosing the whole map as the zloc. Yes, that's the issue. I'm not sure but I think this behavior was related with completion issues, if you change that, take a look if could be any completion issues related to that (the tests should help though)

As usual, you have great foresight @UKFSJSM38. 🙂 I have a patch that makes this behavior change: the parser will move onto whitespace zlocs. As you predicted, I had to be careful to keep the completion tests passing. Once I had that figured out, there were surprisingly few breakages. So, I think my patch is worth discussing. There are a few different possible implementations, so I’ll put together a PR and we can go over the options there. The patch is built on top of this other https://github.com/clojure-lsp/clojure-lsp/pull/707 (which is just some test cleanup) so it’ll be easier to look at after that’s merged. Will you review it when you have time? (I know there’s a lot of other stuff going on, so no rush.)

Sure, I'm still fighting with windows CI to release a new clojure-lsp version with the fix from yesterday, but I'll take a look at your PR after that! Yeah, besides the completion issues I don't see why we can't follow this approach, looks a valid option indeed to me

@UKFSJSM38 Thanks for merging https://github.com/clojure-lsp/clojure-lsp/pull/716. I’ll follow up on your questions there. I also plan to merge or rebase https://github.com/clojure-lsp/clojure-lsp/pull/756, to resolve those conflicts. And I have a few other branches built on 716 that I’ll submit as PRs soon. One of them is the fix to stabilize the cursor positioning when moving collection entries. I’ll put the changelog entry in that one. 716 is a big change, so I’ll try to keep an eye on things over the next few days. Let me know if any surprises come up.

Yeah, 716 was quite big, maybe we should that kind of changes but I know it was necessary for that issue, thank you for all the help!