If nothing immediately jumps to mind I am of course happy to do a minimal repro, file an issue and so on, but I don't really have time until the weekend.

good gif, but I never saw this behavior

a repro where I could try would help a lot

Oh, could you try with 1 or 2 previous versions? We changed something related with zipper parser of comments c/c @U07M2C8TT

Yeah, weird. I’ll research a bit

@UKFSJSM38, 2022.02.01-20.02.32 doesn’t actually have the change to comment parsing, right?

hum, maybe related with the z/next* change we did recently?

Yeah, perhaps. Let me build that release and see whether I can reproduce

Err… actually, no. z/next* was part of the comment parsing

My changes as of that release are just the addition of move-coll-entry and the removal of resolve-code-action

Hum, :thinking_face: @U07M2C8TT can you repro the issue while writing comments for you?

Not really. Although @U6T7M9DBR’s gif shows the completion code offering MANY more suggestions than I get. For example the letter w has at least 20 completions whereas for me, on 2022.02.01-20.02.32 , I only get one

The gif shows the completion code offering functions and other completion types, whereas I only get a “word” type. That is… the editor has seen the word “with” before in the code, so it’s offering that

good catch, @U6T7M9DBR could you try disabling vim's completion to confirm that?

But no functions like “with-open” or anything like that

Maybe I have a different completion setting?

maybe, I know emacs lsp-mode already tweaked a lot the completion performance as well

mmm… right, I am using emacs

Watching the gif closely, it’s offering completions for single characters. For example c completes to cond and many others. If it’s searching for every symbol that begins with c, and serializing all that data, I can imagine it’d be slow. And especially so when writing comments, where you reach the beginning of a word more often. Emacs doesn’t even seem to try to request completions for single letters

actually, company has a setting for that, I think its default is 2 or 3 chars

Actually, there’s a chance I’m not using clojure-lsp’s completion at all. My editor might be configured to use CIDER’s completion. I’m gonna try to figure that out before I offer any more thoughts

company-minimum-prefix-length

Yeah, company-minimum-prefix-length is 2 for me, which is the length at which I start getting completions

I have so many questions…

1. There are two clojure-lsp processes running. Why is that?

maybe 2 opened projects

but both are using a ton of cpu at the same time

2. The gif doesn’t show log lines about completion

3. It does show log lines about codeLens, which report a decent number of milliseconds. I don’t get those in my editor. Maybe I’ll make my editor use codeLens more aggressively

CodeLens are enabled by default on lsp-mode

4. rewrite-clj parses an entire comment, not a single word within a comment. How is it even getting single words?

I get more CPU usage if I turn on codeLens, but not as much as the gifs

5. (related to 3) the gif about comments reports many milliseconds running codeLens and didChange. But the gif about regular code show all the same stuff, plus completion, resolveCodeLens, and resolveCompletion. And yet, the regular code uses less CPU. I’m really confused

one thing that is tricky, is that didChange most of the time report 0ms because it runs async, but usually takes way more time than other features :(, we should metrify didChange better

Is it up to the client to decide how often to call didChange?

yes, we already do a debounce on server side, client can and should call didChange as many times it wants

OK... I don’t know how to debug further. I’ve turned on codeLens and completion. And I’ve set company mode to attempt completion after 1 character. Together, those increase CPU usage, but by similar amounts in comments and in code. If anything, it uses less CPU in comments because completion isn’t triggered.

The obvious difference between my setup and the gifs is that I’m running Emacs. So, I wouldn’t be surprised if the problem is somewhere within vim. I’m not familiar with that infrastructure, so can’t really help debug there.

I initially thought completion was the problem, but at least according to the log in the gifs, vim isn’t asking clojure-lsp to run completion in comments. Vim is, however, offering completions while in comments. (This happens in Emacs too. I think company-mode, the thing that manages completions uses several different completion backends simultaneously. So while in a comment it’s getting completions from somewhere besides clojure-lsp. I suspect vim is doing something similar.). Anyway, perhaps completion isn’t where we should focus.

The log isn’t definitive about what’s using so much CPU. That said, top shows two clojure-lsp processes, but the gif has only one log. It would be helpful @U6T7M9DBR to tail the other log. It may be tricky to figure out which file it is, but usually you can look in the log directory for the most recently changed files. Or you could try adding :log-path “/tmp/clojure-lsp.out” to .lsp/config.edn, so that both processes log to the same path. Edit: log path may be configured differently with coc.nvim

One thing we forgot to ask @U6T7M9DBR... are you building clojure-lsp locally, or using the official release?

Thanks for the help @U07M2C8TT, let's see what @U6T7M9DBR thinks about those questions

Yeah, sounds good. I’ve done a bit more research but don’t have much to report. I got nvim running with coc. I haven’t figured out how to configure it to show the code lenses. Any pointers there would be helpful. It does request completion from within comments. And clojure-lsp returns valid completions. It takes over a second, probably because it invokes completion for individual characters, which means a lot of data is returned. @UKFSJSM38 do you know how that’s even possible? AFAIK, clj-rewrite parses entire comment blocks, not individual words within comments. Is there something special about the completion code that lets it handle words within comments? Anyway, it uses a lot of CPU just from the completion. And I suspect if I could turn on code lenses, it would use more, since that’s invoked a lot in the gifs. But I still have no idea why it would use more CPU in comments for @U6T7M9DBR. That doesn’t really happen for me. All the completion stuff as well as did-change happens just as often in real code as in comments. Maybe it’s all from codelens, although I don’t know why that’d be different while in comments. Anyway, I’m still stumped there. @U6T7M9DBR, I don’t know how, but perhaps you can configure coc and/or nvim to invoke completion only after 2 characters are available? Or maybe increase the delay before completion is requested?

I know that a fix in clojure-lsp would be preferable, but that would at least let us confirm what’s taking so much CPU

@U07M2C8TT the completion is kind of special, the client send the literal text that is being completed, so it "skips" the zipper on clojure-lsp side (but we also check the zipper to know what to complete and etc). So it's sending the comment chars indeed, maybe automatically completing a word of 1 length is a little too much? I'd love to hear options to improve performance if you think of anything, I already spent a lot of time improving that code, but maybe I'm missing something

> the client send the literal text I believe you, but I can’t see in https://github.com/clojure-lsp/clojure-lsp/blob/0efef5c1ea266ac4af0c44f4c802185191a81580/lib/src/clojure_lsp/feature/completion.clj#L352-L430 where it uses the literal text, which makes me think I’m looking at the wrong thing. I see where resolveCompletionItem , uses the literal text, but not completion > options to improve performance if you think of anything, I already spent a lot of time improving that code I’m not surprised that’s been heavily optimized already! It’s just a hard problem… If clients can ask for “anything in this project or its libraries or clojure.core that begins with the letter C”, that’s going to be a lot of data. Perhaps there are faster algorithms for finding those results, but just serializing them could be slow (I’ve seen that be the case when there’s a lot of data, especially for JSON which is pretty verbose). I suppose there could be some server-side restrictions, like the input text has to be 2 characters or more. Or if serialization is the problem perhaps if the result set is too large the server could truncate it or return nothing. Otherwise, I’d have to look at it a lot more closely to come up with any suggestions

Here’s a thought… Perhaps in comments clojure-lsp is returning ALL the completions, as https://github.com/clojure-lsp/clojure-lsp/blob/master/lib/test/clojure_lsp/features/completion_test.clj#L97-L98 suggests it does. And then the editors re-filter that list to only things that match the literal text. So that giant list (almost 1000 items in a nearly empty project) gets calculated and serialized every time completion is invoked from within a comment. And in regular code, because a zloc can be found, clojure-lsp finds and returns a much smaller result set, making it use less CPU. Does that sound possible @UKFSJSM38?

Yes, totally possible @U07M2C8TT, I always tried to avoid limiting the results or length as this is something clients should do probably as default (like don't request at all if length < 2), but maybe we should have a setting on completion regarding the minimum length or try to not return anything on comments

Yeah, inside comments, I see two options. The server could return nothing. Or it could use the word that the client provides (which I still don’t think clojure-lsp does) and filter on that. In any case, I don’t think clojure-lsp should ever return every symbol in clojure.core, java.lang and java.util, beginning with any character, which is what it seems to do now. I also think that whether in a comment or not, it’d be reasonable for the server to require a minimum request length. Providing completions for a single character provides too many options to be helpful, IMO.

Agree, I think we can probably implement all of those, first don't return anything on comments, I don't see why one would want completion on comments, second, don't compute core symbols if length is 1 and we can think if those would not be enough we can think about having a setting to not complete X length

https://github.com/clojure-lsp/clojure-lsp/issues/750

Feel free to complement that issue and LMK if interested on working on that, otherwise I will find sometime until next release, thanks for the help!

I have a lot of context so I can take that Issue. I’m pretty busy this weekend, so won’t get to work on it until next week.

There was some work on that part of the code in the parse-comments branch, which will mean merge conflicts. What are your thoughts on getting that branch into master for this release? It’s a big change, so I’d prefer to have it merged earlier in the release cycle, so it has time to be tested.

Also, we should remember that we aren’t 100% sure this explains the original problem. @U6T7M9DBR, in addition to the above questions, it’d be helpful to know whether the problem gets better or goes away if you change the settings to A) not invoke completion until 2 characters have been entered, B) not invoke completion until after a slight delay (a few hundred milliseconds) and C) not invoke completion at all. (The last one is to know whether the culprit is completion, or something else like code-lens or did-change.)

@U07M2C8TT I intend to review again and merge this weekend!

OK, great. Thanks! If it’s merged before I work on the Issue, that’ll avoid merge conflicts.

Great digging so far, I feel bad for leaving you with only a gif and some whining to work off of now! The chance that I have something messed up in my setup is a lot higher than 0%. I mean look at these great suggestions from my autocomplete:

tian_hieroglyphs?

What the hell is going on

😄

I mean I wouldn't sweat this too much until I have a minimal repro (sorry if that ship has sailed), what the heck can be going on here...

I use the official release (I think, installed via dharrigans AUR build).

turning off autocomplete suggestions via let b:coc_suggest_disable = 1 in the vim config fixes the problem immediately

yeah, I'm no vim user, but it seems your completion backend is using lsp and something else

which is possible on emacs as well, we call it company-backends

I wonder how it messes with the LSP tho

yeah, I guess is not related with clojure-lsp topping cpu

but it'd be good to test without those weird completions enabled just to make sure

Haha what do you mean weird

I need those for stuff

trakana trokes thipic

And toneletters

😂 I never saw something like that

whattt is going ooon 🙂

vim is for pros indeed 😂

I wonder if they are in the dict or something 😄

Yeah I'll figure out wtf is going on with those:

I think that little tag in brackets points to the source, and CLJ here I think is Conjure's clojure client

it seems to be something built-in on clojure "mode" for vim

Yeah maybe, I'll dig around. This is a hilarious mess.

I got rid of the cool completions with CocUninstall coc-conjure Now I only have LSP suggestions, the CPU hogging remains tho.

good, one less thing to think about while debugging 😅

😄

I'll try setting the coc config to as default as possible.

I think it's worth to add the details on that issue we created

If your editor is suggesting hieroglyphs you got bigger problems than clojure lsp?

😂 I think so

🙂 I'll mess around a bit more and see if I can get some sort of hypothesis formed at least

I did try to make a from zero reproduction, and everything worked fine when I only had clojure core and a few random libraries on the path.

This is a large ish (for a clojure project) codebase

it "feels" like the whole classpath is getting analyzed every keystroke

Sounds unlikely from LSP side

I'll add the few details I've found so far, but I'll try to do an open source repro too somehow.

Great effort tho, thanks so much for looking into it.

2022-02-06T20:59:13.689Z DEBUG [clojure-lsp.server:?] - :completion 12836ms

Something is happening atleast when I'm just writing in the comments, but god knows what

Is it normal that I have two LSP processes running?

if you have 2 protjects opened, yes

Hmmm

Well I need to go to sleep 🙂 I updated the issue, I think I might have some cleanup left to do in my vimrc / coc config, or that coc nvim might be misbehaving a bit maybe?

I'll try the other nvim lsp client tomorrow and see if the issue persists.

This doesn't seem right then:

Please, make sure you have your project root correctly configured

I feel like this guy: https://c.tenor.com/krfBwLFmB50AAAAC/drinking-water.gif

it's client responsibility (coc/neovim side)

"languageserver": {
        "clojure-lsp": {
            "command": "clojure-lsp",
            "filetypes": ["clojure"],
            "disableDiagnostics": false,
            "rootPatterns": ["deps.edn", "project.clj"],
            "additionalSchemes": ["jar", "zipfile"],
            "trace.server": "verbose"
        },

@U6T7M9DBR just FYI, we merged a pr from @U07M2C8TT which increase performance a lot on completion, maybe you should try a nightly build from #clojure-lsp-builds

Thanks - I will!