Wait. Re-downloading nightly it started to work as before again. Never mind me!

Yeah, I noticed that too, some build broke that not sure where or what, but was probably relate switch some kondo bump

Anyway, seems fixed on nightly as you mentioned indeed

Yeah, I'm happy. 😃 Will remember to restart clojure-lsp to check if what I see on nightly is latest nightly.

sorry for not using a thread :)

I wish I intended that pun!

It's ok, I knew you were excited :)

I've tested it and it still isn't nearly as responsive as the lsp4j one

in squint compiler.cljc I'm typing x-es on line 144 here:

(str sym-ns "_" xxxxx(munged-name sn))

Hum, I was having the same slowness as you but now it's gone, could you confirm your clojure-lsp --version ?

when I type x-es and then quickly backspace and then x-es again, it blocks my buffer

$ clojure-lsp --version
clojure-lsp 2022.09.24-13.47.55-nightly
clj-kondo 2022.09.09-SNAPSHOT

Just tried your code and can't repro the slowness, is your editor freezing or only diagnostics are taking some time to show?

Let me try to make a gif, like last time

Bummer

Hum, it's really smooth to me, I can't make emacs block no matter how much x or how fast or slow I type :thinking_face:

@jacob.maine do you think the parallelism level would make that big difference? as your was 3 and for mine was 7 IIRC, maybe borkdude has 3 as well

You have to type a few x-es and then stop for 1 second and then type again, then you'll notice the micro-freeze

yes, it definitely makes a difference

yeah, for me it's perfect :/ @U04V15CAJ could you confirm that emacs is using that clojure-lsp, checking the version on lsp-clojure-server-info

yes, I checked it, it's using the nightly and when I swap it back with the one from June it behaves smoothly

@jacob.maine since you have a low parallelism level, could you try borkdude's repro?

@U04V15CAJ I suppose you are using M1, using aarch64 binary right? maybe just give a try on the macos amd64 one or build clojure-lsp from master, since aarch64 uses a different CI, maybe could have a CI bug or something and it's using other version

it is reporting the correct version, right? yes, I can try the amd64 one

yeah, it's just that I changed how that works in the last week and made me think that we might have a bug on the Cirrus release, probably it's not a issue, but I think worth the try

Same issue with amd64. The trick is to type a few x-es, then wait a little bit and then type again.

I’m going to be offline for the next several hours. I can reproduce small lags if I type, but only if I type very slowly. @U04V15CAJ, have you changed your lsp-idle-delay away from the default of 0.5? I’m not sure it’s possible, but I’d love to have your server logs @U04V15CAJ right around the moment that the freeze happens. What I really need is the logs with tracing turned on, which means uncommenting the tracing lines in clojure_lsp/server.clj, and making a new build. If that’s too much hassle, I’ll plan to analyze my logs as soon as I can

This is my config and has been for a while: https://github.com/borkdude/prelude/blob/18fb017b90c3acc8e62ea25edb3a9e5b404e7066/personal/init.el#L365-L392

• https://github.com/clojure-lsp/clojure-lsp/blob/master/cli/src/clojure_lsp/server.clj#L548 • getting server logs: lsp-clojure-server-log

You do have a lsp-idel-delay of 0.5, so probably not related (I have mine of 0.2)

Maybe you can make a configuration option so I can set tracing on in there?

and then download a nightly

yeah, I was thinking about that to do soon

the issue is that at that time we don't have access to settings :/ as we don't even know the project-root

Sorry, have to sign off. Will check in later

Thanks @jacob.maine!

I'll think about something else to make the trace enable easier, meanwhile, let us know if you can do the change manually @U04V15CAJ otherwise we wait for @jacob.maine logs

@ericdallo there’s mechanics for this in LSP, something like setTraceLevel. when starting the server we’ll always pass a trace-ch We need to take that level and pass it through to lsp4clj. Then lsp4clj needs to respect a level like :off.

Sorry, that’s a little incoherent. We can talk about it later

Yeah, I was thinking about a variable which makes client send a setting at start

I can wait for this change

or maybe you can make a branch so I can download the changed binary from cirrus

Cirrus just build the binary on master

ok, I'll try a local build

trying:

bb native-cli

I have the local build now with trace on

here is a log

could it be this?

2022-09-24T14:53:55.751Z  DEBUG [clojure-lsp.server:40] - [Trace - 2022-09-24T14:53:55.751Z] Sending response 'textDocument/documentHighlight - (33)'. Request took 592ms.

that buffer is not small so it's expected that I think, I the issue probably won't be a specific request but the group of multiples during a small window which makes a issue hard to debug

also this one:

2022-09-24T14:53:55.751Z  DEBUG [clojure-lsp.server:40] - [Trace - 2022-09-24T14:53:55.750Z] Sending response 'textDocument/codeLens - (35)'. Request took 588ms.

and this one:

2022-09-24T14:53:55.752Z  DEBUG [clojure-lsp.server:40] - [Trace - 2022-09-24T14:53:55.751Z] Sending response 'textDocument/documentSymbol - (34)'. Request took 592ms.

2022-09-24T14:53:55.783Z  DEBUG [clojure-lsp.server:40] - [Trace - 2022-09-24T14:53:55.782Z] Sending response 'textDocument/codeAction - (32)'. Request took 623ms. Request failed: The request {:id 32, :method "textDocument/codeAction"} has been cancelled. (-32800).
Error data: {
  "id" : 32,
  "method" : "textDocument/codeAction"
}

I'll be afk now for a bit

Latest nightly accepts a --trace , along with latest lsp-mode , you can now (setq lsp-clojure-trace-enable t) and trace should be enabled

Awesome!

Thanks for the traces @U04V15CAJ. Like me, you’re seeing many requests that take longer than expected, because they wait for a long time to get their turn in the pool. I found one reason for this, which is fixed in https://github.com/clojure-lsp/clojure-lsp/pull/1276. @ericdallo if you could review that, @U04V15CAJ can test it. I’m not sure it will really resolve things, but it could help. You’re also seeing way more Received cancellation notification for unmatched request traces than I am. I’m not sure whether that’s a cause, a symptom, or a side-effect. I’ll have to think about whether there’s a clue there.

@jacob.maine Thanks! Note that I reproduced this by typing a few times, then pausing briefly and then typing again, pausing briefly (until I see the status bar updated) then hitting backspace a few times, etc.

I'll test

The PR looks good to me, I tested manually and it's working as good as before to me, I'm ok merging, or we can wait for borkdude testing it

OK, merge, then I'll download the nightly

Here’s a screen recording of my tests. I have KeyCastr turned on, so you can see how the cursor generally keeps up with the keyboard input. I can maybe detect a small delay while deleting, but it’s subtle. I’ve been trying lots of variations, typing at a pace of 2, 3, or 4 characters per second, as fast as I can, and just holding the key down.

And is it different from before your PR?

This is the one right? https://github.com/clojure-lsp/clojure-lsp-dev-builds/releases/download/2022.09.25-19.10.39-nightly/clojure-lsp-native-macos-aarch64.zip

No, I’m just demonstrating that I’ve been trying to follow your instructions for how to reproduce it, but haven’t been able to

ok

Yes, that’s the right nightly

Sadly, I can still reproduce the problem. It is subtle, I agree. I cannot repro it by typing steadily the x-es. But I can repro it by this: Type steadily, then wait .5 seconds, not too long, until lsp has almost finished processing, and then start typing steadily again. This is what you're seeing here: https://clojurians.slack.com/archives/CPABC1H61/p1664029519569659?thread_ts=1664028339.428649&cid=CPABC1H61

is it exactly as that gif? if not could you record a new one please?

I can record a new one.

Note when I backspace, sometimes 4 x-es are removed at once

after freezing

I wonder if Calva behaves the same, we could remove editor issues possibilities here even knowing this used to work better on previous versions

even so, pretty weird how your repro is way slower than mine

That’s a good idea @ericdallo… do you get freezes in Calva @U04V15CAJ? (And btw, thanks for helping us test this!)

I guess I should embark on simulating your system as closely as possible @U04V15CAJ. I don’t have the same hardware as you, but I can do the best I can. You sent me your Emacs config. Do you happen to know which versions of Emacs and lsp-mode you have?

Check if emacs was compiled with native compile support as well, the output of lsp-doctor would help with that

I will try Calva. I run Emacs 28.1 (9.0) downloaded from http://emacsforosx.com. My emacs config is here: https://github.com/borkdude/prelude (note, I've checked all my packages into source control as well, as a de facto way of having version control for my packages... I know)

lsp-doctor:

Checking for Native JSON support: OK
Check emacs supports `read-process-output-max': OK
Check `read-process-output-max' default has been changed from 4k: OK
Byte compiled against Native JSON (recompile lsp-mode if failing when Native JSON available): OK
`gc-cons-threshold' increased?: OK
Using `plist' for deserialized objects? (refer to ): OPTIONAL
Using emacs 28+ with native compilation?: OPTIONAL

Note that I didn't have the microfreezes on lsp4j with clojure-lsp (but this was clear already)

will try Calva now

in Calva I can't reproduce this problem

@jacob.maine Using promesa we are supposed to be handling requests/responses/cancels as the same lsp4j was doing before right? or are we doing something different?

I tried to follow lsp4j pretty closely. I could try stripping out promesa. It adds some wrappers on top of CompletableFuture, so I could drop down to the lower-level CompletableFuture code that lsp4j was using. There are also a few small ways that the new code flow differs from the old flow. Those differences seem minor to me, but I could try matching everything up even more closely. None of that would hurt, but it’s flying blind if I can’t reproduce the minifreezes locally. Maybe if you audited the differences @ericdallo you’d see something I’m not seeing? The message processing code starts https://github.com/eclipse/lsp4j/blob/89b5cd23def0abbd36e020440c26196c4a5ddde9/org.eclipse.lsp4j.jsonrpc/src/main/java/org/eclipse/lsp4j/jsonrpc/RemoteEndpoint.java#L184, continues https://github.com/eclipse/lsp4j/blob/89b5cd23def0abbd36e020440c26196c4a5ddde9/org.eclipse.lsp4j.jsonrpc/src/main/java/org/eclipse/lsp4j/jsonrpc/RemoteEndpoint.java#L217 and https://github.com/eclipse/lsp4j/blob/89b5cd23def0abbd36e020440c26196c4a5ddde9/org.eclipse.lsp4j.jsonrpc/src/main/java/org/eclipse/lsp4j/jsonrpc/RemoteEndpoint.java#L233 for cancellations, and does the main request processing https://github.com/eclipse/lsp4j/blob/main/org.eclipse.lsp4j.jsonrpc/src/main/java/org/eclipse/lsp4j/jsonrpc/RemoteEndpoint.java#L257. At some point we have to start researching whether it’s outside of the CompletableFuture code, in the message i/o.

@jacob.maine Are you using emacs?

Yes

So, one more attempt at explaining how I can reproduce it: type a series of x-es, then stop briefly and then hold backspace. If you can reproduce it, you will see several x-es at once be removed. Try this with very brief pauses (100-200ms, this probably relates to some delay that requests are being kicked off). Also try the same with lsp-mode turned off, it will feel more smoothly

Just hold the x button so multiple x-es will appear

> Try this with very brief pauses which part?

hold x button: 15 x-es or so will appear. Then pause briefly (100-200ms or so). Then hold backspace.

Repeat this several times

I really appreciate the efforts guys

and I think it already feels a bit better than before

you might need to take longer pauses than 200ms, just try different things, but holding x (so multiple xs appear) and then holding backspace is what helps me to reproduce it

So strange. I can’t reproduce it. I copied your LSP config as closely as I could and I have the same lsp-doctor output as you. Have to be afk for a bit

But you were able to reproduce the earlier problems right?

Going to sleep here and will do my work with the nightly tomorrow to see if the problem annoys me in real life usage

Btw, I'm using lspmode LSP :: lsp-mode <tel:202206202056|20220620.2056>, Emacs 28.1, darwin

Not sure if this matters?

could try an upgrade

latest lsp-mode has lsp-clojure-trace-enable which can help with future debuggings

ok, installed that, no difference. the delay is really subtle now, maybe not enough to continue down the rabbit hole, but I'll see tomorrow.

My hardware is m1 macbook air btw

I haven't tried asahi linux yet ;)

Btw, now that we're trying out nightlies, please try the new :unused-value linter as well ;)

> you were able to reproduce the earlier problems right? Sort of. I could very occasionally get a small amount of lag. But it was less reproducible than what you’re experiencing now. Now I can’t really get any lag.

@jacob.maine is there any way to get the parallelism level for each one of us and compare?

I also tried without lsp now, I might be seeing something else, so let me just try tomorrow with some fresh eyes, with and without lsp and then see if I will notice anything in real usage. I might have been focusing on some emacs artifact right now.

What I noticed when trying to disable lsp, is when I type lsp-mode lsp mode is disabled, but lsp-lens-mode is still enabled, this was a bit confusing

What is the output of (java.util.concurrent.ForkJoinPool/getCommonPoolParallelism) for your mac?

I know mine is 7 and Jacob's 3

sounds like a minor bug on lsp-mode, where it's not disabling lens-mode as well

Ah darn, after executing lsp-workspace-shutdown typing (defn xxxxxxxxxxxxxx) is so much smoother :(

I get 7

Chip:	Apple M1
  Total Number of Cores:	8 (4 performance and 4 efficiency)

so there is really something there... we have the same parallelism level, my emacs runs smoothly with no freeze during typing/backspacing, and I have lsp-idle-delay set to 0.2 which should make things slower in tradeoff of a better UI xp

one thing I do have differently is emacs 28.2 with native compile enabled, but I really don't think that would make that different for this specific issue.

I went back to the lsp4j version and it's as smooth as when not running lsp

@U04V15CAJ something to check tomorrow… I sometimes see momentary freezes when Emacs does garbage collection. (I’m watching top while holding x or delete.) Is that when your freezes are too? I’m not sure why the lsp4j version would generate less garbage, but it’s worth checking.

Along this same line what’s your gc-cons-threshold set to? I know your config says 100MB; I have the same thing in my config. But when I check its value after startup, it’s about 30MB, so something else must be resetting it. Perhaps yours is getting changed too

@jacob.maine

ELISP> gc-cons-threshold
104857600 (#o620000000, #x6400000)

If I run lsp-server-clojure-log I can reproduce serious freezes, like the ones you’re seeing @U04V15CAJ… My guess is that creates a huge amount of garbage, and the freezes are Emacs GC pauses. To corroborate that, Emacs takes ~200MB of RAM without the log tail, but about 1.1GB with it. It must be under a lot more GC pressure. Are you tailing the log in Emacs? Are you starting clojure-lsp with tracing? If so, that’d be one reason why the lsp4j version generates much less garbage—it doesn’t have tracing turned on.

I don't think I have tracing on...

but I am logging

I don't have the log opened

This is my config:

{:copy-kondo-configs? false
 :log-path "/tmp/clojure-lsp.out"}

yep, it should not be a problem unless you manually open that buffer or run lsp-server-clojure-log

ELISP> lsp-clojure-trace-enable
nil

Correct… that’s config for the server and tells it where to put the log. The client can ask the server what its log path is, so that it can open a buffer for it, which is, I think, what lsp-server-clojure-log does. There’s also another log, the log of what the client is doing. @ericdallo is that lsp-workspace-show-log? They contain similar information, but with different perspectives. A message that the client sends is logged on both sides, on the client side as a Sending… trace line, and on the server side as a Received… line

lsp-log-io for client logging from lsp-mode

What command are you using to start clojure-lsp? There was a moment there where I think you were adding --trace. What’s that variable called @ericdallo?

lsp-clojure-server-command ?

One thing that is not totally clear to me (might of missed it, apologies if it is stated above) is if you are all trying to reproduce the problem on the same source that borkdude is. Are you all trying the xxxxxx test on analyzer.clj from clj-kondo?

When you folks talk about garbage collection, is this at all related to to JVM GC? (or rather GraalVM native image GC?). Or are you only talking about Emacs GC?

I tested on compiler.cljc of squint as his original post

@UE21H2HHD Emacs's GC 😅

Thanks! Is there a theory as to why the switch to lsp4clj might have affected emacs GC? Or is this at the stage of looking at symptoms and just trying to diagnose what’s up?

Yeah, I can't see why changing from lsp4j would affect emacs GC

I think this question is the thing that Jacob and I were left with last time since we could not reproduce the lag with VSCode

(BTW just tested the xxxxx with delays and quick typing on clj-kondo analyzer.clj and can't repro the issue as well)

Are lsp4j and lsp4clj implementing exactly the same protocol? Could stress/flow tests be built and run against each to see where they behave differently?

(sorry I don’t know much about lsp4j or lsp4clj yet)

no, lsp4j uses futures and it's java way while we do in clojure using promesa. There was a missing place where me and @jacob.maine talked about taking a look that was the IO processing, we don't think that is the issue, but it's the only thing a little bit more different than lsp4j

btw, this problem does strongly correlate with the size of the buffer for me

yeah, makes sense as we send whole buffer to clj-kondo analyze when changing, but we do cancel futures now which we were not doing on latest release

I can even reproduce this by just typing () (the last paren is inserted automatically) and then backspace and then again, in a big buffer.

yeah, your gif is really different than mine, it must be something with emacs :thinking_face:

A while back I wanted to test rewrite-clj against a large source file. The biggest one I found at the time was: https://github.com/clojure/clojurescript/blob/master/src/main/cljs/cljs/core.cljs. It is currently over 12k lines long. Might be a good candidate for large file testing?

metabase is usually a good candidate as well, but squint seems enough

> it must be something with emacs Then why are things very smooth with the version from before lsp4clj?

good question 😂

maybe try increase lsp-idle-delay for you

Switching to lsp-clojure-custom-server-command '("/Users/borkdude/bin/clojure-lsp-stable") is very noticeable...

I'll try that

@U04V15CAJ if you had time and patience, you could maybe try with @ericdallo’s https://github.com/ericdallo/dotfiles/tree/master/.doom.d and see if you get different results?

sure. if it's just something I can clone and doesn't have any system-specific assumptions...?

upping lsp-idle-delay to 1s doesn't help

Or… maybe @ericdallo if you have time and patience, you could try @U04V15CAJ’s emacs config?

I see the delay just after this lsp-idle-delay has passed, when I start typing just after that interval though, I think

Yes, my config should be pretty good for starters IIRC, but I can try yours later tonight

Time and patience is probably a deciding factor for exploration here.

maybe could be other emacs package, I know that only happens changing clojure-lsp releases though

So we are all on the same page exactly what file are we doing the xxxxxx test on?

The xxxx test is: typing xxxxx, then pause briefly (I think the same period of time that lsp-idle-delay is set to), then continue typing. The xxxxes often don't appear smoothly, but are buffered and then appear in one go. Similarly when typing () and then backspace and then repeat.

And maybe we should be doing it all in the same place-ish? Near the top or bottom of the file? Does that matter?

The place does not matter for me, but the buffer size does

> The xxxx test is: typing xxxxx, then pause briefly (I think the same period of time that lsp-idle-delay is set to), then continue typing. The xxxxes often don’t appear smoothly, but are buffered and then appear in one go. Similarly when typing () and then backspace and then repeat. Cool, and which source file? We should probably all be using the same one.

analyzer.clj in clj-kondo or src/squint/compiler.cljc will do

I think the best way to describe this is that it's as if you're typing with sticky fingers

@UE21H2HHD, can I invite you to https://clojurians.slack.com/archives/CPABC1H61/p1663620137210819. It’s the same discussion, but I’d prefer to have one thread to monitor