Fork me on GitHub

eric, some time ago I reported having issues with a large project where typing keywords would cause clojure-lsp to end up eating my cpu core(s) and the logs would report really long (hundreds of seconds) of :completion times. As this seems to happen only with large projects, it's not trivial to reduce this problem to a simple reproduction. So instead I built a really quick-and-dirty babashka script ( to generate leiningen projects with lots of dependencies and a configurable amount of files. At least on my machine I can use this to quite consistently trigger the problem (at least when generating 1000+ source files).


Nice, I can take a look and try the repro


I generated a project and everything is working good, how can I repro the issue with the generated project?


I found that completing without any prefix takes ~6s but it's makes sense since the project is big and there is no prefix to the completion


besides that, writing keywords works without no issues for me


is there anything specific you are doing on that generated project that makes that high consume of CPU?


what I usually do is a) use something to monitor cpu usage, I particularly like the cpu monitor widget of kde/plasma, which shows the individual core usage with a bar chart, b) start typing some nonsense expression where you quickly refer to multiple keywords, e.g. something like (get-in foobar [:a :b :c :d :e :f]) c) stop typing d) look at the cpu usage, usually at this point I have at least 2-3 cores at 100%, depends a bit on how fast I typed the keywords and how many e) depending on the size of the project, after about 20+ seconds I see a bunch of :completion <some value> ms prints in the clojure-lsp log file and the cpu usage drops back to ~zero


the high cpu usage is just a consequence of probably clojure-lsp handling a lot of requests like empty completions which doesn't seems to happen to me My guess, is that you have some configuration in your emacs that is calling completion too soon, or something


for exmaple, typing (get-in foobar [:a :b :c :d :e :f]) doesn't trigger any completion for me as I didn't trigger that


what is your company-minimum-prefix-length ? I have mine set to 2


interesting. Normally when I type something like :foo emacs gives me a little dropdown of possible alternatives, and I've assumed this is something that clojure-lsp provides


yeah it is, is just that for some reason your emacs is requesting completions more times


I'm not certain, but I assume my company-minimum-prefix-length is 1, because that's "suggested" here:


hum, the default value is 3 and doom-emacs sets it to 2, I'm not sure why I configured as 1 there 😅


what I see is: • you can fix this increasing to 2 probably improving the performance on server since it will filter less items • we can improve performance on server somehow


since the issue are keywords, not sure what we could improve on server


I don't have my dev machine online just now, but if I can mitigate this issue by just increasing that prefix length then that's more than fine for me 🙂


yeah, that probably will help, if so, I will update the tutorial later :)


sure, I'll check tomorrow what happens with my actual project if I alter the prefix length and report back any findings. and thanks again for your help (and this awesome tool) 👍


Sweet, thanks!


based on a quick test using the dummy project and with a prefix-length of 3 I no longer can trigger the issue (as easily as before). If I try to intentionally write longer nonsense keywords (`(get-in foobar [:asdsadsa :asdsadfe ....`), I can see that a single cpu core goes to 100% utilization and then after a while I get something like :completion 25821ms in the log. However, previously (with a prefix length of 1) I could easily get several cores (sometimes all of them) to 100% so this is a good improvement 👍


good, still is not the ideal as I can't get those :completion high ms for some reason


Can someone help me get my mental model correct for how clojure-lsp determines which directories it scans to find source files to lint? I’m clearly missing something. Context: I have a ClojureScript project, built with Figwheel Main via deps.edn, and I’m trying to both ensure that all the relevant source files are linted, while not getting lots of spurious errors because generated files are scanned. Here’s where I think I’ve got to (although I suspect I’m wrong somewhere along the line): • clojure-lsp has the related, but different, concepts of both a classpath and a set of source-paths : ◦ The classpath is the result of running clojure -Spath (i.e. it’s constructed from both the :deps and the top-level :paths and :extra-paths within deps.ednsource-paths is constructed by looking at both the top-level :paths and :extra-paths plus any :paths and :extra-paths found within aliases defined by :source-aliases (by default #{:dev :test}). • Any Clojure file found within either the classpath or source-paths is linted. ◦ Unless :ignore-classpath-directories is set to true in which case only Clojure files found within source-paths are linted. The documentation for :ignore-classpath-directories says: > will not consider clojure files within the directories specified by your classpath. This is needed, for instance, if your build puts artifacts into `resources` or `target` that you want lsp to ignore. And this is where I get confused. Because the only way (as far as I know?) for resources or target to get added to your classpath is by having them within :paths or :extra-paths , in which case they’ll be linted. Sure enough this is what I see when I set :ignore-classpath-directories . So how can :ignore-classpath-directories be used to get lsp to ignore resources or target?


You are almost correct, actually I made an improvement yesterday for deps.edn projects to use the dev and test aliases during the classpath scan, but probably no related with your issue


ignore-classpath-directories is a old flag and tricky, probably we can improve the docs or how it works


Thanks @UKFSJSM38. If I’m “almost” correct, where am I going wrong? Is there an example of using :ignore-classpath-directories to avoid scanning resources or target? Or, do I need to take a different approach? I guess I could add resources to an alias which isn’t included within :source-aliases?


I'm not used to work on cljs projects, but recently I was working in a one and ignore-classpath-directories didn't work as expected as well, maybe we can improve that behavior for cljs/compiled folders like target or resources. This is how that works AFAICS:


clojure-lsp during startup make 2 scans, : • for the external classpaths which mostly are related to external libs via the return of clojure -Spath (I changed that yesterday to be clojure -A:dev:test -Spath) • For the source-paths of the project, the one that is resolved from the ignore-classpath-directories is considered before calling kondo in the first case, the external classpath, where we just remove the folders from the result of the clojure -Spath , so, IMO for that works, you need to not include your resources/target into the source paths discovery (from 2 bullet) and it should work (?)


sorry if I'm not clear, but that flag is a really old one before I start working in the project and it's not something even I understand perfectly


OK, I think that that mirrors where I got to. Thanks!


Feel free to create a minimal repro where I can try too and we can think on something to improve that behavior


An observation: Perhaps as well as :source-aliases there should be :classpath-aliases too? Assuming :dev and :test will work for many projects, but not all?


yeah, we already have the :project-specs where you can define that


but you override the whole project-specs with that, probably a flag just for the extra alias would work as well

👍 1

I could create a minimal reproduction, but the Figwheel Main example “Flappy Birds” might serve just as well?:


if you confirm your issue happens there,seems enough indeed


Yup. 1766 warnings from clojure-lsp diagnostics 😉


(after building the Clojurescript, obviously)


and those warnings come from the resources/target ?


alright, I ll take a look


I confirmed the issue after building the project indeed, let me try understand how would be the fix with the current clojure-lsp


So this is what I’ve come up with for my project (which is rather more complex than the flappy birds because I have both a Clojure server and a Clojurescript app living in the same repository): Disclaimer: I haven’t finished testing it completely yet, but it seems to do the right thing:

{:deps {...}}

 :paths ["src/cljs" "src/cljc" "src/clj" "test/clj" "test/cljs"]

 :aliases {:dirac {:extra-paths ["src/dirac" "src/dev" "resources"]
                   :main-opts ["-m" "dirac-figmain.repl"]}
           :figwheel {:extra-paths ["src/dev" "resources"]
                      :main-opts ["-m" "figwheel.main" "-b" "dev" "-r"]}
           :build {:extra-paths ["src/prod"]
                   :main-opts ["-m" "figwheel.main" "-O" "advanced" "-bo" "prod"]}

           :serve {:extra-paths ["src/prod" "resources"]
                   :main-opts ["-m" "race-and-improve.core"]}
           :serve-dev {:extra-paths ["src/dev" "resources"]
                       :main-opts ["-m" "race-and-improve.core"]}

           :test {:extra-deps {com.cognitect/test-runner {:git/url ""
                                                          :sha "dd6da11611eeb87f08780a30ac8ea6012d4c05ce"}}
                  :main-opts ["-m" "cognitect.test-runner"]}}}


I’ve basically just moved “resources” out of paths and into more other aliases.


that's what I was about to say: there is this workaround of moving resources or/and target to :extra-paths of the build task


It results in rather more duplication than I would prefer, but it works for the time being 👍


this is something that will fix the issue, but probably bad since you need to change your project


(I wish that deps.edn allowed one alias to refer to another!


we can probably create a flag on clojure-lsp with some exclude defaults like:

:exclude-source-paths #{"resources" "target"}

👍 2

and then clojure-lsp doesn't consider those folders as source-paths as default