This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2022-05-23
Channels
- # announcements (3)
- # asami (5)
- # babashka (1)
- # beginners (38)
- # biff (4)
- # calva (12)
- # cider (2)
- # clj-commons (6)
- # clj-kondo (46)
- # clj-on-windows (1)
- # clojure (50)
- # clojure-europe (41)
- # clojure-nl (3)
- # clojure-norway (2)
- # clojure-uk (16)
- # clojured (3)
- # clojurescript (49)
- # conjure (1)
- # cursive (29)
- # datahike (6)
- # datascript (4)
- # emacs (70)
- # funcool (1)
- # google-cloud (12)
- # graalvm (10)
- # graalvm-mobile (7)
- # gratitude (4)
- # hyperfiddle (1)
- # joyride (26)
- # lsp (16)
- # malli (23)
- # nbb (5)
- # off-topic (23)
- # polylith (32)
- # re-frame (23)
- # releases (3)
- # remote-jobs (1)
- # reveal (3)
- # tools-build (16)
- # xtdb (50)
is it possible to modify the context passed to sci when running a hook? i.e this def
https://github.com/clj-kondo/clj-kondo/blob/17c64ded054edf0b5fe87b26ac07d4043538285c/src/clj_kondo/impl/hooks.clj#L148
@biscuitpants Currently not
ok cool. so i'd have to maintain a fork of Kondo if i wanted to do that (which i totally don't intend to do)
Before talking about maintaining a fork, perhaps you could first explain what your use case / requirement is.
its for supporting slurp
in a hook, which i mentioned last week. just thinking of sane ways to support what i need to do, which is read a file to get information to use in linting
I then replied: we could support it, but the challenge is that clj-kondo isn't always called from the same working directory by different tools. And then you replied: it's ok, I found a different solution.
yeah that solution requires running a script, which works ok for now. just thinking out loud really 😄
we could maybe call this function differently like: read-file-in-project
or so which starts at the root of the project
that would work nicely
i can work on a PR if you'd like @borkdude?
In a call right now, but first read this please: https://github.com/clj-kondo/clj-kondo/blob/master/doc/dev.md#workflow
@biscuitpants Back from the call. So let's first write up an issue and think about alternatives, etc. I think this is important to do before writing any code, so we don't add things unnecessarily
I have a single Java process that uses clj-kondo (via clj-kondo.core/run!
) to analyze different sub-projects in a monorepo (e.g., lib/a
, lib/b
, lib/c
. It currently does so sequentially (first lib/a
, then lib/b
, then lib/c
). To speed this up, I tried to run these in parallel (naïvely using pmap), but got lock contention on the clj-kondo cache. The job isn’t to refresh the respected clj-kondo caches: I’m writing out the clj-kondo results separately (along with some other data I’m collecting). Is this a case where I’d want to set :cache
to false
? Or perhaps set :cache-dir
per-lib to isolate the caches? Or is there a better approach I’m not seeing?
@grzm you could do :cache false, but this has the result that clj-kondo will not see connections between namespaces, e.g. it won't be able to say that you are calling a function with the incorrect amount of args
But perhaps that's not important to your use case. If you can tell more about your use case, maybe I could give more appropriate advice
It would surprise me if the cache lock contention was the reason that pmap wasn't much faster: clj-kondo doesn't spend a lot of time in reading/writing the cache
I’m using the clj-kondo analysis to understand namespace and var dependencies between the subprojects. I’m essentially creating a big map {lib/a {:clj-kondo {,,,}}, lib/b {:clj-kondo {,,,}}
Oh, it’s just stopping completely when it finds more than one thread is trying to read the cache file.
I think I can repro your problem with:
user=> (require '[clj-kondo.core :as clj-kondo])
nil
user=> (do (pmap #(clj-kondo/run! {:lint [%]}) (repeat 10 src")) nil)
:/But then again, this also doesn't work:
(do (pmap #(clj-kondo/run! {:lint [%] :cache false}) (repeat 10 src")) nil)
So how do you know it's contention on the cache?Here's the stack dump I'm seeing: https://gist.github.com/borkdude/9627c3ae595c2595e75e711d633d8fdc
I was seeing this error Clj-kondo cache is locked by other thread or process.
and assumed it was due to locking of the cache file?
yes, but one thread shouldn't hold it so long that other threads can't access it, unless you have many many threads
Yeah, there are 70+ modules with a 6-core box, so there are potentially a lot of threads.
oh lol, I just forgot a closing "
, this works fine :)
(do (pmap #(clj-kondo/run! {:lint [%] :cache false}) (repeat 10 "src")) nil)
nil
Oh right, with 10 I can trigger it but I should not set the cache to off user=> (do (doall (pmap #(clj-kondo/run! {:lint [%]}) (repeat 10 "src"))) nil) Execution error at clj-kondo.impl.cache/sync-cache (cache.clj:166). Clj-kondo cache is locked by other thread or process.
(loop [retry# 0
backoff# 25]
(if-let [lock#
(try (.tryLock channel#)
(catch java.nio.channels.OverlappingFileLockException _#
nil))]
(try
~@body
(finally (.release ^java.nio.channels.FileLock lock#)))
(if (= retry# ~max-retries)
(throw (Exception.
(str "Clj-kondo cache is locked by other thread or process.")))
(do (Thread/sleep backoff#)
(recur (inc retry#)
(* 2 backoff#)))))))))))