Fork me on GitHub
#clj-kondo
<
2022-05-23
>
biscuitpants07:05:16

ok cool. so i'd have to maintain a fork of Kondo if i wanted to do that (which i totally don't intend to do)

borkdude07:05:54

Before talking about maintaining a fork, perhaps you could first explain what your use case / requirement is.

biscuitpants07:05:43

its for supporting slurp in a hook, which i mentioned last week. just thinking of sane ways to support what i need to do, which is read a file to get information to use in linting

borkdude07:05:58

I then replied: we could support it, but the challenge is that clj-kondo isn't always called from the same working directory by different tools. And then you replied: it's ok, I found a different solution.

biscuitpants07:05:58

yeah that solution requires running a script, which works ok for now. just thinking out loud really 😄

borkdude08:05:12

we could maybe call this function differently like: read-file-in-project or so which starts at the root of the project

biscuitpants08:05:28

that would work nicely

biscuitpants08:05:56

i can work on a PR if you'd like @borkdude?

borkdude09:05:31

@biscuitpants Back from the call. So let's first write up an issue and think about alternatives, etc. I think this is important to do before writing any code, so we don't add things unnecessarily

biscuitpants09:05:13

makes total sense and i agree 🙂

🎉 1
borkdude09:05:51

OK, issue welcome then :)

grzm13:05:50

I have a single Java process that uses clj-kondo (via clj-kondo.core/run! ) to analyze different sub-projects in a monorepo (e.g., lib/a, lib/b, lib/c. It currently does so sequentially (first lib/a, then lib/b, then lib/c). To speed this up, I tried to run these in parallel (naïvely using pmap), but got lock contention on the clj-kondo cache. The job isn’t to refresh the respected clj-kondo caches: I’m writing out the clj-kondo results separately (along with some other data I’m collecting). Is this a case where I’d want to set :cache to false? Or perhaps set :cache-dir per-lib to isolate the caches? Or is there a better approach I’m not seeing?

borkdude13:05:36

@grzm you could do :cache false, but this has the result that clj-kondo will not see connections between namespaces, e.g. it won't be able to say that you are calling a function with the incorrect amount of args

borkdude14:05:14

But perhaps that's not important to your use case. If you can tell more about your use case, maybe I could give more appropriate advice

borkdude14:05:27

It would surprise me if the cache lock contention was the reason that pmap wasn't much faster: clj-kondo doesn't spend a lot of time in reading/writing the cache

borkdude14:05:53

Maybe the overhead of pmap was just not worth it vs sequentially linting

grzm14:05:04

I’m using the clj-kondo analysis to understand namespace and var dependencies between the subprojects. I’m essentially creating a big map {lib/a {:clj-kondo {,,,}}, lib/b {:clj-kondo {,,,}}

grzm14:05:03

Oh, it’s just stopping completely when it finds more than one thread is trying to read the cache file.

grzm14:05:36

(Apologies for not having the error at hand)

borkdude14:05:55

perhaps a better idea is to use (run! {:parallel true :lint ["lib/a" "lib/b"]})

borkdude14:05:51

I think I can repro your problem with:

user=> (require '[clj-kondo.core :as clj-kondo])
nil
user=> (do (pmap #(clj-kondo/run! {:lint [%]}) (repeat 10 src")) nil)
:/

borkdude14:05:43

But then again, this also doesn't work:

(do (pmap #(clj-kondo/run! {:lint [%] :cache false}) (repeat 10 src")) nil)
So how do you know it's contention on the cache?

grzm14:05:19

I was seeing this error Clj-kondo cache is locked by other thread or process. and assumed it was due to locking of the cache file?

borkdude14:05:09

yes, but one thread shouldn't hold it so long that other threads can't access it, unless you have many many threads

grzm14:05:20

Yeah, there are 70+ modules with a 6-core box, so there are potentially a lot of threads.

borkdude14:05:59

oh lol, I just forgot a closing ", this works fine :)

(do (pmap #(clj-kondo/run! {:lint [%] :cache false}) (repeat 10 "src")) nil)
nil

borkdude14:05:00

It still works for me for 100

borkdude14:05:12

Now I'll try 1000 and perhaps I'll trigger the error

borkdude14:05:21

we should maybe make this time out configurable

borkdude14:05:27

there is a back-off mechanism

borkdude14:05:20

Oh right, with 10 I can trigger it but I should not set the cache to off user=> (do (doall (pmap #(clj-kondo/run! {:lint [%]}) (repeat 10 "src"))) nil) Execution error at clj-kondo.impl.cache/sync-cache (cache.clj:166). Clj-kondo cache is locked by other thread or process.

borkdude14:05:46

Check defmacro with-cache

borkdude14:05:58

(loop [retry# 0
                  backoff# 25]
             (if-let [lock#
                      (try (.tryLock channel#)
                           (catch java.nio.channels.OverlappingFileLockException _#
                             nil))]
               (try
                 [email protected]
                 (finally (.release ^java.nio.channels.FileLock lock#)))
               (if (= retry# ~max-retries)
                 (throw (Exception.
                         (str "Clj-kondo cache is locked by other thread or process.")))
                 (do (Thread/sleep backoff#)
                     (recur (inc retry#)
                            (* 2 backoff#)))))))))))

grzm14:05:07

Yup. I’ll take a look. With :cache false you didn’t run into that issue, correct?

grzm14:05:31

Cool. I’ll play around with this. Cheers!

borkdude14:05:50

You can also turn on :skip-lint true if you're not interested in lint warnings

borkdude14:05:54

This might speed up things a little bit

grzm14:05:12

I run it all, baby. java-class-* stuff, too 🙂