Fork me on GitHub
#clj-kondo
<
2022-01-05
>
imre16:01:14

We bumped into an interesting issue with clj-kondo hooks vs tools.namespace/refresh. A one-liner repro case is:

clj -Srepro -Sdeps '{:deps {org.clojure/tools.namespace {:mvn/version "1.2.0"} seancorfield/next.jdbc {:git/url "" :git/sha "24bf1dbaa441d62461f980e9f880df5013f295dd"}}}' -M -e "((requiring-resolve 'clojure.tools.namespace.repl/refresh-all))"

imre16:01:26

result:

:reloading (next.jdbc.protocols next.jdbc.prepare next.jdbc.result-set next.jdbc.datafy next.jdbc.types next.jdbc.transaction next.jdbc.default-options next.jdbc.connection next.jdbc.sql.builder next.jdbc.sql-logging next.jdbc next.jdbc.sql next.jdbc.specs hooks.com.github.seancorfield.next-jdbc next.jdbc.quoted next.jdbc.date-time next.jdbc.optional next.jdbc.plan)
:error-while-loading hooks.com.github.seancorfield.next-jdbc
#error {
 :cause "Could not locate hooks/com/github/seancorfield/next_jdbc__init.class, hooks/com/github/seancorfield/next_jdbc.clj or hooks/com/github/seancorfield/next_jdbc.cljc on classpath. Please check that namespaces with dashes use underscores in the Clojure file name."
 :via
 [{:type java.io.FileNotFoundException
   :message "Could not locate hooks/com/github/seancorfield/next_jdbc__init.class, hooks/com/github/seancorfield/next_jdbc.clj or hooks/com/github/seancorfield/next_jdbc.cljc on classpath. Please check that namespaces with dashes use underscores in the Clojure file name."
   :at [clojure.lang.RT load "RT.java" 462]}]
 :trace
 [[clojure.lang.RT load "RT.java" 462]
  [clojure.lang.RT load "RT.java" 424]
  [clojure.core$load$fn__6856 invoke "core.clj" 6115]
  [clojure.core$load invokeStatic "core.clj" 6114]
  [clojure.core$load doInvoke "core.clj" 6098]
  [clojure.lang.RestFn invoke "RestFn.java" 408]
  [clojure.core$load_one invokeStatic "core.clj" 5897]
  [clojure.core$load_one invoke "core.clj" 5892]
  [clojure.core$load_lib$fn__6796 invoke "core.clj" 5937]
  [clojure.core$load_lib invokeStatic "core.clj" 5936]
  [clojure.core$load_lib doInvoke "core.clj" 5917]
  [clojure.lang.RestFn applyTo "RestFn.java" 142]
  [clojure.core$apply invokeStatic "core.clj" 669]
  [clojure.core$load_libs invokeStatic "core.clj" 5974]
  [clojure.core$load_libs doInvoke "core.clj" 5958]
  [clojure.lang.RestFn applyTo "RestFn.java" 137]
  [clojure.core$apply invokeStatic "core.clj" 669]
  [clojure.core$require invokeStatic "core.clj" 5996]
  [clojure.core$require doInvoke "core.clj" 5996]
  [clojure.lang.RestFn invoke "RestFn.java" 421]
  [clojure.tools.namespace.reload$track_reload_one invokeStatic "reload.clj" 35]
  [clojure.tools.namespace.reload$track_reload_one invoke "reload.clj" 21]
  [clojure.tools.namespace.reload$track_reload invokeStatic "reload.clj" 52]
  [clojure.tools.namespace.reload$track_reload invoke "reload.clj" 43]
  [clojure.lang.AFn applyToHelper "AFn.java" 154]
  [clojure.lang.AFn applyTo "AFn.java" 144]
  [clojure.lang.Var alterRoot "Var.java" 308]
  [clojure.core$alter_var_root invokeStatic "core.clj" 5499]
  [clojure.core$alter_var_root doInvoke "core.clj" 5494]
  [clojure.lang.RestFn invoke "RestFn.java" 425]
  [clojure.tools.namespace.repl$do_refresh invokeStatic "repl.clj" 94]
  [clojure.tools.namespace.repl$do_refresh invoke "repl.clj" 83]
  [clojure.tools.namespace.repl$refresh_all invokeStatic "repl.clj" 162]
  [clojure.tools.namespace.repl$refresh_all doInvoke "repl.clj" 147]
  [clojure.lang.RestFn invoke "RestFn.java" 397]
  [clojure.lang.Var invoke "Var.java" 380]
  [user$eval1 invokeStatic "NO_SOURCE_FILE" 1]
  [user$eval1 invoke "NO_SOURCE_FILE" 1]
  [clojure.lang.Compiler eval "Compiler.java" 7181]
  [clojure.lang.Compiler eval "Compiler.java" 7136]
  [clojure.core$eval invokeStatic "core.clj" 3202]
  [clojure.main$eval_opt invokeStatic "main.clj" 488]
  [clojure.main$eval_opt invoke "main.clj" 482]
  [clojure.main$initialize invokeStatic "main.clj" 508]
  [clojure.main$null_opt invokeStatic "main.clj" 542]
  [clojure.main$null_opt invoke "main.clj" 539]
  [clojure.main$main invokeStatic "main.clj" 664]
  [clojure.main$main doInvoke "main.clj" 616]
  [clojure.lang.RestFn applyTo "RestFn.java" 137]
  [clojure.lang.Var applyTo "Var.java" 705]
  [clojure.main main "main.java" 40]]}

borkdude16:01:28

Where does clj-kondo come in into the above?

imre16:01:54

this is caused by a combination of: • referring a lib by git coordinates, which means that it will appear as a directory on the classpath • said lib exporting clj-kondo hooks • clj-kondo hooks themselves being .clj files • those files normally having a namespace inside them that does not appear in code proper (and this is totally fine)

imre16:01:19

cause "Could not locate hooks/com/github/seancorfield/next_jdbc__init.class

imre16:01:35

I ended up writing a somewhat hacky helper function which tries to exclude clj-kondo.exports directories from the default refresh-dirs of tools.namespace:

(defn remove-clj-kondo-exports-from-tools-ns-refresh-dirs
  "Opinionated helper that addresses the problem that happens when:

  - a lib is referenced via a git dependency
  - said lib exports clj-kondo hooks (which are clojure source files)
  - and the namespace in those files is not something that also exists in a proper source file
    (this is normally not the case at the time of writing this, see
     for example)
  - and clojure.tools.namespace.repl/refresh(-all) is used

  Call this in your user.clj to (hopefully) fix the problem.

  A potential issue from using this is that if the directory containing the clj-kondo.exports folder
  also directly contains to-be-reloaded clojure source files, those will no longer be reloaded."
  []
  (->> (clojure.java.classpath/classpath-directories)
       (mapcat
        (fn [^File classpath-directory]
          (let [children   (.listFiles classpath-directory)
                directory? #(.isDirectory ^File %)
                clj-kondo-exports?
                           #(= "clj-kondo.exports" (.getName ^File %))
                has-clj-kondo-exports
                           (some (every-pred clj-kondo-exports? directory?) children)]
            (if has-clj-kondo-exports
              (->> children
                   (filter directory?)
                   (remove clj-kondo-exports?))
              [classpath-directory]))))
       (apply clojure.tools.namespace.repl/set-refresh-dirs)))

imre16:01:12

and then (remove-clj-kondo-exports-from-tools-ns-refresh-dirs) in user.clj

borkdude16:01:22

There was a similar issue recently with better-cond and cursive where cursive mistook exported hooks as application code

borkdude16:01:36

I'm not sure what the right answer here is. I would say it's a tooling problem

imre16:01:42

Yes, I've bumped into that too

borkdude16:01:24

We could support some extension .cljk or so to prevent other things from loading it... but then highlighting etc won't work as nicely. Tooling should probably have ways to exclude things from loading

imre16:01:28

Certainly a tooling problem. Whether the tool is clj-kondo or tools.namespace here, I'm not sure 🙂

imre16:01:41

Yeah, either a custom ext or nicer exclusion mechanisms for tools.namespace would help

imre16:01:12

I would actually vote for the latter, given hooks are already out there and you'd have to support the current .clj extensions

imre16:01:24

Since there isn't a tools.namespace channel here, I guess I'll refer to ask.clojure

imre22:01:30

So @U064X3EF3 commented on this raising the question of why are these files under the lib's (next.jdbc's) path? What are the location requirements for clj-kondo hooks?

borkdude22:01:43

The requirements are that these files are on the classpath under a clj-kondo.exports directory so clj-kondo knows what to copy to the local config directory

borkdude22:01:17

It encounters these files while linting dependencies

borkdude22:01:27

if they are not on the classpath, clj-kondo will never see them

imre23:01:16

I'm not sure how to resolve this, then.

imre09:05:53

so Alex recently commented on this again on ask.clojure, it appears a non-clj extension would be the "officially supported" solution: https://ask.clojure.org/index.php/11434/could-namespace-refresh-sophisticated-directory-filtering?show=11882#c11882

borkdude09:05:15

@U08BJGV6E I think there might be another workaround

imre09:05:43

interested

borkdude09:05:46

We could make a clj-kondo hooks "shim" library so the namespaces are defined, but they don't do actually anything

borkdude09:05:04

you could do this in your own project to start

borkdude09:05:14

Just define:

(ns clj-kondo.hooks-api)
yourself

borkdude09:05:27

if you are sure that you're not using clj-kondo as a dependency in your project

borkdude09:05:30

although I think having a filter in that reloading library would be a better solution

borkdude09:05:38

a different extension would give different problems I'm sure

imre09:05:43

For now I think I'd prefer my current workaround I detailed above tbh. While the tools.ns filter would be welcome, Alex seems to be on the position that distributing .clj files with a lib means you're distributing those namespaces

imre09:05:32

And as such if you give an instruction to load all namespaces from the lib, clojure isn't at fault for trying to load them

borkdude09:05:37

What if we supported .cljk ? But then we'd have to be sure there would be not other language in the future adopting that extension :/

imre09:05:53

His suggestion was to just use .edn

borkdude09:05:17

That's not correct in the case of clj-kondo hooks. Did you explain this was used for clj-kondo hooks?

borkdude09:05:13

Anyway, perhaps you can PR your current workaround as documentation for clj-kondo hooks? At least we've got something

imre09:05:24

It's all in the question on ask.

imre09:05:57

I can for sure.

borkdude09:05:11

A link to the ask issue is also fine

imre09:05:13

Why would .edn be a problem for clj-kondo?

borkdude09:05:34

it would be a problem for tooling, an .edn file cannot contain language constructs that .clj can

borkdude09:05:43

so clj-kondo itself would already vomit warnings all over your hooks code

borkdude09:05:22

or at least, not right now, but probably in the future when we would include such warnings

imre09:05:25

what about a file extension that is based off edn and not clj? Like ednk or something along those lines? Or even .clj-kondo or .hook

borkdude09:05:32

e.g. #(foo %) and @foo are not supported in .edn

imre09:05:45

Something that doesn't imply "clojure source" to Clojure itself

borkdude09:05:26

foo/bar/hook.kondo

imre09:05:32

I have to agree with Alex on hooks themselves being only data from the library consumer's point of view

imre09:05:57

And since clj source files imply code to Clojure, perhaps this data shouldn't be in clj files

borkdude09:05:16

They are Clojure files executed by clj-kondo

borkdude09:05:34

so I don't agree. A library could also bundle scripts that it would execute using load-file in some special context

imre09:05:52

Would you mind commenting on the question there, then? I don't think I have enough weight in the community to have an effect on this either way

imre09:05:17

I'd dig a .kondo extension, that would be very specific

borkdude09:05:51

Let's support the .kondo extension, or maybe even .clj_kondo or .clj-kondo?

borkdude09:05:43

if we will support another extension, the other uphill battle would be to convince other hook authors to use that extension

borkdude09:05:49

so I think you would still need your workaround

borkdude09:05:04

linking to the ask issue + workaround in the docs seems like the best overall solution

borkdude09:05:54

We could also support .cljc

imre09:05:54

Where in the docs you think this would be best placed?

borkdude09:05:02

and then you could use a reader conditional for clj-kondo ;)

imre09:05:05

I think one of the problems here is that these files are not on the classpath where clojure would look for them if it tried to require one of these namespaces

borkdude09:05:26

isn't the problem that they are on the classpath?

borkdude09:05:44

and that it tries to load a namespace that doesn't exist in the JVM classpath

borkdude09:05:25

e.g.:

(ns my-hooks #?(:clj-kondo (:require [clj-kondo.hooks-api])))

borkdude09:05:31

but that seems like a terrible workaround as well

imre09:05:50

; clj -Srepro -Sdeps '{:deps {org.clojure/tools.namespace {:mvn/version "1.2.0"} seancorfield/next.jdbc {:git/url "" :git/sha "24bf1dbaa441d62461f980e9f880df5013f295dd"}}}' -M -e "((requiring-resolve 'clojure.tools.namespace.repl/refresh-all))"

:reloading (next.jdbc.protocols next.jdbc.prepare next.jdbc.result-set next.jdbc.datafy next.jdbc.types next.jdbc.transaction next.jdbc.default-options next.jdbc.connection next.jdbc.sql.builder next.jdbc.sql-logging next.jdbc next.jdbc.sql next.jdbc.specs hooks.com.github.seancorfield.next-jdbc next.jdbc.quoted next.jdbc.date-time next.jdbc.optional next.jdbc.plan)
:error-while-loading hooks.com.github.seancorfield.next-jdbc
#error {
 :cause "Could not locate hooks/com/github/seancorfield/next_jdbc__init.class, hooks/com/github/seancorfield/next_jdbc.clj or hooks/com/github/seancorfield/next_jdbc.cljc on classpath. Please check that namespaces with dashes use underscores in the Clojure file name."
 :via
 [{:type java.io.FileNotFoundException
   :message "Could not locate hooks/com/github/seancorfield/next_jdbc__init.class, hooks/com/github/seancorfield/next_jdbc.clj or hooks/com/github/seancorfield/next_jdbc.cljc on classpath. Please check that namespaces with dashes use underscores in the Clojure file name."
   :at [clojure.lang.RT load "RT.java" 462]}]
 :trace
 [[clojure.lang.RT load "RT.java" 462]
  [clojure.lang.RT load "RT.java" 424]
  [clojure.core$load$fn__6908 invoke "core.clj" 6161]
  [clojure.core$load invokeStatic "core.clj" 6160]
  [clojure.core$load doInvoke "core.clj" 6144]
  [clojure.lang.RestFn invoke "RestFn.java" 408]
  [clojure.core$load_one invokeStatic "core.clj" 5933]
  [clojure.core$load_one invoke "core.clj" 5928]
  [clojure.core$load_lib$fn__6850 invoke "core.clj" 5975]
  [clojure.core$load_lib invokeStatic "core.clj" 5974]
  [clojure.core$load_lib doInvoke "core.clj" 5953]
  [clojure.lang.RestFn applyTo "RestFn.java" 142]
  [clojure.core$apply invokeStatic "core.clj" 669]
  [clojure.core$load_libs invokeStatic "core.clj" 6016]
  [clojure.core$load_libs doInvoke "core.clj" 6000]
  [clojure.lang.RestFn applyTo "RestFn.java" 137]
  [clojure.core$apply invokeStatic "core.clj" 669]
  [clojure.core$require invokeStatic "core.clj" 6038]
  [clojure.core$require doInvoke "core.clj" 6038]
  [clojure.lang.RestFn invoke "RestFn.java" 421]
  [clojure.tools.namespace.reload$track_reload_one invokeStatic "reload.clj" 35]
  [clojure.tools.namespace.reload$track_reload_one invoke "reload.clj" 21]
  [clojure.tools.namespace.reload$track_reload invokeStatic "reload.clj" 52]
  [clojure.tools.namespace.reload$track_reload invoke "reload.clj" 43]
  [clojure.lang.AFn applyToHelper "AFn.java" 154]
  [clojure.lang.AFn applyTo "AFn.java" 144]
  [clojure.lang.Var alterRoot "Var.java" 308]
  [clojure.core$alter_var_root invokeStatic "core.clj" 5535]
  [clojure.core$alter_var_root doInvoke "core.clj" 5530]
  [clojure.lang.RestFn invoke "RestFn.java" 425]
  [clojure.tools.namespace.repl$do_refresh invokeStatic "repl.clj" 94]
  [clojure.tools.namespace.repl$do_refresh invoke "repl.clj" 83]
  [clojure.tools.namespace.repl$refresh_all invokeStatic "repl.clj" 162]
  [clojure.tools.namespace.repl$refresh_all doInvoke "repl.clj" 147]
  [clojure.lang.RestFn invoke "RestFn.java" 397]
  [clojure.lang.Var invoke "Var.java" 380]
  [user$eval1 invokeStatic "NO_SOURCE_FILE" 1]
  [user$eval1 invoke "NO_SOURCE_FILE" 1]
  [clojure.lang.Compiler eval "Compiler.java" 7194]
  [clojure.lang.Compiler eval "Compiler.java" 7149]
  [clojure.core$eval invokeStatic "core.clj" 3215]
  [clojure.main$eval_opt invokeStatic "main.clj" 488]
  [clojure.main$eval_opt invoke "main.clj" 482]
  [clojure.main$initialize invokeStatic "main.clj" 508]
  [clojure.main$null_opt invokeStatic "main.clj" 542]
  [clojure.main$null_opt invoke "main.clj" 539]
  [clojure.main$main invokeStatic "main.clj" 664]
  [clojure.main$main doInvoke "main.clj" 616]
  [clojure.lang.RestFn applyTo "RestFn.java" 137]
  [clojure.lang.Var applyTo "Var.java" 705]
  [clojure.main main "main.java" 40]]}

imre09:05:11

"Could not locate hooks/com/github/seancorfield/next_jdbc__init.class, hooks/com/github/seancorfield/next_jdbc.clj or hooks/com/github/seancorfield/next_jdbc.cljc on classpath. Please check that namespaces with dashes use underscores in the Clojure file name."

imre09:05:32

cause they are under .....exports somewhere

borkdude09:05:48

then why does it load this file at all

imre09:05:49

but my issue is that Clojure even attempts to load them

borkdude09:05:23

it's the same issue with a test runner, it loads a file to scan for tests. every sane test runner has a config option to skip certain namespaces / directories

borkdude09:05:36

to prevent loading

imre09:05:02

Afair tools.ns looks for clojure source files on directory paths, then parses out the namespace name from their ns forms, then tries to load those namespaces

borkdude09:05:40

yes. it should have the option, is my opinion, but I'm not going to spend any time trying to convince the core team

borkdude09:05:21

I'll leave it at this for now: • document your workaround • if more related issues come up, we can consider the extension

imre09:05:41

Would you like me to record an issue for the latter?

borkdude09:05:02

One related issue in the past has been that Cursive also analyzes hook code and when you have same-named hooks then you end up navigating to the wrong file

imre09:05:18

Yep, I also had that in Cursive

borkdude09:05:44

The solution in Cursive was to just use a different ns for the hook code, but the extension would also save the day here I guess

imre09:05:08

If there's an issue about it, I can also link to that

imre09:05:21

Need to head out now, will do these in the afternoon

borkdude09:05:21

the issue is in the better-cond repo

imre09:05:24

Thank you for your time

👍 1
imre09:05:32

Oh actually, one more reason for the custom extension: tools.build compilation: https://clojurians.slack.com/archives/C02B5GHQWP4/p1651685386479149

imre09:05:49

I ran into that just last week

borkdude09:05:10

ok, the argument is getting more compelling now

borkdude13:05:26

Thanks, merged :)

imre13:05:24

Thank you

imre13:05:45

And here's the issue, I tried to collect everything without being too wordy: https://github.com/clj-kondo/clj-kondo/issues/1685