Clojurians Log v2

@qythium and others: please try ac41fb65b8d0a8df104e2d907614b971c2c8df3b for the new "macros from source" feature.

(defmacro deffoo 
  {:clj-kondo/macro true}
  [x]
  `(def ~x :foo))

(deffoo x)

x ;; x is recognized as var now!

(ns emi.enclojures.kondo-xp
  (:require [clojure.string :as ztr]))

(declare -not-in-kondo)

(defmacro if-kondo
  {:clj-kondo/macro true}
  [kondo clj]
  (if-not (resolve '-not-in-kondo)
    kondo
    clj))

(defmacro resolves
  {:clj-kondo/macro true}
  []
  (if-kondo
    (throw (ex-info (str ztr/blank?) {}))
    nil))

(defmacro doesnt-resolve
  {:clj-kondo/macro true}
  []
  (throw (ex-info (str `ztr/blank?) {})))

(resolves) ;; ok

(doesnt-resolve) ;; kondo gives back `ztr/blank? while clj gives back `str/blank?

more details about that = better

(resolves) raises an error with text clojure.string$blank_QMARK_@3ccb0625, i.e. kondo manages to grab the clojure.core/str? object as I would expect, while (doesnt-resolve) can't resolve the ns through the quasiquote and raises an error with content ztr/blank?.

this should already work. have you tried the newest commit?

I'm on ac41fb65b8d0a8df104e2d907614b971c2c8df3b , will look now

ok try dc52bbb36589cc4c21391df94fa41de9d914ac91 (this is only a cleanup commit though) and make sure you restart clojure-lsp or whatever you're using

or better, use the command line

will do

fwiw this is the first time I'm trying the feature so there shouldn't be anything stale

$ cat src/emi/enclojures/kondo_xp.clj
(ns emi.enclojures.kondo-xp
  (:require [clojure.string :as ztr]))

(declare -not-in-kondo)

(defmacro if-kondo
  {:clj-kondo/macro true}
  [kondo clj]
  (if-not (resolve '-not-in-kondo)
    kondo
    clj))

(defmacro resolves
  {:clj-kondo/macro true}
  []
  (if-kondo
    (throw (ex-info (str ztr/blank?) {}))
    nil))

(defmacro doesnt-resolve
  {:clj-kondo/macro true}
  []
  (throw (ex-info (str `ztr/blank?) {})))

(resolves)
(doesnt-resolve)

$ cd /tmp/kondo-xp && clojure -M:clj-kondo/dev --lint src
src/emi/enclojures/kondo_xp.clj:25:1: error: clojure.string$blank_QMARK_@3a027c11
src/emi/enclojures/kondo_xp.clj:26:1: error: clojure.string/blank?

neat, let me just set things up and will try

yes, looks fixed both in cmdline and in the clojure-lsp build using that hash

you are running clojure-lsp in a jvm or do you compile it ?

jvm

makes sense. me too

neat if-kondo trick btw

written badly lmao, should resolve the ns-qualified symbol as in the-util-ns.not-in-kondo`

?

otherwise it breaks outside or not-in-kondo's ns

at any rate, this feature is great

yeah, I don't mind the details, but this trick will work for stuff that should not run in kondo

I'm waiting for @qythium to report back and then I'll probably merge if there aren't any other complaints

Tried out the new commit, but it still reported a Unresolved symbol: b on the above expr, which I then found was independent of the xref issue - seems that it only analyses up to one level of expansion? ie. with the {:pre ...} commented out so it's self-contained - (when-every [a 12, b 34] (+ a b)) expands to (when-let [a 12] (when-every [b 34] (+ a b))) and the inner form isn't getting sci-macroexpanded.

Another example - notice if you swap the order of clauses the _ and n bindings get recognized correctly.

(defmacro cond+
  "clause => test expr | [binding-form test] expr
  Like [[clojure.core/cond]], but literal vectors in test position
  are interpreted as if-let bindings.
  Also supports :let and :do clauses as in better-cond"
  {:arglists '([& clauses])
   :clj-kondo/macro true}
  ([] nil)
  ([test expr & clauses]
   (cond
     (vector? test) `(if-let ~test ~expr (cond+ ~@clauses))
     :else `(if ~test ~expr (cond+ ~@clauses)))))

(let [x "a123b"]
  (cond+ (number? x) x
         [[_ n] (re-matches #"a(\d+)b" x)] (parse-long n)
         :else 0))

oh and both these macros are handled correctly if I use the existing :hooks :macroexpand config and simply copy them verbatim

Seems like a syntax quote resolution issue - if I edit the recursive calls to use fully qualified heads like (my-ns/cond+ ~@clauses) then it works out. I thought this meant it could be solved by generating another mapping {clj-kondo.gen-macros.my-ns/cond+ clj-kondo.gen-macros.my-ns/cond+} to the config.hooks.macroexpand but that didn't seem to work either

Cool if you can repro this to the smallest possible issue I'll take a look on Monday-ish. In the middle of conferences right now

Amazing, been wishing for something like this for some time~ It seems like cross-ns references aren't supported though, the analysis only extends to 'helper fns' within the same file? repro:

(ns blub.utils)

(defn binding-vec?
  {:clj-kondo/macro true}
  [v] (and (vector? v) (even? (count v))))

(defmacro dummy {:clj-kondo/macro true} []) ; to generate the file

(ns blub.kondo
  (:require [blub.utils :as u]))

(defmacro when-every
  "Like [[clojure.core/when-let]], but takes any number of binding-test pairs.
  Only executes body if all tests are logical true, otherwise nil.
  Short circuits on first failure."
  {:clj-kondo/macro true}
  [bindings & body]
  {:pre [(u/binding-vec? bindings)]}
  (if (seq bindings)
    `(when-let ~(subvec bindings 0 2)
       (when-every ~(subvec bindings 2)
         ~@body))
    (cons 'do body)))

(when-every [a 12, b 34] (+ a b))

$ clj-kondo --lint src/blub/kondo.clj
WARNING: file blub/utils not found while loading hook
WARNING: error while trying to read hook for blub.kondo/when-every: Could not find namespace: blub.utils.
src/blub/kondo.clj:17:14: error: Unresolved symbol: a
src/blub/kondo.clj:17:20: error: Unresolved symbol: b
linting took 18ms, errors: 2, warnings: 0

Not supported .. yet :)

let me try to fix it

can you try commit 70523461a0e6d578acedb7bf9b226307842564bc?

nice, seems to be working on the previous examples, I'll try it out against some hairier macros and report back in a bit.

Have you settled re. the naming/API though - I found the use of :clj-kondo/macro true meta a bit odd how it's playing double duty for both the helper defns and the macros themselves

I found that a bit odd too. would you prefer :clj-kondo/macro-helper true for the other stuff?

why do the helpers need metadata?

because clj-kondo needs to know they should be copied along into the config code

yeah I was curious about that too - isn't clj-kondo's analyzer technically able to trace the functional dependencies or something

you're absolutely right that clj-kondo can analyze this itself but it complicates stuff

I wanted to release this first and maybe make the other annotation optional later

I think the annotation might be better so people are more aware of that it's still limited to what clj-kondo can execute - not arbitrary things that pull in libs

wondering if it makes sense to extend the existing :clj-kondo/lint-as meta key, since they're sort of mutually exclusive(?) Something like

(defmacro foo {:clj-kondo/lint-as :self} ...) ; or :macroexpanded etc

neh

my two cents are on :clj-kondo/for-macroexpansion , or even :clj-kondo/macroexpand-helper uniformly

getting a false positive unused-binding here

(defmacro mylet
  {:clj-kondo/macro true}
  [bindings & body]
  `(let ~bindings ~@body))

(mylet [x 123] `(list ~x))

$ clj-kondo --lint src/blub/kondo.clj
src/blub/kondo.clj:10:9: warning: unused binding x

also ran into a bit of a hiccup with some macros that close over their helper fns - there's no var in this case to attach metadata to

(letfn [(helper [x] x)]
  (defmacro foo
    {:clj-kondo/macro true}
    [x]
    (helper x)))

first issue has been a thing for a while even in the main

please post the first as a separate issue

if it is one

should we support putting :clj-kondo/macro true on top level forms?

oh and I guess this is more of a DX issue but when something like this goes wrong with the SCI evaluation side of things, it doesn't get surfaced in-editor when being used as a flycheck linter? ie. I can only see it printed as a WARNING: error while trying to read hook for blub.kondo/foo: Could not resolve symbol: helper at the top of the call to clj-kondo --lint as a CLI tool

oh and valid forms further down in the copied ns don't get run as a result, which can be quite confusing

good feedback, I'll take a look at this soon

issue created for the false positive of unused-var

I'm going to look into your macro now @k13gomez

what is parse-args, can you please make it self-contained?

ok, I can reproduce with:

(defmacro defdata
  {:clj-kondo/macro true}
  [& vargs]
  (let [[sym args] vargs]
    `(def ~sym (cons 'do '~args))))

(defdata some-random-id
         "c44eb844-c10e-4726-a529-b7d68452452f")

wait, it did work, just needed to restart lsp for some reason, perhaps local dev changes. @k13gomez perhaps you can test this from the command line and make sure you are using the right SHA.

your issue may be that parse-args is a function separately from the macro. this isn't supported yet. let me try to work on a solution for that.

Yeah I figured, I’ll see if I can make it self contained for some of the macros I maintain, this will be a huge help even if only for self contained macros

I was testing it from the command line, built the binary from your branch using GraalVM

ok, no need to build the command line, you can just invoke the main function with clj -M -m clj-kondo.main --lint ...

I have a global dev alias for this so I can invoke it from any dir

@k13gomez this example now works:

(defn parse-args
  {:clj-kondo/macro true}
  [args] args)

(defmacro defdata
  {:clj-kondo/macro true}
  [& vargs]
  (let [[sym args] (parse-args vargs)]
    `(def ~sym (cons 'do '~args))))

(defdata xsome-random-id
         "c44eb844-c10e-4726-a529-b7d68452452f")

xsome-random-id

(exact details are up for debate)

ohh nice, Ok let me give that a try

Ok, progress report, I couldn't get my exact use case to work initially because it was using clj-commons.digest/md5 for something, so I would get this error:

WARNING: file clj_commons/digest not found while loading hook
WARNING: error while trying to read hook for rules.data/defdata: Could not find namespace: clj-commons.digest.

It should work. I just fixed that! :) are you using the newest SHA?

8e31cccb30ff957c66e135ef51de35a80e990bf9

Let me see if I can reproduce again

Ok, I reproduced it again and what appears to not be updating is the ns declaration of the generated gen-macros namespace, for example after removing clj-commons.digest it still appears in the ns decl.

(ns clj-kondo.gen-macros.rules.data
  (:require [clj-commons.digest :as digest]
            [clara.rules :as-alias r]))

the body of the macro was updated and the digest use was removed, but the top level :require in the auto-gen'd namespace is what is out of sync

good feedback, let me look at this

should be fixed, try again

Confirmed, that is fixed, however ran into another edge case, if I add {:clj-kondo/macro true} to a macro, but later remove it, it remains in the auto-gen'd namespace

in this example, the generated code below remained the same even after removing the macro meta from defun

(ns clj-kondo.gen-macros.rules.data)

(defmacro defun "defines a new function in the current namespace with the specified symbol name and body value using intern data so that it can be serialized" {:clj-kondo/macro true} [vargs] (let [[sym & args] (parse-args vargs &form)] `(defun.core/defun ~sym ~@args)))

(defmacro defdata "defines a new data var in the current namespace with the specified symbol name and body value in an implicit do using intern data so that it can be serialized" {:clj-kondo/macro true} [sym & args] `(def ~sym (do ~@args)))

The macros look like this currently:

(defmacro defdata
  "defines a new data var in the current namespace with the specified symbol name and body value in an implicit do using intern data so that it can be serialized"
  {:clj-kondo/macro true}
  [sym & args]
  `(def ~sym (do ~@args)))

(defmacro defun
  "defines a new function in the current namespace with the specified symbol name and body value using intern data so that it can be serialized"
  [vargs]
  (let [[sym & args] (parse-args vargs &form)]
    `(defun.core/defun
       ~sym ~@args)))

try again

Ok, I tested it again, read the code a bit to understand better what is happening, this is what I am observing: • the manifest file is being updated after removing the :clj-kondo/macro meta from a macro • the generated namespace is still not being updated and still contains the faulty macro A bit verbose perhaps but see the state of files below:

$ tree .clj-kondo/clj_kondo
.clj-kondo/clj_kondo
└── gen_macros
    └── rules
        └── data.clj

3 directories, 1 file

$ tree .clj-kondo/inline-configs
.clj-kondo/inline-configs
└── rules.data.clj
    ├── config.edn
    └── gen-macros.edn

2 directories, 2 files

$ cat .clj-kondo/clj_kondo/gen_macros/rules/data.clj
(ns clj-kondo.gen-macros.rules.data)

(defmacro defun "defines a new function in the current namespace with the specified symbol name and body value using intern data so that it can be serialized" {:clj-kondo/macro true} [vargs] (let [[sym & args] (parse-args vargs &form)] `(defun.core/defun ~sym ~@args)))

(defmacro defdata "defines a new data var in the current namespace with the specified symbol name and body value in an implicit do using intern data so that it can be serialized" {:clj-kondo/macro true} [sym & args] `(def ~sym (do ~@args)))

$ cat .clj-kondo/inline-configs/rules.data.clj/config.edn
{:hooks {:macroexpand {rules.data/defdata clj-kondo.gen-macros.rules.data/defdata}}}

$ cat .clj-kondo/inline-configs/rules.data.clj/gen-macros.edn
[{:orig-ns rules.data, :fn-name defdata, :gen-ns clj-kondo.gen-macros.rules.data, :form-str "(defmacro defdata \"defines a new data var in the current namespace with the specified symbol name and body value in an implicit do using intern data so that it can be serialized\" {:clj-kondo/macro true} [sym & args] `(def ~sym (do ~@args)))", :aliases {}}]

what commit SHA are/were you on?

on 9c39b1ee6e8b1a3304ae1d85cd311b8d6eb063b1

can you try 80ef37e08e3191f2cd9163af55c0744f9ed725e3 - not sure if it fixes this specific problem, if not, I'll try to reproduce yours locally, if I have the code

testing it now, if this still won't fix it I will put together the code in a separate micro repo and push it to github

that seems good, thanks!

it's fixed! works for me now, the files are updated correctly each time

Clj-kondo loading a macro from source using the :clj-kondo/macro annotation. Helper functions that are used in the macro should also be annotated. The only restriction is that vars used in the macro at compile time are available in the SCI env where the macro runs (so clojure.core, set, walk, etc are available).

Merged this to master now. Please re-test. I documented the caveats in doc/hooks.md. Won't fix these for next release, so anything beyond that is expected.

heads up that macro was changed into macroexpand-hook

yes

{:clj-kondo/macroexpand-hook true} for both macros and helpers

looks good to me. BTW, re: the if-kondo docs, was it intentional to leave the version that tries to resolve an unqualified symbol?

PR welcome if you want to change it, I didn't think very hard about this one

will do

on which branch?

master

it's been merged already?

alright, I'll do a simple PR; the test does pass as before. However, of course testing this requires to run a program because the bug really comes up at runtime where the unqualified var isn't resolved properly across namespaces.

riiight

merged. Thanks for testing everyone!

YOOOOOOOOOOOOO

is this on a dev build?

it's locally in a branch :)

90% of the linters I end up writing are because of macros

me too

looking forward to it!

one issue I'm hitting is this and not sure if it's a bug or feature.

(ns foo)
(require '[clojure.string :as str])
(defmacro dude [& xs]
  `(str/join "," xs))

(ns bar (:require [foo]))
(foo/dude 1 2 3) ;;=> Unresolved namespace `clojure.string`, did you require it?

i you want to test, pick the newest commit from this branch. https://github.com/clj-kondo/clj-kondo/tree/macros-from-source Would appreciate it if you could find some edge cases :)

i'll try it out with my complex macros lol

As long as your macro is kind of a pure function it should kinda work ;)

i won't use it with the one that relies on tools.analyzer 😉

that will certainly not work

ah, I was planning to try it with the macros inside my clara-rules fork

I guess i'll try it anyways and let you know how it fares

try it out and report issues, we'll see where we end up :)

I've sort of reduced this macro to a simple example to show, but basically these don't work:

(defmacro defdata
  {:clj-kondo/macro true}
  [& vargs]
  (let [[sym args] (parse-args vargs &form)]
    `(def ~sym (cons 'do '~args))))

(defdata some-random-id
  "c44eb844-c10e-4726-a529-b7d68452452f")

Unresolved symbol: some-random-id