Fork me on GitHub
#rewrite-clj
<
2023-02-14
>
Noah Bogart20:02:21

I just wrote a linting engine using rewrite-clj (actually clj-kondo's rewrite-clj output), and it runs really fast compared to the hand-rolled stuff I was playing with beforehand. i'm reading through various defunct clojure projects that do hand-rolled parsing, and it's really surprising to me how simple they would be if they had relied on rewrite-clj (kibit and marginalia specifically). thanks to y'all for working so hard on this library.

lread20:02:28

Thanks for the kind words @UEENNMX0T!

❤️ 2
lread20:02:05

Is it kibit you are working on?

Noah Bogart20:02:12

sort of, yeah

Noah Bogart20:02:43

I originally was building an s-expression pattern matching library for use in clj-kondo, but i think it's not quite appropriate given how @U04V15CAJ likes to write his code, and then i saw how slow kibit is and in the process of taking over the project, i ended up finishing my own version, which lints much faster lol

lread20:02:51

cool! looking forward to seeing the fruits of your experiments!

Noah Bogart21:02:33

initial report:

$ time lein kibit
...
real    41m41.018s
user    42m18.347s
sys     0m3.742s
vs
$ time clojure -M:run ../netrunner/
...
real    0m14.986s
user    1m47.699s
sys     0m1.404s

lread21:02:46

Yoiks! That's more than a tad faster!

borkdude21:02:56

> but i think it's not quite appropriate given how @U04V15CAJ likes to write his code Not sure what you mean by that :P

Noah Bogart21:02:03

sorry, i don't mean to insult or denigrate. you've said that you prefer to perform each lint check exactly where it needs to happen as you process the files in an effort to keep speed as fast as possible, instead of pre-processing a given hunk of code and running over it a second time to find errors

borkdude21:02:54

that's correct. this is not a matter of preference, but a matter of performance. clj-kondo has to be fast to be a viable option to execute on every keystroke

👍 2
borkdude21:02:34

I'm still thinking about a way to let people have the whole file's rewrite-clj structure and do whatever they want, as a hook. But having kibit as a hook, is probably a little bit on the heavy side, right

borkdude21:02:16

Also kibit's matching is a little bit course, since it doesn't really see if a symbol is a local or a var reference, for example, right?

Noah Bogart21:02:44

yeah, it doesn't care about that, it's purely focused on shape. if i were to attempt to integrate this into clj-kondo, a bunch of stuff would need to change

Noah Bogart21:02:28

it is pretty fast tho. linting the https://github.com/mtgred/netrunner/ with native clj-kondo takes 28 seconds, and linting the same repo with my library as an uberjar takes 10 seconds.

borkdude22:02:29

nice. what makes your lib faster than kibit?

Noah Bogart22:02:24

macros to generate functions that compare the shape: given (not (empty? coll)), it'll generate something like (and (= :list (:tag form)) (let [children (:children form)] (and (= 1 (count children)) (= 'not (:sym (nth children 0))) ...)

Noah Bogart22:02:38

wraps it in a function, and then calling that function is pretty quick

borkdude22:02:19

but you could do the same on raw s-expressions right?

borkdude22:02:45

does it come down to: not using core.logic is faster than rolling your own matching logic? that's what I would have expected :)

Noah Bogart22:02:36

Haha yes, it is due to that

borkdude22:02:36

I'm not sure how rewrite-clj makes stuff simpler for you compared to working with s-expressions directly. in clj-kondo I only use rewrite-clj so I have more control over the locations (e.g. numbers can have location metadata) and invalid expressions still parse, etc.

borkdude22:02:06

you are aware of grasp right? https://github.com/borkdude/grasp it works directly on s-expressions using clojure.spec expressions

👍 2
Noah Bogart22:02:11

I relied on clj-kondo’s parser because i started this to satisfy my curiosity about alternate methods of writing clj-kondo linters lol

😄 2
borkdude22:02:57

well, clj-kondo still has access to the rewrite-clj nodes so in theory it could give you access to every s-expression it lints to do your custom kibit like matching. Are you aware of the macroexpand hook? It turns a macro rewrite-clj node into an s-expression, which the user can then transform (or lint) and after that it's turned back into a rewrite-clj node.

borkdude22:02:10

What you're doing currently is probably more performant though.

Noah Bogart22:02:19

I just switched from using clj-kondo's rewrite-clj to the rewrite-clj library itself and running takes longer: 7.6 seconds to 19.2 seconds

Noah Bogart22:02:51

i suspect because i had to add a small bit of logic to skip over comment and whitespace nodes

Noah Bogart22:02:39

(from (let [~children-form (vec (:children ~new-form))] to (let [~children-form (vec (remove n/whitespace-or-comment? (n/children ~new-form)))]

borkdude22:02:26

I've alluded to it before but why aren't you just matching against s-expressions instead of using rewrite-clj?

Noah Bogart22:02:34

using edn/read-string?

borkdude22:02:01

Oh god no, edn/read-string doesn't work for clojure code. That is very close to what grasp is doing (which I also mentioned before in this thread), if you want to take that approach, you can take a look there.

😂 2
👍 2
Noah Bogart22:02:45

i don't want to fall down the well of having to update my code to handle new edge cases in clojure parsing, you know? better to rely on the existing libraries with nice apis

Noah Bogart22:02:06

kibit does that, and is currently broken if you use :as-alias lol

borkdude22:02:13

grasp uses edamame to parse clojure code: https://github.com/borkdude/edamame just look at its source, it's very simple.

Noah Bogart22:02:24

ah excellent, i forgot about that library

borkdude22:02:54

I mean source as in, look at how grasp is using edamame

Noah Bogart22:02:30

i'll read it over

Noah Bogart22:02:47

warning, i wrote it for myself lol

borkdude22:02:52

I'll take a look when I'm not close to going to bed :)

borkdude22:02:02

your lib is also rewriting the original source?

borkdude22:02:01

I didn't know that was something that kibit did, but I think you could take the same approach with this library: the matching can happen on regular s-expressions like kibit does (which will make it faster than working with rewrite node, I think and the matching logic becomes simpler) and the rewriting can be done using rewrite-clj

Noah Bogart23:02:31

I’m only rewriting in the printing to out, not in the file itself

Noah Bogart23:02:21

Interesting, I’ll try it on edamame output, see how that feels. The primary logic doesn’t change much thankfully, the output of the macro barely relies on the rewrite-clj form