Clojurians Log v2

For posterity: • I actually found this when attempting to remove double expansion for doseq. I was writing tests, but I realized any auto-boxed variables again doubled the number of expansions https://github.com/frenchy64/fully-satisfies/blob/5d0a1ec5ed51f7c8ec1c206fd8db4e12c2dbfde7/test/io/github/frenchy64/fully_satisfies/linear_expansion_test.clj#L9-L11 • Here's the loop where Clojure iteratively analyzes the same loop, throwing away expansions until all loop args are inferred https://github.com/clojure/clojure/blob/4dffbdb20d8038c399490dccbf975a3819d8d2d5/src/jvm/clojure/lang/Compiler.java#L6939

thanks for your detective work!

So, Clojure does even more than at-most-twice. Indeed, we have no real guarantee for how many times macros are invoked.

Excellent work, Ambrose. This adds even more weight to my question: What is considered correct macroexpansion, in a dialect? I don't think we have the answer to that right now, but it's a great question.

yep, great news for compiler implementers!

what's the great news?

I'm just being sassy, but this is almost license for alternative clojure implementers to expand macros however they please.

hahaha. In truth, this is not great news. lolcry is the apt emoji.

it seems like this is a quirk of auto-boxing tho, right?

well if you avoid state in your expansions, then you don't care 😉

lol the cat's out of the bag for that one

it's almost like avoiding state makes things easier

i'm uploading environment secrets in all of my macros

in terms of educating users, there's at least two good reasons to avoid side effects in macros (beyond avoiding state) 1. the side effects will not fire at all in AOT compiled code 2. Clojure expands macros repeatedly in an undefined order I think idempotent side effects are fine (like compiling a Java class) but side effects that assume macro expansion order just don't work. Obvious in hindsight, but had I known about this loop I could have deleted that entire chapter in my dissertation 🙂

@jeaye is it not good? if it's undefined behavior, then compiler implementers can go to town with runtime optimizations via extra analysis.

I wouldn't say so. I think we should design languages for user experience, not ease of compiler implementation. Undefined behavior is not a good thing, in my opinion. Coming from C++, I'm all too familiar with it. For jank, I will aim to have an exactly-once macroexpansion guarantee, since chances are every normal Clojure dev I know already has that as their mental model.

does Clojure run arbitrary macros twice inside of doseq or was that about doseq itself?

doseq itself duplicates the body in the expansion. but doseq expands to loops, that if they themselves auto box, then they expand twice. I don't think that's reproducible with doseq right now, but I wrote my own doseq to fix it that accidentally auto boxed a loop.

user=> (defmacro it [x] (prn :x x) x)
#'user/it
user=> (doseq [i [1 2 3]] (it i))
:x i
:x i
nil

Ambrose showed a working reproduction above, using loop. https://clojurians.slack.com/archives/C06E3HYPR/p1776483620027079?thread_ts=1776462647.589469&cid=C06E3HYPR Since doseq expands to a loop, it's probably more helpful to just talk about loop.

@jeaye if you're not aware there's a ton of work in the racket community around this subject. this is a decent starting point https://stackoverflow.com/a/13109177

I was not aware! Thank you. I've saved this along with my notes.

Fascinating! My latest scenario also had to do with loop bindings. Thanks so much for chiming in with this, Ambrose. You're a treasure trove of knowledge. 🙂

Macroexpansions happen at compile time, not eval time so they would always happen once, right?

Hmmm. We may not be using the same terminology. I am building on these phases of compilation: • Lexing • Parsing • Semantic analysis • Codegen Macro expansions happen during semantic analysis. Something like clojure.core/eval would take a parsed form (already gone through lexing + parsing) and would do semantic analysis + codegen on it. Thus, when I do this:

user=> (eval '(when true 5))
5

why would you expand a macro twice if you can help it?

Macros may have access to and use state, so I would think doing it more than once would be unexpected and potentially bad

I agree, but it led me to consider if this has been stated anywhere and, if not, if it should be.

Or perhaps this is the sort of thing which is so obvious like (foo 1 2) should call foo once and not twice and so it doesn't need explicit mention? 😄 In jank, I realized that there were multiple places I analyzed things twice and it was generally because it was easier to analyze, learn about the code, change some inputs and analyze again so I can do even better than to analyze once and then use the learnings to modify the AST expressions I've already built. Only today, when such a scenario came up again did it occur to me that macros would be expanded twice and that sounded really not-good. So it's clear that I shouldn't be doing this, but a morbid curiosity led me to see if there's anything in Clojure's docs which actually prevents it.

What I've seen in various places is that macros generally expand once, but it isn't guaranteed to be done by the compiler. E.g. some macros like core.async/go will expand macros in the go body and then return some transformation on the expanded stuff. Thereafter the compiler won't process those macros again.

In ClojureScript macroexpansion happens once and often is cached (for performance reasons), unless a .clj file changes

Interesting, so that's not exactly-once per use.

It's more likely at-most-once.

right

Which is arguably better than at-most-twice. 😂

The Clojure compiler expands macros twice like you do in jank in the case where a loop arg is boxed:

Clojure 1.12.4
user=> (defmacro side-effect [] (prn "expanded"))
#'user/side-effect
user=> (set! *warn-on-reflection* true)
true
user=> (set! *unchecked-math* :warn-on-boxed)
:warn-on-boxed
user=> #(loop [a 1] (side-effect) (recur nil))
"expanded"
NO_SOURCE_FILE:1 recur arg for primitive local: a is not matching primitive, had: Object, needed: long
Auto-boxing loop arg: a
"expanded"
#object[user$eval143$fn__144 0x7a639ec5 "user$eval143$fn__144@7a639ec5"]
user=> 

If you're morbidly curious I wrote a whole section in my dissertation speculating about Clojure's policy about macro side effects: https://thesis.ambrosebs.com/#x1-10400019

It was basically exploring whether it was justifiable to write analysis passes like this, tho I was more interested in reordering macro side effects.

I seem to remember finding a jewel on the mailing list where Rich said something about macro effects in the early days that gave me hope for these kinds of approaches, but I didn't save the link.

> > According to Paul Graham's On Lisp, macroexpanders should be purely > > functional, and you should not count on how often a macro gets > > expanded. This seems like a reasonable restriction for Clojure too. > > However, Chouser posted an example that shows the expansion of proxy > > does have a side effect [2]. > > > > Should macros written by ordinary mortals follow PG's rule? > > Yes. https://groups.google.com/g/clojure/c/Ch4aaR_pTD0/m/Gpiqi4JJeK8J

> No. There's no significant difference between CL and Clojure in this area. Same post, this ^ was more convincing because CL does not define the order of expansion.