Clojurians Log v2
clojure-dev
Hi folks. 🙂 I am hoping someone can shed some light on when/where Clojure evaluates metadata. For example, if we take this code:
user=> (defn foo
"Doc here"
{:fn (fn [])}
[])
#'user/foo
user=> (meta #'foo)
{:arglists ([]), :doc "Doc here", :fn #object[user$fn__134 0xd1a10ac "user$fn__134@d1a10ac"], :line 1, :column 1, :file "NO_SOURCE_PATH", :name foo, :ns #object[clojure.lang.Namespace 0x71391b3f "user"]}(fn []) into a function object. However, we can see from the definition of defn that it expects :inline metadata to not yet be evaluated, since it looks inside the list to see if the fn has name and gives it a name if it doesn't. So this implies that metadata evaluation happens after defn, but I haven't spotted it within DefExpr, since the eval method in there just calls meta.eval(). Assuming meta is a MapExpr, that also won't end up calling Compiler.eval .
So, I have two specific questions about all of this:
1. When/where does Clojure end up evaluating this meta?
2. How does Clojure end up not losing the original meta data (fn in source form) for use with AOT generation, since the #object cannot be used?
I think there must be some subtle rule here I'm missing.
There is a lot to unpack here.
The way the metadata map is evaluated(when it is) is just like any other map
So it compiles the map into a method call that builds the map, and compiles the fn into code that creates the fn object that is executed and then passed to the method call (RT.map maybe?) that builds the map
the eval methods in the compiler are not used when compiling(generating bytecode)
.eval is only used when evaluating a restricted subset of expressions in the repl, and it interprets
For more complicated expressions, the repl does the same thing aot compilation does, generates bytecode then runs it
Annoyingly there isn't a nice simple rule for when metadata gets evaluated, in the beginning it just sort of happened where ever, but these days a little more attention is paid when new features are added to keep new ways it is evaluated
The other thing is defn is def + fn, DefExpr only covers the compilation of def
The metadata usage you have your example, I think., is actually a feature of fn not def
So you get to figure out how fns are compiled, which is basically a whole subsection of the compiler if you haven't looked at it yet
The metadata that gets attached to the var created with def is also a different thing from the metadata attached to the fn object, and I don't recall but it would not surprise me if they had different evaled vs not evaled behavior. I think maybe metadata attached to vars via def has been historically evaluated (for resolving type tag classnames) but maybe most other is not
I think I am not correct about that metadata syntax being part of fn
what happens if you have (defn foo "doc here" {:fn (do (prn (ex-info "" {})))} [])?
Thanks for the info. It has not matched my research that this is part of fn. There are two parts here, as I can see it.
1. DefExpr handles its own meta
2. defn uses with-meta on the symbol of the generated def so the map never appears in the analyzer again, since it's tucked away in the meta of the symbol
So these are two different code paths:
(defn ^:foo foo [])
(defn foo {:foo true} [])[[clojure.lang.AFn applyToHelper "AFn.java" 156]
[clojure.lang.AFn applyTo "AFn.java" 144]
[clojure.lang.Compiler$InvokeExpr eval "Compiler.java" 4222]
[clojure.lang.Compiler$InvokeExpr eval "Compiler.java" 4221]
[clojure.lang.Compiler$BodyExpr eval "Compiler.java" 6715]
[clojure.lang.Compiler$MapExpr eval "Compiler.java" 3568]
[clojure.lang.Compiler$DefExpr eval "Compiler.java" 468]
[clojure.lang.Compiler eval "Compiler.java" 7762]
[clojure.lang.Compiler eval "Compiler.java" 7712]
[clojure.core$eval invokeStatic "core.clj" 3232]
[clojure.core$eval invoke "core.clj" 3228]
[clojure.main$repl$read_eval_print__9248$fn__9251 invoke "main.clj" 437]
[clojure.main$repl$read_eval_print__9248 invoke "main.clj" 437]
[clojure.main$repl$fn__9257 invoke "main.clj" 459]
[clojure.main$repl invokeStatic "main.clj" 459]
[clojure.main$repl_opt invokeStatic "main.clj" 523]
[clojure.main$main invokeStatic "main.clj" 668]
[clojure.main$main doInvoke "main.clj" 617]
[clojure.lang.RestFn invoke "RestFn.java" 400]
[clojure.lang.AFn applyToHelper "AFn.java" 152]
[clojure.lang.RestFn applyTo "RestFn.java" 135]
[clojure.lang.Var applyTo "Var.java" 707]
[clojure.main main "main.java" 40]]So this is indeed part of MapExpr.
So {:foo (fn [])} must be a MapExpr with a FnExpr as a value.
So this does actually just come from DefExpr evaluating its meta via meta.eval().
Ok, so that would answer the first question, but the second question regarding AOT remains. If the var holds {:fn #object[...]} as its meta, how does Clojure do the AOT generation of the pre-eval meta?
As I said, .eval is only used in a restricted set of circumstances
You have to look at .emit if you want to understand bytecode generation
Yep, I understand the difference between these two.
But, during AOT generation, we also do eval.
Not with .eval
Aot compilation, and non-trivial expressions are evaluated via compilation to bytecode then executing the bytecode
And that all happens via .emit and friends
Sure, I'm familiar with this. I think I must be missing something that you're saying, so let me provide an example. Let's say I want to AOT compile this code:
(def foo 2)DefExpr which will have a NumberExpr as init. Normally, I would just eval this with no need to emit anything. My understanding is that during AOT we'll eval to do the effect and then we'll also emit to write out our .class file.
I believe you're telling me that during AOT we don't do the eval and instead just do an emit and we use that for our effect and also our generated .class file?No
Aot compilation emits the bytecode and then runs the bytecode
No calling eval
Ok, so that matches what I believed you were telling me.
Like, forget aot compilation, when you type something like (let []) into the repl, the way that is evaluated is the compiler wraps it in a thunk like (fn* (let [])) then compiles that to a class, then instantiates the class that calls IFn.invoke on it
Yep, I'm familiar with that.
Yeah, so aot compilation is the same thing, it just writes the class to disk as well
The same applies to fn, let, loop, try, and throw.
Yep, there is no need to explain further. 🙂
In short, Clojure gets around the second issue I brought up by not combining eval and emit but instead only using one or the other.
I appreciate the answers, @hiredman. So far, my mental model of Clojure JVM has been incorrect, thinking that we combine these two during AOT compilation. That's what jank does, since there's a big difference between codegen for AOT compilation and codegen for JIT compilation. So I'll need to tackle this differently than Clojure, but that's on me to sort out.
There is a vaguely related thing clojure does where if you pass an object (like a fn) to the compiler (like if you have a macro that expands to a fn object instead of a form that evaluates to one) it compiles it as a call to a 0 arg constructor on the class of the object.
That kind of thing could be a way to turn an already evaluated metadata map into code, but is also a source of bugs and fails if the fn closes over anything
Clojure also has this kind of side table https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/DynamicClassLoader.java#L84 that can be used to pass objects through bytecode, I am not sure if the compiler uses it these days
Thanks! I think this can be avoided, on the jank side, by relying on meta from the AST expression directly, rather than the var. That way the var can have the evaluated meta and the AST expression will still have the original, unevaluated meta. This will allow for codegen from the AST expression without #object shenanigans.