Clojurians Log v2

here is an example of the problem I was describing yesterday in de the Clojure compiler itself : https://github.com/clojure/clojure/blob/1e835a9d9fb4498a5cf643df861565c07701b18b/src/jvm/clojure/lang/Compiler.java#L4094-L4097 if you add any extra meta at the reader the compiler will fail in a obscure way unless you remember to add them in this enumerations because there is logic depending on it which is kind of brittle. There is one more here : https://github.com/clojure/clojure/blob/1e835a9d9fb4498a5cf643df861565c07701b18b/src/jvm/clojure/lang/LispReader.java#L1125-L1128 and there is also I think an attempt to deal with this stuff? not sure : https://github.com/clojure/clojure/blob/1e835a9d9fb4498a5cf643df861565c07701b18b/src/jvm/clojure/lang/Compiler.java#L287

I have similar issues in SCI: it should also know when to evaluate metadata vs metadata it added itself when reading. But why is this your problem?

I'm hitting this because I'm working on a dev compiler, which is a patch on top of the Clojure compiler that emits instrumented code. I had to extend the LispReader to on top of line and column also provide a sexp coordinate (coord in the tree), and as soon as I added the extra meta the compiler didn't work anymore

and after fixing those two places I hit it again by breaking now the ClojureScript compiler, all by just adding a extra meta at the reader

does it make sense?

yes. maybe you could use a custom reader :)

wdym?

I mean, I have everything working now, just reporting that this is kind of brittle

ah so you if you add metadata during the read phase, the compiler breaks. yeah, that won't work with a custom reader then either, without changing the compiler

exactly

I think the idea is that compiler works in sync with the reader. making changing in one place will also mean changes in another place. it's kind of expected?

well, IMHO adding somithing as simple as a extra meta key shouldn't break completely unrelated stuff (and hard to figure out) in an open map world system. and second it also broke the ClojureScript compiler and any other project that is removing reader keys to figure out if read returned any meta

can you tell what exactly broke? What I would expect is that the metadata would be evaluated. Maybe adding extra quoting should help, if the expression contains symbols or lists.

you mean what brokes the Clojure compiler or other projects like ClojureScript?

yes, both

for example, if you add a extra key at the reader phase then this goes wrong : https://github.com/clojure/clojure/blob/1e835a9d9fb4498a5cf643df861565c07701b18b/src/jvm/clojure/lang/Compiler.java#L4094-L4097

what do you mean by "goes wrong", what is the error you are seeing?

oh, I'll have to make it happen again, let me see

what I would expect is that the fn has metadata, so it will be evaluated at runtime, but adding extra quoting can prevent this

user=> (meta ^{:x (+ 1 2 3)} (fn []))
{:x 6}

user=> (meta ^{:x x} (fn []))
Syntax error compiling at (REPL:1:7).
Unable to resolve symbol: x in this context

user=> (meta ^{:x 'x} (fn []))
{:x x}

by adding a extra key to the reader I got :

Exception in thread "main" java.lang.ExceptionInInitializerError
	at clojure.main.<clinit>(main.java:20)
Caused by: Syntax error compiling at (clojure/core.clj:340:44).
	at clojure.lang.Compiler.analyze(Compiler.java:6955)
	at clojure.lang.Compiler.analyze(Compiler.java:6892)
	at clojure.lang.Compiler$InvokeExpr.parse(Compiler.java:3894)
	at clojure.lang.Compiler.analyzeSeq(Compiler.java:7284)
	at clojure.lang.Compiler.analyze(Compiler.java:6936)
	at clojure.lang.Compiler.analyze(Compiler.java:6892)
	at clojure.lang.Compiler$InvokeExpr.parse(Compiler.java:3970)
	at clojure.lang.Compiler.analyzeSeq(Compiler.java:7284)
	at clojure.lang.Compiler.analyze(Compiler.java:6936)
	at clojure.lang.Compiler.analyze(Compiler.java:6892)
	at clojure.lang.Compiler$InvokeExpr.parse(Compiler.java:3970)
	at clojure.lang.Compiler.analyzeSeq(Compiler.java:7284)
	at clojure.lang.Compiler.analyze(Compiler.java:6936)
	at clojure.lang.Compiler.analyze(Compiler.java:6892)
	at clojure.lang.Compiler$InvokeExpr.parse(Compiler.java:3970)
	at clojure.lang.Compiler.analyzeSeq(Compiler.java:7284)
	at clojure.lang.Compiler.analyze(Compiler.java:6936)
	at clojure.lang.Compiler.analyze(Compiler.java:6892)
	at clojure.lang.Compiler$InvokeExpr.parse(Compiler.java:3970)
	at clojure.lang.Compiler.analyzeSeq(Compiler.java:7284)
	at clojure.lang.Compiler.analyze(Compiler.java:6936)
	at clojure.lang.Compiler.analyze(Compiler.java:6892)
	at clojure.lang.Compiler$BodyExpr$Parser.parse(Compiler.java:6241)
	at clojure.lang.Compiler$LetExpr$Parser.parse(Compiler.java:6573)
	at clojure.lang.Compiler.analyzeSeq(Compiler.java:7282)
	at clojure.lang.Compiler.analyze(Compiler.java:6936)
	at clojure.lang.Compiler.analyzeSeq(Compiler.java:7270)
	at clojure.lang.Compiler.analyze(Compiler.java:6936)
	at clojure.lang.Compiler.analyze(Compiler.java:6892)
	at clojure.lang.Compiler$BodyExpr$Parser.parse(Compiler.java:6241)
	at clojure.lang.Compiler$FnMethod.parse(Compiler.java:5559)
	at clojure.lang.Compiler$FnExpr.parse(Compiler.java:4111)
	at clojure.lang.Compiler.analyzeSeq(Compiler.java:7280)
	at clojure.lang.Compiler.analyze(Compiler.java:6936)
	at clojure.lang.Compiler.analyzeSeq(Compiler.java:7270)
	at clojure.lang.Compiler.analyze(Compiler.java:6936)
	at clojure.lang.Compiler.access$1(Compiler.java:6895)
	at clojure.lang.Compiler$DefExpr$Parser.parse(Compiler.java:587)
	at clojure.lang.Compiler.analyzeSeq(Compiler.java:7282)
	at clojure.lang.Compiler.analyze(Compiler.java:6936)
	at clojure.lang.Compiler.analyze(Compiler.java:6892)
	at clojure.lang.Compiler.eval(Compiler.java:7364)
	at clojure.lang.Compiler.load(Compiler.java:7826)
	at clojure.lang.RT.loadResourceScript(RT.java:381)
	at clojure.lang.RT.loadResourceScript(RT.java:372)
	at clojure.lang.RT.load(RT.java:459)
	at clojure.lang.RT.load(RT.java:424)
	at clojure.lang.RT.<clinit>(RT.java:338)
	... 1 more
Caused by: java.lang.RuntimeException: No such var: clojure.core/apply
	at clojure.lang.Util.runtimeException(Util.java:221)
	at clojure.lang.Compiler.resolveIn(Compiler.java:7573)
	at clojure.lang.Compiler.resolve(Compiler.java:7543)
	at clojure.lang.Compiler.analyzeSymbol(Compiler.java:7504)
	at clojure.lang.Compiler.analyze(Compiler.java:6915)
	... 48 more

and what does the extra key contain?

the clojurescript issue was even worse. but let me try with a clojure clone from scratch, because I have other stuff modified

hmm I'm not being able to reproduce it now with that simple change, but you get the idea, there is logic on different applications that depend on knowing the exact keys the reader is using, which I think is brittle

I'll not spend too much time on this because it isn't a problem I have now, is just something I noticed and wanted to report here

@jpmonettas yes, but the reason I asked for the exact error/repro is because I think it can be fixed without changing the LispReader/Compiler, by adding quoting. But maybe I'm wrong ;)

not sure I follow, I'm not working with Clojure, everything is in Java, I'm modifying the compiler itself, so I don't see how something like quoting fits. And what I'm talking about here is a more general problem. If a underlying library that provides you data (in this case the reader) is going to add to your data (in this case line, column, etc) if it doesn't provide a way for you to tell apart the original data from the enhanced one and you need to tell it apart ,you will have to enumerate the keys, which is brittle because it will break when more keys are added

don't know if that makes sense, but thanks a lot for your time!

and all the good stuff you write btw!

thanks :) can you give an example of what kind of stuff you are adding on top of the existing metadata?

sure, so it is the forms coordinates, so if you have (+ a b ( 3 c)) the c will have :coord [3 2] while the ( 3 c) list will have :coord [3] etc

alright, so just vectors and numbers, no lists or symbols?

then modifying macroexpansion to preserve that meta, you can link the expanded code to the original one

yeah, just a vector of numbers

ok, then my theory doesn't hold

@jpmonettas FWIW, edamame has a postprocess hook in which you can choose what metadata and location info you can add to the read form. You can also configure the location keys separately. It doesn't add *file* metadata to read forms.

user=> (map meta (e/parse-string "[x y]" {:row-key :foo}))
({:foo 1, :col 2, :end-row 1, :end-col 3} {:foo 1, :col 4, :end-row 1, :end-col 5})
user=> (map meta (e/parse-string "[x y]" {:location? (constantly false)}))
(nil nil)
user=> (e/parse-string "[x y]" {:postprocess (fn [{:keys [loc obj]}] [loc obj])})
[{:row 1, :col 1, :end-row 1, :end-col 6} [[{:row 1, :col 2, :end-row 1, :end-col 3} x] [{:row 1, :col 4, :end-row 1, :end-col 5} y]]]

nice you can configure that there, I guess those are the two options, or provide readers what keys you want in meta, or have a way to ask it to subtract or return the ones it added

but I think it is important to cover that in the main reader so it can grow without breaking other programs

why do you need to subtract the reader's meta?

I don't, but if you look at the ClojureScript compiler code I linked they are doing it, and I guess you do it if you want to check that the forms you are reading comes with any meta, and you are probably not interested in readers meta

in most cases I guess you can go the other way around, and check for the meta you are interested in, but there will be no way of answering "give me all the meta in the form I just read (not counting readers one)"

the forms are coming with meta

"what you are interested in" is entirely contextual

I mean, if I'm reading some forms, it can happen that I want to know if the original data contains any meta (I think that is why projects like the ClojureScript compiler are subtracting readers one)

well I guess you can work around it by not providing a LineNumberingPushbackReader, since it is what it's using to add line and column

I understand where you're coming from, but this is a world of open maps where tools and users are free to add things in the process. Reading is an operation that annotates what is read. If anything, I think an enhancement to add an option to the reader to not annotate would be maybe a possibility, not sure exactly what that would need to cover

yeah, so I'm in full support of open map world, and I just came here to report a issue with this not being the case, because I added a extra key to the LispReader meta and by doing that I broke the ClojureScript compiler

so is your complaint with the Clojure reader or the ClojureScript compiler? If the latter, then #cljs-dev is a better place to discuss

I figured out that is because the ClojureScript compiler is enumerating the readers keys, so I first thought on creating a issue on ClojureScript, but then thought how would they fix it, if the reader doesn't provide a way for you to tell apart the keys it read from the keys it added

does it make sense?

it makes sense, but still unclear what options we could consider

I think for the nicer one its already late, which is maybe namespaced keys. And a quick low risk one could be to maintain that enumeration inside the reader, bu yeah, not easy, since in the example of the ClojureScript compiler they are subtracting other keys like :source, :end-line, :end-column that aren't coming from the LispReader, so don't know 🤷‍♂️

Thanks anyway @alexmiller

FYI when using tools.reader, you can control whether it adds or not the meta by using (or not using) an indexing-reader as the source

@bronsa nice, I guess in that respect is similar to the LispReader.java by using (or not using) a LineNumberingPushbackReader, but more of an accident than a designed feature. And there is still no way of asking for the meta (in case you need it) but also being able to tell it apart from the read data meta, without enumerating it in your source code