Clojurians Log v2

Do you mean something that fundamentally maps arbitrary Clojure subforms to the part of the bytecode they compile into?

yes

yeah I figured

i'll have to see how exactly clj-java-decompiler works to see if i can do something similar for each subform

or maybe modify the compiler itself, create a standalone program like flowstorm

feels like that would probably be your best bet, I don't think there's much in the way of inspecting the compiler's behavior in vanilla Clojure

dang actually, i bet flowstorm would be a good place for this

I'd bet that clojure to java decompiler simply compiles into a tempdir and reads the bytecode off of it

or does flowstorm already do this... i'm gonna feel mighty silly if it does lol

all flowstorm does is add instrumentation to the emitted bytecode doesn't it?

I don't think you can by decompiling, because the class file only supports adding "line markers", and not columns, while in Clojure we have a bunch of expressions on the same line. But make something like ClojureStorm generate a source map is maybe possible?

i suspect you could hack the line maker support by multiplying every number by 1000, using the lower 4 digits to store the column number lol

must use Cantor pairs, columns may exceed 1k

I played with that already, but it uses an int if I don't remember wrong and it wasn't enough

of course you did lol

glad i'm asking before spending a bunch of time to retread this ground

But for your idea I guess you don't need to add anything to classes, you could generate a separate source map I guess

Just emit the regular bytecode, and add a tail of trash insns to encode stuff, duh

does bytecode support something like a comment?

What I would try is on Compiler.java, to make sure all ... class XXX implements Expr { has fields for line and column (some of them have I think), then being sure you fill them on the parsing part, and on the emit map the blocks of generated bytecode to those line,cols

@borkdude I don't think so

I assume this is also a problem the Java world has, src <-> bytecode source map-ish stuff?

Looking at the class spec, I guess you could have a custom attribute, and put that mapping there, but not sure of the benefit vs a separate source map file since only your tool will work with it

@borkdude classes have already a https://docs.oracle.com/javase/specs/jvms/se7/html/jvms-4.html#jvms-4.7.12 attribute, so only a way of mapping lines to bytecode, which I guess works for languages like Java

which is probably sufficient?

you can kind of hack stuff with SMAP via the SourceDebugExtension (JSR 45)

although not sure how debuggers like IntelliJ that allows you to step into multiple expressions on the same line works

jruby does some crazy stuff where they insert virtual stack frames to delineate expression boundaries, not sure at what level they do that

> which is probably sufficient? when I looked at it it wasn't, since that line info is an int

right

I guess this SourceDebugExtension (JSR 45) Alex mentioned is probably the way to go, since it is a source map thing, but better because the info there could be leveraged from other tools also

https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/Compiler.java#L4896-L4910 is a use of smap in clojure to attach debug info

nice!

what is the target of that info Alex? What tool is making use of it?

existing java debuggers like in IntelliJ

it's only line level though, I don't think you can get to columns

Perhaps some heuristics can be applied to get the relevant form

you could reformat the code to have one expression per line :)

(I'm
  (sure
    (most people
      (will be 
        (happy with that))))))))))

closing parens can still be on the same line, so it's only half bad

you don't have to show it to people :)

Does anybody knows where this SMAP grammar is specified? Can't find it

do you mean, reformat, then recompile?

SMAP is defined in JSR 45 - you can download the spec from the jsr page

it is somewhat arcane

yeah, found it

but yeah I don't see any column thing, I guess you could store there your own format instead of this SMAP

> do you mean, reformat, then recompile? yeah, reformat, remember a line mapping from old to new, compile, then use the smap info and the reverse line mapping to tie bytecode back to source

yeah SMAP is only valuable because it's embedded in the class and JVM tools understand it. if you're hooking the compiler itself you could be more direct

I guess they added this https://docs.oracle.com/javase/specs/jvms/se7/html/jvms-4.html#jvms-4.7.11 because https://docs.oracle.com/javase/specs/jvms/se7/html/jvms-4.html#jvms-4.7.12 wasn't enough for other languages but they created this SMAP DSL to put there that only support lines 🙄

I think the big advantage is the layers idea

Which no one uses

I mean you could probably treat an intermediate expr per line as a layer?

I guess if your tool would be the only one understanding that then you could just store edn instead of this SMAP string instead

Or because this SMAP is a string, I guess your line numbers there can be bigger? So you could encode both lines and columns into the line fields of a stratum for Clojure ?

nah, it says : >

InputStartLine: NUMBER

well maybe that doesn't mean tools should parse that as ints, so no idea

or use one range of the number for the column?

you could use number = 2^line * 3^column

no those numbers grow too large ;)

darn, reality messes with math

depends on how big "number" is, here. if it's a java int, that gives you 2 billion lol

well if you have line number 1051 and column number 50, it will be a number of 300+ digits

so, scratch that idea ;)

first 32 bits for line, other 32 bits for column, should work

i suggested just multiplying row by 1000 and using the lower 3 digits for column, but jp said he ran into problems trying something like that

oh sorry I missed that

no worries

wait

what about two lines annotations per line+column pair

because it says just Integers there, and it doesn't define any JVM types, not sure if you can break existing tooling by overflowing there. Or by putting a non SMAP string.

When I get some time I'll experiment in ClojureStorm with mapping on the heap while loading and compiling stuff, instead of adding it to the class itself, because that should be enough for a godbolt like tool I guess

that would be........ next level, truly marvelous work

@borkdude haha, I vividly remember suggesting that 2^x * 3^y as a pairing function in a uni class back in the day Funnily enough, the Cantor pairs I mentioned above would kinda suck too: they'd fail as soon as one of the numbers hits about 2^(maxbits/2), in fact slightly earlier than smooshing two maxbits/2 ints as you suggested afterwards, so it'd be much better to assume a sane column length and then encode as (nlines * (maxcol + 1)) + ncols, which is more or less what Noah suggested

Using 3 bits as a "form index" - just identifying which form incrementally - would give 8 forms/line and leave 8K lines

clever

ah yeah, counting nested forms per line would be good

I'm really interested in these techniques for my IDE since I have a lot of control over what gets sent to the compiler and how exceptions, warnings, etc are presented to the user.

Line numbers are already annoying since you can eval a function, add a new function above it, and then the line numbers are already wrong. I like the idea of autoincrementing numeric IDs that I can tie to vars/functions/forms more directly.

i think that's a separate issue/idea than what i was proposing above, tho it would be cool to have in an editor

Can you also stuff info into the filename to improve source mapping?

the discussion here only reminded me of some obfuscators back in the beginning of 2000s that took perfectly normal compiled java classes and "harmed it" just enough to break decompilers but not enough to break the execution in the jvm ... no idea if these things are still around

and when your eclipse or netbeans debugger stumbled on those then you better had your fingers crossed

I think they are still around - stuff like yGuard

@nbtheduke I gave this a try in the latest FlowStorm https://clojurians.slack.com/archives/C06MAR553/p1784301480567749

oh my god, i can't wait to try this

feedback is welcome!