macros take an implicit &env parameter

Thanks for answering all my questions! Although I should have watched your talk on tools.decompiler (which was great btw) first and then could have answered the part about that myself 😛

Since they meet in a similar AST form I should try and just work with that unless decompilation is absolutely necessary. I'm not sure what you mean about &env, though. Are you saying if I call jvm/analyze on one I'll be able to see it? The :env key in ASTs isn't quite what I'm looking for.

user=> (defmacro foo [] (list 'quote (keys &env)))
#'user/foo
user=> (foo)
nil
user=> (let [a 1] (foo))
(a)

Oh huh. That seems pretty much like a local version of let/cc in Scheme.

in forms macros expanded through t.a.jvm, &env is bound to the :env value of the analyzer at the point of macroexpansion, in forms macroexpanded through clojure it's a map of local symbol -> clojure.lang.Compiler$LocalBinding nodes for that local

no, let/cc captures the current continuation, &env is just a mapping of the compiler lexical state

I think that's what I meant by "a local version." &env is always delimited by the scope of the macro whereas let/cc is global.

I think my confusion is how :env differs from recursively applying mapcat to all values in :params below the top-level and scraping the bodies for occurrences of them (since they can occur in nested scope). It seems like there would be an AST key that specifically tags bound variables, though. Am I just missing it?

I'm not sure I understood your question

I'm not interested in this as a calling convention, but from the perspective of capturing the structure of nested locally bound variables. This actually relates exactly to our discussion about nested #(..) the other day.

I'm assuming now I'm discounting lets but otherwise would like to just have a map that captures the structure of where function arguments (both in a top-level defn and any functions inside it) are bound to where they're used.

are you looking for :binding nodes?

I'm wondering whether there are AST keys for where parameters are used that I'm just not noticing or whether I'd have to filter the bodies of all children for them. Or alternately, whether I could obtain information like this by actually running a function. I think it's my line of thought surrounding this last possibility that's confusing everything.

references to local variables are :op :local

I still don't understand what you are asking for, sorry, if you give me an example of an input and what the output you want should be maybe I can help better

To clarify first, any values keyed with :op :local should correspond to essentially arguments in things that desugar to fns (I'm actually unsure whether that's the case for let in Clojure, but would assume those are included as locals as well)?

fn, let, letfn, deftype/reify arguments, loop, catch

any special form that introduces a binding, that binding is keyed by :op :binding, any time a reference to that binding is made, that is keyed by :op :local

so in (let [a 1] a), the first a is a :binding, the second is a :local

That sounds like everything I should need.

The example I was going to use is something like:

(defn succ   [n] (fn [f] (fn [x] (f ((n f) x)))))
=> {:n {:f {:x [(f ((n f) x))]}}}

the :env val holds all the local variables that are available at this point + metadata about the node

Oh, okay. So my confusion was looking at :env in reverse order. I should have been going from the inner most scope out.

(wrt to the above example, I'm not fixed on maintaining the structure of locals as I demonstrated there. it was just a convenient example, consisting only of locals)

This is a case where, since the above wasn't immediately obvious to me in the AST structure, I explored different options to try and get at that information and they were so perverse I wanted to continue down that road. So I found a well-documented CPS library, although the implementation is rather naive and it also doesn't support delimiting them other than at the boundary of all functions you don't CPS, then got into running no.disassemble on the continuations themselves...which obviously were so long as to overflow an emacs buffer so then I started parsing the strings to examine the stacks. I'm not even sure what the locals in the all the different stacks correspond to, though, since they didn't seem to be affected by what functions I was CPSing.

"disassembling" a continuation won't do what you think, what you were doing was disassembling the code that compiled to the function the continuation is an instance of

you won't have access to locals by looking at the static bytecode for a funcntion

let me know if there's some better documentation that would've helped you figure this out earlier

One part was what you just said about "disassembling the code that compiled to the function the continuation is an instance of." I ended up with things like this: https://gist.github.com/Sophia-Gold/c11b4b942f6981849c57fbb635696339

The downside of not being forced to take that ludicrous approach is I wanted to title a blog post on this "Decompiling with Continuations"

Although, I may try out both approaches to my initial problem: one top down where I recursively filter for :op :binding and :op :local at each level and get back a nice nested map, and one bottom up where I just have the :env values from the leaves.

>I ended up with things like this yeah so in clojure every function is compiled to a class and an instance of that class is made for the function

if you're decompiling the class you're not seeing any property about the runtime object of that function

like the closure objects

Yup, I realized it was a Clojure vs. Scheme issue in my expectations and had a hunch about what you just told me.

nah it's not a clojure vs scheme, it's that that you were asking a completely different question to clojure

if you disassemble a function in scheme you'll get the same result, altho in some different representation, it will be the bytecode/machine code for that function

hopefully this cleared up a bit of the confusion

Well, first of all I just wanted to comment on something previously.

In the latter approach I mentioned above, I'd have to reconstruct the binding sites from the :env :locals {}, right? This segues into answering your actual question about documentation: there's a ton on types of nodes, but only short verbal descriptions of the keys themselves so I have to infer a lot from the soft docs, i.e. "example usage."

Just since you asked: relatively the tools.analyzer docs are quite good.

>I'd have to reconstruct the binding sites from the :env :locals {}, right yes

ok, I'll try to write a bit more of a tutorial-like introduction when I have some time

Got it. I was unclear whether that information would be preserved in the keys to the hash-map somehow.

I did give a talk a few years ago about some of the main ideas in tools.analyzer if you're interested

and tim baldridge gave one the year before that is an interactive kind of walk through on how to implement transformation passes with t.a

Yes, I should probably dig into that. Tbh, I think I was turned off at first by it essentially breaking homoiconicity. That's really a property of Clojure, though.

it doesn't break homoiconicity

homoiconicity doesn't mean that the shape of the code is exactly the shape of the AST

I just mean that the ASTs aren't simply the s-expressions in user land.

they almost never are, even in a lot of schemes

e.g. in the nanopass framework the AST is a record hidden behind a syntax-object

Right. Basically what you said in your talk on tools.decompiler about needing to work at the AST level to implement any interesting transformations. That's true in other lisps as well.

right

I did watch Tim's talk quite a while ago and I think I conflated a lot of his use case with simpler ways I could use those ASTs. A lot of it is probably writing functions to reduce the verbosity to just what I care about for different purposes.

Anyway, so just two more things. First, regarding my adventures in CPSing Clojure functions: I'm still unclear what you meant by, "if you're decompiling the class you're not seeing any property about the runtime object of that function."

brb

np

right, so

when you decompile, say (let [a 1] (fn [] a))

that (fn [] a) needs to be first compiled into executable code

then instantiated for a=1

when you decompile it, you get the machine/bytecode for the (fn [] a) function

but you don't get the closure associated with the intance with a=1 or its stack

is this clearer?

behind the scenes of (let [a 1] (fn [] a)) you can imagine it being transformed into something like

(deftype my_fn_123 [a] clojure.lang.IFn (invoke [] a))
(let [a 1] (my_fn_123. a))

and you're decompiling the my_fn_123 type/class, not anything about any of its instances, like (my_fn_123. a)

there are ofc horribly broken or hackish ways of getting at the stack or the closure object (or if you have primitive support for continuations you don't need any of that), but decompilation is not what you're looking for

(csp transform is one way to reify and access the call stack )

Sorry, had to take a call and just reading this

I think I was just confused about compiler passes and what gets compiled to its own class

The reason I was decompiling the cc specifically was because I assumed that would be a "horribly broken or hackish ways of getting at the stack or the closure object." As you mention, cps transform is a way of capturing the call stack but you can't inspect it because it itself is already compiled. I assume the final piece in this is that if I decompile it I'm still only getting at the top-level class, i.e. the implementation of the continuation rather than the actual frame objects. Is that right?

And looking at that decompiled continuation, I assume I would have to decompile several classes down to get to any clojure.lang.Var$Frame object.

>I assume the final piece in this is that if I decompile it I'm still only getting at the top-level class, i.e. the implementation of the continuation rather than the actual frame objects. yes

Ah gotcha. So I do have a third option, which is the truly perverse one: recursively disassembling the continuation until I reach frame objects.

no

the frame objects are not code you can decompile

they are runtime values

Ah ok. They're basically just hooks

So I can use them, but not inspect them

I still wonder if there's a way I could inspect them through instrumentation

I don't want to belabor that, though. The other thing I was going to ask was just about compiling tools.decompiler with javac. Their docs are actually rather sparse. Is there an example project file somewhere?

>compiling tools.decompiler with javac

?

I was under the assumption it has to be AOT compiled using lein-javac

no

javac is for java files

tools.decompiler works on clojure compiled forms

I was referring to this line in your README: "Use lein javac to AOT compile clojure.tools.decompiler.RetrieveClasses then you can use lein repl or clj to launch a repl"

ah, to compile tools.decompiler itself, sure, you literally just need to type lein javac, what's the issue?

i don't need anything in :javac-source-path?

it's already https://github.com/Bronsa/tools.decompiler/blob/master/project.clj#L6

Ok. I think it was actually including that field in my own project.clj that screwed me up

Sorry, I'm multitasking and realize it's evening for you. I do really appreciate your help today. And everything you've done with these libraries is great and right up my alley. I should probably think about contributing in some form eventually.

you're welcome

Let me know if you'd rather me open an issue about this, but I had to revert to 1.8 due to the removal of bigdec? and now for some odd reason this let binding isn't being resolved: https://github.com/Bronsa/tools.decompiler/blob/master/src/clojure/tools/decompiler/bc.clj#L120

that's because of a 1.8 bug

tools.decompiler only supports 1.9

bigdec? is not in 1.8 either

it was just in some alpha versions of 1.9 so I don't know what issue you're having

1.9 is backwards compatible with 1.8

i'm using clojure 1.9 and your version 0.1.0-alpha1. the error message makes it seems like the version on clojars was before this commit: https://github.com/Bronsa/tools.decompiler/commit/9f2c29492ed26182f96b805cde86583c362f6abc#diff-192c7acd38e01ad43d78f10f3b6adc1e

?

i’ve used java executors when i wanted control the max number of threads

and also for monitoring purposes

using https://github.com/ztellman/dirigiste

Not exactly what you're asking, but a transformation like that would be trivial with XSLT. For example:

<xsl:stylesheet version="2.0"
  xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
  xmlns:xs="http://www.w3.org/2001/XMLSchema"
  exclude-result-prefixes="xs">

  <xsl:output method="xml" indent="yes"/>

  <!--
  Copy every input node that doesn't match any other <xsl:template> into the
  output XML tree as is.
  -->
  <xsl:template match="@* | node()">
    <xsl:copy>
      <xsl:apply-templates select="@* | node()"/>
    </xsl:copy>
  </xsl:template>

  <!--
  Increment the value of every <number> element by one.
  -->
  <xsl:template match="number">
    <xsl:copy>
      <xsl:value-of select="xs:int(.) + 1"/>
    </xsl:copy>
  </xsl:template>

</xsl:stylesheet>

Unfortunately the project is limited to using Clojure. Thanks though!

https://github.com/eerohele/sigel 😛

Ohh interesting. Thank you! 🙂

If you want to stick with Clojure, another option would be to use Specter (https://github.com/nathanmarz/specter).

(use 'com.rpl.specter)

(transform (walker number?) inc xml-document)

=>
#xml/element{:tag :parent,
             :content [#xml/element{:tag :number, :content [2]}
                       #xml/element{:tag :child, :content [#xml/element{:tag :number, :content [3]}]}]}

That worked, thank you! 🙏

duh, that’s great! 😅

How did I not know that sort-by has an additional arity 😄

Ah, now I see why you said that. :-)

IIRC, it was because of the packaging and deploy situation. I never worked on JRuby, but I think it wasn't just "bundle install" or "cap deploy". Stuff breaking, and dependencies failing to work on JRuby.

ahh, makes sense. IIRC at my old job we used Traveling Ruby (which is now dead) and locked down our versions pretty hard especially of native gems to work around that

I did manual timing in my code to eliminate JVM startup overhead, but Ghadi pointed out that there is additional overhead from jitting that's hard to nail down. So yeah, that makes sense.

@U1B1E4ZDM Application code and library code will, and should, look very different. My best advice would be to look at the code in web frameworks templates, such as luminus, chesnut, duct etc - they are designed to be exemplary application code.

Agreed. There's a huge difference between the style of code in my favorite libraries and anything you'd want to be writing on a production team. Since you mentioned ClojureScript in particular (and btw there's a channel dedicated to that) I would mention Om.Next as something I think is amazing and relatively few people use in production (Reagent/Re-Frame might be good to look at instead). Another example is Specter, which has quickly become probably my favorite Clojure library. It's by definition a non-idiomatic way to write Clojure and additionally the source is largely macros that have had a ton of performance tuning. That's the kind of stuff you wouldn't want to write in a setting like this.

I'd recommend looking at @U064WBEPL's blog and github for "best practices" type stuff.

Also: https://github.com/bbatsov/clojure-style-guide

@U06GS6P1N Thanks a lot, good point about application vs library style. And I'm a fan of Bhozidar, I'll have a look at that style guide.

@U2TCUSM2R I'll check it out thank you.