Fork me on GitHub
#clojure
<
2020-01-18
>
Ivan Koz05:01:02

do we have any in-depth material on clojure compiler and evaluation process?

Matti Uusitalo07:01:17

The source code is on github

seancorfield06:01:42

@nxtk What sort of things are you looking for? Most of the books explain it at a high-level.

seancorfield06:01:35

Each form is read and compiled to bytecode (functions become classes with static invoke methods; namespaces become classes with static initializers). When a program starts up, main is run (either clojure.main or a designated, compiled "main" class -- namespace with a -main function), and as each namespace is required, it is loaded (and compiled if not already) and then the namespace's (class's) static initializer is run. Then functions are invoked.

seancorfield06:01:28

The process is essentially the same for the REPL, reading each form and compiling it and then executing the bytecode.

Amit Rathore06:01:41

hello world!

8
Ivan Koz06:01:23

@seancorfield exactly that, but more and in-depth maybe there is blogs about compiler implementation and design choices or something like that

Ivan Koz06:01:56

i just want to understand compiler more, reading java sources is not enough, because i don't see the full picture

seancorfield06:01:29

I guess I'm not sure what "big picture" there is... Rich has talked about some of his choices and his overall goals in conference talks -- which are all online, many with transcripts.

Ivan Koz06:01:10

alright may be i'm looking for non existing thing, and just need more time

seancorfield06:01:22

The compiler is fairly simple really -- since Clojure as a language is pretty simple.

seancorfield06:01:01

But maybe that's because I come from a compiler background... I co-wrote one of the first ANSI-validated C compilers and I wrote a C++ compiler front end. Clojure is a much simpler language (thank goodness!).

Ivan Koz06:01:42

@seancorfield most likely i just need a crash course into compilers, any books you can recommend maybe?

Ivan Koz06:01:34

i'm doing sicp and mit 6.001 currently

seancorfield06:01:44

The "dragon" book is the classic work on that, by Aho and Ullman, as I recall.

4
Ivan Koz06:01:43

yes i found it, thank you

seancorfield06:01:51

You might also find this interesting https://en.m.wikipedia.org/wiki/Lispkit_Lisp -- see also the FP book linked in the page by Henderson. It's a good read.

seancorfield06:01:17

Those two works by Henderson were the inspiration for my PhD work back in the mid-'80s (the design and implementation of functional programming languages).

lispyclouds06:01:21

@nxtk when talking about implementing lisps, https://github.com/kanaka/mal is amazing too if you already havent checked it out 🙂

Ivan Koz06:01:43

i did actually, great stuff ty

Bobby Towers10:01:19

I'm involved with a project building an online coding platform for learning Clojure. One thing we need to do is represent their solution as an AST for it to be analyzed for possible automated suggestions. As we are just beginning the design, I'm looking for suggestions for "inspiration".

vemv10:01:07

depends on whether the suggestion is stylistic ("prefer if-not seq to if not-empty") or if it goes beyond that, e.g. a suggestion related to getting an algorithm right

vemv10:01:01

don't know for the latter; for the former you can use kondo or eastwood using eastwood repeatedly is not a particularly explored area, but one can certainly make it work

Bobby Towers10:01:00

well I imagine to be really complete it would include all syntactical information, like formatting

Bobby Towers10:01:32

yeah I was thinking of starting with clj-kondo

👍 4
dominicm06:01:51

You probably want to look at term rewriting and origin tracking

Bobby Towers06:01:13

Like meander? That would be fun, I've been looking for some excuse to try that

Bobby Towers06:01:43

clj-kondo led me to rewrite-clj and rewrite-cljs

p-himik11:01:59

A question about reflection. From a comment on https://clojure.atlassian.net/browse/CLJ-2400 by Alex Miller:

My guess is that reflection is picking an unintended override, probably because it's choosing based on some arbitrary ordering that happens to be different than other jdks.

As with any reflective case, you can stop getting arbitrary results by providing type hints on a and b
[...]
I don't think there's really anything to do here, and I don't think it's actually specific to J9.
Does it mean that when I have to interop with Java, I have to avoid reflection as much as possible? Because otherwise, it seems that it can be an undefined behavior given that just switching a JDK leads to different results.

vemv12:01:52

I had it in my wishlist to run all my builds (for context: ~50 libs/projects/...) against a grid of JVMs and also clj compiler flags (e.g. direct linking, no read-eval, elide-meta...) i.e. if a given test suite only passes against a very basic combination (jdk 8, no flags) that would probably indicate a problem

p-himik12:01:35

That would definitely be interesting to see.

Alex Miller (Clojure team)12:01:01

in practice, this kind of ambiguity is rare in my experience. we do run the Clojure test suite in a matrix build against different jdk versions and vendors, and this is typically not a problem

Alex Miller (Clojure team)12:01:10

J9 is particularly different than the openjdk-based family (which is what most people use)

p-himik12:01:10

Is it different in a way that could affect more Clojure programs than a switch to a different JDK with the OpenJDK family? I'm guessing the answer is "yes". Right now, I'm trying to understand why running shadow-cljs (which doesn't really use CLJ magic AFAIK) on OpenJ9 intermittently leads to errors caused by improperly used KeywordLookupSite or its anonymous inner class. Sometimes it's used as a number, sometimes as a seq, sometimes something else.

Alex Miller (Clojure team)15:01:08

it's definitely more different than the OpenJDK variants

Alex Miller (Clojure team)15:01:31

since they all share a code base

pez13:01:05

Is there a way I can hide some arities from the public interface of a function?

yuhan13:01:34

Like overriding the :arglists metadata on the var?

👍 8
Ivan Koz13:01:13

do we know how :arglists is used by runtime?

yuhan13:01:49

(defn foo
  {:arglists '([a] [a b] [a b & cs])}
  [& args])

pez13:01:10

Use case: I often find I create an arity that I use for recursion, but that does not make sense at the outer calling level… I’ll try if that :arglists thing works for me…

Ivan Koz13:01:53

often you can define recursion as a local bound step function

pez13:01:22

Yay, works! Like so:

(defn wheres-waldo-bonus
  "Find the path to waldo"
  {:arglists '([waldo vektor])
   :test (fn []
           (is (= [7 1 0]
                  (wheres-waldo-bonus :W
                                      [[:A :B :C]
                                       [:D :E :F]
                                       [:G :H :I]
                                       [:J :K :L]
                                       [:M :N :O]
                                       [:P :Q :R]
                                       [:S :T :U]
                                       [:V [:W] :X]
                                       [:Y :and :Z]])))
           (is (= nil (wheres-waldo-bonus :W
                                          [[:A :B :C]])))
           (is (= [1]
                  (wheres-waldo-bonus :W
                                      [:V :W :X])))
           (is (= [1 2 1]
                  (wheres-waldo-bonus [:i :am :waldo]
                                      [[:a :b "ceee"]
                                       [[:i :am :not :waldo] [:i :am :spartacus] [:hello [:i :am :waldo] [:who :are :you?]]]
                                       ["d" "e" 6]]))))}
  ([waldo vektor]
   (wheres-waldo-bonus waldo vektor []))
  ([waldo vektor path]
   (when (vector? vektor)
     (let [i (.indexOf vektor waldo)]
       (if (> i -1)
         (conj path i)
         (->> (map #(wheres-waldo-bonus waldo %1 (conj path %2)) vektor (range))
              (remove nil?)
              first))))))
(Which is a solution to a challenge in the latest Purely Fubctional new letter.)

pez13:01:16

Any feedback on this is very welcome. The challenge was about finding the path to a value in an arbitrary nested vector such that the path could be used with get-in.

pez13:01:21

Curious about that local bound step function, @nxtk. Mayvbe it is cleaner than dabbling with arglists.

Ivan Koz13:01:05

(let [step (fn step ... ] ...)

yuhan13:01:55

same here, I usually call it "helper" if it isn't necessarily a step-wise recursion

Ivan Koz13:01:16

yeah it's helper function in general

pez13:01:32

What’s a step-wise recursion?

yuhan13:01:33

hmm just an informal term, most often I see "step" used to mean functions from state->state

❤️ 4
yuhan13:01:02

it also looks like the waldo argument is just being passed around unchanged, so that could simply be closed over in a local helper function

yuhan13:01:49

(defn wheres-waldo-bonus
  "Find the path to waldo"
  [waldo vektor]
  (let [helper
        (fn helper [vektor path]
          (when (vector? vektor)
            (let [i (.indexOf vektor waldo)]
              (if (> i -1)
                (conj path i)
                (->> (map #(helper %1 (conj path %2)) vektor (range))
                  (remove nil?)
                  first)))))]
    (helper vektor [])))

metal 8
pez13:01:12

This is using a private function so losing the closing of waldo:

(defn- wheres-waldo-helper
  [waldo vektor path]
  (when (vector? vektor)
    (let [i (.indexOf vektor waldo)]
      (if (> i -1)
        (conj path i)
        (->> (map #(wheres-waldo-helper waldo %1 (conj path %2)) vektor (range))
             (remove nil?)
             first)))))

(defn wheres-waldo-bonus
  "Find the path to waldo"
  [waldo vektor]
  (wheres-waldo-helper waldo vektor []))
I feel like I should be able to separate out the step of adding one more waypoint to the path, but right now the how eludes me…

p-himik13:01:19

Usually I see just two functions. One is a public one and the other is a private one. It could also be argued that it's a more obvious way to convey the intention, and a bit more DRY (after all, you have to repeat the arguments in your solution).

pez13:01:17

and arglists make me need to repeat it once again…

p-himik13:01:38

Yes, that's what I meant. :) If you want to have two functions/signatures, you're bound to have some repetition. But with :arglists, there's one more.

pez13:01:12

Gotcha, I’ll see what it’ll look like with doing the work in a private function instead. Feels like that will open up an extra level of unit testing…

p-himik13:01:44

Not really - you don't need to test the private function. It's just a matter of organizing the code - all variants (`:arglists`, a local binding, an extra function) should behave identically unless there's some issue with Clojure itself. (Or unless I don't know something - always a possibility).

pez14:01:46

I don’t think I have to test each provate function. It’s just that… It might be a mirage, but I “see” a step where each waypoint is added to the path, and that it could be tested separately from the outer “can I find waldo” task. But right now the outer function just dispatches the whole task to the private one, so there is no point in adding separate tests.

Jacob Emcken18:01:13

Anyone have an idea why the namespace i defined as :aot would compile twice when running lein uberjar?

bibiki18:01:51

in cases like these, first thing I do is question assumptions: how do you know it is compiling twice?

Jacob Emcken18:01:35

I guess it don't but it writes it twice to stdout like so:

Compiling my-ns.core
Compiling my-ns.core

bibiki18:01:56

right, that could be it

Jacob Emcken18:01:26

that could be what?

bibiki18:01:51

that it is not compiling twice, but for some reason you see on your terminal that string twice

Jacob Emcken18:01:07

that is just going to change my question 🙂

Jacob Emcken18:01:25

My other projects doesn't... so what triggers it to write it twice?

bibiki18:01:26

yeah, I have no answer to that...

Jacob Emcken18:01:56

also it takes equal amount of time between the print lines on stdout. So it seems to do an equal amount of work behind the scenes.

bibiki18:01:57

but, do you see anything between the two lines? what do you see after the second line? is there any other line that shows twice? is there a pattern?

Jacob Emcken18:01:08

nothing between the 2 lines no 😞

bibiki18:01:29

well, if it compiled twice you should see same thing between two lines as after the second line, no?

bibiki18:01:22

let me lein uberjar in one of my projects to see what I get

Jacob Emcken18:01:53

Compiling my-ns.core
Compiling my-ns.core
Created /home/je/Projects/my-project/target/my-project-1.0.0-SNAPSHOT.jar
Created /home/je/Projects/my-project/target/my-project-1.0.0-SNAPSHOT-standalone.jar

bibiki18:01:41

now it looks like you really have two builds going lol

Jacob Emcken18:01:37

one jar is the standalone and the other one is just the slim one... but usually it looks like so:

Compiling my-ns.core
Created /home/je/Projects/my-project/target/my-project-1.0.0-SNAPSHOT.jar
Created /home/je/Projects/my-project/target/my-project-1.0.0-SNAPSHOT-standalone.jar

Jacob Emcken18:01:08

it shouldn't need to compile twice... that jar stuff is just packaging

bibiki18:01:47

no actually hold on

bibiki18:01:04

those are two different jars, not same jar twice

bibiki18:01:34

so, you only have one build, not two... but why you get that line twice... hmmm.

Jacob Emcken18:01:14

yes two different jars

bibiki18:01:49

I did lein uberjar in my project and I got a "Compiling ns-name-here" for each clj file anywhere inside my src or its children dreictories

Jacob Emcken18:01:28

you probably have :aot :all in your project.clj

Jacob Emcken18:01:50

you can choose which namespaces you want to ahead-of-time compile.

Jacob Emcken18:01:45

I did it with :aot [my-ns.core]

bibiki18:01:54

yes, I do have that

bibiki18:01:18

let me put aut [some name space] and see if I get it twice

bibiki18:01:10

no, I only have one "Compiling ..." line in this case. hhmmm

Jacob Emcken18:01:32

I created a new empty project myself... and I'm seeing weird behavior here as well... though totally different 😬

Jacob Emcken18:01:07

this is going to be a fun night... I'll see if I can reproduce in a more simple setting and get back

Jacob Emcken18:01:36

I just hoped that someone had experienced it before and knew what weirdness could trigger such

Jacob Emcken18:01:40

anyways thanks so far

bibiki18:01:55

yeah, I am trying that to. I googled it and found some github issue so I am trying based on that. If I reproduce, will let you know

bibiki18:01:59

and actually, I just did

bibiki18:01:10

give me a sec

bibiki18:01:20

check what your :main inside project.clj points at and compare to your other project

bibiki18:01:14

when I removed ^:skip-aot from :main I got "Compiling namespace core" twice

bibiki18:01:49

and here is the github issue that pointed the way https://github.com/technomancy/leiningen/issues/2521

Jacob Emcken18:01:55

thanks a lot... it sounds like the bug I caught 😉

bibiki18:01:49

👍 yeah thank you, too. I am new to clojure and it's tools so I stick my nose in other people's issues to prompt me to learn. I hope I was not too annoying

Jacob Emcken18:01:08

nope not at all

sogaiu18:01:41

@nxtk i was also interested in the clojure compiling topic recently and in addition to what @rahul080327 mentioned, suggest at least checking out (if you haven't already done so): * @gfredericks' "what are all these class files even about?" conj talk: https://www.youtube.com/watch?v=-Qm09YiUHTs wonderful slides (e.g. see at around 11:50 and other places for a nice diagram of some of the relevant functions and code paths), though i haven't found those online anywhere. * @guilespi's http://blog.guillermowinkler.com/blog/2014/04/13/decompiling-clojure-i/ (and 2 more parts) also covers some similar material. nice to be able to compare.

sogaiu18:01:01

tyvm for the slides and the talk!

👍 8
seancorfield18:01:49

@sogaiu Ah, good memory! Yes, Gary's talk is good material! Now you've mentioned that, I seem to recall @ghadi also did a Conj talk about Clojure internals a while back...?

seancorfield19:01:30

Hmm, I think I was remembering this one from Conj '14 https://www.youtube.com/watch?v=S1mzUi_zbEs

sogaiu20:01:17

@seancorfield thanks for pointing out that talk. it seems the first 9 minutes or so has some coverage of what a typical clojure function turns into as jvm bytecode with some accompanying description. the rest of the talk (invokedynamic and truffle) looks interesting too -- should make a note to come back to it 🙂

thumbnail22:01:44

Hi 👋:skin-tone-2: I’m trying to read a ns declaration including metadata like row/column info. is there any clojure.tools.* lib i can use?

borkdude22:01:26

@jeroen.dejong You could try https://github.com/clojure/tools.namespace and/or clojure.tools.reader for this I guess. I'm not sure about preservation of row/column. If that doesn't work: https://github.com/borkdude/edamame is something I wrote for interpreting clojure code.

thumbnail22:01:16

Cool :thumbsup::skin-tone-2: was playing around with tools.namespace but the metadata isn’t present (albeit there’s some references in the source) Will give edamame a try, seems to fit 🙂