Fork me on GitHub
#cljs-dev
<
2017-02-08
>
mfikes18:02:32

There’s some evidence that self-hosted compiler perf has eroded significantly over time. One measure (perhaps a little inaccurate owing to adding a few new test namespaces over time), is how long it takes to run script/test-self-parity. Here are some timings over time:

2/6      HEAD 2m47.490s
 1/27   1.9.456 2m06.745s
12/16 CLJS-1873 2m13.174s
10/19   1.9.293 1m53.280s
 10/8 CLJS-1815 1m56.592s
  9/5 CLJS-1773 1m41.529s
 8/19 CLJS-1760 1m36.202s
 4/24    1.8.51 1m16.113s
There is perhaps a decent amount of stuff outside of direct self-hosted compilation going on in the above. Another measure is how long it takes Planck to load the cljs.core.async namespace (which I suspect is largely up to ClojureScript—not much Planck itself is doing once the require has been issued), and this takes 3x longer when comparing Planck 1.17 (ClojureScript 1.9.229) and Planck 2.0.0 (ClojureScript 1.9.468). I think I’ll try to produce a more fine-grained timeline for script/test-self-host in order to hopefully find the handful of commits that hurt perf the most.

dnolen18:02:08

@mfikes that would be useful, likely some of these things are simple

anmonteiro18:02:53

Some of those test timings are also related to us including more test namespaces in self-parity

anmonteiro18:02:03

we weren't testing all of them until the end of summer

anmonteiro18:02:36

but there might also be something going on with .cljc files

anmonteiro18:02:48

^ this is a thing, btw /cc @mfikes

dnolen18:02:32

yes, but we definitely need to look at serial compile perf too

dnolen18:02:40

there’s probably a lot of gains if we spent some time on it

anmonteiro18:02:32

definitely. I'm not disregarding Mike's observations

anmonteiro18:02:46

just introducing some more variables for context 🙂

mfikes18:02:22

@anmonteiro Here is another bit of corroborating evidence, comparing the perf of Lumo 1.0.0 vs. 1.1.0:

time lumo -c andare-0.4.0.jar -e'(ns foo.core (:require cljs.core.async))’
With 1.0.0: 0m40.883s With 1.1.0: 1m45.191s

anmonteiro18:02:45

definitely slower 🙂

anmonteiro18:02:49

I wonder what we messed up

mfikes18:02:01

Yeah. My hunch is that there is no single commit that explains all of it.

anmonteiro18:02:03

did you bisect?

anmonteiro18:02:11

there could be one

mfikes19:02:54

There might be a single commit recently that does account for most of it; I agree. But we probably also have a bit of lingchi going on.

thheller19:02:49

one thing that comes to mind is that the externs inference always runs, although that is by no means that much work

anmonteiro19:02:56

@mfikes something came to mind

thheller19:02:25

@anmonteiro unlikely given that there are very few ns forms

anmonteiro19:02:50

right. I thought about that right after seeing the changeset 🙂

anmonteiro19:02:06

nothing else comes to mind though

anmonteiro19:02:33

there should at least be one commit that causes 80/90% of the perf degradation

anmonteiro19:02:37

^ that's my hunch at least

mfikes19:02:14

I’m wondering how hard it might be to write a script that marches through the change sets, running script/test-self-parity and recording the time taken (if the script runs for a given commit), and then making a chart to look for jumps in that chart. I bet that could be done with a few hours work, and if successful, it would be a very useful tool. (The thing that gets executed could be anything for each commit, even the regular tests.)

mfikes19:02:40

I might work on that, presuming that it would pay off for the work put into it.

dnolen19:02:25

@mfikes that would be a useful tool

mfikes19:02:38

Some jumps would be easily explained by adding new test namespaces to script/test-self-parity but others might visually jump out and not be explainable other than a perf regression for a commit.

favila19:02:46

(for clojure)

mfikes19:02:10

Yeah, we need to grow, steal, borrow from existing work in the area 🙂

mfikes19:02:43

Rohit did some work in that area, IIRC.

rohit19:02:36

@favila: I really need to pick up on that thread.

rohit19:02:42

I do have a proper setup.

rohit19:02:15

Need to do something about it!

mfikes21:02:48

I’m running a simple script across all the commits since 1.8.51. It should take less than 24 hours. 🙂 https://gist.github.com/mfikes/2db7e33c6494e6b0950b02e7dc5b0df6

thheller21:02:21

that poor machine it is running on 🙂

darwin21:02:42

btw. git bisect can work with a script, if you have consistent threshold number for your machine, you could get faster to the exact commit

mfikes22:02:12

Ahh cool. The real issue here is that various commits over time have regressed perf. (There is no binary good/bad.)