Fork me on GitHub
#cljs-dev
<
2015-11-17
>
mfikes02:11:06

Parallel compilation does not result in excessive swap! retries (confirmed with a bit of diagnostic logging https://github.com/mfikes/clojure/commit/39777c472436ae3778668834207e4f2071ff24cf)

mfikes04:11:41

[Parallel compilation] After closely watching correlation of ns compilation times and the YourKit memory view, I’m now fairly convinced that parallel compilation stops scaling as GC starts to dominate. In other words, you start to become ultimately limited by GC bandwidth, no matter how many cores you throw at it.

dnolen05:11:31

@mfikes: and I guess bumping up the max heap doesn’t help?

dnolen05:11:40

like setting it to 4gb or something?

thheller08:11:21

@mfikes did you try switchting to a different GC?

thheller08:11:51

maybe -XX:+UseG1GC works better

thheller08:11:48

doesn't do anything one my machine though

thheller08:11:57

increasing heap doesn't change anything either

mfikes11:11:40

@dnolen, @thheller, right. I set it to 9GB and I also tried G1. G1 made it avoid pauses, but the throughout was a little slower than the default GC. FWIW YourKit wasn't showing "time in GC" like I expected. But I could see pauses when it cleaned up and the graph dropped, and blocks of ns compilations all slowed down together. Even without the pauses, I suspect everything related to memory just has to slow down as a result.

bensu16:11:40

Is there any reason why .clj files are not watched by cljs.closure/watch? https://github.com/clojure/clojurescript/blob/master/src/main/clojure/cljs/closure.clj#L1981

thheller16:11:30

@bensu that would trigger recompiles of cljs when editing clj files

thheller16:11:11

don't think that any clojure changes are actually handled

thheller16:11:50

last time I checked macro reloading wasn't actually a thing a cljs.closure

bensu17:11:17

@thheller: yes, but isn't that what you want? if you change a macro file, you want all the dependents recompiled.

dnolen18:11:23

@bensu hrm I’m not sure that if we did that it would actually work

dnolen18:11:32

not sure if we track clj -> cljs dependencies very well

dnolen18:11:20

@bensu and for that to really work we would need to track clj -> clj deps

bensu18:11:43

@dnolen: you just interrupted yourself (I'm watching today's talk simple_smile )

bensu18:11:25

even if the ground work is not there, the question is it a behavior that we want?

dnolen18:11:50

yeah need to think about it

mfikes23:11:48

Interestingly the curve in https://clojurians.slack.com/archives/cljs-dev/p1447683898000164 matches what you'd get from Amdahl's law if about 15% can't be parallelized.