Fork me on GitHub
#boot
<
2015-08-30
>
juhoteperi09:08:41

@micha: btw. there is work going on to decouple Cider from nREPL so that it can work with Clj 1.8 socket/network repl

micha20:08:48

"Explanation?"

micha20:08:13

boot sucks is the explanation, and we suck

micha20:08:20

individually and collectively

martinklepsch20:08:25

There’s a character limit, didn’t read it as very rude myself

micha20:08:06

that's why i don't twitter simple_smile

martinklepsch20:08:16

And let’s not assume bad intentions from those couple of words simple_smile

micha20:08:38

haha yeah, i'm not being 100% serious

micha20:08:12

but with boot it's been a constant demand to explain its existence

micha20:08:32

"another build tool. explanation? kill yourself"

micha20:08:37

basically

micha20:08:20

fwiw i think backing out the directories change will make it fast again

micha20:08:27

it was compiling everything every time

juhoteperi20:08:53

I guess we could revert that change now

juhoteperi20:08:12

As it looks like it will be fixed on Cljs side

juhoteperi20:08:21

Though I don't know how long that will take

micha20:08:45

we could have a "hacks" option

micha20:08:03

boot cljs --hacks test-refresh ...

micha20:08:16

like feature flags for pending upstream workarounds

juhoteperi20:08:19

The issue is here: http://dev.clojure.org/jira/browse/CLJS-1437 recompile-dependents looks to be quite broken even with multiple directories

martinklepsch20:08:39

How about we push a version of that and ask him to try again/report back

micha20:08:19

i can test here if you push to master

micha20:08:30

if you want some extra eyes on it

micha20:08:53

or to a branch, whatever

juhoteperi20:08:37

Shouldn't affect advanced build?

micha20:08:19

well not after i made my patch

micha20:08:30

remember the patch that special cased :none

micha20:08:37

i think that was after the tweet

juhoteperi20:08:38

Yeah that's what I mean

micha20:08:53

yeah advanced should be identical

juhoteperi20:08:11

Not sure, I pushed -3 24th and tweet is from 25th

micha20:08:41

as the doctor says in the simpsons... "you may have two things"

micha20:08:02

so maybe there is also another issue

juhoteperi20:08:17

25s -> 21s :none build time after removing the other dirs from build call

micha20:08:06

what about incremental?

micha20:08:14

incremental should be fixed for sure

micha20:08:24

also what's your setup?

juhoteperi20:08:27

I have try with larger project

juhoteperi20:08:43

I strill used old boot-cljs

micha20:08:18

lol the suspense is building

juhoteperi20:08:25

well, 20s with proper version

micha20:08:13

lein-cljsbuild uses cached stuff

juhoteperi20:08:40

the tweet mentioned cold build, e.g. cleaned target dir

martinklepsch20:08:30

what’s the difference for incremental compiles?

juhoteperi20:08:05

Those times are quite useless without knowning how they are measured etc.

micha20:08:36

i am pretty sure it has nothing to do with filesets

juhoteperi20:08:37

Passing only the one dir to build has no affect on incremental compile times

juhoteperi20:08:01

But I'll try with larger project

micha20:08:04

manipulating filesets is way faster than anything else that's happening in there

juhoteperi20:08:21

I also have lein setup for that project so I can see the differences

martinklepsch20:08:40

@micha: but where does it come from then?

micha20:08:51

the cljs compiler

micha20:08:58

we triggered something deep inside

micha20:08:05

and it has to think about it

micha20:08:27

maybe something is having it's last modified time updated or something like that

micha20:08:00

you can test by just commenting out the call to build

juhoteperi20:08:02

Well in fact boot is faster for me... at least in one case

juhoteperi20:08:20

Though that's probably only because leiningen startup is really slow

juhoteperi21:08:47

No much dfferences other than for incremental recompile for me

micha21:08:22

not much difference boot vs lein?

juhoteperi21:08:25

And the difference there is probably mostly caused by measurement differences

micha21:08:27

or -3 vs HEAD?

martinklepsch21:08:31

probably JVM parameters that are passed to boot and lein are something that should be listed in such comparisons

juhoteperi21:08:31

Luckily Cljs has :compiler-stats which can be used to display how long compilation takes, that should offer uniform measurement

micha21:08:46

a couple of hundred millisec for fileset manipulation is expected

micha21:08:54

and various boot overhead

juhoteperi21:08:05

I don't see differences in none compile times with or without additional directories

micha21:08:40

you have additional source files in the additional directories?

micha21:08:52

like ones that aren't :require'd by the program you're compiling

micha21:08:23

it was definitely compiling things it shouldn't have been compiling when directories were being added

juhoteperi21:08:03

https://github.com/adzerk-oss/boot-cljs/issues/93 added compiler measurements to incremental compile logs, show there's no differenence in cljs compiler times between two

micha21:08:25

why do you say no difference? i must be reading it wrong?

micha21:08:53

boot incremental compiles look like ~2x longer elapsed time

juhoteperi21:08:54

Well it's pretty close to 300ms for each recompile

micha21:08:13

vs like 150ms for cljsbuild

micha21:08:32

the add dependencies log lines

juhoteperi21:08:32

To compare time taken only by cljs compiler, sum basic and add dependencies times

juhoteperi21:08:08

for boot basic sources takes only 5ms because our only file in input folders is the main ns and all others are compiled in dependencies step

juhoteperi21:08:04

And basic sources in lein includes all cljs files in source-paths

juhoteperi21:08:54

The difference between time used by cljs compiler and boot elapsed time is probably overhead from boot

micha21:08:17

ah i see now

micha21:08:06

and this is adding all directories in the call to build, this benchmark you just did?

juhoteperi21:08:31

only the main ns dir

micha21:08:11

pretty sweet writeup btw :thumbsup:

micha21:08:18

actual numbers otg

juhoteperi21:08:24

But that didn't have difference in my case because my project doesn't have additional files

juhoteperi21:08:08

But looks like boot overhead for recompiles is relatively large

juhoteperi21:08:32

I wrapped build in time and it definitely shows it's not cljs compiler which is taking the time 😞

micha21:08:29

yeah 300ms is boot, more or less

micha21:08:46

we could shave that, for sure

micha21:08:56

like we could write to the target dir asynchronously

micha21:08:07

or not write there at all while boot's running

micha21:08:29

just batch filesets and write only the last one before the pipeline ends

micha21:08:00

i bet most of the boot overhead is writing to target

juhoteperi21:08:07

or maybe just provide option to disable target dir completely, no need when using watch and serving from classpath etc.

micha21:08:08

becauseit has to actually write files there

micha21:08:14

right yeah

micha21:08:20

i like that for the default

juhoteperi21:08:32

Or even doing that as default and then adding option to enable target dir

micha21:08:33

writing to the target dir is silly while boot's running

micha21:08:50

i think we don't want to support that even with an option

juhoteperi21:08:52

Because it looks like target dir is confusing many people

micha21:08:01

you can always write a task to do it if you really need to

micha21:08:27

i think it just came over from boot1 when we didn't have filesets

micha21:08:34

i'll push to a branch, one sec

alandipert21:08:40

separating target into a task, love this concept

juhoteperi21:08:07

Makes sense, the only time when target is needed is when build a jar

juhoteperi21:08:10

or debugging

onetom21:08:14

it was a lot to read back, good morning!

alandipert21:08:17

right, it's like 'export'

martinklepsch21:08:18

Could target be a link to an existing tmp-dir?

juhoteperi21:08:45

@martinklepsch: There is no one tmp-dir which would contain all the files

micha21:08:26

that's an interesting concept though

micha21:08:36

we don't actually need to copy the files out of the fileset anymore

micha21:08:51

since they only get written to target after the pipeline is finished

micha21:08:02

so we can just have hard links in there that point to things in temp dirs

micha21:08:22

because those temp dirs are now immutable, since there are no references to them in any tasks anymore

martinklepsch21:08:23

Removing target seems like a thing that might confuse new-comers that are used to this kind of thing

micha21:08:39

i think we should write to target when the pipeline finishes

micha21:08:48

instead of after every incremental build

micha21:08:18

if someone wants to have the target populated each time, they can write a simple task for that

micha21:08:25

and add it to the pipeline

onetom21:08:23

(target --show), (target --dump) or rather (output --show), (output --dump)

juhoteperi21:08:25

@martinklepsch: True but seems also that target being available confuses many new-comers as they try to serve files from target (or even include target in source-paths)

martinklepsch21:08:12

right. also true. Just want to make sure things stay approachable simple_smile

micha21:08:09

actually i like the (target) task idea

micha21:08:37

or (dump :dir "target")

onetom21:08:56

for understanding what happens along the pipeline - as a newcomer - i was using (show -f) quite a lot (but it didnt tell me which files will i see in target at the end)

micha21:08:22

@onetom: raelly? it should be showing what will be in the target at the end

micha21:08:37

depending on where in the pipeline you use it of course

micha21:08:58

like boot show -f cljs will show you what would be send to target before the cljs task runs

micha21:08:08

but the cljs task adds a lot of files simple_smile

micha21:08:16

boot cljs show -f

onetom21:08:26

then i was just confused by seeing stuff there which shouldn't have been there

micha21:08:26

that should correspond exactly with the target dir

onetom21:08:34

eg i was not expecting the out/ dirs present in case of simple or advanced optimizations

onetom21:08:10

i even had a (sift :invert true :include #{#"^out/"}) to remove that

micha21:08:12

are you sure you weren't doing show -f before the cljs task in the pipeline?

onetom21:08:54

cant remember. it was weeks ago. as i said, i used it to understand more, so at that time i was not very clear about the filesets

onetom21:08:23

btw, how come u ended up with hoplon/javelin vs javelin/javelin?

micha21:08:09

do we prefer javelin/javelin?

onetom21:08:13

and also not boot-hoplon.core but hoplon.boot-hoplon

onetom21:08:32

i thought that was the conclusion we arrived at in https://hoplon.hackpad.com/Project-structure-GgBekBLIaIA

micha21:08:36

having all the things in the same clojars group will make it easier to manage for admin

onetom21:08:17

that was my counter argument against the otherwise very beginner friendly structure

onetom21:08:06

but then i said it's probably a minor inconvenience

onetom21:08:01

and u seemed to agree simple_smile

micha21:08:04

@alandipert: what do you think?

onetom22:08:06

clojure.lang.Compiler$CompilerException: java.io.FileNotFoundException: Could not locate cljs/analyzer__init.class or cljs/analyzer.clj on classpath., compiling:(javelin/core.clj:1:1)

alandipert22:08:41

most useful will be aligning clojars/github groupid under hoplon

micha22:08:26

@onetom: does your project have a cljs dependency?

onetom22:08:06

yeah, i was trying to put hoplon.javelin there

juhoteperi22:08:45

I added OS X results, overhead looks to be a bit larger.

micha22:08:25

i will have a boot update shortly

onetom22:08:38

has anyone tried to run clojure on http://c9.io? it would be super easy to peek into each other's projects there and see it for yourselves whats wrong and how long things take

juhoteperi22:08:47

I do have a Windows laptop also at hand but I doubt boot-cljs would work at all there…

onetom22:08:47

@alandipert: not sure if that guest on the hackpad is you but we had this conclusion when thinking about the naming: > Not throwing javelin, castra and boot-hoplon under the hoplon/ artifact group id makes the dependency specifications very beginner friendly.

juhoteperi22:08:44

(My desk is pretty crowded at the moment, 32+24 displays and three laptops)

martinklepsch22:08:09

Where does the 80ms difference during incremental compiles (boot overhead subtracted)

juhoteperi22:08:58

@martinklepsch: How did you calculate that?

xificurC22:08:02

@juhoteperi: I compiled a small cljs file like 2 days ago on windows with boot-cljs

xificurC22:08:18

why would you think it wouldn't work?

juhoteperi22:08:20

@xifi: There are still few hairy path/url manipulations going on. Could be they work or could be they break only in certain cases. But to my knowledge no-one is using boot-cljs on Windows actively?

xificurC22:08:03

@juhoteperi: well you can't call me active yet but I hope to become

martinklepsch22:08:25

@juhoteperi: took the time from Add dependencies, elapsed time: 256.398092 msecs and the incremental times from cljsbuild it’s closer to 150ms though, misread

xificurC22:08:26

if things break on win I'll just set up a vm. But my final artifacts should work on win

juhoteperi22:08:24

@martinklepsch: Compare basic sources + add dependencies times between boot and lein

martinklepsch22:08:01

oh, there boot-cljs is like crazy faster?

juhoteperi22:08:51

@martinklepsch: No. 5+294=300 and 120+203=320, they are nearly as fast

juhoteperi22:08:16

The difference in distribution between basic sources and dependencies is caused by different configuration

alandipert22:08:52

@onetom: what about not having hoplon/ makes things beginner friendly in your opinion?

onetom22:08:15

there is less confusion between the github -> group id -> namespace structures hoplon/hoplon -> hoplon(/hoplon) -> hoplon.core hoplon/javelin -> javelin(/javelin) -> javelin.core hoplon/castra -> castra(/castra) -> castra.core hoplon/boot-hoplon -> boot-hoplon(/boot-hoplon) -> boot-hoplon.core

onetom22:08:41

there is a higher chance of name collision on clojars of course but micha checked and everything was available...

onetom22:08:56

tho we should discuss this on #C08BDAPRA probably

xificurC22:08:53

they both look symmetric to me 😮 Personally I like hoplon/ more

xificurC22:08:30

if you think about it as organization/project

micha22:08:59

there we go!

juhoteperi22:08:56

Hm with that boot shouldn't write to target while watch is going on?

micha22:08:23

if you don't use the dump task it won't write

micha22:08:27

if you do then it will

micha22:08:35

do we want a mode where it only writes at the end?

juhoteperi22:08:44

How do I make sure I'm using right version of boot?

juhoteperi22:08:20

ah one sec, target could have been from previous run

juhoteperi22:08:44

hmm no, still writing to target

micha22:08:58

boot cljs

micha22:08:01

like that?

juhoteperi22:08:00

I do have 2.2.0 on boot.properties and make install should have been enough?

micha22:08:23

that's what i did

juhoteperi22:08:25

lol, I forgot to checkout the branch

juhoteperi22:08:54

I should be sleeping... but I'll just test this fast...

juhoteperi23:08:42

Okay, no target dir but no much difference in incremental compile times either.

juhoteperi23:08:58

Debugger or just adding loads of time calls should tell what is taking the time

micha23:08:47

yeah java io is so fast these days

micha23:08:02

probably the least slow part of the whole thing

onetom23:08:30

@juhoteperi: u r not using the https://github.com/ptaoussanis/timbre lib for such profiling?