Fork me on GitHub
#boot
<
2015-10-22
>
martinklepsch08:10:25

I wonder if we could take some inspiration from tools like gulp for the above problem.

danielsz13:10:57

Hi, so if you want to shell out from a boot task, you'd use the dosh function in boot.util, right? But when the command needs sudo privilege, is there going to be a prompt? I haven't tried it yet. Wondering out loud.

danielsz13:10:48

I'm not explaining myself well, I'll do some tests and rephrase my question. Please ignore for the moment.

danielsz14:10:09

OK, so my question is, is it possible in boot to shell out interactive commands, ie. commands that interact with the user, prompting for input, like a sudo prefixed command.

micha14:10:57

@danielsz: dosh doesn't handle interactive commands, it was made to stream output from the process as it's produced rather than after the process has exited

micha14:10:16

i think handling input correctly is a tricky thing

danielsz14:10:22

Thanks, @micha. I'll rethink my approach.

martinklepsch15:10:25

@micha: I have two things where I wanted to ask you for your advice: 1. How would you go about debugging a memory leak in Boot (specifically pod-pools) 2. In general I’d like to get a better understanding of where time is spent in Boot — just use something like yourkit?

micha15:10:49

for 1. i used visualvm in the past and was able to track down issues with pods pretty well

micha15:10:04

that's what i used when the pod pool thing was in development

micha15:10:45

for 2 yourkit is probably the thing to use, we could maybe get a sponsored license for boot

martinklepsch15:10:55

it’s free for OSS (I think)

micha15:10:09

yeah we just need to register and stuff and they need to approve it

juhoteperi15:10:47

I tried using Visualvm some time ago and it was SUPER SLOW in compared to YourKit

juhoteperi15:10:37

It had to instrument each class and with cider-nrepl and etc. there's like 50k classes in classpath. With some filters I managed to get it to something like 5000 classes and then it was only 5 minutes instead of 30 minutes to start CPU profiling.

martinklepsch15:10:15

yourkit seems pretty snappy

martinklepsch17:10:08

@micha: when using make install can I be certain that stuff is rebuilt?

micha17:10:51

@martinklepsch: you can do make clean first

martinklepsch18:10:19

Getting No such task (./pod-loop) when just running a script file similar to this one: https://github.com/alda-lang/alda/blob/master/bin/alda

martinklepsch18:10:16

even when using the very same file…

micha18:10:26

how are you running it?

martinklepsch18:10:45

also tried adding the dir to PATH and running without ./

micha18:10:58

what's pod-loop though?

micha18:10:07

is that a file?

micha18:10:38

can you paste the contents of the pod-loop file?

martinklepsch18:10:23

I also tried with the contents of the alda script linked above

martinklepsch18:10:19

(.clj was appended by emacs)

micha18:10:35

you must have a build.boot file in the directory perhaps?

micha18:10:45

i just ran that and it worked fine for me

micha18:10:38

what does boot --version say?

micha18:10:56

i guess you're building from master probably?

martinklepsch18:10:11

lib 2.3 app 2.2

martinklepsch18:10:26

should probably try with app 2.3 although that hasn’t caused any trouble until now

micha18:10:34

yeah it shouldn't matter

micha18:10:58

maybe your boot.properties is interfering?

micha18:10:02

or something?

micha18:10:10

i mean profile.boot

martinklepsch18:10:33

deleted that, same

martinklepsch18:10:51

hm. now it’s getting weird.

micha18:10:26

man, something weird is going on

martinklepsch18:10:45

wow. it was app 2.2

martinklepsch18:10:51

works fine with 2.3

micha18:10:55

that's very weird

martinklepsch18:10:39

I’ll check again if that really was the difference or if I maybe did something else too which I forgot about

dave20:10:50

@martinklepsch: by any chance do you use windows? i've had reports from windows users trying to run the alda boot script of a similar error message

dave20:10:00

"No such task 'alda'"

martinklepsch20:10:58

can I invoke a file using boot similar to how it’s done when using #!?

micha20:10:33

@martinklepsch: how do you mean?

micha20:10:47

like type boot ./foo?

martinklepsch20:10:47

boot —file pod-loop

micha20:10:55

yeah that should work

micha20:10:30

no --file option though simple_smile

martinklepsch20:10:36

@micha: do you have 10min to chat?

micha20:10:59

sure 5 min from now?

martinklepsch20:10:16

sure, will dump some thoughts here

martinklepsch20:10:05

I’ve been looking into some of the memory issues that were reported with boot-garden (https://github.com/martinklepsch/boot-garden/issues/10) and also with boot in general (https://github.com/boot-clj/boot/issues/268). While all of these cause OutOfMemory exceptions it seems that they’re solely related to PermGen space or “Metaspace" in Java 8. While increasing PermGen size fixed the issue for Java 7 the new behavior in Java 8 is to have unlimited “Metaspace”. So if there are lots of classes generated this space can fill up your entire RAM & crash the process. With PermGen this problem surfaced earlier but still should have ultimately caused OutOfMemory exceptions. Now reading up on when Metaspace things are GC’d it seems that it’s tricky and “subtle references may leak and prevent the Classloader from being reclaimed” (http://stackoverflow.com/questions/2344964/when-and-how-is-a-java-classloader-marked-for-garbage-collection). There are some potential workarounds mentioned in the above SO post but before I dig deeper I’d make sure I’m not running in the wrong direction full speed. Does all this sound reasonable to you?

micha20:10:37

i thought permgen was gone in java 8

martinklepsch20:10:40

The setting for it is gone but there’s still a special place for class definitions, now called Metaspace

micha20:10:25

yeah i spent a lot of time in visualvm to try to find any pod class leaks

micha20:10:41

i thought id eliminated them

micha20:10:59

but maybe we have a regression that was introduced at some point

micha20:10:05

all of the work i did was pre-2.0

micha20:10:20

back when i was evaluating the boot server concept

martinklepsch20:10:42

using -XX:MaxMetaspaceSize=200m you can make the occur faster: https://gist.github.com/martinklepsch/ac4c4680e3f47c5c2df9

micha20:10:52

your script doesn't hold a reference to anything

micha20:10:21

so i can't really think of any reason why the classloaders wouldn't be eligable for GC

micha20:10:41

there is a jvm option for enabling GC on classes, maybe we need to use that?

micha20:10:50

actually let me run your script and see

martinklepsch20:10:33

I found a few things that are not being closed properly along the way ^

martinklepsch20:10:05

I’m not sure if this is correct but probably someone else can review (it works ;P)

micha20:10:03

i don't think that's related to the pod issue though, do you?

micha20:10:23

i think the pod pool is the cause

micha20:10:58

this runs for me without OOM issues

martinklepsch20:10:15

@micha: right, should have checked that earlier, will give it a try with yourkit and ping back

micha20:10:16

hmm maybe not

micha20:10:33

i forgot to set that JVM option lol

micha20:10:56

it still crashes 😕

martinklepsch21:10:18

was just going to say that

jfntn21:10:01

Is it reasonable to try boot-clj for a project that essentially requires sub-projects packaging like lein-sub does?

martinklepsch21:10:10

@jfntn: I’m not doing it myself but would say yes, see https://github.com/decomplect/ion/blob/master/build.boot for an example

jfntn21:10:29

@martinklepsch: oh thanks that looks just like what I was going for!

jfntn21:10:42

I don’t see any inter-project dependencies though, would you just be able to add a deps entry for previously built artifacts and make sure that the pipeline builds them in topological order?

martinklepsch21:10:44

@micha: if you scroll to the “AnonymousClassLoader” section that sounds relevant: http://blog.headius.com/2008/09/first-taste-of-invokedynamic.html

martinklepsch21:10:19

@jfntn: you mean like you have subprojects A, B & C and A & B depend on C so C should be built first when building all?

martinklepsch21:10:52

as long as you don’t want to AOT or uberjar A or B this shouldn’t be an issue at all I think

martinklepsch21:10:05

if you want to do that you’ll have to take care of it yourself

jfntn21:10:16

Right, but in theory you should be able to uberjar C first, and make sure the artifact ends up on the classpath for A & B when you uberjar those down the pipeline?

martinklepsch21:10:54

@jfntn: if A or B depend on C you probably don’t want to uberjar C but just use regular resolution via maven. In this case you can just specify C as a regular dependency

jfntn21:10:16

ah yes, good point!

martinklepsch21:10:34

I understand that as “a Clojure var holds a reference to the CL” but I don’t see where that would happen

alandipert22:10:26

@martinklepsch, @micha re: mem core.async might be a culprit, https://github.com/projectodd/shimdandy#preventing-memory-leaks