Fork me on GitHub
#boot
<
2015-11-22
>
bensu16:11:45

hi! can I get some guidance on making an uberjar with boot? I'm following saapas structure and doing (comp (cljs-build) (aot) (pom) (uber) (jar)). The final jar has everything needed but in the wrong dir structure (i.e, no libs/), when I do java -jar target/asterion-0.1.0-SNAPSHOT.jar it can't find clojure in the classpath

bensu16:11:03

any clues on what could be messing the jar's internal dir structure?

martinklepsch16:11:21

@bensu: do you have clojure specified as a dependency?

bensu16:11:06

@martinklepsch: yes, and I can see its jar inside my jar, but in top dir instead of under libs/

martinklepsch16:11:57

is libs/ a standard thing in uberjars? probably can’t help with anything more specific...

bensu16:11:30

maybe it's not. it is in uberwars. but I do see clojure-1.7.0.jar inside my jar yet when I run it, it can't find clojure.lang.IFn: Exception in thread "main" java.lang.NoClassDefFoundError: clojure/lang/IFn

micha16:11:45

are you trying to make an uberwar?

bensu16:11:49

no, a jar

bensu16:11:04

I gave up with the uberwar after 2 days 😞

micha16:11:13

you should not have jars in jars

micha16:11:28

the idea of the uberjar is to merge all the dependency jars into a single jar file

micha16:11:50

it explodes all the individual jars into the fileset, that's what the uber task does

micha16:11:59

then the jar task builds a single jar from that

micha16:11:48

if you want to build an uberwar you can do that too

micha16:11:01

using the :as-jars option to ub er

micha16:11:16

because war files can contain other jar files

micha16:11:36

which will end up in theWEB-INF/lib directory in the war

micha16:11:50

and the servlet container will load those on the classpath at boot

micha17:11:29

can you paste your build.boot file please

micha17:11:16

if you make a war file you will need to have a web.xml for it

micha17:11:28

the web task provides that functionality

micha17:11:48

that's where you specify the servlet handler and stuff

bensu17:11:25

the jar now works, I had :as-jars on from trying to make an uberwar

bensu17:11:30

I was using web to make the uberwar, but tomcat in elastic beanstalk wouldn't understand my config and return 404 for every request

micha17:11:59

ah yeah it's a pain to debug tomcat problems

micha17:11:08

did you try using the boot-jetty task?

micha17:11:23

it runs a real tomcat server locally

micha17:11:35

and if your thing works there it will work in production

micha17:11:59

the boot-http task is slightly different from the production environment

micha17:11:42

i think boot-jetty is probably the best way to debug tomcat issues

bensu17:11:45

I was using boot-http. great! I'll try with boot-jetty

micha17:11:06

you should be able to reproduce the 404 that way

micha17:11:11

in your local env

bensu17:11:19

do I want it in :scope "test"?

micha17:11:02

boot tasks should always be :scope "test", especially when you're making an uberjar

micha17:11:06

or an uberwar

micha17:11:23

unless you need those tasks at runtime of course

bensu17:11:31

is this how you pipe to serve? (comp (cljs-build) (web) (uber) (war) (serve :port 8080))

micha17:11:47

yeah that'll work

micha17:11:09

actually one sec

micha17:11:44

i'm not sure about the war task

micha17:11:48

you don't want that one

micha17:11:21

like when you upload your war file to the tomcat server and deploy it

micha17:11:30

the tomcat server explodes the war into a directory

micha17:11:40

that's what boot-jetty wants i think

bensu17:11:30

ok, trying that

bensu17:11:56

interesting, it started the server before all other operations.

bensu17:11:11

(even though it was at the end of the comp)

micha17:11:26

yeah it only starts the server once

micha17:11:32

when the task is constructed

micha17:11:55

i.e. when (serve :port 100) was evaluated

micha17:11:59

that's when the server was started

bensu17:11:20

right, the "tasks as functions" is very good abstraction but sometimes I find it hard to follow when it leaks. I'm guessing this is an instance of that.

micha17:11:28

hah actualy i was wrong about that

micha17:11:36

it's just printing that it's starting the server

micha17:11:42

it's not really doing it then

bensu17:11:24

ahh ok, it should do it inside the delay

micha17:11:17

but that's an example of how you can do some pre-pipeline prep in a task

micha17:11:31

mostly it's used to allocate stateful things

micha17:11:40

like creating that tempdir

micha17:11:10

the expressions before the (with-pre-wrap ...) can perform side effects and allocate resources

pesterhazy17:11:27

I'm trying to speed up my repl restart times. I've followed https://github.com/boot-clj/boot/wiki/Improving-startup-time which helps tremendously

micha17:11:27

and those are evaluated before the pipeline starts

pesterhazy17:11:40

Are there any other things that can help?

micha17:11:49

how long does it take to start a repl?

micha17:11:58

ah yes, one thing you can do

pesterhazy17:11:05

@micha: in my project, it takes 12s

micha17:11:10

boot -B repl

pesterhazy17:11:14

with a bunch of dependencies

micha17:11:26

yeah the -B option ignores your build.boot

micha17:11:46

you can do (load-file "build.boot") whnever you want, with the repl already started

pesterhazy17:11:49

@micha: good call, but I've done that already --- removing CIDER cut the time in half

micha17:11:13

you should get a repl in like 5 seconds with boot -B repl

micha17:11:24

or more extreme: boot -BP repl

micha17:11:37

that ignores your profile.boot file too

micha17:11:46

ah here's another idea

micha17:11:52

you can make a repl serve

micha17:11:59

*repl server

pesterhazy17:11:17

boot -BP repl < /dev/null takes 10s

pesterhazy17:11:01

do you mean setting :server true for the repl task?

micha17:11:26

no, like a server of servers

micha17:11:48

the runboot task runs boot in boot

micha17:11:13

so you start a repl with boot -BP repl

micha17:11:37

then once in there you can use the runboot stuff to start repl servers in a boot-in-boot

micha17:11:45

which will be a lot faster probably

pesterhazy17:11:58

that's too crazy for me simple_smile

micha17:11:00

yeah you can run boot inside of boot

pesterhazy17:11:54

other than setting better JVM settings, disabling CIDER was the biggest win so far

micha17:11:58

do you know what's taking the time?

micha17:11:10

i wonder what cider was doing?

pesterhazy17:11:28

I looked into removing dependencies, but that didn't seem to have much of an effect

micha17:11:52

how about making a repl task that doesn't use nrepl?

micha17:11:07

you can use rlwrap to get readline support

pesterhazy17:11:37

that's now confirmed: boot -BP takes 10s, whereas boot dev-repl takes 12s, so most of the seems to be spent inside general boot start-up

pesterhazy17:11:07

that'd be an interesting experiment, but an nREPL is important for my work flow

pesterhazy17:11:46

how long does time boot -BP repl < /dev/null take for you?

bensu17:11:18

thanks @micha! I now have a setup to locally diagnose tomcat config problems.

bensu17:11:07

it's not quite working since boot-jetty pins a certain version of jetty which seems to be incompatible to my ring dependency but the big parts are in place

micha17:11:33

real    0m4.130s
user    0m7.404s
sys     0m0.220s

micha17:11:23

barp $ time boot -BP repl < /dev/null 2>&1 > /dev/null
nREPL server started on port 51682 on host 127.0.0.1 - 
real    0m4.246s
user    0m7.936s
sys     0m0.200s

micha17:11:41

sometimes it takes like 10s the first time for some reason

pesterhazy17:11:53

that's certainly faster than here simple_smile (OSX, boot 2.4.2)

micha17:11:05

i'm using boot 2.5.0-SNAPSHOT

micha17:11:14

but it shouldn't make a difference there

micha17:11:32

i'm also on a machine with 16G memory

micha17:11:44

but only 2 cores with 4 threads

micha17:11:47

or whatever

pesterhazy17:11:49

I've got a macbook air 2012, maybe CPU is the bottleneck

micha17:11:45

2.5.0 has a lot of performance things involving caching, maybe there's something we can cache to make repl startup faster

micha17:11:07

i guess we need to profile it to see what's taking so long

micha17:11:40

if you could make a flame graph or something that would be a good place to start

pesterhazy17:11:02

which tool would you suggest for profiling?

micha17:11:29

i've used visualvm, and there are good tools that come with the jdk too

micha17:11:10

we could get an open source license for it i think

pesterhazy17:11:47

boot -vv doesn't print any more helpful information

micha17:11:38

yeah boot isn't really doing much when you have -BP set

micha17:11:52

so you won't see anything interesting in the debug log

micha17:11:33

visualvm is probably the easiest way to start profiling

pesterhazy17:11:23

boot --help takes 4.2s, so that's probably the baseline

micha17:11:46

yeah that's the minimum amount of work that boot can do

micha17:11:25

is ~/.boot/cache/cache populated with files and stuff?

micha17:11:49

boot caches jars and things in there

micha17:11:56

that will definitely affect startup

pesterhazy17:11:19

boot --help takes 2.29s with BOOT_JVM_OPTIONS set to the expected settings

pesterhazy17:11:28

s/expected/recommended/

micha17:11:47

the insane mode options, like -XTieredCompilation:stopat...

micha17:11:23

yeah those options will cut the startup time in half

pesterhazy17:11:36

the cache is pretty much empty, yeah

micha17:11:30

this is what mine looks like:

micha17:11:36

tree ~/.boot/cache/cache/boot/default/
/home/micha/.boot/cache/cache/boot/default/
├── 1.6.0
│   ├── 2.5.0
│   │   └── deps.cache
│   └── 2.5.0-SNAPSHOT
│       └── deps.cache
└── 1.7.0
    ├── 2.4.2
    │   └── deps.cache
    ├── 2.5.0
    │   └── deps.cache
    └── 2.5.0-SNAPSHOT
        └── deps.cache
7 directories, 5 files

micha17:11:58

this is when BOOT_LOCAL_REPO isn't set

micha17:11:06

i'm assuming you're not setting that

micha17:11:32

the deps.cache files are the thing you want to be there

micha17:11:40

they are pointers into the maven cache

micha17:11:50

so boot doesn't need to resolve its own jars

pesterhazy17:11:53

no, I'm not setting any additional env vars

micha17:11:20

it's like ~/.boot/cache/cache/<local repo or default>/<clojure version>/<boot version>/deps.cache

micha17:11:49

it needs a cache for each permuatation of those parameters

pesterhazy17:11:16

well I always re-run the same command and only look at the second run, so it should be able to cache all it needs

micha17:11:39

yeah it only needs to cache it the first time you run that version of boot with that version of clojure

pesterhazy17:11:09

time env -i PATH=$PATH BOOT_JVM_OPTIONS="-client -XX:+TieredCompilation -XX:TieredStopAtLevel=1 -Xmx2g -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -Xverify:none" $(which boot) -BP repl < /dev/null 
<- 5.98s total

micha17:11:10

the cache is invalidated though if any of the jar files change their mtime

micha17:11:40

that normally won't happen

micha17:11:15

but like if somehow the jars in your maven repo are altered, boot will invalidate its cache

pesterhazy17:11:39

yeah, that's probably not the problem

pesterhazy18:11:01

well I'm happy for now, I've understood things a bit better

micha18:11:16

i think 5.98 sec is about what i was getting back when i had a 2012 MPA

micha18:11:35

actually i think it was more like 7sec

micha18:11:55

i think the next step is probably to profile it

micha18:11:20

and see if there's some shoelace untied that we can just make into a bow and get fast perf

pesterhazy18:11:09

time java -cp lib/clojure-1.6.0.jar clojure.main < /dev/null takes 1.5s, so we won't be able to do better than that

micha18:11:27

yeah probably not

micha18:11:36

do you use the nrepl/reply functionality?

micha18:11:48

or are you ok with a repl with fewer features

micha18:11:07

you might try making a task that does the clojure.main repl

pesterhazy18:11:42

I do use nREPL because I connect from emacs

pesterhazy18:11:04

but there are circumstances where a bare-bones (rlwrap/jline) repl is useful as well

pesterhazy18:11:25

a bare-repl tasks would a useful addition to boot

micha18:11:39

i think there is some room for optimizing how the repl server and client are synchronized when they start in the repl task

pesterhazy18:11:09

yes, just from looking at the stdout it seems like there's some unnecessary waiting involved

micha18:11:03

there is some "ack" mechanism in nREPL that i don't understand

micha18:11:09

i think it's for that purpose

micha18:11:17

but i couldn't figure out how it works

micha18:11:44

do you know anything about node-gyp by any chance?

pesterhazy18:11:32

yes I'm using it

pesterhazy18:11:56

well I don't really know anything about it simple_smile

bensu19:11:33

quick question, if I am composing tasks and each steps generate files the fileset gets bigger on each step. Is there a way to filter the files that will make it to the final task?

bensu19:11:56

(comp (add-repo) (uberjar) (filter "somehow") (zip))?

bensu19:11:20

otherwise the final task (`zip`) gets a lot of files I'm not interested in

micha19:11:16

the sift task is probably going to do what you want

micha19:11:28

2.5.0 has a much faster implementation of that btw

micha19:11:55

so if you build from master you can speed up that process, and all the other uberjar stuff too, by orders of magnitude

bensu19:11:06

cool, will do.

bensu19:11:26

when I initially read sift I thought it was for moving things, I see now that it does much more

micha19:11:41

yeah it's a swiss armyknife type of thing

micha19:11:49

for filset manipulations

jaen19:11:00

@micha: I've asked once about boot classpath and you replied that I have to specify it through the JVM option since boot can't really do that. But how would I know what path to give if it's boot that resolves dependencies? Woudl I have to hard-code a path in my script, or?

jaen19:11:12

Or is there some way to ask boot for a path to some jar?

micha19:11:46

@jaen: maybe if i know more about what you're wanting to do

micha19:11:06

i feel like i'm missing something obvious simple_smile

jaen19:11:26

Basically for HTTP2 to work it has to have an ALPN implementation

jaen19:11:34

And it works only when it's on boot class path

jaen19:11:32

But if you said I can't do what lein does with it's boot classpath plugin due to how boot works (single JVM)

jaen19:11:42

But I don't want to check-in a jar into source control either

jaen19:11:42

So I'm wondering if there's at least a way to query boot for path to a certain jar, so I don't have to do it by myself

jaen19:11:38

Basically what I want is to reproduce this with boot, since it's my main build tool - https://github.com/jcrossley3/immutant-repro/blob/master/project.clj

micha19:11:08

hm, i see now

micha19:11:37

interesting problem

micha19:11:50

i think we can make something for this

jaen19:11:12

That would be neat if possible

micha19:11:34

in the meantime you can get the jar locations from boot show -c i believe

micha19:11:52

but yes, i think we want a more elegant mechanism for this

micha19:11:27

maybe a task that creates a wrapper script

micha19:11:30

or something

jaen19:11:34

Sure, I guess I can split that by colons and grep the one I need, sounds reasonable for nwo

micha19:11:12

i wonder why it needs to be on the bootclasspath really?

micha19:11:32

maybe we can do it a simpler way if it can be added to a classloader dynamically

jaen19:11:01

I wouldn't know, but all the docs say it's required to be there

jaen19:11:01

I think it patches core Java classes

jaen19:11:05

And that's why it has to be

jaen19:11:25

ALPN is a functionality that's going to be added in JDK9, but for older JVMs it requires patching of core classes

micha19:11:50

ok that makes sense