Fork me on GitHub
#boot
<
2017-02-08
>
chooie00:02:22

@pesterhazy @pithyless @bhagany @ska @alandipert Thank you guys for your responses. I tried time BOOT_JVM_OPTIONS="-client -XX:+TieredCompilation -XX:TieredStopAtLevel=1 -Xmx2g -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -Xverify:none" boot -BPh and indeed I was able to cut down on the time: BOOT_JVM_OPTIONS= boot -BPh 3.55s user 0.30s system 54% cpu 6.989 total However, it is still taking much longer than a number of you guys are getting. One thing I did discover was in my boot.properties file - key values are not properly read if the string sprawls multiple lines: is this a bug? boot.properties

BOOT_CLOJURE_NAME=org.clojure/clojure
BOOT_CLOJURE_VERSION=1.8.0
BOOT_VERSION=2.7.1
BOOT_LOCAL_REPO=./third_party_dependencies/
BOOT_HOME=/.boot
BOOT_AS_ROOT=yes
BOOT_JVM_OPTIONS="-client-XX:+TieredCompilation -XX:TieredStopAtLevel=1
-Xmx2g
-XX:+UseConcMarkSweepGC
-XX:+CMSClassUnloadingEnabled
-Xverify:none"
Output from boot show β€”env
:config
 {"BOOT_AS_ROOT" "yes", "BOOT_CLOJURE_NAME" "org.clojure/clojure", "BOOT_JVM_OPTIONS" "\"-client-XX:+TieredCompilation -XX:TieredStopAtLevel=1", "BOOT_FILE" "build.boot", "BOOT_CLOJURE_VERSION" "1.8.0", "BOOT_VERSION" "2.7.1", "BOOT_LOCAL_REPO" "./third_party_dependencies/"},

chooie00:02:08

Making the BOOT_JVM_OPTIONS value all one line fixes this issue

chooie01:02:22

^^ Turns out I should have written it like this:

BOOT_CLOJURE_NAME=org.clojure/clojure
BOOT_CLOJURE_VERSION=1.8.0
BOOT_VERSION=2.7.1
BOOT_LOCAL_REPO=./third_party_dependencies/
BOOT_HOME=/.boot
BOOT_AS_ROOT=yes
BOOT_JVM_OPTIONS=-client \
                 -XX:+TieredCompilation \
                 -XX:TieredStopAtLevel=1 \
                 -Xmx2g \
                 -XX:+UseConcMarkSweepGC \
                 -XX:+CMSClassUnloadingEnabled \
                 -Xverify:none

chooie01:02:12

I managed to reduce my boot time to a more reasonable level:

boot -BPh  10.12s user 0.48s system 285% cpu 3.711 total
time BOOT_JVM_OPTIONS="-client -XX:+TieredCompilation -XX:TieredStopAtLevel=1 -Xmx2g -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -Xverify:none" boot -BPh

boot -BPh  3.61s user 0.26s system 192% cpu 2.015 total

chooie01:02:33

I uninstalled boot with Brew. Furthermore, I deleted the ~/.boot directory. Could the slowdown be due to the cache getting too large?

seancorfield02:02:57

Hmm, -Xverify:none... hadn't tried that...

richiardiandrea02:02:41

Oh BOOT_AS_ROOT is new to me, we'd better add tests πŸ˜‰

seancorfield02:02:45

Heh, that not being the default has saved me more than once...

richiardiandrea03:02:40

Oh I bet, and <paranoia>I would review the code twice if I had to enable something like that </paranoia>

kenbier08:02:00

The clojure 2016 survey results are out

kenbier08:02:14

> boot made significant advances this year, moving from 13% usage to 22% usage

pesterhazy10:02:33

After not touching leiningen in 1.5 years, I just tried lein-figwheel and it's great! It's made so much progress since I last looked at it.

pesterhazy10:02:13

My favorite feature now is that it automatically opens a repl prompt in the terminal - that's great usability.

pesterhazy10:02:42

I'd love to see this in boot as well, but can't see an easy way to make this happen

moxaj11:02:03

@pesterhazy you need 2 separate terminals right?

pesterhazy11:02:11

one thing I thought would be useful would be to start a socket repl that provides a cljs repl

pesterhazy11:02:38

then open another terminal window (shell out to xterm or whatever) and have that telnet into the socket repl

dominicm11:02:08

@pesterhazy There is a slight limitation in boot's model in that you can't have a REPL-y running & a watch (this may be wrong, but I know no workaround) But if you're looking at that sort of craziness, is it that much more helpful than having the user hit Ctrl-Z then running boot repl -c to connect back to their REPL? Then there's a function (cljs-repl) I believe which gets them the REPL prompt for cljs.

pesterhazy11:02:54

yes but that's super hacky

pesterhazy11:02:04

and worst of all, not automated

pesterhazy11:02:21

it's better to be able to point people in your team to a single command (`boot dev`) that does all the work

pesterhazy11:02:03

@micha mentioned a way to combine watch with repl -c.. you can run boot repl and then run the main pipeline from there, in a future

pesterhazy11:02:35

still that seems sub-optimal as you lose the error output from the cljs compilation

dominicm11:02:43

I don't think that is hacky. The nREPL isn't a single interface. It's two things, a server and a client. I launch a server as part of my development, if I want a TUI to interface with the REPL I can launch one (conveniently) from boot! Or I could use something like parle, or the traditional editor integrations too.

dominicm11:02:22

Yeah, I really can't see why you consider that hacky. Automation can be handled by this: boot dev & && boot repl -c

dominicm11:02:13

Also, I'm pretty sure when a backgrounded process prints to stdout, you get a message. Not sure how REPL-y handles that, but you might still get the cljs build failure.

pesterhazy12:02:44

IMO Unix shell job handling just doesn't work particularly well

pesterhazy12:02:43

what's parle?

dominicm12:02:33

It's a node.js based REPL-y type thing. Faster to start than REPL-y (jvm vs node)

dominicm12:02:59

@pesterhazy I've had opposite experiences with unix shell job handling. I run my entire status panel using a series of HUP'd tasks and FIFO pipes across processes. I've never had an issue with it.

pesterhazy12:02:02

I'll check out parle

pesterhazy12:02:38

just tried out opening a cljs repl in a separate terminal window, it's pretty easy

pesterhazy12:02:36

i.e. socket server functionality makes this pretty easy

miikka13:02:14

Hey, if I launch a repl with boot repl, can I somehow reload build.boot?

micha13:02:43

hi @miikka ! sure, you can do (clojure.core/load-file "build.boot")

miikka13:02:19

oh, right! thanks

pesterhazy14:02:20

I think I can also just reload the buffer in cider, not sure why that works

richiardiandrea15:02:03

@pesterhazy also in the latest cider on jack-in-clojurescript it opens two repl buffers one of which is directly in cljs.user -> https://github.com/clojure-emacs/cider/pull/1928

richiardiandrea15:02:31

That's great , I knew it would have happened sooner or later https://github.com/kanej/parle

tmarble15:02:22

hey folks are cribbing my naming-projects-in-french thing πŸ™‚

mobileink20:02:32

the jar task makes it easy to write an entry to the manifest. what's the best way to read an entry? i need to pass some metadata to the jar consumer (another boot task).

micha20:02:02

@mobileink you can use the java.util.jar.Manifest class

mobileink21:02:45

thanks, didn't know about that. wonder if it's worth adding some convenience fns for this to boot. e.g. (defn manifest-map [jar] ... ) or sth like that.

geoffs21:02:17

I don't think it's that common of a need TBH

geoffs21:02:46

The only thing I use a JAR manifest for is the Main attribute... I imagine that's probably true of many clojure devs

mobileink21:02:37

@geoffs: agreed, not so common, probably. but underappreciated, imho; really useful for meta-programming with boot tasks!

geoffs21:02:14

really? do you have an example project that does that? I'm not really sure I understand what you're talking about πŸ™‚

mobileink21:02:01

not quite ready for the masses, but: i have a clojure lib

miraj.polymer.paper
that "wraps" the paper components of https://www.polymer-project.org/1.0/.

mobileink21:02:36

the consumer is a webpage project that does

(:require [miraj.polymer.paper :as paper])
. my
compile
task turns that into <link ... > elements in a generated html file.

mobileink21:02:18

for this to work, the href in the link elements must be resolvable. that's where the manifest comes in. it contains an entry specifying the bower cmd needed to download the polymer assets. so the

compile
task must also read the manifest to get the bower cmd, run bower, and put the obtained assets into the fileset.

mobileink21:02:05

this way the consumer of the lib does not have to worry about asset config - the writer of the wrapper lib, plus the structure of the boot task lib, handles everything, magically.

mobileink21:02:46

btw this assumes that the webpages will be static, generated from clojure code at build time but served directly by the http server at runtime.

geoffs21:02:13

interesting

mobileink21:02:38

make sense? i guess it's a little complicated, but in the end the complexity will be hidden.

mobileink21:02:23

i'm really liking it so far. should have sth to share in next few days.

mobileink21:02:32

anyway, key point being is that the manifest is a nice jar-level task communication mechanism. in earlier versions i relied on edn files, so the webpage programmer had to configure stuff to make the lib/jar work. then i realized the manifest would obviate the need for that. luv zero-config stuff!

mobileink22:02:17

i hate needing to run bower, npm, etc. and then having to make sure my <link...> elts are correct, etc., just to set up a web app before i even start coding the app. i don't have to do that to write a clojure app, :require et al. just work; why shouldn't webapp dev be just as easy?

geoffs22:02:14

I like the goal πŸ™‚ still a bit unclear on the details, but it definitely sounds interesting

mobileink22:02:15

not to mention sharing assets across webapps.

mobileink22:02:12

stay tuned, this is about my 3rd iteration so i may even be able to explain it clearly. πŸ˜‰

mobileink22:02:59

meanwhile this might help a bit: i put some metadata on the namespace var of miraj.polymer.paper. that's what allows my compile task to do the right thing; it filters the result of all-ns to find the stuff it understands.

mobileink22:02:32

i love metadata.

mobileink22:02:55

shite and onions. just realized i do not need the manifest. i can just add the info needed to the ns metadata. amazing what you learn about your own stuff when you try to explain it to others!

geoffs22:02:32

happens every time πŸ™‚

mobileink22:02:06

wait. i swear this was broken before i showed it to you. i can't understand why it's working now!

richiardiandrea22:02:12

the biggest boot command I have ever run:

boot -B --source-paths "." watch -e "^mvn-target" sift -i "src" -i "pom.xml" mvn -W `pwd` -A "-Pboot-clj clean install" target

mobileink22:02:48

howsabout "most granular boot tasklib"? i got about a dozen tasks for boot-gae, and i'll bet i could get some more. πŸ˜„

mobileink23:02:18

or "best integration": boit tasklib that makes the best use of other tasklibs"?

mobileink23:02:53

when i write boit i mean boot. just sayin'.

mobileink23:02:36

semi-serious about that. boot is wonderfully compositional.

mobileink23:02:15

i have several boot libs in the works designed to work together via composition. wondering how many boot tasklibs are independent v. depending on other boot tasklibs.

mobileink23:02:55

more generally: what does it take to make your boot tasks play nice with tasks from another boot lib?

richiardiandrea23:02:22

@mobileink just call the next handler with the fileset should suffice

mobileink23:02:36

idempotence? not so easy, in my expeience with boot-gae.

richiardiandrea23:02:24

ah idempotence, depending of what yes

mobileink23:02:36

sure, but tasks have assumptions about incoming (filesets, but also caches).

richiardiandrea23:02:17

yes ok, but that is more of task logic

richiardiandrea23:02:27

if you look at the boot side of things

richiardiandrea23:02:37

the only thing you need to do is to call the next handler

mobileink23:02:40

maybe what i'm wondering is more along the lines of "how to make public the assumptions your tasks make".

richiardiandrea23:02:14

I think that's the power of a middleware, no assumption should leak

mobileink23:02:55

e.g. i have some libs that work together, but only because they assume there is an edn file of particular form in the fileset.

richiardiandrea23:02:05

if two tasks need to communicate, there is some leakage, but if you talk about one task none should happen

richiardiandrea23:02:22

yep in that case you need to coordinate

alandipert23:02:33

task ordering, also can be tricky

alandipert23:02:37

its a tradeoff tho

mobileink23:02:02

well, the main mech of comm is the fileset. but that could be anything.

richiardiandrea23:02:18

yes, I remember we spoke here a couple of times about using something like a tree structure for composing tasks

richiardiandrea23:02:40

but it complicates things

richiardiandrea23:02:55

I wish Slack had history now πŸ˜„

qqq23:02:41

the one feature I really want from boot .-- a HUD like control display implemented in the browser

qqq23:02:47

I feel like the pure text interface is limiting

qqq23:02:05

I want boot interfaces that I can, drag/drop, toggle, expand/collapse, and visualize statistics about

mobileink23:02:57

@alandipert: yeah, ordering. idempotence. you can do it, but it involves overhead. in my case, tasks look for a master config edn. to get idempotence, if they don't find it they must create it. cost/benefits i guess.

richiardiandrea23:02:20

boot-cljs and boot-reload kind of do the same thing

mobileink23:02:41

and then, if i want to use tasks from different libs in one pipeline, they must agree about dealing with the master confog edn and its structure.

mobileink23:02:18

a job for spec i suppose.

richiardiandrea23:02:19

I mean, you could have a task which creates a bus/queue and then your tasks will grab/put things on there...but is it worth it?

mobileink23:02:40

what if we could annotate tasks somehow to stipluate what they expect to see in the incoming fileset?

alandipert23:02:04

could do that now, with preconditions?

alandipert23:02:27

ie a precondition in the fileset-receiving fn that a task returns

richiardiandrea23:02:46

preconditions or spec, it is kind of a similar approach I guess

alandipert23:02:00

i guess the precondition would be the place to put the calls to spec, maybe

alandipert23:02:09

along with some boot code to pull the file out that you expect

richiardiandrea23:02:10

yep I was thinking the same

richiardiandrea23:02:42

there could be spec in the meta of the task var

richiardiandrea23:02:11

little bit of automation

richiardiandrea23:02:37

but it is not a user-facing feature

mobileink23:02:00

so let's back up a bit. does it make sense to expect that any boot task can be composed with any other boot task? prolly not. but there is a subset, where tasks from different libs should be expectedvto play nice together. puts me in the mind of protocols.

mobileink23:02:17

so e.g. assume boot-servlets. then we might have boot-gae and boot-tomcat, etc.

mobileink23:02:09

and we want anybody to be able to write boot-foo to work with boot-servlets.

richiardiandrea23:02:11

yes in that case you would have maybe a compatibility or spec register for this or that task

mobileink23:02:54

boot-servlets would need to publish something that tell the dev what needs to be done to interop.

mobileink23:02:29

fileset, not task.

mobileink23:02:37

it would need to say, somehow, here's what the fs should contain, with this format.

richiardiandrea23:02:32

I think at the moment not many tasks have this necessity so that's maybe why it has not been developed yet

mobileink23:02:29

early days! πŸ˜‰ once people begin to understand that boot obsoletes all other such tools we'll see many more use cases. let's not forget that boot is not really a build tool, it's more of a job control language.

mobileink23:02:03

build stuff is just a special case.

richiardiandrea23:02:10

ah ah well yeah, I have for example the with to convert as many namespaces as I can to .cljc and see it running on node.js....maybe another life πŸ˜„

mobileink23:02:00

so here's a nutty proposal: boot should always put a distinguished edn file in the fileset that can be used by tasks to pass data. it would not need a name, just make it available by a fn call, like master-edn.

mobileink23:02:48

that would save me the trouble of always looking for such a file and creating it if i don't find it.