Fork me on GitHub
#boot
<
2016-03-02
>
seancorfield02:03:35

Our build.boot file is over 750 lines now… and as of today we no longer need Ant for our regular build/test process! Yay!

seancorfield02:03:46

We still have some old Selenium (Selenese) tests that are run via Ant, and our deployment process still leans heavily on Ant’s tar/untar stuff and ability to delete whole swathes of files, but we’re working on that.

seancorfield02:03:18

Things our Boot build script does includes: unpacking test files, running Grunt(!), tearing down and rebuilding various databases, running CFML (ColdFusion) scripts via HTTP(!), and the normal stuff that you’d expect (running tests, creating JAR files etc).

seancorfield02:03:01

We’re also starting to use pods more to help isolate things from each other...

isaac02:03:04

macro changed not recompiler. platform: Mac OS X

boot -v
#
#Wed Mar 02 10:52:27 HKT 2016
BOOT_CLOJURE_NAME=org.clojure/clojure
BOOT_CLOJURE_VERSION=1.8.0
BOOT_VERSION=2.5.5
[adzerk/boot-cljs          "1.7.228-1"   :scope "test"]

tolitius02:03:02

@seancorfield: that's great simple_smile do you guys use uberjars at all?

seancorfield03:03:24

@tolitius: No, not at present. We probably will as we move more of our processes to Docker.

seancorfield03:03:46

Our biggest pain point so far has been boot-test so expect some tickets coming your way @micha et al simple_smile

alandipert03:03:20

@seancorfield: : your build.boot sounds amazing, congrats

escherize05:03:47

Sorry: How do I call a shell command (like ls!) from a boot task?

escherize05:03:28

oh i can just: boot cljs && rsync....!

martinklepsch06:03:54

Does someone know if webjars provides 1) an API to initiate builds 2) a way to build all dependencies of a project?

juhoteperi06:03:59

@martinklepsch: It builds dependencies automatically and there is undocumented endpoint to initiate build (check network log after triggering build from browser)

martinklepsch06:03:16

@juhoteperi: thanks, should dependencies show up in the "Deployed NPM webjars" section then?

martinklepsch06:03:21

@juhoteperi: I deployed basscss but none of it's deps (https://github.com/basscss/basscss/blob/master/package.json#L17) are listed there...

martinklepsch06:03:11

Boot also comes with it's own dosh somewhere simple_smile

seancorfield06:03:30

I had a feeling it might @martinklepsch but it would have take longer to find that than to write what I needed I suspect! 😸

escherize07:03:53

Who came up with speak? 👍 🍎

Petrus Theron11:03:12

I’m using Cursive for dev, and I’m unsure about browser nrepl usage with the cljs nREPL that is started by boot. When I run boot dev, it does (comp (serve) (watch) (cljs-repl) (reload) (speak) (cljs)), which compiles my cljs and starts a repl. When I connect from Cursive, should I run (start-repl)? It seems like this would start two cljs compilation steps. /n00b

escherize12:03:51

I could be wrong but I don't think it starts 2 compilation steps. rather (start-repl) jacks into the brepl, instead of a clojure repl into the jvm process running the boottask.

jethroksy14:03:40

hey! just gave my boot talk, and a few questions popped up that I couldn’t really answer properly.

jethroksy14:03:19

1. Given that boot is turing complete, does boot do anything to ensure that nothing malicious is being done to the system?

jethroksy14:03:11

2. Is there a function that gets all dependencies, including those in pods?

alandipert14:03:45

1. no (at least, nothing more than what the JVM does)

alandipert14:03:15

what do you mean by get all dependencies? like list all the dependencies that have been loaded?

jethroksy14:03:24

that would potentially be loaded as well

jethroksy14:03:28

their use case is very specific

jethroksy14:03:41

their startup deals with deps

jethroksy14:03:13

they check if these deps have security concerns

alandipert14:03:47

so there isn't a way to know definitively what dependencies are loaded

alandipert14:03:03

you could survey all the pods and look, but as the program runs more could be added since this part is dynamic

alandipert14:03:15

the standard way for JVM applications to display dependencies is via pom

alandipert14:03:33

and boot supports/condones creating and sharing .pom files describing what libs built with boot need to run

jethroksy14:03:55

ah right in tasks

alandipert14:03:31

take boot-template for example

alandipert14:03:39

it uses stringtemplate in a pod to do string substitution at "build time"

alandipert14:03:49

i use it for filling out xml etc. before things get uberjar'd

alandipert14:03:59

however the application i built doesn't depend on stringtemplate

alandipert14:03:26

so i don't really care if the stringtemplate dep i used incindentally at build-time is compromised (that much)

alandipert14:03:05

so the short answer is no, boot isn't amenable to any static analysis of the deps at all

alandipert14:03:25

but the long and nuanced answer is it can be done by convention, using the pom task, and depending on what the application author's concerns are

jethroksy14:03:49

thanks for the answers simple_smile

jethroksy14:03:01

they really liked the abstractions boot provided

alandipert14:03:35

what is their startup btw?

jethroksy14:03:42

one of the main hurdles they mentioned was converting their giant project.clj file into build.boot

alandipert14:03:45

i started a startup at one time to do the same thing and decided not to lol

jethroksy14:03:53

sourceclear iirc

alandipert14:03:48

definitely one of the hurdles of programming is finding the time to program, i battle with this daily 😼

jethroksy14:03:04

i thought programmers just programmed all day

jethroksy14:03:18

weekends -> techno and code

alandipert14:03:40

the more you do, the more bugs you find and improvement ideas you have

mobileink14:03:22

@jethroksy: in boot-bowdlerize I have a show task that acts like builtin show, except it's driven by a config file. The latter contains info about deps that are used by other tasks, and show extracts the coordinates and uses aether/dep-tree to show dependencies that are used during build but not at runtime. Would something like that help?

jethroksy14:03:26

the rabbit hole in all programming projects 😞

escherize14:03:52

Uh, hey booters! Is boot taking over the way i feel like it is?

jethroksy14:03:58

@mobileink: right, but you’d have to specify all the deps in a file

mobileink14:03:22

that's true, but you have to specify them somewhere anyway.

jethroksy14:03:26

the guy wants to parse build.boot and say, hey, I have these deps, are there secure?

escherize14:03:30

Also, I have put out an early version of a.. cljs fiddle, and im going to spam you now - im sorry.

jethroksy14:03:02

so for lein, bundler etc it’s really easy

jethroksy14:03:06

because everything is declarative

mobileink14:03:47

but the drawback is then all the deps are in one space.

jethroksy14:03:59

right, so he’s asking

jethroksy14:03:06

can I get all deps, including those in pods statically

alandipert14:03:08

definitely boot pom is the 99% solution i think

alandipert14:03:23

since most people don't use pods in their app at runtime

alandipert14:03:13

i do sometimes, so to play with their thing i'd make a file with all the deps i'd want audited... and create a pom from it for them with a task

alandipert14:03:47

then in my build.boot and where i make my pods i'd use the deps from this file

mobileink14:03:56

yeah, that's why i wanted the ability to show build-time v. runtime deps. divide and conquer. plus that way the build-time only deps are not included in the build.

mobileink14:03:41

haven't looked closely at boot.pom, though. guess i'd better do that.

jethroksy14:03:08

so build.boot reads this deps file to load deps as well

alandipert14:03:51

@escherize: the cljs fiddle awesome!

mobileink14:03:10

no, just tasks that need them. originally I make them .clj files with defs, then I saw the light so I'm switching to edn files.

alandipert14:03:57

@escherize: feature request, add javelin 😃

alandipert14:03:24

i have wanted to make something like this for hoplon for a long time

jethroksy14:03:34

is it the traffic or is cljsfiddle not loading for me

jethroksy14:03:57

ahh ok i see it now

jethroksy14:03:04

I would RT but I have no twitter simple_smile

jethroksy14:03:32

I think the pom file is the way to go

jethroksy14:03:44

you can’t be sure that all authors specify their deps in an edn file

mobileink14:03:46

@jethroksy: my use case was fetching webjars deps containing web resources and then unpacking them into resources/public/. once that's done they're not needed at runtime. having a show task for them shows any transitive deps and also allows multiple edn files - it just saves you the trouble of going through a bunch of files to see what deps were used during build.

alandipert14:03:37

i wonder about a boot shim

alandipert14:03:46

that hooks into boot at the maven/aether level and disallows downloading flagged jars

alandipert14:03:02

or maybe gives you a hook to log something

alandipert14:03:17

then their service could tell you things about deps dynamically

jethroksy14:03:34

in short it’s just something they’d have to write

mobileink14:03:36

yeah, edn files might not work for a general tool, you'd have to be able to deal with anything

jethroksy14:03:58

edn files are definitely great for personal use.

escherize14:03:19

the cljsfiddle thing is quite slow to load, no illusuion, @jethroksy

mobileink14:03:35

yeah, anywhere you have to luxury of setting the rules

jethroksy14:03:38

I guess if people expect to get stuff out of their service, then they should put in some effort to specify their deps

jethroksy14:03:09

I’m not even remotely familiar with what they do simple_smile

alandipert14:03:08

http://projars.com/ is me and micha's thing that's on a hiatus, but i'd be curious to hear if anyone's interested in it

alandipert14:03:43

the idea there was to do the analysis at the maven server level. so you'd point your boot projects to our private mirror, and then you'd have visibility into the deps your apps are bringing in across build/run times

mobileink14:03:43

sounds like the real problem is general: how to convert a leiningen project map (or similar for other build systems) into boot stuff. i'm inclined to think step one is to identify exactly when each dep is used, build time or run time, for dev, test, prod, etc. and then keep them segregated somehow. i have some projects i'm planning to convert so i'll soon find out.

mobileink14:03:56

this is kinda sorta related to supporting build "flavors" and "types" as used in the Android gradle plugin - I'm staring at it and thinking boot would make things so much simple.

jethroksy14:03:15

projars looks great

jethroksy14:03:31

but i’m years from the level where I can start writing libraries 😄

escherize14:03:08

aw c'mon, you can write a library

jethroksy14:03:56

i’m sure some sort of static analyses can be done to convert project.clj files to build.boot

jethroksy15:03:06

but I think one of the greatest joys come from writing build.boot on your own

jethroksy15:03:28

intimacy with build.boot is one of the key factors that bring clarity to the project

mobileink15:03:51

@alandipert: how would it work? is the idea that it could tell you what pods you have and what their deps are?

jethroksy15:03:40

I would write a library if there weren’t so many out there already

jethroksy15:03:16

I wanted to release my slack client as a library, but there’s more than enough out there

jethroksy15:03:40

dilution of the clojure ecosystem is a problem that both node and clojure are struggling with imo

mobileink15:03:02

more is better! just look at all the build systems out there. 😉

jethroksy15:03:06

I’d rather put in effort pushing a library forward than reinvent the wheel

jethroksy15:03:23

unless of course it offers a whole new perspective (;

dominicm15:03:42

I'm trying to figure out how I can parse a fileset, where the user says "I have my files in xs directory" I can't filter input-dirs by name, so I'm a little unsure how to map original name to the temp name.

escherize15:03:06

I've got a truly noob question for you guys

escherize15:03:24

I want to pass an option to boot cljs with -c

escherize15:03:38

i want to set static-fns to true

escherize15:03:57

boot cljs -c '{:static-fns true}' ?

dominicm15:03:20

This may not be possible actually, which is why boot-cljs uses those .edn files.

alandipert15:03:55

@mobileink: how would which part work? sorry, was in meeting

mobileink15:03:16

was wondering what projars adds to clojars etc. the analytics and auditing stuff mainly? going back to @jethroksy 's question "Is there a function that gets all dependencies, including those in pods?" - could projars tell me something like "your build uses n pods and here are their deps"?

mobileink15:03:54

that would be cool

alandipert15:03:47

@mobileink: oh - not quite. the main thing is clojars mirror for your company. the secondary things are reports. it couldn't show youa bout pods, but it could do like "the application asking for jars from this IP in this time period requested these jars, 2 of which we've flagged because of vulnerabilities - here are alternatives"

alandipert15:03:27

flagging things with no/incompatible licenses in their poms is another angle

mobileink15:03:50

that would be useful. but it would only happen at fetch time, correct? after that the resolution mechanism would not talk to projars since the jars would be available locally? so you'd look at some kind of report to see if you have problems, rather than be notified each time you build?

alandipert15:03:37

@mobileink: correct, tho it would be easy to blow the maven cache away on e.g. CI

alandipert15:03:21

altho that could have problems too. the lowest denominator is definitely the humble .pom

mobileink15:03:08

sounds promising.

mobileink15:03:43

a related question: how hard would it be to add some kind of -verbose option for pods, so that when you run one you get a classpath dump?

mobileink15:03:12

not that I'm in dire need of such a thing, but I like knowing exactly what's going on...

alandipert15:03:47

@mobileink: you mean like an option to make-pod that causes it to log or something?

alandipert15:03:27

@escherize: btw you're on a roll, keep crushing it my friend metal

mobileink15:03:37

something like that, not really sure. maybe it sets a switch so with-call-in etc. log something. you wouldn't want to use it much but it might be very useful for troubleshooting

mobileink15:03:06

esp. since pods are first class. i can imagine having a dynamically configured pod where i'm really not so sure what's in it. 😉

mobileink15:03:44

not to mention it would be great for learners

alandipert15:03:36

@mobileink: i see. i can definitely imagine maybe a library for inspecting pods, that could use the existing stuff for running code in pods... and the more recent stuff that lets you run code in any existing pod, even ones you didn't create

mobileink15:03:33

shall i open an issue? not a pressing issue for me right now but good to have on the list?

alandipert15:03:36

sure, maybe mark it help-wanted and it can be a plase to discuss

curtosis17:03:05

is there a recommended best-practice for handling profiles and synchronizing them with a project.clj? I want to put credentials in a (non-scm) file -- which I have working fine -- but I also want to pick them up in project.clj so Cursive can see them.

curtosis17:03:05

I should add that the goal is to not dump the credentials into the project.clj -- I'd like to be able to check that one into scm to simplify getting started for new devs.

curtosis17:03:23

(I also realize this somewhat raises the question of whether boot is the right solution for a Cursive-native project...)

jethroksy17:03:29

how do you "pick them up" in project.clj?

jethroksy17:03:50

boot's default file for adding stuff not committed to scm is profile.boot

jethroksy17:03:56

lein has its own conventions

martinklepsch17:03:01

@curtosis: project.clj can have slurp and def and other things before the defproject — maybe that already gets you the dynamic-ness you need?

jethroksy17:03:18

but you can always declare it in edn format in a separate file and slurp from boot

curtosis17:03:47

@jethrosky: I have a lein-generate task that writes a project.clj, and I can pull in :repositories

curtosis17:03:53

I set-sys-env! the credentials in profile.boot currently.

curtosis17:03:10

and then build the repo map in build.boot.

curtosis17:03:36

@martinklepsch: hmmm... if I change profile.boot to just def the credential vars, then I could write project.clj to load profile.boot first.

curtosis17:03:25

though the generate-lein-project-file! code (from the wiki) is a bit complex to rewrite to do that...

martinklepsch17:03:01

@curtosis: just pretending something would do it no?

martinklepsch17:03:21

Prepending might work better 😊

curtosis17:03:36

lol -- no, pretending works flawlessly every time! simple_smile

curtosis18:03:50

but yes, I can tack on a load-file and that seems to do the trick. thanks!

curtosis18:03:18

though it means I can't reuse the :repositories map -- boot has already loaded the creds

curtosis18:03:20

not too difficult, just assoc in new forms, of course. nice to have a real language instead of a dsl. simple_smile

curtosis18:03:45

when is the configure-repositories! callback actually run?

mobileink18:03:11

a general question before i start implementing something: i have a task that downloads bower packages, i want that to happen only once, and I want the downloaded stuff to be available across projects. the obvious thing to do is mimic maven: put the packages in e.g. ~/.bower/bower_components and then soft-link them into the project. i have the download bit working, but before i go on to add more logic, is this the Right Way to go about this? it does go outside of the fileset; is there a more bootish way to do this?

micha18:03:36

@mobileink: boot provides a persistent cache

micha18:03:39

that you can use

micha18:03:53

but they're not named places on the filesystem, they're still managed temp dirs

micha18:03:10

but tasks can store files in those dirs that persist across invocations of boot

micha18:03:28

there are global caches and per-project caches

mobileink18:03:38

even across projects?

micha18:03:43

the global caches are for things that can be cached across projects yes

micha18:03:50

like things a task would cache

micha18:03:09

that could be used by the same task but for any project

mobileink18:03:16

so in effect if i did this with bower i would be putting the bower repo somewhere in /.boot/cache instead of /.bower/bower_components?

micha18:03:29

boot.user=> (doc cache-dir!)
-------------------------
boot.core/cache-dir!
([key & {:keys [global]}])
  Returns a directory which is managed by boot but whose contents will not be
  deleted after the build is complete. The :global option specifies that the
  directory is shared by all projects. The default behavior returns different
  directories for the same key when run in different projects.

mobileink18:03:51

i like it. always wondered what the cache stuff was intended for, now i know.

mobileink18:03:23

glad i asked, i'll try it with caches...

curtosis19:03:02

argh. leiningen won't resolve variables in repo credentials!

curtosis19:03:14

broke down and write out a profiles.clj with the auth info. oh well.

micha20:03:28

you should be able to ~ the username and/or password you read from a file with that technique

mobileink20:03:53

cordwainer: somebody who makes shoes from new leather. cobbler: somebody who repairs old shoes. you cud google it!

mobileink20:03:45

last: a mechanical form that has the shape of a human foot. forget "template"! you make boots with lasts! ;)

micha20:03:21

hahahhahah

mobileink20:03:49

somewhere knuth said in so many words that metafont may be harmful because once you get the hang og typeface design it may take over your life. I'M LOOKING AT YOU, BOOT! I HAVE A FAMILY TO FEED! (not rilly, just a boss who's wondering if boot really is the greatest thing since sliced bread.)

micha20:03:47

boots are a delicious and nutritious substitute for food!

mobileink20:03:50

@micha get your marketing people on it!

alandipert20:03:19

i'm sure whatever your boss does could be factored into a few boot tasks

mobileink20:03:27

hahaha - my all time favorite s energy is Charlie Chaplin make a meal out of his boots. I think it was in Gold Rush. If you haven't seen it it's the funniest ever.

alandipert20:03:42

be sure to let him know that if he ever questions your tools boot-clj

alandipert20:03:27

ok probably not, and that's bad advice

micha20:03:56

i remember seeing that film

micha20:03:22

also the one where he's stuck in the machine cogs with just his head sticking out of the machine

micha20:03:38

and they funnel soup into his mouth at lunchtime via a chicken

mobileink20:03:26

my boss is a finance guy - his job is to disappoint, probably. you got a boot task for that we'll buy!

mobileink20:03:03

I do nah unnestand why Chaplin is not better known these days. I think he's the greatest artist os 20th cent.

mobileink20:03:14

I wonder what other famous boot episodes in history or mythology are out there. fun.

micha20:03:38

sword and sandal films

micha20:03:42

the worst

mobileink20:03:10

of course, if Achilles had had a decent pair of boots. ..

mobileink20:03:00

... he probably would have been cut down on day one. ;)

mobileink20:03:37

sorry for the silliness, started working about 3 am. now on my 3rd glass of vino.

mobileink20:03:36

forgot the rule: don't drink and slack

mobileink20:03:34

ok, in all seriousness, I really do think boot is about as close as we get to a killer app for clojure. I'm planning out (ok, that's an exaggeration) some blog posts about it. one thing I'd like to do is properly introduce the devs. who the heck are you guys, how the heck did you come up with this idea, when, where, etc.

mobileink20:03:51

are you game? if so we could so some kind interview privately or here on slack, which I'm sure others would like.

mobileink20:03:14

FWIW there no money involved, I'm not a professional journalist, just a hacker.

micha20:03:57

sure, no problem

mobileink20:03:28

I mean for my own purely selfish reasons I'd be happy to ask away. but boot is Serious. If you folks are willing to do some kind of q and a maybe we should announce it , at least on the clojure list.

mobileink20:03:13

I'll bet there are lots of people who would like to know more about it.

mobileink20:03:25

you do know you're about to become superstars, right? not that you're not already super, I'm just sayin'.

curtosis20:03:59

@micha: that would work fine for boot, but I need to put the creds somewhere leiningen can find them as well; lein only supports string literals and environment variables, and I couldn't figure a better way to do that than just writing the creds to a profiles.clj that can be excluded from scm.

micha20:03:28

hahaha alan is deep in some work stuff at the moment

micha20:03:47

@curtosis: that link i pasted is to the project.clj for boot/pod

micha20:03:02

it reads a properties file and injects the values from that into the project.clj

micha20:03:15

i was thinking that seems like exactly what you want

micha20:03:28

i've done the creds thing that way before i think

micha20:03:32

with leiningen

mobileink20:03:02

no prob, I'm about to get very busy but maybe sometime in the next few months.

mobileink21:03:26

meantime I promise I'll do some blog posts. you guys, well, playing hard to get is often a winning strategy. we all love a mystery.

micha21:03:25

sorry, i caused a bit of a commotion at work this morning, still cleaning up the damage

micha21:03:49

it was a rough day so far

mobileink21:03:17

it'll all be good when you hit vino #3. promise

micha21:03:30

i may go direct to the dirty gin martinis

mobileink21:03:54

liquor's quicker ;)

curtosis22:03:04

@micha: oh, I see -- the trick there is unquoting to get ~version inserted. I'll have to give that a shot; I know the :repositories stuff is a little finicky (special handling for the value :env etc.)

micha22:03:27

yeah i just did something like

micha22:03:03

:repositories [["foobar" :username ~user :passphrase ~pass] ...

curtosis22:03:07

excellent! thanks!