Fork me on GitHub
#boot
<
2016-03-03
>
onetom02:03:01

Finally we are trying deraen.boot-less. When I change a .less file, it gets recompiled nicely, BUT (cljs) also kicks in AND recompiles AND reloads my Hoplon app unnecessarily. Is there a simple way to get around this problem?

onetom02:03:01

My pipeline looks like: (comp (serve) (from-cljsjs) (sift) (watch) (less) (hoplon) (reload) (cljs) (speak))

alandipert03:03:56

anyone built up tools for working with webjars? i'm looking for a form of sift -j for exploding webjars deps into my resources

alandipert04:03:51

thusly computed:

(require '[boot.pod         :as pod]
         '[boot.core        :as core]
         '[boot.from.digest :as digest])

(core/deftask webjars
  "Unpack all webjars into the fileset."
  []
  (let [webjars (->> (pod/resolve-dependencies (core/get-env))
                     (filter #(= "org.webjars" (namespace (first (:dep %)))))
                     (map :jar))]
    (with-pre-wrap [fs]
      (reduce #(core/add-cached-resource
                %1
                (digest/md5 %2)
                (partial pod/unpack-jar %2)
                :include [#".*"]
                :mergers pod/standard-jar-mergers)
              fs
              webjars))))

seancorfield04:03:52

You know, it's s**t like this that makes me love Boot... It's all "just Clojure"...

seancorfield04:03:35

When I was writing boot-new there were several times where I was like "Wow, it was that easy?"...

onetom05:03:09

@alandipert: it's funny u mentioned this webjar explode function because i was just thinking about utilizing the org.webjars.bower/semantic-ui "2.1.8" instead of maintaining our own cljsjs package of it

alandipert05:03:28

yeah i'm working on emulating bower in boot basically

alandipert05:03:41

so we can use webjars and they get added to the classpath at /bower_components

alandipert05:03:04

definitely will share what i come up with

alandipert05:03:18

i'm doing it to more easily use polymer from hoplon

alandipert05:03:36

the idea with the move-webjars task is that it identifies "bower webjars" and groups their files under /bower_components/<project>/<project-file> in the fileset

alandipert05:03:15

anyway now i must sleep, will pick it up later. also feel free to run with it

escherize06:03:38

here, it says to add some boot tasks to my project: https://github.com/cljsjs/packages/tree/master/codemirror

escherize06:03:49

but, where does that go?

dominicm07:03:45

@alandipert: That's a neat idea. Most people in the js world consider bower "dead" though.

dominicm07:03:02

Npm is supposed to be where it's at now, even for css. Which spins my head still a little.

mitchelkuijpers09:03:56

Hi I have a question we recently moved our from leiningen to build. And our buildserver uses a SAN which seems to make boot very slow when creating a uberjar. Does anyone here has any ideas for creating some sort of uberjar with non unpaked jars and starting up with boot or something like that? Or is that a crazy idea?

mitchelkuijpers09:03:13

Our build went from 4 minutes to 11 that kind of sucks 😞

dm309:03:48

it can create a 'fat' jar of jars which will get unpacked when the application is run

dm309:03:05

no plugin for boot exists for that that I'm aware of though

mitchelkuijpers09:03:58

Aha but they do have a leinigen plugin, thnx @dm3 might be an idea to create a boot plugin myself

dm309:03:07

IIRC this reduced our build time from >10 min to <2 min

dm309:03:51

also you can trade faster AOT compilation for slower startup by creating an empty main_aot ns and delegating to the actual main

mitchelkuijpers10:03:48

@dm3: So you guys are using this already but without boot?

dm310:03:12

yes, that project is lein

dm310:03:39

but it amounts to building a MANIFEST file I think

mitchelkuijpers10:03:56

Aha ok I will check out the leiningen plugin

alandipert14:03:39

i wonder about adding :dirs as an alias

alandipert14:03:43

to the target task

dm314:03:44

@alandipert, just wondering, what do you guys use for ops? Provisioning, setting up servers, etc? Did you manage to sneak clojure into these processes too?

alandipert14:03:54

historically and still mostly today we use a mix of chef and AWS stuff but that's changing fast

alandipert14:03:28

micha has been working on a project to replace it all with a few small shell scripts

alandipert14:03:10

basically we're backing out of chef and most of the AWS stuff and falling back to haproxy, shell scripts, and EC2 API

alandipert14:03:12

i've been working on reporting and batch job type stuff and i've been making heavy use of lambda + clojure

alandipert14:03:59

oh a cool outcome of micha's work so far has been https://github.com/micha/json-table

alandipert14:03:14

we use that to gather information from AWS apis and process it in shell

dm314:03:06

interesting simple_smile thanks for the details

dm314:03:34

we're running on hosted hardware so a bit more involved

dm314:03:02

been using pallet, but I'm finding it very complicated, even after a year of using

alandipert14:03:15

yeah i used it way back, not a fan

nha14:03:16

What would you use now for a project from scratch ? (I used fabric in python at work, played a bit with ansible but none really convinced me)

alandipert14:03:06

it would depend on lots of things

nha14:03:52

Oh ? like I deploy on digital ocean droplets (but wouldn't mind moving to EC2)

nha14:03:06

And it is usually an uberjar

mobileink14:03:30

does the following return a new fileset? the doc is not clear on this. (-> fileset (boot/add-resource tmp-dir) boot/commit!))

nha14:03:37

(with pm2 at work, since I am the only one doing Clojure, and pm2 is used by all the other node.js servers)

alandipert14:03:42

@nha: are you on a team? also do you have more than one app deployed there?

alandipert14:03:50

@mobileink: correct, returns fileset

nha14:03:15

@alandipert: small team (2) and we have < 5 apps

nha14:03:54

but I would experiment on a personal project first anyway

mobileink15:03:32

but is it a new fileset? it looks like commit! mutates its argument, and the doc is not clear on add-resource: "Add the contents of the java.io.File dir to the fileset's resources." I assume it returns a new fileset, just like clojure.core/assoc etc.

alandipert15:03:34

i've found that all the approaches are the same until the number of people, churn, and apps involved exceeds some magic number

alandipert15:03:38

then they all seem to fall down

alandipert15:03:11

@mobileink: the add-resource part returns a new fileset

mobileink15:03:22

thought so, thanks

nha15:03:39

Interesting. Well I'm not there yet so I may as well just pick one ^^

alandipert15:03:49

@nha: yup! maybe a general thing to avoid is anything that claims to be one of comprehensive/declarative

alandipert15:03:17

regardless of scale, building your ops around programs you made and understand seems almost always better than building it around someone else's declarative framework

nha15:03:38

I tried chef and something else I can't remember. It was the worst.

alandipert15:03:38

which is odd because that's exactly what all the devops marketing says is terrible

nha15:03:16

Right, I fell into that at some point.

alandipert15:03:39

but yeah i suppose it's less that boot is helping us do ops

alandipert15:03:54

and maybe more that experience failing at ops w/ frameworks helped inform the boot principle of Programming First

nha15:03:54

Neat. Really neat. I remember doing makefiles, then gruntfiles then gulpfiles in the .js world, but because of mutability those files quickly ended up being somewhat messy.

alandipert15:03:02

yeah, i suppose being able to program assumes you have the right primitives to build on, the fileset for whatever the domain is

alandipert15:03:46

i think micha's ops scripts are really clean because they build on AMIs, ELBs, ASGs, and EC2 instances, primitives provided by aws

alandipert15:03:43

he has identified the ASG as the principle collection abstraction, a first-class way to name a set of instances

alandipert15:03:09

speaking of, i need to deploy some things, bbl

dm315:03:39

yep, you need to define some abstractions to work with

dm315:03:00

which various configuration management frameworks define for you

dm315:03:18

but mostly only allow to configure, not program

alandipert15:03:25

it sucks to use chef atop AWS, since the chef abstractions aren't the real ones

alandipert15:03:47

it's like the ORM dirty-check problem kinda

dm315:03:07

pallet is way ahead in that regard

dm315:03:18

however the way it executes the pipeline is crazy

alandipert15:03:24

in some way, pallet is a set of clojure functions yuo can use to drive whatever you want right?

alandipert15:03:32

i've definitely used it for the libraries it bundles

alandipert15:03:40

which was comprehensive last i checked

dm315:03:12

it still ends up pretty declarative although you seem to be writing normal clojure code

dm315:03:26

as everything is compiled to shell scripts and executed remotely

dm315:03:48

and your code operates inside an implicit session context

martinklepsch16:03:20

(deftask from-jars
  [i imports IMPORT #{[sym str str]} "Tuples describing imports: [jar-symbol path-in-jar target-path]"]
  (let [add-jar-args (into {} (for [[j p]  imports] [j (re-pattern (str "^" p "$"))]))
        move-args (into {} (for [[_ p t]  imports] [(re-pattern (str "^" p "$")) t]))]
    (util/dbug "Importing from jars: %s\n" (pr-str add-jar-args))
    (util/dbug "Moving into locations: %s\n" (pr-str move-args))
    (util/info "Importing %s files from jars...\n" (count imports))
    (sift :add-jar add-jar-args
          :move move-args)))

(deftask import-css []
  (comp (from-jars :imports #{['org.webjars.npm/basscss "META-INF/resources/webjars/basscss/8.0.1/css/basscss.css" "_bass.scss"]
                              ['org.webjars.npm/normalize.css "META-INF/resources/webjars/normalize.css/3.0.3/normalize.css" "_normalize.scss"]})))
@alandipert:

martinklepsch16:03:18

@alandipert: I move files into appropriate places and import them as partials in SASS/less liking it so far

alandipert16:03:58

yeah i got something similar kinda working for polymer "bower webjars" but there are just too many

alandipert16:03:28

so i need to do some computing to put all the jar contents in the right places

alandipert16:03:57

i guess the actual problem is i want to depend on webjars that have transitive dependencies that i also need to add to fs

martinklepsch16:03:59

one thing that's still annoying me is that webjars doesn't seem to build the deps of basscss (npm) and so I'd have to do all of these manually 😕

alandipert16:03:21

that is lame 😞

martinklepsch16:03:49

@juhoteperi mentioned that webjars should build dependencies but doesn't seem to for me

juhoteperi16:03:38

@martinklepsch: Only tested in once with a bower package I think, I thought it did package deps but not sure

mobileink17:03:54

hi folks. is there any way to build a pipeline dynamically in a task? i've been trying but I'm stumped. here's an abbreviated version:

mobileink17:03:57

(boot/deftask meta-config [p pkg-mgrs PGKMGRS #{kw} "package mgr keywords, :bower, :npm :polymer, :webjars; default is all"] (let [pkg-mgrs (if pkg-mgrs pkg-mgrs #{:bower :npm :polymer :webjars})] (for [pkg-mgr pkg-mgrs] (meta-config-impl :pkg-mgr pkg-mgr))))

mobileink17:03:46

meta-config-impl is a private task. this for loop does not work. it does work if I call meta-config-imple once at the top level

mobileink17:03:08

a recur loop to pass the fileset? is there a standard way to do this? some bit of clojure that's hiding from me, prolly.

juhoteperi17:03:12

Task must return a function. for returns a sequence.

mobileink17:03:06

yeah, that for loop is just to illustrate what i want to do - run the task once for each keyword, so it can default to running all of them

juhoteperi17:03:32

Just combine the functions with apply comp or such.

mobileink17:03:45

sorry for being dense, but I don't see how to get the args. generically, i have one function f that i want to compose, once for each of n args: (comp (f a) (f b) ...) but my n args are in a set. i can't see how to do it. not a boot-specific question i guess, maybe i'll try on the general mailing list...

mobileink18:03:26

in case anybody's wondering:

mobileink18:03:32

`(boot/deftask meta-config [p pkg-mgrs PGKMGRS #{kw} "package mgr keywords, :bower, :npm :polymer, :webjars; default is all"] (let [pkg-mgrs (if pkg-mgrs pkg-mgrs #{:bower :npm :polymer :webjars})] (apply comp (map #(meta-config-impl :pkg-mgr %) pkg-mgrs))))`

mobileink18:03:22

@alandipert: FYI you might find it worth a few minutes to look at the webjars support i've added to https://github.com/mobileink/boot-bowdlerize. Doco's out of date, but see the webjars testapp for an example, it's pretty straightforward. My approach is a bit different, the idea is that we use edn files (one per pkg mgr) to construct a master edn file (bowdlerize.edn) that gets passed through the pipeline. Installation is then a separate task that is driven by the master edn, not the pkg mgr edns. So to installing webjars looks like this: $ boot b/meta-config -o webjars b/install-webjars target Creating bowdlerize.edn Elaborating bowdlerize.edn with :webjars stanza Writing target dir(s)...`

mobileink18:03:54

installation unpacks the webjars into a global boot cache but first checks to see if they're already there and skips unpacking if so.

mobileink18:03:34

the nice thing also is that the config files can be extended and used in other tasks. man, i love boot!

mobileink18:03:13

fwiw the code is still a work in progress but I welcome any critique/comment/advice.

mobileink18:03:41

ps. an entry in webjars.edn looks like this: {polymer.paper {button [org.webjars.bower/github-com-PolymerElements-paper-button "1.0.11"] card [org.webjars.bower/github-com-PolymerElements-paper-card "1.0.8"]}}

mobileink18:03:41

pps. i'm not sure i'm using pods optimally. i'm at the stage where i do stuff just because i see its possible and i want to see how it works, like making a ->pkg-mgr-pod that makes a pod based on data in an edn file. very interested in advice as to how best to use pods...

alandipert19:03:11

@mobileink: that looks great! i'll dig in later

arnaudsj22:03:43

hi, has anobody already made a boot flavored npm module template for ClojureScript?