Fork me on GitHub
#boot
<
2016-06-09
>
micha00:06:26

that task stores the previous fileset in an atom

micha00:06:42

then the next time it runs it can compare against the new fileset

micha00:06:55

to see which source files need to be recompiled

mhr00:06:34

doesn't fileset-diff take two args?

micha00:06:34

the ones that do not need to be recompiled correspond to artifacts that are cached privately in the task, so it can add those to the fileset without doing the work to recompile them

micha00:06:46

yes it's using the ->> thread-last macro

micha00:06:03

so it's like (boot/fileset-diff @prev-fileset fileset) really

micha00:06:22

(->> foo (bar baz)) is same as (bar baz foo)

micha00:06:00

also (fileset-diff nil fileset) returns fileset

mhr00:06:11

aha, this looks just like I imagined it would be. when I tried to make the task myself, I also was going to use an atom, but I first asked here.

micha00:06:42

it's excellent that it's how you imagined 🙂

mhr00:06:58

so say that fileset-diff finds no difference

mhr00:06:09

what happens then? does the task return nil?

micha00:06:14

then that means that no files changed

mhr00:06:15

how do I make the pipeline stop?

micha00:06:27

you can just not call the next task handler

micha00:06:59

that's why we use the middleware pattern there

micha00:06:11

so your task can call the next task once, many times, or no times

mhr00:06:13

this is a brilliant design

mhr00:06:38

I'm going to go make my task, I'll come back if I get stuck

micha00:06:56

for performance reasons the fileset-diff keys on last modified time

micha00:06:15

but it is efficient to loop over that smaller set to compare hashes

micha00:06:41

and you don't even need to read the files, because the fileset organizes them by hash anyway

micha00:06:52

for example:

micha00:06:13

(->> fileset (fileset-diff @prev) (map :id))

micha00:06:25

that will give you all the md5 hashes of the files that changed

mhr00:06:25

is the state defined in a task persisted?

mhr00:06:05

if I define an atom referring to the previous fileset

micha00:06:27

@mhr: not automatically, you would do that yourself, like this: https://github.com/boot-clj/boot/wiki/Tasks#task-anatomy

micha00:06:43

see item #4 in that example

micha00:06:23

the outermost let binding is where you would allocate your task's persistent state, like your atom that will persist across pipeline runs

mhr00:06:46

so let defines a closure sort of environment?

micha00:06:51

exactly yes

mhr00:06:08

ah, and that's a feature of clojure in general?

micha00:06:48

so that closure is created when the task is constructed, which happens separately from when the handler it returns is composed with the next task

micha00:06:50

if that makes sense

micha00:06:58

for example in the repl you would do

mhr00:06:02

I saw the use of let when reading examples of tasks and assumed it was some sort of pattern with boot. I recalled from dabbling with common lisp that it defined local variables, but I didn't realize they stuck around

micha00:06:14

(boot (my-task :option1 "foo") (other-task))

micha00:06:30

boot doesn't add any new things to clojure really

mhr00:06:58

oh and because the next task is called from within that let environment, it has access to those variables defined!

micha00:06:12

"lexical binding"

micha00:06:32

meaning expressions "inside" that one can see them, but expressions "outside" cannot

micha00:06:56

those bindings are immutable, which is why we use the atom

micha00:06:17

so in the expression i pasted above

micha00:06:39

the subexpression (my-task :option1 "foo") is evaluated before the (boot ...) one

micha00:06:53

the boot function composes the tasks into a pipeline

micha00:06:55

and runs it

micha00:06:17

so your let binding is created before the tasks are composed

micha00:06:30

so it's the place to put your persistent state

micha00:06:08

deftask is just defn with some syntax sugar for the options parsing

micha00:06:41

fancy destructuring sort of

mhr00:06:04

it's clear it's a macro, I wasn't sure exactly what it did on top of defn though when I was studying examples

micha00:06:58

it does two things: it generates the argument parsing stuff for the command line etc., and it adds metadata so that boot can list all the available tasks

micha00:06:25

but what you get is a regular clojure function

micha00:06:30

a little farther down in that wiki page is the thing you're doing: "control tasks" is the heading

micha00:06:57

that will show the extra things you'd need to do to update boot's state when you call the next handler multiple times etc

micha00:06:05

and make a new fileset

mhr00:06:51

why doesn't reset-fileset have bang notation to denote side effects? does it not have side effects?

mhr00:06:08

or oh, it's just passed on to the next function, I see. never mind

micha00:06:29

yeah it's just constructing a new fileset, which is immutable

micha00:06:39

commit! is wher ethe side effects happen

micha00:06:08

that's where you sync the immutable fileset object with the temp dirs boot maintains in the classpath

micha00:06:42

like when you call reset-fileset the classpath is still unchanged

micha00:06:10

but when you call commit! on it then it's updating the classpath and so on

micha00:06:59

you want to call commit! before passing the fileset on to the next task, because the next task will expect that the fileset it gets reflects the current JVM and filesystem configuration

mhr01:06:14

mapping :id to a fileset returns a set nils for me 😕

micha01:06:49

ah sorry, my bad

micha01:06:20

@mhr: to get a seq of the files in the fileset you can use boot.core/output-files or boot.core/input-files

micha01:06:36

the fileset object itself has a :tree field

micha01:06:01

which is a clojure map of path (relative to the classpath) => TmpFile object

micha01:06:24

the TmpFile object has the :id field, which is the md5 hash of the file's contents

micha01:06:57

there is also boot.core/ls which will give you a set of all the files in the filest

micha01:06:04

all the TmpFile objects

micha01:06:08

so like this:

micha01:06:25

(->> fileset ls (map :id))

micha01:06:38

sorry it's coming back to me now

richiardiandrea02:06:17

@micha cool stuff, I followed the conversation and good job with this task! It also shows once more how powerful boot is/can be.

mhr02:06:42

what does fileset-diff actually return? because

(println
                    (->> fileset
                        (fileset-diff @prev-fileset)
                        ls
                        (map :id)))
will print a file's id when I save it without changing, not only when I make a change and save it.

seancorfield02:06:25

@micha: Could you push a version with the task exposed so folks can just do boot -d org.clojars.micha/boot-cp some-context-task with-cp -o cp.edn

seancorfield02:06:58

It just needs {:boot/export-tasks true} right?

micha02:06:40

@seancorfield: ah should work now, thanks

micha02:06:59

i just pushed a new version with the correction you wrote there

seancorfield02:06:36

Hmm, I got an empty classpath file… odd...

seancorfield02:06:05

boot -d org.clojars.micha/boot-cp with-cp -o cp.edn => empty cp.edn file...

micha02:06:20

did you set dependencies?

micha02:06:28

the -d option?

seancorfield02:06:46

how about boot -d org.clojars.micha/boot-cp worldsingles with-cp -o cp.edn where the worldsingles task loads all the dependencies?

seancorfield02:06:56

That still produces an empty file.

micha02:06:09

right, the dependencies are passed as an option to the with-cp task

micha02:06:13

if you want to write the file

micha02:06:37

because presumably you wouldn't want to load them via set-env! since that would preclude using the -i option

seancorfield02:06:42

Oh, it won’t pull them from the environment?

micha02:06:01

like this:

micha02:06:04

boot -d org.clojars.micha/boot-cp with-cp -d '[[ring "1.4.0"]]' -l lib -o cp

seancorfield02:06:20

Not really practical for how we deal with dependencies

richiardiandrea02:06:42

probably the file input is more practical yes

seancorfield02:06:03

I thought this was the first step to what we discussed earlier about a task that produces a classpath without the Boot / system dependencies stuff. Sorry, misunderstood.

micha02:06:19

this would do that, no?

micha02:06:27

it produces a classpath of whatever you like

micha02:06:50

it only includes jars that you specified, or their dependencies

seancorfield02:06:59

We dynamically load dep coords from a series of EDN files and then munge the versions based on a .properties file.

seancorfield02:06:27

Right now we hack the classpath for loading all that by using show -C and and translating the : to newline.

seancorfield02:06:40

boot worldsingles show -C|tr : '\n'|wc => 135 lines

seancorfield02:06:09

But that includes JDK paths and all Boot’s own JAR files.

micha02:06:51

yeah i will fix that

seancorfield02:06:47

It’s workable… but kinda icky 🙂 But World Singles’ CFML web apps totally depend on it. That’s how they figure out all the libraries and Clojure source code to load into the web app at startup (since we still do a source deploy right now — because that’s pretty much forced on us by CFML anyway — and it has benefits: we can deploy just one updated file and reload the code).

seancorfield03:06:23

Ultimately we’ll move away from it. As we’re starting to create more standalone Clojure apps, we’ve moving to JAR-based deployments (finally!).

seancorfield03:06:20

Although, to be honest, having a build.boot file full of tasks that we can easily use from cron jobs etc and as part of our general build / deploy process...

seancorfield03:06:40

…that’s just golden (and why we switched from Leiningen to Boot six months ago).

micha03:06:09

we have a ton of clojure scripts we use for various admin tasks at adzerk now

seancorfield03:06:16

OK, other stuff to do… time to feed the cats, for example 🙂

seancorfield03:06:46

We’ve almost completely replaced Ant with Boot / Clojure. Very exciting for us. No more XML!

micha03:06:04

haha 👍

micha04:06:05

@seancorfield: it now defaults to :dependencies if you don't use the -d option

micha04:06:32

so what you did above should work

seancorfield04:06:55

Cool... I'll try that out...

seancorfield05:06:36

Ah, it's fussy about dependency conflicts 🙂

seancorfield05:06:07

I'll have to tidy things up in our dependency files. We have conflicts but show -p says they all resolve the right way (and Boot doesn't warn about the overrides) so we haven't bothered adding all the exclusions yet...

mhr06:06:52

@micha, since fileset-diff wasn't doing what I thought it should, I decided to roll my own implementation using the hash of the file contents, and I think it works! I figure this was a good opportunity to get my hands dirty and learn some Clojure. Now my concern has shifted. Is this idiomatic?

(deftask diff
    "Call next handler when file contents differ from before"
    []
    (let [prev-ids (atom (vec []))]
        (fn [next-task]
            (fn [fileset]
                (defn remove-period [ids] (get (split ids #"\.") 0))
                (def ids (->> fileset ls (map :id) vec))
                (cond (not= (set (map remove-period ids)) (set (map remove-period @prev-ids)))
                    (next-task fileset))
                (reset! prev-ids ids)))))

mhr06:06:38

I have diff coming after watch in the pipeline. So watch checks for saved files, and diff further filters

dm310:06:50

defn and def inside the function is not idiomatic

dm310:06:01

also cond with one clause is really an if

dm310:06:49

or rather when in this case, as there's no else

dm310:06:05

otherwise it's a nice task! 🙂

dm310:06:42

(fn [fileset]
          (let [remove-period #(first (split % #"\."))
                ids (->> fileset ls (map :id) vec))
             (when (not= (set (map remove-period ids)) (set (map remove-period @prev-ids)))
                 (next-task fileset))
             (reset! prev-ids ids)))))

dm312:06:09

did anyone have a need to deploy multiple artifacts for the same groupId/artifactId?

dm313:06:57

like what's done by

mvn deploy:deploy-file
-DgroupId=com.soebes.test
-DartifactId=x1
-Dversion=2.7.5-SNAPSHOT
-Dfile=TheMainArtifact.jar
-Dclassifiers=first,second
-Dfiles=firstFile,secondFile
-Dtypes=zip,xml
-DrepositoryId=RepositoryId 
-Durl=URLOfTheRepository

micha13:06:12

@dm3: i'd like to support this, but i'm not familiar enough with how it works

micha13:06:49

PRs welcome if you think of a good way to implement this 🙂

dm313:06:15

I'm yet to check how it stuffs all that information into a single pom file

micha13:06:13

yeah the pom file is the unifying abstraction, i think

dm313:06:57

then I also have

(defmacro pipelines
  "Run pipelines in separate boot cores. Pipeline commands can be expressed
  either as Clojure forms or as strings, e.g.:

    (pipelines
      (bump-version :release \"1.0.0\")
      (comp (pom) (jar) (push)))

  equivalent to:

    (pipelines
      \"bump-version --release 1.0.0\"
      \"pom -- jar -- push\")"

dm313:06:02

is that of interest?

dm313:06:45

it's doing reverse parsing of tasks into strings

micha13:06:01

wow interesting

micha13:06:25

these are running in separate pods?

dm313:06:46

(defn pipelines* [commands]
  (par/runcommands :commands commands :batches 1))

(defmacro pipelines
  "Run pipelines in separate boot cores. Pipeline commands can be expressed
  either as Clojure forms or as strings, e.g.:

    (pipelines
      (bump-version :release \"1.0.0\")
      (comp (pom) (jar) (push)))

  equivalent to:

    (pipelines
      \"bump-version --release 1.0.0\"
      \"pom -- jar -- push\")"
  [& commands]
  `(pipelines* (mapv compile ~commands)))

dm313:06:54

in boot.parallel

micha13:06:25

we could add something to the built-in install and push tasks to accomodate some kind of artifact map, btw

micha13:06:18

the machinery underneath the task is ready for it, but i couldn't think of a good way to implement it other than just a blob of data, which doesn't seem very nice

micha13:06:47

and not open, in that if you provide a blob of data to the task then other tasks can't know about it

micha13:06:47

the current system uses the pom as the data that describes the artifacts, which is nice because you can have any number of tasks that read and/or modify the pom

micha13:06:56

before the install/push task is run

micha13:06:30

so you can have other tasks to automate the configuration for you by modifying the pom

dm313:06:41

I agree with that

micha13:06:22

i feel like there must be an elegant way to support the multi-artifact maven deploy

micha15:06:22

@seancorfield: refactored the thing so it should work for your use case better

richhickey16:06:49

@micha thanks, I’ll look at boot-cp

seancorfield16:06:00

@micha: Nice! A clean classpath with none of the Boot or Java system stuff on!

seancorfield16:06:23

Are you considering merging that into the show task in some form?

seancorfield16:06:40

or making with-cp a build-in task?

wamaral17:06:45

hey guys, I'm new to boot, need help to debug this error I'm getting: java.lang.NoClassDefFoundError: boot/App

wamaral17:06:59

let me find somewhere to paste my build.boot

wamaral17:06:08

it actually builds, but I get that error when trying to run the uberjar

mhr17:06:37

now that I have my task, how can I make it available to future programs without having to copy and paste it to build.boot in each of my projects?

richiardiandrea17:06:39

@wamaral: it looks like scope provided is the cause

richiardiandrea17:06:56

for boot.core, ops @micha go ahead you know better 😉

wamaral17:06:38

humm, good point, let me try

micha17:06:15

@mhr you can define your task in a namespace and package it as a jar on clojars

micha17:06:28

you can use this as a template: https://github.com/micha/boot-cp

wamaral17:06:01

still same error :(

richiardiandrea17:06:56

does it say who is causing it?

micha17:06:00

@wamaral: are you trying to use boot namespaces in your uberjar?

micha17:06:19

@wamaral: does report.core try to :require boot.core?

wamaral17:06:20

@micha: I defined a task with deftask inside my application's core

micha17:06:31

ah that is the cause

wamaral17:06:39

yes, I have boot.core :refer [deftask]

wamaral17:06:57

can't do that? :(

micha17:06:01

that can only be run when the JVM is launched via the boot shim

micha17:06:30

hm no sorry

micha17:06:37

that's a different thing

wamaral17:06:36

hmm I see... I tried doing that to start the webserver defined in core, while in development

wamaral17:06:46

but I didn't want to require too many things in build.boot

wamaral17:06:51

guess I'll have to :)

wamaral17:06:03

yup, got it to work again

richiardiandrea19:06:45

just discovered this great plugin for lein: https://github.com/webnf/lein-collisions do we have something like this in boot?

seancorfield19:06:45

Looks easy enough to rebuild as a Boot task… The only thing it seems to rely on is get-classpath from Leiningen.

richiardiandrea19:06:09

yes true, but now we have boot-cp 😉

seancorfield19:06:59

So send @micha a Pull Request to add a collisions check to boot-cp 😆

richiardiandrea19:06:17

(!> tasks-chan "send Micha a PR") 😄

richhickey20:06:23

@micha trying out boot-cp, I’m wondering how do I resolve a conflict? I presumed supplying a specific version of a dep would win, e.g. core.async has dep on old clojure version:

micha20:06:34

@richhickey: ah sorry, that's a bug, there is a filtering step i forgot to add, where it filters out conflicts that were fixed by overrides like what you show

micha20:06:12

i used a function that's normally used for the show -p task

richhickey20:06:02

when accepting edn args does boot have any facilities for getting from file, vs the cat thing I had to do above?

micha20:06:37

the task could be modified to do that

micha20:06:55

you can also specify it in the build.boot file

micha20:06:57

if you have one

micha20:06:41

(task-options!
  micha.boot-cp/with-cp
  {:dependnecies '[...]})

micha20:06:04

that "curries" the task fn

micha20:06:41

you can override that on the command line or in the repl, as kwargs are overridden

richhickey20:06:47

I saw that in the readme, but as I said I consider the deps list to be data and would rather not have to replicate a boilerplate ‘program’ like this in every project. I’m trying to see if I can get a set of ambient boot tooling that, once present, would have me making only a deps.edn in each project

richhickey20:06:16

at least for simple things

micha20:06:27

will you have a build.boot file at all?

richhickey20:06:39

not for simple things

micha20:06:12

are you okay with loading a task via boot's -d option?

richhickey20:06:22

I’d rather have shared utilities in profile.boot

micha20:06:31

ah okay, then we can program in there

micha20:06:42

so everything is easy

richhickey20:06:09

right, I have this in there now:

micha20:06:11

in profile.boot you can slurp "deps.edn" and do the task-options! above

micha20:06:04

you can add to that:

richhickey20:06:12

profile.boot gets run for all uses of boot though, right?

micha20:06:32

yes, you can see exactly what it does with boot -vb task task

richhickey20:06:46

so one thing to map code i nthere, another to do work (which might fail for other uses)

micha20:06:59

that will print an annotated version of the generated boot.user ns that boot evaluates

micha20:06:44

you can have default configurations in profile.boot

micha20:06:49

like this perhaps:

micha20:06:55

(require '[ :as io])
(let [depsfile (io/file "deps.edn")]
  (when (.exists depsfile)
    (task-options!
      with-cp {:dependencies (read-string (slurp depsfile))})))

micha20:06:27

then when you're in a project dir with a deps.edn present

micha20:06:45

you can do boot with-cp -wf cp.out

micha20:06:15

you could also override the deps on the command line still

seancorfield20:06:50

Thanks for boot-cp — just used it to bludgeon all of our dependency conflicts into submission! Had to add a lot of :exclusions and nearly all of them were for [org.clojure/clojure] — which leads me to wonder why every project doesn’t use "provided" as the scope for Clojure itself, so it doesn’t end up as a transitive dependency?!?! 😠

richhickey21:06:23

got it. I was wondering though about the other uses of with-cp, since boot has already started the jvm and clojure, and deps will usually have clojure in them, what will aether do with second/maybe-conflicting clojure? with-cp is still using aether to load deps right?

seancorfield21:06:24

I ended up having :exclusions [org.clojure/clojure] on every single contrib library dependency in the end.

micha21:06:36

@seancorfield: the thing it's really doing is calling boot.pod/resolve-dependency-jars, which you can use in your own tasks

seancorfield21:06:05

We may well add a step to our build/test process that "breaks the build" if there are conflicts.

richhickey21:06:55

@seancorfield: absolutely I prefer that as the default

richhickey21:06:19

automatic conflict resolution == problems

richhickey21:06:50

a maven misfeature IMO

seancorfield21:06:59

I’m beginning to agree with that position (I thought it was unnecessarily extreme until the last few days taught me otherwise — with Clojure 1.9.0 Alpha 5 🙂 )

richhickey21:06:06

so @micha I think the mode is not -pedantic but -safe

micha21:06:28

i like it 🙂

micha21:06:41

@richhickey: with-cp isn't using aether when you use --read, it just adds JARs directly to the URLClassloader

micha21:06:25

you can resolve dependencies with aether without adding them to the classpath. boot has functions to help with that, functions that return clojure data

micha21:06:00

like in boot-cp, it resolves the dependencies using maven and then extracts from that data just a list of jar files

richhickey21:06:38

cool, but clojure will still end up in the mix twice though, right?

richhickey21:06:10

@micha that slurp strategy works great btw, thanks, and for with-cp!

micha21:06:05

yeah we really need a way to safely patch the classpath

seancorfield21:06:14

@richhickey: As a philosophical issue, given the above, do you think that Clojure libraries — including contrib libs — should ensure that Clojure itself doesn’t become a transitive dependency? i.e., they should mark org.clojure/clojure as "provided"?

micha21:06:37

we can do it by keeping track of all loaded dependencies, and not allowing new dependencies to be loaded with different versions

richhickey21:06:56

@micha that seems necessary to avoid a mess

micha21:06:10

yes, definitely

richhickey21:06:21

that’s what I’m always afraid of with aether

richhickey21:06:41

but any dynamic loading is subject to these problems

micha21:06:30

if you only add dependencies by jar URL i think it's safe

micha21:06:17

because you get exactly what you ask for there, with no aether to do unpredictable things

richhickey21:06:26

so rockin here with minimal setup - make a src dir and a deps.edn, call boot with-cp -w and launch repl with -cp

richhickey21:06:05

@micha: yes, much better when manual URLs vs aether, you can control given the check you describe above

richhickey21:06:55

I will end up making a real clojure repl task though so it can follow with-cp

micha21:06:09

you can use boot.pod/resolve-dependency-jars to get the list of jars, and you can then use boot.pod/add-classpath to add the jars to the cp

micha21:06:17

like if you wanted to do this in the repl or something

seancorfield21:06:25

FWIW, I just added this task to our build.boot file:

(deftask check-conflicts
  "Verify there are no dependency conflicts."
  []
  (with-pass-thru fs
    (require '[boot.pedantic :as pedant])
    (let [dep-conflicts (resolve 'pedant/dep-conflicts)]
      (if-let [conflicts (not-empty (dep-conflicts pod/env))]
        (throw (ex-info (str "Unresolved dependency conflicts. "
                             "Use :exclusions to resolve them!")
                        conflicts))
        (println "\nVerified there are no dependency conflicts.")))))

richhickey21:06:38

@seancorfield: I’d rather tools get better/safer than try to get everyone to be consistent

seancorfield21:06:39

Fair enough. I definitely think I’ll start marking org.clojure/clojure as a "provided" dependency in my projects from now on tho’, based on the pain I just went through today! 😸

richiardiandrea21:06:44

@micha sorry to barge in, doesn't set-env! :exclusions already take care of removing the deps globally? just asking confirmation because it is what I am doing in order to be sure to have only the one org.clojure/clojure coming from :dependencies in the classpath

micha21:06:54

@richiardiandrea: yes, that just adds :exclusions to all of your individual dependencies when it resolves

micha21:06:13

it does exactly (or should do) what you said

micha21:06:40

@richhickey: i am unsure of how to implement the clojure jar url scheme in boot

micha21:06:25

because of the way maven coordinates are normally used to specify dependencies, and the direct jar file setting wouldn't necessarily have a version associated with it

micha21:06:08

i was thinking maybe if you have BOOT_CLOJURE_VERSION=<file://home/me/work/clojure.jar> set then it would know to load the jar directly

micha21:06:00

that would make it consistent, in that you could use the existing machinery to starts pods with that or any other version of clojure, etc

seancorfield21:06:30

Thanks for that insight @richiardiandrea — I had no idea about that… would have saved me a lot of pain 😈

micha21:06:55

@richiardiandrea: i rarely use :exclusions because usually i would add a direct dependency on specific version of the thing i want, rather than excluding all versions of a thing i don't want

richiardiandrea21:06:13

@seancorfield: I actually discovered it recently myself (and updated the wiki yesterday)

seancorfield21:06:29

For org.clojure/clojure tho’, it makes a lot of sense — since pretty much every project drags it in as a transitive dependency.

micha21:06:54

and sometimes there is a dependency that has a ton of unnecessary transitive dependencies that i don't want on the classpath, but then i add :exclusions to that specific dependency, not globally

richiardiandrea21:06:10

Yesterday I discovered (not a big surprise I guess) that some clj/cljs/cljc library brings in ClojureScript and the Google Closure compiler so that's why I used global :exclusions...of course depends on your use case but it is good to have it

micha21:06:44

ah and you don't want closure or cljs in all pods

richiardiandrea21:06:00

In general it should never go in your uberjar as it is used only for compiling right?

micha21:06:00

that's a good use case for the global :exclusions

richiardiandrea21:06:10

So yes and it filters also transitive deps which is exactly what you want for the case above

micha21:06:40

yeah it's completely ignored by aether then

richhickey21:06:52

@micha does that syntax work now? BOOT_CLOJURE_VERSION=

micha21:06:25

@richhickey: no, currently it represents the maven version

richhickey21:06:37

that’s what I thought