@mhr: https://github.com/hoplon/boot-hoplon/blob/master/src/hoplon/boot_hoplon.clj#L131

that task stores the previous fileset in an atom

then the next time it runs it can compare against the new fileset

to see which source files need to be recompiled

doesn't fileset-diff take two args?

the ones that do not need to be recompiled correspond to artifacts that are cached privately in the task, so it can add those to the fileset without doing the work to recompile them

yes it's using the ->> thread-last macro

so it's like (boot/fileset-diff @prev-fileset fileset) really

ah

(->> foo (bar baz)) is same as (bar baz foo)

also (fileset-diff nil fileset) returns fileset

aha, this looks just like I imagined it would be. when I tried to make the task myself, I also was going to use an atom, but I first asked here.

it's excellent that it's how you imagined 🙂

so say that fileset-diff finds no difference

what happens then? does the task return nil?

then that means that no files changed

how do I make the pipeline stop?

you can just not call the next task handler

nice

that's why we use the middleware pattern there

so your task can call the next task once, many times, or no times

this is a brilliant design

I'm going to go make my task, I'll come back if I get stuck

for performance reasons the fileset-diff keys on last modified time

but it is efficient to loop over that smaller set to compare hashes

and you don't even need to read the files, because the fileset organizes them by hash anyway

for example:

(->> fileset (fileset-diff @prev) (map :id))

that will give you all the md5 hashes of the files that changed

is the state defined in a task persisted?

if I define an atom referring to the previous fileset

@mhr: not automatically, you would do that yourself, like this: https://github.com/boot-clj/boot/wiki/Tasks#task-anatomy

see item #4 in that example

the outermost let binding is where you would allocate your task's persistent state, like your atom that will persist across pipeline runs

so let defines a closure sort of environment?

exactly yes

ah, and that's a feature of clojure in general?

yes

so that closure is created when the task is constructed, which happens separately from when the handler it returns is composed with the next task

if that makes sense

for example in the repl you would do

I saw the use of let when reading examples of tasks and assumed it was some sort of pattern with boot. I recalled from dabbling with common lisp that it defined local variables, but I didn't realize they stuck around

(boot (my-task :option1 "foo") (other-task))

boot doesn't add any new things to clojure really

oh and because the next task is called from within that let environment, it has access to those variables defined!

exactly

"lexical binding"

aha

meaning expressions "inside" that one can see them, but expressions "outside" cannot

right

those bindings are immutable, which is why we use the atom

so in the expression i pasted above

the subexpression (my-task :option1 "foo") is evaluated before the (boot ...) one

the boot function composes the tasks into a pipeline

and runs it

I see

so your let binding is created before the tasks are composed

so it's the place to put your persistent state

deftask is just defn with some syntax sugar for the options parsing

fancy destructuring sort of

it's clear it's a macro, I wasn't sure exactly what it did on top of defn though when I was studying examples

it does two things: it generates the argument parsing stuff for the command line etc., and it adds metadata so that boot can list all the available tasks

but what you get is a regular clojure function

neat

a little farther down in that wiki page is the thing you're doing: "control tasks" is the heading

that will show the extra things you'd need to do to update boot's state when you call the next handler multiple times etc

and make a new fileset

why doesn't reset-fileset have bang notation to denote side effects? does it not have side effects?

or oh, it's just passed on to the next function, I see. never mind

yeah it's just constructing a new fileset, which is immutable

commit! is wher ethe side effects happen

that's where you sync the immutable fileset object with the temp dirs boot maintains in the classpath

like when you call reset-fileset the classpath is still unchanged

but when you call commit! on it then it's updating the classpath and so on

you want to call commit! before passing the fileset on to the next task, because the next task will expect that the fileset it gets reflects the current JVM and filesystem configuration

mapping :id to a fileset returns a set nils for me 😕

ah sorry, my bad

@mhr: to get a seq of the files in the fileset you can use boot.core/output-files or boot.core/input-files

the fileset object itself has a :tree field

which is a clojure map of path (relative to the classpath) => TmpFile object

the TmpFile object has the :id field, which is the md5 hash of the file's contents

there is also boot.core/ls which will give you a set of all the files in the filest

all the TmpFile objects

so like this:

(->> fileset ls (map :id))

sorry it's coming back to me now

lol

@richhickey: https://github.com/micha/boot-cp

@micha cool stuff, I followed the conversation and good job with this task! It also shows once more how powerful boot is/can be.

@richiardiandrea: thanks 🙂

what does fileset-diff actually return? because

(println
                    (->> fileset
                        (fileset-diff @prev-fileset)
                        ls
                        (map :id)))

@micha: Could you push a version with the task exposed so folks can just do boot -d org.clojars.micha/boot-cp some-context-task with-cp -o cp.edn

It just needs {:boot/export-tasks true} right?

@seancorfield: ah should work now, thanks

i just pushed a new version with the correction you wrote there

Hmm, I got an empty classpath file… odd...

boot -d org.clojars.micha/boot-cp with-cp -o cp.edn => empty cp.edn file...

did you set dependencies?

the -d option?

how about boot -d org.clojars.micha/boot-cp worldsingles with-cp -o cp.edn where the worldsingles task loads all the dependencies?

That still produces an empty file.

right, the dependencies are passed as an option to the with-cp task

if you want to write the file

because presumably you wouldn't want to load them via set-env! since that would preclude using the -i option

Oh, it won’t pull them from the environment?

like this:

boot -d org.clojars.micha/boot-cp with-cp -d '[[ring "1.4.0"]]' -l lib -o cp

Not really practical for how we deal with dependencies

probably the file input is more practical yes

I thought this was the first step to what we discussed earlier about a task that produces a classpath without the Boot / system dependencies stuff. Sorry, misunderstood.

this would do that, no?

it produces a classpath of whatever you like

it only includes jars that you specified, or their dependencies

We dynamically load dep coords from a series of EDN files and then munge the versions based on a .properties file.

ah

Right now we hack the classpath for loading all that by using show -C and and translating the : to newline.

boot worldsingles show -C|tr : '\n'|wc => 135 lines

But that includes JDK paths and all Boot’s own JAR files.

yeah i will fix that

OK, cool.

It’s workable… but kinda icky 🙂 But World Singles’ CFML web apps totally depend on it. That’s how they figure out all the libraries and Clojure source code to load into the web app at startup (since we still do a source deploy right now — because that’s pretty much forced on us by CFML anyway — and it has benefits: we can deploy just one updated file and reload the code).

Ultimately we’ll move away from it. As we’re starting to create more standalone Clojure apps, we’ve moving to JAR-based deployments (finally!).

Although, to be honest, having a build.boot file full of tasks that we can easily use from cron jobs etc and as part of our general build / deploy process...

…that’s just golden (and why we switched from Leiningen to Boot six months ago).

we have a ton of clojure scripts we use for various admin tasks at adzerk now

OK, other stuff to do… time to feed the cats, for example 🙂

We’ve almost completely replaced Ant with Boot / Clojure. Very exciting for us. No more XML!

haha 👍

@seancorfield: it now defaults to :dependencies if you don't use the -d option

so what you did above should work

Cool... I'll try that out...

Ah, it's fussy about dependency conflicts 🙂

I'll have to tidy things up in our dependency files. We have conflicts but show -p says they all resolve the right way (and Boot doesn't warn about the overrides) so we haven't bothered adding all the exclusions yet...

@micha, since fileset-diff wasn't doing what I thought it should, I decided to roll my own implementation using the hash of the file contents, and I think it works! I figure this was a good opportunity to get my hands dirty and learn some Clojure. Now my concern has shifted. Is this idiomatic?

(deftask diff
    "Call next handler when file contents differ from before"
    []
    (let [prev-ids (atom (vec []))]
        (fn [next-task]
            (fn [fileset]
                (defn remove-period [ids] (get (split ids #"\.") 0))
                (def ids (->> fileset ls (map :id) vec))
                (cond (not= (set (map remove-period ids)) (set (map remove-period @prev-ids)))
                    (next-task fileset))
                (reset! prev-ids ids)))))

I have diff coming after watch in the pipeline. So watch checks for saved files, and diff further filters

defn and def inside the function is not idiomatic

also cond with one clause is really an if

or rather when in this case, as there's no else

otherwise it's a nice task! 🙂

(fn [fileset]
          (let [remove-period #(first (split % #"\."))
                ids (->> fileset ls (map :id) vec))
             (when (not= (set (map remove-period ids)) (set (map remove-period @prev-ids)))
                 (next-task fileset))
             (reset! prev-ids ids)))))

did anyone have a need to deploy multiple artifacts for the same groupId/artifactId?

like what's done by

mvn deploy:deploy-file
-DgroupId=com.soebes.test
-DartifactId=x1
-Dversion=2.7.5-SNAPSHOT
-Dfile=TheMainArtifact.jar
-Dclassifiers=first,second
-Dfiles=firstFile,secondFile
-Dtypes=zip,xml
-DrepositoryId=RepositoryId 
-Durl=URLOfTheRepository

@dm3: i'd like to support this, but i'm not familiar enough with how it works

@dm3: there is support for classifiers and packaging: see https://github.com/boot-clj/boot/blob/master/boot/aether/src/boot/aether.clj#L298-L319

PRs welcome if you think of a good way to implement this 🙂

ok

I'm yet to check how it stuffs all that information into a single pom file

yeah the pom file is the unifying abstraction, i think

then I also have

(defmacro pipelines
  "Run pipelines in separate boot cores. Pipeline commands can be expressed
  either as Clojure forms or as strings, e.g.:

    (pipelines
      (bump-version :release \"1.0.0\")
      (comp (pom) (jar) (push)))

  equivalent to:

    (pipelines
      \"bump-version --release 1.0.0\"
      \"pom -- jar -- push\")"

is that of interest?

it's doing reverse parsing of tasks into strings

wow interesting

these are running in separate pods?

(defn pipelines* [commands]
  (par/runcommands :commands commands :batches 1))

(defmacro pipelines
  "Run pipelines in separate boot cores. Pipeline commands can be expressed
  either as Clojure forms or as strings, e.g.:

    (pipelines
      (bump-version :release \"1.0.0\")
      (comp (pom) (jar) (push)))

  equivalent to:

    (pipelines
      \"bump-version --release 1.0.0\"
      \"pom -- jar -- push\")"
  [& commands]
  `(pipelines* (mapv compile ~commands)))

in boot.parallel

ah i see

we could add something to the built-in install and push tasks to accomodate some kind of artifact map, btw

the machinery underneath the task is ready for it, but i couldn't think of a good way to implement it other than just a blob of data, which doesn't seem very nice

and not open, in that if you provide a blob of data to the task then other tasks can't know about it

the current system uses the pom as the data that describes the artifacts, which is nice because you can have any number of tasks that read and/or modify the pom

before the install/push task is run

so you can have other tasks to automate the configuration for you by modifying the pom

I agree with that

i feel like there must be an elegant way to support the multi-artifact maven deploy

@seancorfield: refactored the thing so it should work for your use case better

@micha thanks, I’ll look at boot-cp

@micha: Nice! A clean classpath with none of the Boot or Java system stuff on!

Are you considering merging that into the show task in some form?

or making with-cp a build-in task?

hey guys, I'm new to boot, need help to debug this error I'm getting: java.lang.NoClassDefFoundError: boot/App

let me find somewhere to paste my build.boot

http://paste.ofcode.org/3679uvrWafsMJS56pWmaqT2

it actually builds, but I get that error when trying to run the uberjar

now that I have my task, how can I make it available to future programs without having to copy and paste it to build.boot in each of my projects?

@wamaral: it looks like scope provided is the cause

for boot.core, ops @micha go ahead you know better 😉

@richiardiandrea: which?

humm, good point, let me try

@mhr you can define your task in a namespace and package it as a jar on clojars

you can use this as a template: https://github.com/micha/boot-cp

you can see lots of examples here https://github.com/boot-clj/boot/wiki/Community-Tasks @mhr

still same error :(

does it say who is causing it?

@wamaral: are you trying to use boot namespaces in your uberjar?

here's the whole backtrace http://paste.ofcode.org/tWZLPWCcdiPv6rqsvsDRAc

@wamaral: does report.core try to :require boot.core?

@micha: I defined a task with deftask inside my application's core

ah that is the cause

yes, I have boot.core :refer [deftask]

yeah..

can't do that? :(

that can only be run when the JVM is launched via the boot shim

aww

@wamaral: there is this: https://github.com/adzerk-oss/boot-ubermain

hm no sorry

that's a different thing

hmm I see... I tried doing that to start the webserver defined in core, while in development

but I didn't want to require too many things in build.boot

guess I'll have to :)

yup, got it to work again

thanks!

@wamaral: great!

just discovered this great plugin for lein: https://github.com/webnf/lein-collisions do we have something like this in boot?

Looks easy enough to rebuild as a Boot task… The only thing it seems to rely on is get-classpath from Leiningen.

yes true, but now we have boot-cp 😉

So send @micha a Pull Request to add a collisions check to boot-cp 😆

(!> tasks-chan "send Micha a PR") 😄

@micha trying out boot-cp, I’m wondering how do I resolve a conflict? I presumed supplying a specific version of a dep would win, e.g. core.async has dep on old clojure version:

@richhickey: ah sorry, that's a bug, there is a filtering step i forgot to add, where it filters out conflicts that were fixed by overrides like what you show

i used a function that's normally used for the show -p task

ok

when accepting edn args does boot have any facilities for getting from file, vs the cat thing I had to do above?

the task could be modified to do that

you can also specify it in the build.boot file

if you have one

(task-options!
  micha.boot-cp/with-cp
  {:dependnecies '[...]})

that "curries" the task fn

you can override that on the command line or in the repl, as kwargs are overridden

I saw that in the readme, but as I said I consider the deps list to be data and would rather not have to replicate a boilerplate ‘program’ like this in every project. I’m trying to see if I can get a set of ambient boot tooling that, once present, would have me making only a deps.edn in each project

at least for simple things

will you have a build.boot file at all?

not for simple things

are you okay with loading a task via boot's -d option?

I’d rather have shared utilities in profile.boot

ah okay, then we can program in there

so everything is easy

right, I have this in there now:

in profile.boot you can slurp "deps.edn" and do the task-options! above

you can add to that:

profile.boot gets run for all uses of boot though, right?

yes, you can see exactly what it does with boot -vb task task

so one thing to map code i nthere, another to do work (which might fail for other uses)

that will print an annotated version of the generated boot.user ns that boot evaluates

you can have default configurations in profile.boot

like this perhaps:

(require '[ :as io])
(let [depsfile (io/file "deps.edn")]
  (when (.exists depsfile)
    (task-options!
      with-cp {:dependencies (read-string (slurp depsfile))})))

then when you're in a project dir with a deps.edn present

you can do boot with-cp -wf cp.out

etc

you could also override the deps on the command line still

Thanks for boot-cp — just used it to bludgeon all of our dependency conflicts into submission! Had to add a lot of :exclusions and nearly all of them were for [org.clojure/clojure] — which leads me to wonder why every project doesn’t use "provided" as the scope for Clojure itself, so it doesn’t end up as a transitive dependency?!?! 😠

got it. I was wondering though about the other uses of with-cp, since boot has already started the jvm and clojure, and deps will usually have clojure in them, what will aether do with second/maybe-conflicting clojure? with-cp is still using aether to load deps right?

I ended up having :exclusions [org.clojure/clojure] on every single contrib library dependency in the end.

@seancorfield: the thing it's really doing is calling boot.pod/resolve-dependency-jars, which you can use in your own tasks

We may well add a step to our build/test process that "breaks the build" if there are conflicts.

@seancorfield: absolutely I prefer that as the default

automatic conflict resolution == problems

a maven misfeature IMO

I’m beginning to agree with that position (I thought it was unnecessarily extreme until the last few days taught me otherwise — with Clojure 1.9.0 Alpha 5 🙂 )

so @micha I think the mode is not -pedantic but -safe

🙂

i like it 🙂

@richhickey: with-cp isn't using aether when you use --read, it just adds JARs directly to the URLClassloader

you can resolve dependencies with aether without adding them to the classpath. boot has functions to help with that, functions that return clojure data

like in boot-cp, it resolves the dependencies using maven and then extracts from that data just a list of jar files

cool, but clojure will still end up in the mix twice though, right?

@micha that slurp strategy works great btw, thanks, and for with-cp!

yeah we really need a way to safely patch the classpath

@richhickey: As a philosophical issue, given the above, do you think that Clojure libraries — including contrib libs — should ensure that Clojure itself doesn’t become a transitive dependency? i.e., they should mark org.clojure/clojure as "provided"?

we can do it by keeping track of all loaded dependencies, and not allowing new dependencies to be loaded with different versions

@micha that seems necessary to avoid a mess

yes, definitely

that’s what I’m always afraid of with aether

but any dynamic loading is subject to these problems

if you only add dependencies by jar URL i think it's safe

because you get exactly what you ask for there, with no aether to do unpredictable things

so rockin here with minimal setup - make a src dir and a deps.edn, call boot with-cp -w and launch repl with -cp

@micha: yes, much better when manual URLs vs aether, you can control given the check you describe above

I will end up making a real clojure repl task though so it can follow with-cp

you can use boot.pod/resolve-dependency-jars to get the list of jars, and you can then use boot.pod/add-classpath to add the jars to the cp

like if you wanted to do this in the repl or something

neat

FWIW, I just added this task to our build.boot file:

(deftask check-conflicts
  "Verify there are no dependency conflicts."
  []
  (with-pass-thru fs
    (require '[boot.pedantic :as pedant])
    (let [dep-conflicts (resolve 'pedant/dep-conflicts)]
      (if-let [conflicts (not-empty (dep-conflicts pod/env))]
        (throw (ex-info (str "Unresolved dependency conflicts. "
                             "Use :exclusions to resolve them!")
                        conflicts))
        (println "\nVerified there are no dependency conflicts.")))))

Thanks @micha !

@seancorfield: I’d rather tools get better/safer than try to get everyone to be consistent

Fair enough. I definitely think I’ll start marking org.clojure/clojure as a "provided" dependency in my projects from now on tho’, based on the pain I just went through today! 😸

@micha sorry to barge in, doesn't set-env! :exclusions already take care of removing the deps globally? just asking confirmation because it is what I am doing in order to be sure to have only the one org.clojure/clojure coming from :dependencies in the classpath

@richiardiandrea: yes, that just adds :exclusions to all of your individual dependencies when it resolves

it does exactly (or should do) what you said

@richhickey: i am unsure of how to implement the clojure jar url scheme in boot

because of the way maven coordinates are normally used to specify dependencies, and the direct jar file setting wouldn't necessarily have a version associated with it

i was thinking maybe if you have BOOT_CLOJURE_VERSION=<file://home/me/work/clojure.jar> set then it would know to load the jar directly

that would make it consistent, in that you could use the existing machinery to starts pods with that or any other version of clojure, etc

Thanks for that insight @richiardiandrea — I had no idea about that… would have saved me a lot of pain 😈

@richiardiandrea: i rarely use :exclusions because usually i would add a direct dependency on specific version of the thing i want, rather than excluding all versions of a thing i don't want

@seancorfield: I actually discovered it recently myself (and updated the wiki yesterday)

For org.clojure/clojure tho’, it makes a lot of sense — since pretty much every project drags it in as a transitive dependency.

Yeah ^

and sometimes there is a dependency that has a ton of unnecessary transitive dependencies that i don't want on the classpath, but then i add :exclusions to that specific dependency, not globally

Yesterday I discovered (not a big surprise I guess) that some clj/cljs/cljc library brings in ClojureScript and the Google Closure compiler so that's why I used global :exclusions...of course depends on your use case but it is good to have it

ah and you don't want closure or cljs in all pods

In general it should never go in your uberjar as it is used only for compiling right?

that's a good use case for the global :exclusions

right

So yes and it filters also transitive deps which is exactly what you want for the case above

yeah it's completely ignored by aether then

Booted 😄

@micha does that syntax work now? BOOT_CLOJURE_VERSION=

@richhickey: no, currently it represents the maven version

that’s what I thought