Fork me on GitHub
#boot
<
2016-08-08
>
danielsz01:08:19

system 0.3.1 is out and here is the official announcement. https://groups.google.com/forum/#!topic/clojure/Td4-b8gDL58

richiardiandrea02:08:43

Oh man I have to take the time for trying mount, sorry about the delay

danielsz02:08:07

@richiardiandrea: Not a problem at all. Only if you have time and motivation to push this forward. The success of the mount branch depends on the people who want to see this happen. We'll get there, eventually. 🙂

borkdude07:08:11

@danielsz: Thank you for this great work. I'm using boot more and more and this only makes it better.

danielsz07:08:27

This is so nice to hear, @borkdude.

cycle33708:08:20

I have a boot task in my build.boot where I simply want to output the project version (i need it propagated to my namespaces, and just want to check if it is accessible in a task)

cycle33708:08:41

(deftask showver
  (println "version is: " version)
  )

cycle33708:08:53

when I run boot showver

cycle33708:08:21

i get this error

cycle33708:08:29

cli: expected short option, got echo
cli: expected long option, got nil
cli: option nil: expected optarg, got "version is: "
 clojure.lang.ExceptionInfo: Wrong number of args (1) passed to: cli/argspec->cli-argspec
    data: {:file "/tmp/boot.user4393831086517409494.clj", :line 27}

cycle33708:08:33

what am I doing wrong ?

pesterhazy08:08:52

you need [] after the task name

cycle33708:08:52

how could I have made such a noob mistake 😞

borkdude08:08:11

@avabinary: because the error messages suck

pesterhazy08:08:01

good news: the error message has been fixed in master

cycle33708:08:53

now it works, thanks @pesterhazy

stuarthalloway14:08:32

questions about util/verbosity • why is it both dynamic and an atom? • what is the idiomatic way to change it from the REPL?

stuarthalloway14:08:53

I want (boot (turn-up-verbosity) (thing-I-am-testing))

martinklepsch14:08:55

Uh, ignore that, doesnt work like that from repl

alandipert14:08:08

@stuarthalloway:

(deftask turn-up-verbosity [] 
    (fn [next-task] 
      (fn [fileset]
         (binding [boot.util/*verbosity* (atom 3)] (next-task fileset)))))

alandipert14:08:27

don't know why micha made it a dynamic atom, but i do that so i can inc/dec a value without rebinding

micha14:08:25

ah i don't remember, it was probably not an ideal way to solve whatever the problem was

stuarthalloway14:08:00

am I “doing it wrong” trying to debug tasks at the REPL this way?

stuarthalloway14:08:26

feels like anybody debugging a task would immediately hit this, so I wonder if my workflow is odd

alandipert14:08:48

i usually start with boot -vvv repl and then (boot ...) and (load-file "build.boot") prodigiously.. so i think what you're doing is par

micha14:08:16

@stuarthalloway: do you still need to be able to use a non-maven version of clojure for boot itself, such as by specifying BOOT_CLOJURE_VERSION=?

micha14:08:30

or do pods provide all you need?

stuarthalloway14:08:21

@micha I am using boot-cp everywhere, and writing my own code for managing classpath

stuarthalloway14:08:55

non-mavening not just Clojure but other things

micha14:08:37

the only thing that doesn't cover is the version of clojure in the boot core pod, which will be loaded from local maven

micha14:08:55

but we could add the BOOT_CLOJURE_VERSION hack if you need it

micha14:08:15

all the other things in boot-cp have been added to master and will ship with 2.7.0

richiardiandrea16:08:40

I have a lil problem in that I receive a Could not find artifact boot:core:jar:2.7.0-rc1 in clojars here even if my boot.properties says 2.6.0...

richiardiandrea16:08:58

shell I delete some cache folder in .boot?

richiardiandrea16:08:02

found the culprit, never mind 😉

levitanong16:08:43

What was the culprit?

richiardiandrea16:08:11

I had the 2.7.0-rc1 version hardcoded in deps for a pod

richiardiandrea16:08:49

in general I use (boot.App/config "BOOT_VERSION") but that was a leftover

levitanong16:08:20

ah, glad you figured it out 😄

stuarthalloway18:08:49

I want to use the Spring S3 wagon, how do I say that in boot?

stuarthalloway18:08:56

(aether/register-wagon-factory! "s3" #(org.springframework.build.aws.maven.SimpleStorageServiceWagon.))

micha18:08:01

@stuarthalloway: the "worker" pod is where the maven machinery is loaded, so you can add the implementation class to the classpath of the worker pod by using boot.pod/with-eval-worker to call boot.pod/add-classpath or boot.pod/add-dependencies etc., then evaluate your code above in the worker pod too

micha18:08:11

or alternatively you can do something like

micha18:08:02

(set-env! :wagons #(conj % '[s3-wagon-private "1.2.0"]))

micha18:08:32

that will work "automatically" because it contains a file that describes the class bindings

micha18:08:02

so to clarify, if you don't have the wagons.clj file in the jar already and the jar in maven, you want to do this:

micha18:08:03

(boot.pod/with-eval-worker
  (boot.pod/add-classpath "path/to/wagon.jar")
  (require '[cemerick.pomegranate.aether :as aether])
  (aether/register-wagon-factory! "s3" #{...}))

stuarthalloway18:08:01

thanks @micha. I have (set-env! :wagons #(conj % '[s3-wagon-private "1.2.0"])) but don’t see the factory being registered, will post more when I narrow it down

micha18:08:01

@stuarthalloway: it should be registered in the worker pod

micha18:08:07

that's the relevant snippet of code to doit

stuarthalloway18:08:17

I see it if I eval in the worker pod directly:

stuarthalloway18:08:23

(boot.pod/with-eval-worker
  (require '[cemerick.pomegranate.aether :as aether]
           '[clojure.pprint :as pp])
  (util/warn "%s" (with-out-str (pp/pprint @#'cemerick.pomegranate.aether/wagon-factories))))

stuarthalloway18:08:37

But not if I look while inside a task

stuarthalloway18:08:50

(deftask demo
  []
  (with-pass-thru [_]
    (util/warn "%s" (with-out-str (pp/pprint @#'cemerick.pomegranate.aether/wagon-factories)))))
(boot (demo))

micha18:08:31

right, you want to wrap the pp/pprint part with with-eval-worker in the task, too

micha18:08:53

because the core pod (the one your task body is evaluated in) doesn't have any aether stuff in it

stuarthalloway18:08:35

i don’t understand that last sentence and I want to 🙂

micha18:08:03

ah sorry, what i mean is that there are a couple of pods that boot starts when it boots up

micha18:08:37

it starts a worker pod that has various dependencies loaded in it, dependencies that boot needs to do its jobs, like the maven aether machinery, things like that

micha18:08:44

many hairy dependencies

stuarthalloway18:08:52

I am writing my own task that wants to run in that worker pod

micha18:08:59

but your build.boot file is not evaluated in that pod

micha18:08:10

because there are all those dependencies that would pollute your build

micha18:08:25

so boot makes a second pod with no dependencies loaded in it, the core pod

micha18:08:33

and it evaluates your build.boot file in that pod

stuarthalloway18:08:02

when I write my own tasks, what pod do they run in?

micha18:08:05

when you call boot.pod/add-dependencies etc it's really calling into the worker pod to resolve the maven coordinates into jar locations

micha18:08:10

they run in the core pod

micha18:08:28

but you can use with-eval-worker for example to evaluate expressions in the worker pod

micha18:08:41

the boot.pod namespace has many wrappers for functionality that is actually provided by the worker pod

micha18:08:06

the wrappers allow all other pods to use the functions in the worker pod via those proxies

micha18:08:23

so you may want to do something similar

micha19:08:14

@stuarthalloway: ah you can also add a :schemes key to the :wagons vector, like this

(set-env!
  :wagons #(conj % '[some-maven-id "1.2.3" :schemes {"s3" '#(org.springframework.build.aws.maven.SimpleStorageServiceWagon.)}]))

micha19:08:15

you'd need to quote the factory fn like above, because it will be passed as data and evaled in the worker

stuarthalloway19:08:04

now I am further down the rabbit hole...

stuarthalloway19:08:35

I have an entire source file that I want to require from inside the worker pod

stuarthalloway19:08:16

but I think that my dev source directory is not in the worker pod classpath

micha19:08:29

ah that makes sense

micha19:08:48

you can use boot.pod/add-classpath to add them, if you want

stuarthalloway19:08:11

and this code is itself a library, so I need an approach that covers my dev use, and later downstream use

stuarthalloway19:08:54

so naive use of boot.pod/add-classpath won’t be enough — right now it is “src”, later it will be “this-lib.jar"

micha19:08:31

it's unusual to need to evaluate your source files in the worker

micha19:08:55

usually what you do is make your own stub fns that act as proxies

micha19:08:15

i mean act as proxies in the core pod for functions you want to call in the worker pod

micha19:08:47

is the thing you need to evaluate in the worker pod an aether wagon?

stuarthalloway19:08:14

no, it is a bunch of code that uses pomegranate and aether

micha19:08:49

i think it's simpler if you do the proxy approach, if possible

micha19:08:23

like what we do in boot is we have the boot.pod namespace that is in all pods

micha19:08:33

some of the functions in that namespace are calling aether functions

micha19:08:44

but they do it via with-eval-worker

stuarthalloway19:08:16

I will take a look

stuarthalloway19:08:06

hm. I have existing libs that make use of aether. and I want to write a boot task shim for them

micha19:08:46

you can make a pod yourself and add aether to it as a dependency, plus your libs

micha19:08:59

that's another option

stuarthalloway19:08:12

that certainly seems most flexible

micha19:08:45

that would avoid any chance of conflicts with other dependencies in the worker pod

stuarthalloway19:08:26

yeah, it seems maybe worker pod should be treated mostly as private to boot itself — use it where it does something you need, but don’t add a lot to it

micha19:08:57

do you want to replace the boot dependency stuff with your own?

stuarthalloway19:08:30

for myself — not to inflict on others 🙂

micha19:08:52

interesting, i'll think about how to do that

micha19:08:23

that's what adds dependencies when you do (set-env! :dependencies...

micha19:08:42

you could make your own defmethod to override that

stuarthalloway20:08:37

is it cool to add user-space things to pre-env!? Should probably be namespaced keys...

micha20:08:57

yeah i can't really fully imagine the whole picture there

micha20:08:15

and what the tradeoffs would be

micha20:08:13

but this is an interesting use case, seems like it will shed light on what the real API should be

sbrady21:08:00

Somewhat related to pre/post-env, I was wondering if there's a good reason not to expose this on-change handler for watching user-dirs: https://github.com/boot-clj/boot/blob/master/boot/core/src/boot/core.clj#L192-L194. I'm currently monkey patching boot.core (inconveniently) to piggieback on my own callbacks whenever there's a file change. I'm doing this so as to not spawn a whole bunch of watcher threads (which for our repo would be a lot given the number of directories we have). It seems like it could generally useful to expose file changes as something that could be observed while in the repl, for example.

micha21:08:32

oh whoops

micha21:08:48

sorry rogue paste

sbrady21:08:11

No worries. I pulled all the secrets from that link 😉

sbrady21:08:32

Yes, I did see that atom, but it doesn't get the actual changes

micha21:08:40

it could serve you an ad 🙂

micha21:08:32

the watch task currently does some things to work around limitations of the fs event APIs

micha21:08:57

the fs events are good to use because you don't need to poll, which uses a significant amount of cpu

micha21:08:10

so like if you're on a laptop that would be kind of a drag

sbrady21:08:19

right, which is why I've gone to piggiebacking on those changes

sbrady21:08:32

So I just reuse what's already captured

micha21:08:34

but the limitations are that the events are a lot of times spurious or incomplete etc

micha21:08:57

so in the watch task it does some debouncing and then compiles its own list of changes

sbrady21:08:23

Right, and I've been intercepting those changes

micha21:08:25

but i think it would be tricky to move that into boot core because you probably don't want to do that change tracking all the time

sbrady21:08:27

It's worked well so far

micha21:08:54

can you describe a bit what your use case is?

micha21:08:04

does boot.core/fileset-diff not help?

sbrady21:08:30

It would, but how do I access that in the context of a repl session?

micha21:08:58

hm that's interesting!

sbrady21:08:03

exactly 🙂

sbrady21:08:24

So my use case is that I recompile java classes on the fly (don't tell anyone :)

sbrady21:08:44

So I'm doing that while I'm coding in the repl

micha21:08:55

wow, i would like to see how you do that

sbrady21:08:59

And so when those java source files change, I want to recompile

sbrady21:08:06

Oh, it's pretty easy

sbrady21:08:21

I cribbed most of it from alan dipert and boot

micha21:08:57

you can run a boot pipeline in a future, perhaps?

sbrady21:08:17

yes, but that's pretty heavy weight, at least relative to my monkey patch

sbrady21:08:33

boot with two pods and what not is already quite heavy as is

sbrady21:08:44

I replace the set-user-dirs! function with this one

sbrady21:08:15

And then add callbacks to FileChangeCallbacks at will while in the repl (or in build.boot)

sbrady21:08:46

I just basically lifted out the anonymous function and extended it will calling the callbacks

micha21:08:24

testing something

micha21:08:13

so what kind of stuff do you do in your callbacks?

sbrady21:08:06

For the javac callback, I see if the file was added or removed and either compile it and add it to Clojure's class cache or remove it

micha21:08:27

why not do that in a task in the pipeline?

micha21:08:45

one more thread isn't a big deal, i wouldn't think

sbrady21:08:46

because I prefer to have one jvm running

micha21:08:58

you can use the same jvm, just another thread

sbrady21:08:37

However, that pipeline would be akin to boot watch javac

sbrady21:08:44

If I read you correctly

micha21:08:49

sure, but that would be exactly what you want

sbrady21:08:02

Almost, I'll get another 200 threads spawned as well

micha21:08:03

instead of like a partial pipeline

sbrady21:08:06

to observe the file system

micha21:08:24

the watch task doesn't add threads for that

micha21:08:33

those threads already exist

micha21:08:00

the watch task just watches the last-mod-time atom

sbrady21:08:04

I don't think that's what I observed, but I could have been doing something off

micha21:08:12

well it does compute a diff

micha21:08:21

which uses many threads

sbrady21:08:23

so it does a full fs scan to produce the diff?

sbrady21:08:32

so perhaps that's what I saw

micha21:08:43

there isn't any way to avoid that as far as i can tell

sbrady21:08:44

I'm pretty sure I saw a lot of threads created being created

micha21:08:53

the fs events aren't reliable

micha21:08:01

so you can't patch state based on fs event deltas

sbrady21:08:04

yes, that I know

sbrady21:08:45

It's close enough for repl development though

micha21:08:43

seems like we'd be better off getting the background pipeline working efficiently than by exposing these internals

micha21:08:48

because then you have like a partial pipeline thing that isn't composable with anything else and goes outside the locking and synchronization of the pipeline

micha21:08:27

there are some semaphores and things in the pipeline because of all those filesystem event watches

micha21:08:30

i've used gary fredricks repl utils macros in the repl, makes things nicer

sbrady21:08:27

I obviously don't have the perspective you have on the boot pipeline, but it does feel like the pipeline API doesn't quite fit the event orientation of the repl. While the repl's running, there are events that are happening. It does seem like I should be able to observe and react to those.

sbrady21:08:35

We've got things like pre-env and post-env, which look the start of being able to bind to events

sbrady21:08:27

I wouldn't use these things in the more command-line-oriented boot tasks that I have, but the repl does have its own orientation that's different

micha21:08:02

yeah the repl is a special case for sure

micha21:08:11

and it's running in a pipeline already!

micha21:08:17

so maybe that's the solution?

micha21:08:22

something with the repl

micha21:08:46

because to start the repl you usually will do so via the repl task

micha21:08:07

that task gets filesets passed to it if you put the watch task in front of it

sbrady21:08:30

I'd love to get access to the fileset while in the repl, for sure

sbrady21:08:52

Having the repl sit at the end of the pipeline and block it, that seems to be the cause

micha21:08:58

hm unfortunately that doesn't work

micha21:08:08

the repl client blocks it

micha21:08:41

so what we really want is that when you change a file in your project source dirs for instance

micha21:08:51

that an atom in boot.core will have the new fileset in it

sbrady21:08:00

that'd be perfect

micha21:08:01

and everything else watches that

sbrady21:08:35

When I first hunted around, I actually did think that it was happening somewhere (with the user dirs sync), and I actually was surprised that the changes delivered by the watch-dirs provided it in a format that was not the fileset format

micha21:08:09

i don't remember exactly what the issues were with that approach

micha21:08:35

seems like maybe boot.core/commit! should actually update a global var

micha21:08:44

with the current committed fileset

micha21:08:32

maybe that's what was missing

sbrady21:08:37

I think so

micha21:08:25

ok cool, can you open a ticket please?

micha21:08:30

we can maybe refactor that part, the part that conveys fs event updates to the boot env

micha21:08:43

probably make it a lot simpler

sbrady21:08:01

thanks for entertaining the idea

micha21:08:19

sure, no problem

micha21:08:28

that would solve your needs too, right?

sbrady21:08:12

Yep, it would

micha21:08:12

(add-watch boot.core/current-fileset ::foop (fn [_ _ _ _] (boot (javac))))

micha21:08:20

or something like that

kenny22:08:11

@micha: Not sure if you saw this as I posted it quite late but here is the first cut of the task we talked about last week: https://gist.github.com/kennyjwilli/97f83cc49934bc08af4d3905487a9b49 Any feedback?

micha22:08:41

@kenny: looks good to me!

kenny22:08:55

Cool. Works as expected

micha22:08:26

what was the use case you're using it for?

kenny22:08:57

(update-file :match-files #{#"ui\.cljs\.edn"}
             :expr (fn [cljs-edn]
                     (update cljs-edn
                             :compiler-options #(-> %
                                                    (dissoc :preloads :external-config)
                                                    (assoc :closure-defines
                                                           {'myns.core            "prod"
                                                            'myns.core/server-url ""})))))

micha22:08:07

ah right!

micha22:08:11

very nice, only one small nitpick

micha22:08:15

if that's ok

kenny22:08:25

That's why I posted it here 🙂

micha22:08:02

liek on line 27, you could do (-> (io/file new-dir (:path tmp-file))

micha22:08:20

using the 2 arg version of io/file

micha22:08:31

so as not to need to do special things for windows

kenny22:08:38

Ah, right

kenny22:08:40

There is one issue though.. As you probably know, in Clojure 1.9.0 namespaced map reader macros were introduced (http://dev.clojure.org/jira/browse/CLJ-1910). So, if you are using Clojure 1.9.0 in your project, the task will spit the updated map using reader macros. This causes a problem in boot-cljs because it doesn't know how to read .cljs.edns with the new reader macros.

micha22:08:42

hmm, pr-str prints the reader macros?

kenny22:08:57

(pr-str {:myns/foo 1})
=> "#:myns{:foo 1}"

micha22:08:23

i wonder why it doesn't know how to read them?

micha22:08:35

perhaps that pod is using an older version of clojure?

micha22:08:48

it's using read-string, which should be symmetric, right?

micha22:08:04

strange that it prints with the reader macros by default

micha22:08:28

strange in general to print reader macros at all

kenny22:08:12

Well, the idea was to remove the verbosity of namespaced maps

micha22:08:31

right, if a human is writing it i can see the logic there

kenny22:08:43

What we really need is an option to disable printing of the reader macros

micha22:08:47

but does the printer have like an inverse reader macro facility?

micha22:08:10

to generate reader macros from the data in memory?

micha22:08:16

that seems odd to me

micha22:08:36

especially since there is no guarantee that there is not more than one reader macro representation for a given piece of data, is there?

kenny23:08:58

Looking at boot-cljs, Clojure is a provided dep so I should be able to set the Clojure version in the project using boot-cljs

micha23:08:43

do you have a clojure dependency in your project?

micha23:08:08

also how do you feel about adding your update-files task to the boot built-in sift task?

kenny23:08:13

And BOOT_CLOJURE_VERSION is set in boot.properties

micha23:08:45

the -u and -U options are available

kenny23:08:20

I think it makes sense there

micha23:08:32

-u, --update-clj and -U, --update-str perhaps

micha23:08:51

-u would get the contents of the file as clj data

micha23:08:59

vs as a string

micha23:08:00

then we could use the normal map of {regex code} for the matcher and expr

kenny23:08:39

That would need to be added, yeah?

micha23:08:31

the two options would need to be added to sift

micha23:08:53

u update-clj MATCH:EXPR {regex code} "The map of regex to update function expression."

micha23:08:57

something like that

kenny23:08:01

Do you think the edn parser makes sense as the default then?

micha23:08:21

at the comand line you'd do boot sift -u '*\.clj$:#(concat % (quote (println "hello")))'

micha23:08:47

having 3 parts would be too much i think

micha23:08:03

that's why i thought of breaking it into two separate options

kenny23:08:11

Yeah, that is more general anyway

micha23:08:26

(sift :update-clj {#"\.clj$" #(concat % [`(println "hello")])})

kenny23:08:40

Looks good

micha23:08:45

ah we do want the read-string to read all forms

micha23:08:15

like (read-string (str "(" (slurp infile) ")"))

micha23:08:50

there is a function in boot already that does that

kenny23:08:22

What is that for?

micha23:08:45

if you just use read-string you get only the first top-level form

micha23:08:57

most of the time you'll want to get all the forms

kenny23:08:34

I think this is a case where you should use the -U option

micha23:08:53

interesting

kenny23:08:08

This is what I was talking about with the edn default

micha23:08:30

yeah maybe you just do that in your function anyway

micha23:08:38

it's easy to do the default thing yourself really

micha23:08:22

so maybe the callback just gets input and output streams

kenny23:08:48

I say we just make the default be the edn reader and if someone wants to use a different parsing function they can use the :update-string option which is passed the contents of the file as a string.

micha23:08:30

now i'm thinking the edn reader is maybe overkill

micha23:08:37

and streams is all we want

micha23:08:38

it would be fine to call clojure.edn/read-string in your updater fn anyway, i imagine

kenny23:08:50

So the update function gets passed the file stream?

micha23:08:03

and the output stream it would write to

micha23:08:25

you can still slurp/spit to them

kenny23:08:45

Why not just have the return value of the function be spit to the file?

micha23:08:15

it would mean that the files would need to be small, no?

micha23:08:47

could be premature optimization to use streams

micha23:08:24

but we could use a simpler thing with streams in the task and have functions that handle what you describe that are alraedy available

micha23:08:35

like if you have edn->edn

micha23:08:32

you could do (sift :update {#"\.clj$" #(with-edn [edn %] (assoc edn :foo 100))})

kenny23:08:04

I think that would be the most general approach

micha23:08:05

the with-edn thing would handle the streams

kenny23:08:49

You would also need to pass in the output stream

micha23:08:01

ah right, yes

kenny23:08:31

Do you want me to submit a PR or are you making the changes?

micha23:08:01

a PR would be most welcome 🙂

micha23:08:37

i can add stuff like caching etc

micha23:08:51

if you get stuck with it (the sift task is weird) you can make an issue on github to track it

kenny23:08:35

Okay. Will dive in further tomorrow

micha23:08:48

awesome 👍

micha23:08:23

i think this will be a cool addition

micha23:08:45

you can do a lot of things you'd normally make a separate task for with this approach i think

kenny23:08:18

Will be interesting to see how it is used

micha23:08:13

perhaps the -U option can be for --no-update-modtime

micha23:08:28

so you can update files in place without bumping the modification time if you want

kenny23:08:26

Not sure how you would do that. Does file have an option to set the modification time?

micha23:08:41

yeah it's part of the java.io.File interface

kenny23:08:24

http://dev.clojure.org/jira/browse/CLJ-1993 may solve the problem with the namespaced maps

micha23:08:54

it's very strange to me that the default would be to print those things

micha23:08:36

i wonder what made that necessary?

kenny23:08:19

I think the consensus was that it just made things cleaner when printed

micha23:08:25

seems like a job for a pretty-printer, not the base printer

micha23:08:37

i thought pr-str was for machines

micha23:08:45

and clojure.pprint/pprint was for humans

micha23:08:25

anyway i'm sure it'll work out somehow

micha23:08:37

by the time 1.9 is released