Fork me on GitHub
#boot
<
2016-02-07
>
michal11:02:13

hey guys, quick question. I’m creating uberjar and resulting file gets created somewhere in ~/.boot/cache. is there any way (other then setting BOOT_EMIT_TARGET) to provide destination dir ?

jethroksy11:02:56

@michal use the target task: boot target -h for more details

michal11:02:21

@jethroksy: great! that’s what I needed. thanks!

andrewboltachev12:02:48

Hi. Is it possible to spit huge EDN object (i.e. huge piece of data) in one task and read it from another?

dominicm12:02:31

Filesets are where my brain goes to.

andrewboltachev12:02:36

@dominicm: I'm using them already. I just wonder should I use them or not if my file is, say, 50 megabytes of Clojure data? EDN parser would reject probably

andrewboltachev12:02:24

i.e. what files (in a fileset, or filesystem or whereever) contain is strings, not data objects

dominicm12:02:10

ah, I'm not sure on that one.

andrewboltachev12:02:50

so, what I wanted Boot to do for me is to watch for changes

mobileink15:02:21

Hi folks. I'm thinking about extending boot to support first-class task dependencies, so e.g. a compile task will only run if sources have changed. I can't see how to do this with handler composition, at least not without embedding the logic in the tasks themselves, which would be bad. Any plans for this sort of things?

ragge15:02:04

@juhoteperi: I did, and yes, the update after separation of boot/boot-bin are later than 15.09 so you need to be on unstable

ragge15:02:21

@juhoteperi: or just override stuff in your .nixpkgs/config.nix

martinklepsch16:02:21

@mobileink: you mean like a cljs task that only compiles if cljs files have changed? why would you want to split the "see if something has changed" bit from the actual task?

jellea16:02:31

I'm building a boot-new template; Is it recommended to add a boot.properties file?

martinklepsch16:02:59

@jellea: I think it's a good idea to avoid confusion over the version of boot to use/have installed

jellea16:02:10

martinklepsch: boot -u > boot.properties will do right?

mobileink16:02:34

@martinklepsch: Well, there's the general principle that a task should do one thing only; checking for file changes and compiling are distinct tasks. But more practically, nested handler composition forces you to invoke every task in the chain. If we have a task chain A -> B -> C, the actual work of A might not be needed, but it has to call B since it cannot determine if B is needed, and so on. To be able to skip unnecessary tasks, the task controller must be separate from the tasks, and the tasks must be annotated with input/output info so the controller can determine which tasks can be skipped. This is roughly what Gradle does, but working with Gradle is a nightmare once you've seen Clojure. In principle is should be possible to have a defjob macro, with first arg {:inputs [..filesets..] :outputs [..filesets..]}, together with a composition op such that the composition mechanism decides whether or not to run tasks. Only data/files would be passed from task to task, not handlers. The task pipeline would be a flat one-way sequence rather than a nest of embedded fns. In other words, first order composition v. the second order composition used by boot, ring, etc. So far this is just an idea, but it seems feasible. I would really like to have the power and flexibility of Gradle in Clojure and boot seems like a good candidate. Make sense? BTW I'm not complaining about handler composition, it works great when that's what you need.

micha16:02:07

@mobileink: there is benefit to running all tasks in the pipeline though

micha16:02:25

they can encapsulate the caching strategy inside their middleware

micha16:02:35

and no other tasks need to participate

micha16:02:59

really all dependency systems like GNU make for example, are really caching strategies

micha16:02:15

you can use a cached result instead of building a new thing

micha16:02:32

boot tasks can also do this, since the fileset is immutable

micha16:02:45

you can hang onto the previous fileset and diff it against the next one

micha16:02:50

to decide whether to do work or not

micha16:02:08

tasks may keep their own caches, which decouples that from the rest of the system

micha16:02:46

so a task may find that the files it's interested in have not changed since the last time it was called, so it will not build anything. instead it will add cached files from the last build to the fileset

micha16:02:10

this is an extremely efficient process because no copying of files or I/O actually takes place

micha16:02:34

it's just a few hard links created if necessary

micha16:02:44

and a new immutable fileset object created

micha16:02:23

i think in most cases it's much harder to specify a correct dependency DAG than it is to just put the tasks in a pipeline in a certain order

micha16:02:44

because to the developer it's usually completely obvious which tasks need to happen before which other tasks

micha16:02:51

in any specific situation

micha16:02:13

but expressing this in the abstract is very difficult

dm316:02:50

kind of related - we've talked with Alan briefly on the merits of https://github.com/ndmitchell/shake here and he brought up the possibility to kind-of JIT-optimize the boot build by noting what files are touched in the tasks

micha16:02:46

i think what boot does is like memoization

micha16:02:54

or a pass-thru cache

micha16:02:43

these are usually superior to other solutions because they are self-contained, they don't need to negotiate with some supervisor

micha16:02:27

imagine if every time you wrote a task you would need to express the relationship it will have to any other task that might be used

micha16:02:33

that's a Hard Problem

mobileink16:02:56

@micha: right, handler composition definitely has its place and you guys have done a great job. i have to think about this some more, but basically i think there are two approaches. one is to make each task responsible for caching, diffing, etc.; this is the approach boot takes. the other is to give that responsibility to a supervisor/controller. there are pros and cons to each approach.

micha17:02:18

the reason we went with the pipeline model was because we were stuck by how obvious the pipeline is in any specific build scenario, versus the complexity of trying to solve the problem in a general way for all build scenarios

micha17:02:59

like building a system that could divine the user's intent from declarative specifications is way more complicated than the user just telling you their intent by putting tasks in a linear pipeline

micha17:02:13

the job of the supervisor, at the end of the day, is to forma linear pipeline

mobileink17:02:40

actually I think removing the caching/diffing decision-making from tasks makes it much easier to build (flat) pipelines. no need to express relations to other tasks, or to negotiate anything - just specify your inputs and outputs, and then construct a pipeline - the controller decides what needs to be done, based on the inputs and outputs. (Note that I'm making a distinction between handler pipelines (which are not really pipelines, they're nestings) and flat pipelines. And I'm not suggesting handler pipelines should be replaced - they're just a mechanism and they work well.)

micha17:02:13

you can't really express in terms of inputs and outputs i don't think

micha17:02:21

there isn't a 1-1 mapping there

micha17:02:32

there could be side effects involved, for example

mobileink17:02:47

In any case, it sounds like what I've described is not being worked on so if I can find the time to do some proof-of-concept work I won't be overlapping.

micha17:02:02

like with boot a task may or may not emit a thing, depending on some external value

micha17:02:18

like "is this already in a CDN" or something

micha17:02:29

and it may be that another task might put the thing in the CDN

micha17:02:21

i think you can definitely do it

micha17:02:34

you could have one supervisorr task perhaps

micha17:02:46

that kicks off the DAG based build

micha17:02:33

that way it can be hooked into the watch task etc

mobileink17:02:36

I should probably write a blog post with some more detailed examples of what I have in mind. Ultimately I think this is the sort of thing for which we need some running code so we can see what would happen with it in actual use. I think it would work but there are undoubtedly hidden landmines. Plus you would want it to work seamlessly with boot as it is now - it's not really a major change of paradigm, just a different mechanism for dealing with task/data dependencies.

pandeiro17:02:41

are boot.properties transitive from a dependency to a project?

pandeiro17:02:53

like will a lib's BOOT_JVM_OPTIONS affect apps that use that lib?

micha17:02:57

@mobileink: totally

micha17:02:12

@pandeiro: they are not

micha17:02:49

@pandeiro: the boot program looks in two places for a file named boot.properties when it starts, it doesn't look on the classpath or anything like that

mobileink17:02:03

@micha thanks much for the feedback. I'll make some time to write up a blog post with more detailed examples and maybe some implementation ideas and then run it past you.

pandeiro17:02:13

@micha: cool, thanks

martinklepsch17:02:30

@pandeiro: also I think JVM options can't be set via boot.properties

martinklepsch17:02:44

(because it's Java reading the properties file)

micha17:02:51

@mobileink: if you need any info on lower-level fileset stuff don't hesitate to ask

micha17:02:11

you'll probably need to do some things to merge filesets etc in your thing

micha17:02:25

resolve merge conflicts etc

pandeiro17:02:40

if a lib specifies a BOOT_VERSION in boot.properties, is there any reason to include boot/core in :dependencies as well?

pandeiro17:02:02

same question re: org.clojure/clojure version in :dependencies

micha17:02:26

the boot.properties of the lib are not packaged in the jar, and they don't appear in the pom

micha17:02:08

i think having an explicit dependency on clojure in a library is good, because it provides information to consumers of the library about which version of clojure it is supposed to work with

micha17:02:24

so if it doesn't work you can look at the boot show -p output and see the conflict

pandeiro17:02:11

ok even if clojure is depended on with :scope "provided"?

micha17:02:14

depending on boot itself is currently somewhat broken, so you would need to use :scope "provided", which isn't ideal

micha17:02:29

so i recommend not depending on a specific version of boot in your library

pandeiro17:02:46

not even in boot.properties? or only there?

micha17:02:47

i think the :scope "provided" doesn't end up being very useful

micha17:02:15

i think having an explicit clojure dependency in your library that will be in the pom is what you want

micha17:02:36

the user will need to declare their own clojure dependency in their build.boot anyway or things will get weird

pandeiro17:02:51

interesting

micha17:02:14

i mean you'll pull in some random version of clojure from one of your dependencies no matter what

micha17:02:26

unless you explicitly depend on some specific version, which is what you should do

pandeiro17:02:35

so i should remove the boot/core dependeny from build.boot and just specify that in boot.properties, and remove scope provided from clojure in build.boot?

micha17:02:41

and that version should be the same as BOOT_CLOJURE_VERSION

micha17:02:54

exactly, yes

micha17:02:17

boot needs to be able to mask boot dependencies

pandeiro17:02:20

clojure with scope test or no scope at all?

micha17:02:30

no scope at all

micha17:02:40

the default is :scope "compile"

micha17:02:58

which means a regular dependency that will be visible transitively

pandeiro17:02:58

but this means that an app using e.g. clojure 1.8.0 will have to possibly use :exclusions [org.clojure/clojure] with my lib to prevent it using 1.7.0 (lib version) instead, right?

micha17:02:23

no, they will need to declare their own explicit clojure dependency in`:dependencies`

micha17:02:35

that will force maven to choose that version over any transitive ones

pandeiro17:02:47

ah right, perfect

micha17:02:59

but you need to do that anyway, because if you have any dependencies at all you will have some random clojure transitive dep

micha17:02:37

but the value here is that someone can do boot show -p and see which version of clojure your library was intended to work with

micha17:02:57

so like if they are using 1.6.0 tand you're using features from 1.7.0

micha17:02:03

and things break they can see why

micha17:02:42

if you had :scope "provided" or no clojure dependency at all in your library then boot show -p wouldn't show anything

pandeiro17:02:31

got it, makes sense. i'm fixing boot-http to work that way now

micha17:02:39

i think a good rule of thumb is that task libraries should have only one dependency: clojure

micha17:02:14

all other dependencies are loaded by the library into a pod, so they are not declared as transitive deps in the pom.xml

pandeiro17:02:37

ok that would go for things like bootlaces and boot-test as well?

micha17:02:48

yes i think so

pandeiro17:02:02

I could make boot-test a dev-dependency that only gets loaded in the test task, but how would I do with bootlaces?

pandeiro17:02:31

i have their scope as "test", if that helps

micha17:02:53

the scope is just to determine how consumers read the pom.xml when they load the jar

micha18:02:15

like if you have :scope "test" on a dependency that will be included in the pom.xml

micha18:02:36

so when a consumer of your library adds your lib to their deps, maven downloads your pom

micha18:02:48

it looks in the pom to see if there are transitive dependencies it needs to also load

micha18:02:15

if the pom has scope test for a given dependency, then maven won't load that dependency into the consumer's classpath

micha18:02:38

because scope test means it's a dependency that the library needs only for development

micha18:02:50

not when you use the library in another project

micha18:02:03

so if i understand you correctly you want to use boot-test while you're developing your library, but you don't want consumers of your library to have boot-test pulled into their project transitively, right?

micha18:02:34

if that's the case then :scope "test" is precisely the thing you want to use

micha18:02:15

that will cause maven to only load that dependency when you're working on your library, that is to say it will only load that dependency in your own build.boot

micha18:02:32

consumers will not even see boot-test in the output of boot show -d

micha18:02:43

they'll just see boot-http and its clojure dependency

micha18:02:43

i don't think there is any real need to have "dev dependencies" that are loaded from tasks

pandeiro18:02:59

yeah ok i see that

micha18:02:14

you can have all your dev dependencies in the :dependencies at the top level, at least for boot-tasks, since they don't have any transitive deps that could cause any problems

micha18:02:23

you can just use :scope "test" on them

micha18:02:29

and consumers won't see them

pandeiro18:02:41

but back to your guideline, maybe it could be libs should depend on clojure + stuff in test scope?

micha18:02:47

so you don't need to load different deps when you're testing and developing from when you are deploying

pandeiro18:02:57

and then anything else, in pods

micha18:02:57

right, yes

micha18:02:06

i meant deps that consumers can see

micha18:02:39

if everything but clojure is scope test then when a consumer depends on your library and does show -d they will only see clojure as a dependency of your library

micha18:02:52

they won't see any of the dependencies you have with scope test

pandeiro18:02:25

makes perfect sense

pandeiro18:02:47

but then how do we debug dependency issues within pods?

micha18:02:09

boot show --pod foo.bar --deps

micha18:02:21

2.5.0 and later supports this

pandeiro18:02:28

foo.bar is a ?

micha18:02:44

pods are named by default after the namespace in which (make-pod ...) was called

micha18:02:19

so like the cljs compiler pod, for example, would be named adzerk.boot-cljs, since that's the namespace that makes the pod

micha18:02:45

you can also do this (doto (make-pod ...) (.setName "foop"))

pandeiro18:02:56

ah ok, awesome

micha18:02:56

to set the name to foop

micha18:02:25

boot show --list-pods

micha18:02:27

also handy

pandeiro18:02:55

btw would this set the stage for being able to walk all a namespaces of a project and its deps and build a complete dependency graph?

micha18:02:08

i should mention that pods can be garbage collected and fall out of scope, so you may need to do something to have access to a given pod

micha18:02:16

like introduce some kind of delay or something

micha18:02:49

i had a sneaky idea about dependency graphs the other day

micha18:02:11

like instead of doing static analysis like tools.namespace etc does

micha18:02:30

what if you add a watch to clojure.core/*loaded-libs*?

micha18:02:41

then you could record the order in which namespaces are loaded

micha18:02:54

then you don't need a dependency graph at all

micha18:02:07

you have the linearized sequence of namespaces in dependency order already

micha18:02:20

with no need to inspect any source files at all

pandeiro18:02:02

is there only one instance of loaded-libs per any clojure process?

micha18:02:14

per runtime, yes

micha18:02:27

it's the global cache for compiled namespaces

micha18:02:45

so when you call (require ...) on a namespace that's already compiled it doesn't compile it again

pandeiro18:02:50

ok but then each pod would have one of those right?

micha18:02:05

each pod can have different dependency orders too

micha18:02:11

since everything in clojure is dynamic

micha18:02:41

seems like you could make a ns-tracker type thing without doing any analysis of source files at all

micha18:02:10

using only information from the runtime environment

dm319:02:55

oh wow, that's interesting

pandeiro20:02:58

@micha I hit one problem with removing boot/core from dependencies: if I try to execute a task within my test NS, it complains that it can't find boot/core boot/core_init etc

micha20:02:12

the tests run in a pod which won't have access to boot.core

micha20:02:36

so loading them there was probably not what you wanted

pandeiro20:02:39

I wanted to be able to call the task, run some tests, kill the task, repeat

pandeiro20:02:08

Otherwise I would just do it the way boot-cljs does, with a task invocation outside the tests

pandeiro20:02:05

I guess I will try just stringing together several serve + boot-test/test calls

richiardiandrea21:02:30

@pandeiro: if you feel adventurous enough there is a branch with that kind of workflow (`clojure.test` embedded in boot tasks) in https://github.com/boot-clj/boot/pull/401

richiardiandrea21:02:50

hopefully it will soon be merged to 2.6.0-SNAPSHOT

richiardiandrea21:02:21

but boot.built-in/runtests allows you to run (in parallel) tasks with is inside

richiardiandrea21:02:35

and collect results and display as usual

pandeiro21:02:10

neat @richiardiandrea thanks

richiardiandrea21:02:28

(if I understood correctly your problem of course)