Fork me on GitHub
#boot
<
2016-11-13
>
zane07:11:12

I'm kind of perplexed as to how to use Boot with the reloaded workflow.

zane07:11:42

It seems like the recommended way to serve static generated files is with boot-http. Does that mean that I don't use my web server component at all in development?

zarkone08:11:29

@zane hi! it's just convenient, because if you have only client code and while development you just plug (serve) and have web-server for static files out of the box

zane08:11:36

So, I do have server code.

zane08:11:55

But I'm having trouble serving the compiled cljs.

zane08:11:13

Should I expect compiled cljs to be available in the classpath after the cljs task has run?

zarkone08:11:24

ah, sorry. So you don't use serve?

martinklepsch09:11:53

@zane yes, should be available on classpath. Location is defined by your *.cljs.edn file (assuming you have one) — if you don't create one as described in the "Multiple Builds" section in the "Usage" page on the wiki

nha14:11:34

Writing a boot task to ass :exclusions for me when there are conflicts. Where can I find helper functions available in boot to manipulate the :dependencies vector (if any)?

nha14:11:02

(I am guessing something that convert :dependencies to a hash-map and from a hash-map back to a vector maybe exists already in the boot codebase?)

nha14:11:31

Ah looks like there are no public fns, just things like (reduce #(if-let [v ( %2)] (assoc %1 %2 v) %1) {})

micha15:11:26

@zane have you seen @danielsz's excellent "system" library? https://github.com/danielsz/system

nha15:11:42

Oh wow there are indeed functions 🙂 I just rewrote some for nothing. Thanks !

zane16:11:05

@micha, I have. I'm using mount.

micha16:11:20

@zane system works really well with boot, it has web server components and so on, perhaps that would at least provide inspiration for how you'd use your components in boot?

zane16:11:51

I've heard that.

zane16:11:13

My problems don't really have to do with how components are structured.

micha16:11:45

i'm just saying, you'd use your web server component in dev with system, which is something you asked about above

zane16:11:30

Right. I resolved that issue, but others remain.

zane16:11:14

Pods just seem like they conflict significantly with how I typically work.

zane16:11:30

clojure.tools.namespace.repl/refresh doesn't work out of the box.

micha16:11:53

each pod is isolated, they're separate clojure runtimes

micha16:11:02

so you would refresh them individually

micha16:11:34

but you also have the option of not refreshing them, you can rebuild them immutably instead

micha16:11:01

you can decide based on the tradeoffs

micha16:11:21

also you don't need to use pods if you don't need them

zane16:11:39

I'm talking about the main pod that everything lives in until you use the target task.

micha16:11:17

how do you mean? the main pod is just a normal clojure runtime

micha16:11:31

tools.namespace and so on works in there

micha16:11:46

a PR to ctn would make it default i suppose

zane16:11:01

I don't think this is CTN's problem?

micha16:11:15

ctn assumes static classpath

micha16:11:19

like lein

micha16:11:28

boot doesn't

micha16:11:40

ctn could discover the classpath

micha16:11:51

but it currently doesn't

micha16:11:54

so you need to tell it

zane16:11:23

I'm making the case that assuming a static classpath is a reasonable default.

micha16:11:39

but boot doesn't have a static classpath

zane16:11:51

Right. So Boot should handle updating the refresh directories.

micha16:11:01

ctn is not a dependency of boot

micha16:11:09

i dont' use it, ffor example

micha16:11:19

boot doesn't know anything about ctn

zane16:11:21

Yes, my sense is that most people who use Boot don't.

micha16:11:29

it's one line of code

micha16:11:48

and ctn could discover the classpath dynamically

micha16:11:57

i have pasted that code in the past, too

zane16:11:09

Right on. I trust you. I don't know the details.

micha16:11:39

boot is designed to be simple, which means we assume the user knows exactly what they want to do

micha16:11:04

once you start doing all kinds of things for the use you make it more complicated to understand

micha16:11:16

because then you need to understand the underlying thing plus the boot facade

micha16:11:57

like if we were to add code to boot that sniffs for people using ctn and then doing things secretly to adjust their ctn configuration

zane16:11:29

Totally up to you.

zane16:11:54

However, my experience of using it so far for this project has been a maddening series of incomprehensible errors like this one: https://github.com/ptaoussanis/sente/issues/250

micha16:11:38

yeah i am not a fan of ctn

micha16:11:40

personally

micha16:11:47

too complicated

micha16:11:00

but a lot of people in this channel do use it

micha16:11:03

and enjoy it

zane16:11:50

I would just argue that having a good story for how to interact with core libraries is probably a good idea, even if that story involves including yet another compatibility library.

micha16:11:42

a agree, and we are definitely open to new ideas on that front

grant16:11:00

@micha What does your workflow look like?

micha16:11:24

sorry i hope my tone didn't come off as being dismissive, i didn't intend that but i think maybe that is how it seems?

micha16:11:58

if you have a recommendation for how to improve the ctn situation please elaborate 🙂

grant16:11:13

It might be interesting for people to see an alternative to ctn that accomplishes the same kinds of productivity for you that they get from ctn.

micha16:11:20

@grant my workflow is minimal, i use vim with fireplace, and i have a separate repl server + client running at the same time in a tmux screen

micha16:11:37

i reload namespaces with simple require :reload

micha16:11:09

when i do cljs i don't use a cljs repl, i prefer using the live reload as my repl

micha16:11:34

i also don't run test continuously while i program

zane16:11:44

Yeah, the above does appear to be the beaten path for Boot folk.

micha16:11:58

my namespaces are generally organized so that reloading is easy without special tooling

zane16:11:03

But it has issues (which are described in the c.t.n README).

micha16:11:24

yeah i personally don't run into those issues

micha16:11:32

i don't mind restarting my repl sometimes

grant16:11:53

So you don't use anything like Stuart Sierra's component or mount or anything like that?

zane16:11:09

> recommendations I'd probably restructure that wiki page a bit and provide a task somewhere that, when run, makes c.t.n compatible with Boot without requiring that you subscribe to automatic refresh.

micha16:11:11

the one thing i can't abide is stale state in my repl that makes me think something is working when it is only working because of some var that is in the repl that doesn't exist in my code anymore

micha16:11:19

or worse issues than that

micha16:11:57

i use boot tasks in the way most people use comonent

micha16:11:01

component

sparkofreason16:11:45

Has anyone used pulsar (http://docs.paralleluniverse.co/pulsar/) with boot? Barring that, is there a convenient way to get the same functionality as lein :java-agents?

micha16:11:53

boot tasks naturally implement the component workflow, ie. start and stop events

grant17:11:33

That would be an interesting example to see. Boot task vs. component.

micha17:11:32

line 4 is where you set up initial state, etc, that's like the start event

grant17:11:42

I think there are some parts of boot that I still don't fully grok. It seems like using boot tasks and getting the same effects as ctn and component working together should be obvious, but it's not clicking for me at the moment.

zane17:11:46

It's worth noting that c.t.n has pretty wide popular support, though.

zane17:11:55

It's built in to Cider, for example.

micha17:11:02

between lines 4 and 5 you could add a call to boot.core/cleanup which is a macro

micha17:11:13

the body is evaluated when the pipeline terminates

micha17:11:17

that is the stop event

micha17:11:24

@grant note that dependency graphs are redundant there, since boot pipelines are already linear

zane17:11:15

That, of course, means that you lose out on any benefits provided by tools that automatically manage dependencies (e.g. mount).

zane17:11:32

Also, the dependencies between tasks are implicit rather than explicit.

zane17:11:41

I get where you're coming from, but that approach probably isn't right for me, personally.

grant17:11:41

Just to check my intuition, boot runs a single pipeline per invocation. The pipeline being a sequence of tasks, that each have the cumulative fileset threaded through them. That sound right?

micha17:11:52

they're explicit, because you explicitly order them

zane17:11:05

The explicit ordering isn't sufficient to infer the dependencies.

micha17:11:27

in practice it's not a problem

zane17:11:33

Knowing that there is an ordering doesn't tell you anything about what the dependencies might be.

zane17:11:42

It has been for me.

micha17:11:53

like when i make the pipeline i know which things need to be done first

micha17:11:12

way easier than reasoning about a graph that might do something surprising

zane17:11:21

Right. The problem I've been trying to highlight throughout this discussion is that boot is pretty hostile to people who don't "just know" things like that.

zane17:11:17

If I built a complex system where components were started and stopped in Boot tasks and introduced a new contributor to that system he/she'd have a much harder time ramping up on what the dependencies are between those components.

micha17:11:30

i don't agree there

micha17:11:45

i think looking at a pipeline of sequential tasks is easier to grok than a web of dependencies

micha17:11:02

consider core.async vs nodejs callback pattern

zane17:11:03

> I think looking at a pipeline of sequential tasks is easier to grok Okay. 😄

micha17:11:37

i found component to be difficult to debug

micha17:11:48

when i had to work on projects using it

micha17:11:27

the dependency graph is easier for simple things, i think

zane17:11:39

I'm just giving you feedback here.

micha17:11:43

but pipeline is better for more complex things

zane17:11:51

It sounds like your response is, "Your experience doesn't match mine."

zane17:11:54

Which, like, okay.

micha17:11:11

yeah i mean boot was intentionally designed to not have dependncy graphs

micha17:11:10

boot tasks are designed to be composable, which means they can't know anything about each other

micha17:11:32

once they get coupled you can no longer compose them

micha17:11:48

like currently you can swap out boot-reload with your own live reload implementation, no problem

micha17:11:01

and without any of the other tasks knowing or caring

zane17:11:21

@dave.dixon: I haven't, sorry.

nha17:11:11

In :exclusions, is is possible to specify a specific version? Or does it have to go at the level of one specific dependency? Ex. (set-env! :exclusions '[[my-lib "2.0.16"]]) does not seem to work for me

micha17:11:01

@nha it's by id, like :exclusions [org.clojure/clojure]

micha17:11:44

however i recommend against using that in most cases

nha17:11:54

So I can't exclude [org.clojure/clojure "1.2.0"], right? I have to put in :dependencies for the lib. that imports it

micha17:11:08

it's sufficient to simply specify directly the version you want in your :dependencies

micha17:11:36

that will prevent the artifact from being loaded via transitive dependency

nha17:11:00

Oh you mean not using exclusions but put something at the top level of dependencies? I thought they were all treated the same

micha17:11:19

if you exclude it then no version will be loaded transitively

micha17:11:30

but you usually want a specific version

micha17:11:07

the only time you'd want to use :exclusions i think is when there are transitive dependencies that are not being used and you want to remove them from the classpath eg. to make an uberjar smaller

micha17:11:53

if you want to prevent the wrong version from being added to the classpath i would recommend just adding the right version as a dependency

nha17:11:13

Ok got it, thanks 🙂

micha17:11:28

also if a particular dependency is bringing in the wrong version of an artifact you could add :exclusions to that dependency

micha17:11:13

that's nice in some ways because then you can then update other dependencies that would pull in a different version of the transitive dep

micha17:11:31

without needing to update your top level dependency

grant17:11:49

@micha You'd only use something like https://github.com/micha/multi-module-build/blob/master/build.boot when you need to run two independent pipelines at the same time. Is that correct? Basically, it just gives you a task that is equivalent to running boot in two terminals?

micha17:11:24

@grant right, plus that example has dependencies between the two, which is a little more than just running in two terminals

grant17:11:43

Ah, I missed the dependencies.

grant17:11:49

Pipelines are not explicit outside of something like task using comp to put other tasks together, are they?

grant17:11:43

i.e. There is no pipeline data structure.

micha17:11:04

no, just functions

micha17:11:30

i guess there is a data structure, but it's formed by the stack

grant17:11:01

Gotcha. ring style vs pedestal style of putting thing together.

micha17:11:09

precisely

grant17:11:58

Has there been any thought/discussion of moving the pedestal direction? Or this works well enough so no need?

micha17:11:32

one thing that i would like to explore is decoupling the work the task does (ie. transforming the fileset) from the control flow (ie. middleware pattern etc)

micha17:11:06

so you could have other ways to schedule the work

grant17:11:39

Ah, cool, sounds like an interesting experiment.

micha17:11:45

but there are a ton of things that introduce coupling when you have for instance a live repl

micha17:11:34

the pipeline model is simple and robust, and it allows many optimizations and things

grant17:11:25

That is what I was trying to wrap my head around. Keeping the notions of the source code, the eval'd code and the running system with all it's state clearly separated. At least as concepts in my thinking, if not artifacts in the build and running processes.

micha17:11:51

that's where the pipeline simplifies everything

micha18:11:05

things happen in the exact order you specify

micha18:11:24

as a series of transformations on immutable fileset objects

micha18:11:58

with the reactor pattern or whatever you lose that

micha18:11:38

but boot doesn't enforce that, you don't need to use the pipeline

grant18:11:45

For building a jar, that seems easy to grok. Adding things like figwheel and updating the backend in a live REPL all running in the same boot process, seem to muddy the waters, at least for me at them moment.

micha18:11:21

figwheel is a monolithic thing that doesn't really integrate with boot because it assumes ownership of named files in the filesystem

micha18:11:32

that's pretty much incompatible with filesets

micha18:11:54

but there are boot tasks that do the same things, but as separate components

micha18:11:05

eg boot-reload, boot-cljs, boot-cljs-repl, etc

micha18:11:16

and boot-http

grant18:11:26

Ah, maybe that is what I'm feeling more than me not getting what I'm doing with boot.

micha18:11:49

it's the unix philosophy really

micha18:11:21

you make apipeline of individual programs that do a single thing, and communicate via stdin/out

micha18:11:31

the stdin/out is the fileset

micha18:11:41

and the pipe is the middleware pattern

grant18:11:53

I try not to fight my tooling. If stuff feels awkward, I try to take a step back and figure out what assumptions or models are not lining up, either in my mind or between two libraries.

micha18:11:37

boot is really a tool to help you make your own tooling

micha18:11:47

a toolkit kind of

grant18:11:07

Yes, and I very much like that.

micha18:11:14

because any one-size-fits-all solution will end up being hopelessly complex

grant18:11:22

Thank you, by the way, for making it.

micha18:11:26

with tons of protocols and so on

micha18:11:35

you're welcome! 🙂

micha18:11:28

also a major goal is that once you have developed your own stuff you can continue to use it for a long time without updating anything

grant18:11:22

You mean in a backward compatibility sense, for tasks?

micha18:11:00

yes and exposing a low enough interface that you can add little things as you need them without disturbing your previous work

micha18:11:33

if boot does everything for you you'd end up needing to modify things rather than compose them

micha18:11:03

i mean if boot made assumptions about what you're trying to do

micha18:11:35

that's the unix philosophy as i see it

micha18:11:45

small pieces that once you write them they're done

micha18:11:55

and to do new things you compose them with other small things

grant18:11:31

I like boot's trend towards simple composability over out of the box doing most things for you. We have lein if we want something easy for the common case.

micha18:11:42

exactly yes

micha18:11:14

and lots of stack overflow answers for how to configure lein, so it's good for beginners in most cases

grant18:11:32

Yeah. It handles the 80% case really well.

grant18:11:03

Is there a good way to think about the live REPL at the end of a pipeline? Say as somewhat similar to a target directory, just files get loaded. Or is there some other mental model I should be trying to apply when thinking about running things vs. just dropping artifacts?

micha18:11:48

the repl task is for side effects only, basically

micha18:11:53

like the webserver task

micha18:11:26

it starts a server once, and that server runs independently until the pipeline ends

micha18:11:44

the classpath changes as the filesets are commit!ed

micha18:11:05

it doesn't really matter where in the pipeline the repl task goes

micha18:11:28

the cljs-repl task is a little different because it generates code that is compiled and sent to the client

grant18:11:35

Ah, that is interesting, it sees everything no matter where it is in line?

micha18:11:53

well it's just concerned with the jvm classpath

micha18:11:00

i mean the classpath of the pod it's in

micha18:11:09

you can start a repl server in any pod

micha18:11:15

but the default is the main pod, of course

grant18:11:31

The one named core, right?

micha18:11:33

the repl server itself doesn't care about filesets or anything

grant18:11:18

It just exists in whatever pod it is started in. And that pod's classpath can be updated by other tasks?

micha18:11:47

whenever you do (commit! fileset) the classpath of the core pod is updated

micha18:11:35

other pods will have their classpaths updated also if they are created the default way, which includes the directories on the core pod's classpath in their classpaths

micha18:11:15

like usually you'd make a pod like (boot.pod/make-pod boot.pod/env)

micha18:11:25

that's like a copy of the current pod

micha18:11:49

or if you wanted to add dependencies to the new pod in addition to the dependencies in the curent pod

grant18:11:30

But they still see whatever gets committed by another pod, as long as they are in the same place in the fielset?

micha18:11:35

(boot.pod/make-pod (update-in boot.pod/env [:dependencies] into '[[foo "1.2"] [bar "3.4"]]))

micha18:11:59

pods are isolated clojure runtimes

micha18:11:15

the mechanism used to achieve isolation is classloaders

micha18:11:25

clasloaders are configured with urls

micha18:11:54

each url must represent either a jar file or a directory

micha18:11:18

the directory in that case could contain the same contests as you'd expect in an exploded jar file

micha18:11:24

*contents

micha18:11:52

if you add a jar file to a classloader the classloader considers it to be immutable

micha18:11:04

however if you add a directory it does not

micha18:11:27

so if you have two pods A and B and both have some directory added to them

micha18:11:34

and you add a file to that directory

micha18:11:43

that new file will be on the classpath of both pods

micha18:11:06

because they both are configured with the same mutable place in the filesysytem

grant18:11:18

So pod's are not meant to be completely independent environments, the fileset is an orthogonal concern? They just isolate the classloaders. With multiple pipelines, you'd have independent filesets? Or not, because they both point at the same place on disk and that gets read in when it changes.

micha18:11:57

remember that filesets refer to anonymous temp directories

micha18:11:03

not named places

micha18:11:10

well they have names, but they're gensym

micha18:11:33

but if you do (get-env :directories) you can see them

micha18:11:45

those are the directories on the classpath

grant18:11:58

The gensym ones.

micha18:11:09

those also are directories in the fileset that are synced when you do commit!

micha18:11:26

if you pprint a fileset object you will see them in there

micha18:11:29

the same ones

micha18:11:48

filesets and pods are orthogonal concerns though

micha18:11:53

like you said before

micha18:11:19

filesets are the way you can manage the directories on the classpath

micha18:11:46

pods are the way to isolate classpaths

micha18:11:55

but the isolation doesn't need to be complete

micha18:11:02

it can be, or not, as you wish

grant18:11:17

Is it wrong to think of filesets as analogous to a ring request? In the way the are what is threaded through the tasks?

micha18:11:29

yes that's how i think of it

micha18:11:42

and it's imutable

grant18:11:45

Ok, good, glad I had that part right. 🙂

micha18:11:56

so you can hold onto a request map

micha18:11:07

and you don't need to worry that some subsequent task will modify it

micha18:11:18

that's the value of filesets

micha18:11:30

they make it possible to rewind the side effects of the pipeline

grant18:11:34

Each commit! gets you a new fileset.

micha18:11:49

well each operation on a fileset produces a new fileset

micha18:11:59

like (add-resource fileset some-dir)

micha18:11:18

(commit! fileset) syncs that fileset object with the underlying directories

micha18:11:27

and returns the fileset for convenience

micha18:11:42

it's like restoring from a snapshot

micha18:11:56

that's how you can rewind the effects of the pipeline

micha18:11:38

you can hold onto a fileset, and later if you commit it then the underlying filesystem will correspond to that fileset object

micha18:11:58

regardless of what's been written, updated, modified, or deleted in the meantime

grant18:11:18

Oh, so you can stomp over stuff by committing an older fileset.

micha18:11:23

you can also hold onto a fileset and later compare it to another fileset

micha18:11:31

get a diff of the two

micha18:11:45

so you can see exactly what has changed, very efficiently

micha18:11:12

this is what we use in boot in place of the dependency graph in say Make or Lein

micha18:11:34

every task runs, but they don't need to do anything unless there is something interesting in the fileset diff

micha18:11:16

instead of the build tool trying to figure out which tasks need to run based on which files changed, like in make, boot lets the task itself decide

grant18:11:42

Every task that you list, right? Not all tasks always run every time?

micha18:11:56

i mean taks in your pipeline

grant18:11:22

Yeah, that is what I thought you meant, just wanted to make sure I wasn't confused.

micha18:11:32

they can just call the next task if there is nothing for them to do

micha18:11:38

usually what you'd do is like this

micha18:11:50

your task will have a cache directory, a temp dir of its own

micha18:11:08

each time the task runs it compares the fileset to the fileset from the previous run

micha18:11:11

gets the diff

micha18:11:24

it can then decide from the diff which files in its cache are invalid

micha18:11:35

it can delete them and rebuild them in its cache

micha18:11:47

or if no files are invalid it can skip that step

micha18:11:57

then it adds the cache dir to the fileset and calls the next task

grant18:11:47

Oh, temp files persist across runs. I had missed that somehow. I thought they were only in place while a pipeline was running.

micha18:11:58

no, you're correct

micha18:11:11

although you can make persistent caches too

micha18:11:21

but most of the time you don't want to

micha18:11:29

because cache invalidation is hard

grant18:11:54

Indeed, I avoid it as much as possible.

micha18:11:09

yeah once you're bitten you end up doing make clean every time anyway

micha18:11:15

in self defense

micha18:11:43

boot doesn't let any stale state accumulate across pipelines

micha18:11:49

unless you specifically make persistent caches

micha18:11:04

like boot does use persistent caches for uberjars for example

micha18:11:15

the cache key is the md5 hash of the jar file

grant18:11:21

So really the cache comes into play with something like watch, were your next task gets called multiple times.

micha18:11:27

it's to make exploding jars more efficient

micha18:11:35

yes precisely

micha18:11:59

during the pipeline the fileset diff makes sense with caching

micha18:11:13

but across pipelines that wouldn't work

micha18:11:25

because you don't know what the correct state is when you start

micha18:11:42

if you only cache within the pipeline you know what the initial state is

micha18:11:49

empty 🙂

grant18:11:23

Haha, rock bottom is real solid ground. 🙂

micha18:11:20

i like that

micha18:11:28

that could be my new motto

grant18:11:54

Thanks for going over some of this stuff with me Micha, I appreciate the time. I think I still have some pieces to put in place, but it definitely helps to be able to check understanding and assumptions.

micha18:11:09

sure, any time