Is it possible to change the boot env for a single task run? E.g. a build task can’t be called from a repl because I might have dev-only source-paths added, so I’d like to be able to make it set a specific source-path for building without affecting the source paths in the repl as a whole. Is this possible?

may be an obvious question, but I’m only a day into boot and haven’t been able to figure out how people handle these situations.

Like-wise for wanting to run tests from the repl without adding the test sources into the main repl

i think what you might want to do is create a separate repl in a pod

and have the main pod have the correct classpath for running tasks

because you need to run tasks in the main pod context

because there is state there that boot needs

you can use the boot.pod/make-pod function to create a new pod

then you can use boot.core/launch-nrepl to launch a repl server in that pod

It may just be that I don’t grok pods, but I thought I’d seen that a REPL can’t be in a pod). Furthermore, I don’t see how that fixes the problem of runnign tasks that have different source-path needs. Like running a (boot (tests)) then (boot (build)) as the former will affect the contents of the built jar

there is a way to run boot in boot

which is the way to do that

but if this is your first day you may not want to get frustrated with that

but you can totally make a repl in a pod, no problem

i mean the repl you're in is in a pod

with boot all clojure runs in pods

so multiple reps are you meaning having multiple processes or having a repl that I exit every time I need to run a boot task in the main repl, then re-enter?

you can have multiple repls, yes

it's unusual to need that though

in the smae process? I don’t visualise how that’s useful

because it's rare to be both developing, testing, and making production builds all at the same time

i'm not following

re: repl in pod, https://github.com/boot-clj/boot/wiki/Cider-REPL seems to allude to repls being special and that they can’t run in a pod because it needs to be in project context AND have access to boot context

@micha: frankly, I’m probably just confused. Boot seems to try to be quite functional but then has a global shared, mutable, env

yes, the repl task can't run in a pod

but you can start a repl in a pod

it's just that tasks can only really run from the main pod

which is the pod your build.boot is evaluated in

@micha: use-case where I first ran into this: a build task which first runs all the tests to check that it’s passing before building. The former corrupts the env which then makes the build artifact include tests

ah you can use the boot-test tasks

that runs tests in a pod

so as not to affect your main environment

I am using boot-test. It doesn’t see my test paths unless I first add them to the env though

it doesn’t appear to have an option to specify test paths either

based on boot bt/run-tests -h, for example

hm

one sec

@micha: I was precisely expecting boot-test to provide a way to declare test paths for it to use (I was also expecting it to have a task that would only call the next-task if the tests passed, but instead I'd just use the version that throws an exception, which seems good enough)

https://github.com/SevereOverfl0w/boot-snippets/blob/master/src/io/dominic/boot_snippets.clj#L14-L43

so I suppose what you are saying is that I could make my own test task which creates a pod, adds the path, then calls boot-test’s task, which I suppose creates yet another pod?

@micha: at a glance, that looks like exactly what I was hoping for

hah except that it mutates the state, then resets it

which seems less nice

that's how dynamic bindings work

it's not really doing anything bad

like with clojure when you do (binding [*foo* bar] ...)

that's mutating state

but clojure guarantees that it's only visible to a single thread

which is also what boot does

the boot pipeline runs in a single thread

micha: fair enough

I was expecting it to surround it with a pod

but I guess that’s probably a bunch slower for no real gain

pods are completely separate runtimes though

so you can't really wrap anything in a pod as such

like a clojure vector in one pod would be just an unknown java object in another pod

any clojure data basically

because each clojure runtime creates the classes and interfaces for core collections etc at runtime, dynamically

so the class clojure.lang.PersistentVector in one pod is totally different from the class named clojure.lang.PersistentVector in another pod

Yeah, that’s how I already understood it, but probably not deeply enough to grok all implications. You earlier stated that boot-test creates it’s own pod so was vamping on that idea a bit

yeah the test pod is like a sort of worker

you can send it a job and it will doit

yeah I was reading the code a bit and did see that

I guess having boot-test just accept some extra source-paths would be neat

yeah that's a good idea

i wonder if there is already an issue for that

ah no

i'm working on boot stuff this weekend, i'll see if i can add that

@micha: in the “boot for lein users” pages I don’t see dev dependencies listed much. From reading a bunch of boot scripts in different projects it seems like some people are just adding deps to the env in certain tasks (which I have been) while others are adding them to the root env but with scope “test”. Are there any implications of one approach or the other or are they specifically for different use cases that I haven’t picked up on yet?

@micha: want me to open an issue?

i myself havent had the need to have separate dev dependencies ever

i'm curious to know about your use case

why do you need to separate the dev deps?

(i use the :scope "test" method myself)

@micha: well for example I don’t need asset autobuilding in production (i’d precompile it all to be served), or in dev I might want a ring middleware which displays extra debug information on exceptions, but in production I wouldn’t expose exceptions at all

normally i'd use environment variables for that kind of thing

like my app wouldn't use that middleware if it's in production according to the env

micha: I do too but it’s common practice at least in lein land to not even include those dependencies in, for example, an uberjar for prod

the uber task will only include certain scopes

test isn't included in uber's explosions

so in my case, I have a dev-only source-path which might expose middleware to be wired up and require the dev-only middleware

yeah that’s what I have inferred from reading other projects’ build.boot files

i too have been adding test scope to those dependencies

it's also quite easy to use the sift task to sift out things you don't want

but since the “for lein users” doc pages don’t explicitly make that connection, I wasn’t 100% sure I was on the right path there

before you make the jar

yeah I am using sift now to clean out source assets and .clj files that are aot’d etc

I think :scope “test” is the clearest way I’ve seen so far and it sounds like there isn’t a gotcha with that that I’ve missed so should be a-ok.

in general we have tried to avoid adding things like directly specifying dependencies to tasks etc

because it starts to get more complicated to think about

yeah

agreed

i like scope test, i can do eevrything i need to do using that technique

i mean as far as test dependencies etc goes

yeah it works fine for “dev dependencies” (which aren’t test in the sense they aren’t strictly there to support any kind of automated tests) it’s just the name (which I understand is a maven thing) through me momentarily

for uberjars i've been having very good results using a very small shim that gets AOT compiled, leaving my actual application as source

this: https://github.com/boot-clj/boot/wiki/Circumnavigating-Watch-and-AOT-Compilation-Limitations

yeah I suppose with :aot :all things you don’t require won’t get aot’d so simply wiping any /.clj[cs]?$/ files is enough to remove dev deps anyway

AOT has all kinds of weird footguns lurking, i try to avoid it

hmm

the main idea there is

(ns foo.main
  (:gen-class))

(defn -main [& args]
  (require 'foo.core)
  (apply (resolve 'foo.core/-main) args))

to AOT compile that namespace instead of foo.core

then you can use the watch task while you're working on your uberjar

micha: yeah I’m actually doing this incidentally anyway, though I’m aot’ing everything atm

and you won't get issues with recompiling a class that's already been loaded

I’m not doing any kind of compilation with watch atm

uberjars get a huge benefit from incremental compilation because boot can cache everything

I can imagine

so like the first compile might take 8-10 seconds to build the uberjar

and then after that it's like 1s

if you’re watch’ing only right? My understanding is that boot doesn’t maintain any kind of cross-process cache

that's right, although uberjar's do use a persistent cache

there is a facility in boot for making persistent caches

both global and per-project

the uber task uses the global one, with the hash key being the md5 hash of the jar contents

i mean of the jar file itself

so once it unzips a jar it can cache all of that

so the second time you build an uberjar will be a lot faster than the first time

even without the watch task

this is possible because boot knows exactly when to invalidate the cache

if the jar file's md5sum hasn't changed then it knows that the cached contents are ok

and it can use them

makes sense

so tl;dr boot will cache things, but only when there is a foolproof way to invalidate the cache

so you never get stale things

can the content-addressed cache strategy be used for :aot?

(none of this is really important to me right now, FWIW, I’m just curious and playing with boot in a fresh project so I don’t have any strict needs or anything)

it's not used for that currently, i'm not sure how that would work

is it possible to know if a compiled class is "correct"?

like how would boot know to evict a compiled class from the cache?

also a single namespace will be compiled into multiple .class files, which is also weird and hard to think about

ah yeah of course

i just remember having a terrible time with caching in every build tool i've ever used pretty much

and you end up running the clean task all the time out of desperation

it's extremely depressing to spend an hour debuggin some crazy problem that turns out to be caused by stale artifacts

or your thing works and you work for a few hours, then you clean and rebuild and everything is broksn

also incredibly depressing lol

yes I too have been there

anyway thanks for the discussion. It’s been very helpful. Unfortunately I have to go to some grocery shopping and other real life things.

later! if you have any issues let us know

i mean with boot

not with groceries

hah

(deftask api-test []
  (merge-env! :source-paths (get-in config [:api :test :source-paths])
              :dependencies (get-in config [:api :test :dependencies]))

  (require 'adzerk.boot-test)
  (let [api-test (resolve 'adzerk.boot-test/test)]
    (println "Hello World")
    (comp
     (watch)
     (api-test)
     #_(report-test))))

Can you run this in the repl? Seems like when I run (api-test), it returns the comp function

I mostly just want to save memory by not having to run a separate jvm instance for testing

@taylor.sando: does (boot (api-test)) not work in the repl?

it looks like it should work ok

oh wait, it's recursive

i'm not sure my brain can handle that

oh nevermind, it's not recursive

whew

I wasn't running the boot before api-test. Can you run the watch task in a future or delay so that it doesn't take over the repl?

you can do (def b (future (boot (api-test))))

then to end it you can do (future-cancel b)

Does boot-test always have a really long delay to run? I'm getting 16 seconds for just one test.

is datomic involved?

It's a dependency, but not involved in the one test I just made

Does loading datomic slow things down?

i think so

you can probably test it by removing things

Ya taking out the requires speeds things up a lot

boot test has a :requires option

you can use that to preload those and warm them up

should help some

> :requires Extra namespaces to pre-load into the pool of test pods for speed.

@taylor.sando: ^^

I added :requires #{'datomic.api}, but it didn't really speed things up.

maybe it's osmething you're doing with datomic that's being slow

The test is just (is (= 5 5)) at the moment. I removed the datomic-specific tests

I'll just run a separate instance and use run-all-tests with the namespaces, it's not absolutely critical that I have this all in one jvm

how does multiple jvms relate to the slowness problem?

wouldn't it be just as slow in another jvm?

I can run clojure.test/run-all-tests without the same slowness. The separate jvm is just because I'm using mount at the moment, and you can't really have two instances of mount going. There is yurt, but it hasn't been updated to use the latest mount changes.

@micha with-env from boot-snippets doesn’t seem to do anything. I did a (show :environment true) inside and outside the block and the env seems unchanged

look at boot.pod/env somewhere in there

it should be different

requiring pedestal seems to really slow things down on the initial run

micha: egh actually I think it’s the way I’m using env. It really has to wrap the invocation of a specific task but I’m wrapping a whole composition of tasks which means when they actually are called the environment is probably already reset

er wtf no even that doesn’t work

tasks are kind of like stateful transducers, like the way the creation of the transducer is done separately from the use of it in a pipeline

tasks normally will create their pods in the creation phase

which should work, but some of them unfortunately do not

this is something that i'm working on fixing in the boot core tasks

and the ones i help to maintain

so you may be running into an issue related to that

I mean the with-env macro is really simple, so I am kinda surprised that this still has a :source-paths of just #{“src”}:

(with-env
    {:source-paths #{"src" "test”}}
    (show :env true))

oh that's differernt

that's showing the env that is in effect when the pipeline runs

so that's working correctly

the with-env macro only affects the environment during the creation of the task

for example:

hmm well this is what I was originally trying, but run-tests still does not use the altered source-paths:

(with-env
    {:source-paths #{"src" "test"}}
    (comp (watch)
       (bt/run-tests)))

I’m guessing that’s because it’s been reset by the time it actually runs as mentioned above

(deftask foo
  []
  (let [p (make-pod (get-env))]
    (with-pre-wrap [fs]
      ...

it might work if I wrapped both watch and run-tests, but that’s annoying

look at that task

if you wrap that with the with-env macro it will affect the env when the pod is created

which is what that macro is intended to do

eg

right

(deftask bar
  []
  (comp
    (watch)
    (with-env the-env
      (foo))))

i've only really used a macro like that when i had some annoying dependency conflicts and i wanted a shortcut to get something working temporarily

yeah I can tell I’m going against the grain here, but I guess I’m just used to a certain workflow.

Maybe I need to learn more abotu pods so I can jsut create my own

that sounds like a good plan

(comp (watch) (with-env {…} (run-tests))) probably might work in some way, but then I imagine the watch isn’t watching the tests, right?

the watch task just watches your source files

hence (let [e {…}] (comp (with-env e (watch)) (with-env e (run-tests))))

it doesn't really know anything about the tasks it will pass the fileset to for the rest of the pipeline

i suspect there is a simpler way to make it work

So recall I’m trying to avoid polluting the caller of a task with changes needed for the task. I started with a (set-env! :source-paths #(conj % “src”)) in the test task

i'd probably start there

but since I’m having a task which is a “dev” mode entry point, which starts a repl with just src and dev src-paths and an auto test runner with src and test paths, it’s getting messy

it's extremely rare to need to do so much shuffling of environment stuff

there is almost certainly a more straightforward way to organize the project

this isn’t about organization, it’s about workflow

what is the workflow you'd like to achieve?

I’ll take anything that lets me have a workflow of something like boot [cider] dev which starts an nREPL server that just has “dev” and “src” on the source-paths and starts a watch and test run with both “src” and “test” in source-paths

what is the purpose of having different classpaths?

that's the part i'm not understanding

It’s a matter of purity I guess. Each task should have the same classpath when run as a group as when run individually

if I just ran boot test it wouldn’t include dev

if I just ran boot repl it wouldn’t include test

but what's the purpose?

OCD?

i don't really have any philosophical preference, i just want to make computer programs

really, it just bothers me. Probably rooted in my background with languages like Ruby where when A loads B and then loads C somehow C knows about B and it means a lot of spooky action at a distance so I have a keen aversion to that kind of pollution

i put all the things i need on the classpath, and just go from there

luckily we don't need to do those kinds of things in clojure, though

micha: thank god, I’m just giving context for this particular tick of mine

in any case, it does seem strange to me that the fileset etc is strictly functional but the environment is not

you should give the straightforward alltogether way a shot and see how it goes, maybe you'll enjoy it

from an ergonomic perspective as a first time user, it was very surprising. I expected a change in env to propagate to a task and it’s sub-tasks but not to the previous task

the fileset isn't strictly functional, though

it's still closely coupled to the classpath

which is definitely mutable shared state

what the fileset gives you is a way to manage that state

Maybe not strictly functional but the method of passing the input and output fileset between tasks is still pretty functional, compared to the environment

right, and the fileset is an immutable value, too

micha: sure but it was surprising to me that there wasn’t a great way to manage the state of the environment

but the immutable fileset object is useless if it doesn't directly relate to the actual mutable classpath

well there is

i mean the classpath isn't a value

it wouldn't work if it was

because the whole purpose of it is to be shared state

if each expression had its own immutable classpath then you couldn't add dependencies

or anything like that

but anyway you're going to be interacting with java things that expect to use a noremal java classpath

so reinventing that wasn't really an option

one of the things about the jvm is that you can't unload classes reliably

so adding a jar to a classloader is not a reversible process

or a directory, for that matter, but boot does a thing that makes it work

Yeah I understand that. My desire for separation of the class path in these different modes is probably because I’ve picked up from projects like luminus and examples in system/mount, etc, patterns like declaring the same namespace in different source-paths to do things like vary configuration (e.g. diff middleware definition) with minimal conditionals. That’s really really nice, but it requires being able to not include a certain path in the source-paths at certain times

you can use pods to have separate cojure runtimes each with their own different dependencies

this first actually bit me because my build task wouldn’t run from the repl because my dev-only user.clj depended on boot but was getting accidentally included into the jar because “dev” was on source-path and then obviously not compiling

why wouldn't it compile?

because boot is not in the uberjar

it’s not a project dependency

are you talking about AOT or just require?

require

why would that be needed in the uberjar?

running (boot (build)) complained about not finding boot.core if AOT or similarly when later run if not AOT

it’s not needed in the uberjar. It’s not wanted. It was a side-effect of having dev dir on the source-path

having the file in the jar isn't a problem though

only if you require it

which is what prompted my quest to have the source paths specifically set during the execution of a certain task without pollution other parts

boot tasks are similar

dev/user.clj requires boot. That’s a file I only expect to ever be relevant when developing locally, where boot is present

if you look at most boot tasks, like boot-test for example

there is a namespace adzerk.boot-test.impl or something like that

it depends on things that are not in the environment that you run the test task from

(which works great IFF “dev” is nto on the source-paths set when uberjarring happens)

it can only run in the pod that the task creates

it's in the jar file

but it's not a problem because you don't require that namespace except in the pod

there is no need for fancy footwork there

if your -main function in the uberjar doesn't require the user namespace there won't be a problem

unless you're doing AOT compilation for every namespace, in which case you'll probably have many weird issues to untangle

hmm

well I’m gonna step back a bit and stop trying to go against the grain

I’ll live as long as I can with the single classpath and just ignore it

haha give it a shot

it’s probably just OCD on my part but it really bothers me that different tasks pollute the env of others

maybe don't think of it as pollution

I wish env passed between tasks just like filesets

though env wouldn’t be the ideal name in that case hah

maybe ctx

one thing about tasks that you will like is that no boot task every has transitiev dependencies

that is, tasks don't pollute the classpath

you can add as many boot task dependencies to your project as you want, you can never have a conflict

that's the place where it's really important to maintain discipline and hygiene

yeah… I probably don’t fully grok that yet.

because when you add a dependency you're using someone else's code

however the stuff you have in your project is controlled by you

so you can organize it so the dev things don't conflict iwth other things

i'm not familiar with the patterns you describe about overloading namespace names and sort of metaprogramming in the project.clj by including or excluding them

It just always feel wrong whenever I have to go change some thing over there to change the behaviour of something over here. I’m chanigng source-paths in a global env so that a task I’m about to call which will check that env for that config can act on it. It’s impossible to tell the watch task, for instance, to watch “src” and “test” without telling everything everywhere that “test” is now a source-path 😞

but i am kind of skeptical of those patterns

well you're setting up the jvm classpath

I know the source-paths affect the classpath at certain points, but watch, for example, doesn’t strictly need to modify the cp

it’s just taking advantage of a known key that has that implication

it's looking at the classpath

that's what it's watching

huh that’s annoying

you can make your own watch task, though

I mean not really annoying I guess

there is also a "manual" option to the watch task

it’s just my mental model is not quite right and is taking a bruising

For bootignore, how would you exclude all subdirectories? I've tried #"^src/dev/path/to/dir/.*?" #"^src/dev/path/to/dir/*" and #"^src/dev/path/to/dir"

the patterns should match paths in the fileset

not paths in the filesystem

What would the pattern look like if I want to include "src/main", but exclude "src/main/do-not-want"?

first do boot show -f

that will show you the paths you will be matching against

do you see src in the output of that?

No

so any pattern with src in it won't match anything

Ok

you can try different patterns until boot show -f produces the result you like

@micha: what's the point of uberjar-ing in dev mode?

we are using the reloaded.repl flow and that ensures <1s reload times

or were u talking about some CI server making the uberjars in 1s?

@onetom: just if i want to run the uberjar while i'm working

for whatever reason

are people typically using boot-reload instead of figwheel?

@bo: i haven't even realized there is a boot-figwheel, but as i see it just brought over the same declarative config approach which proved to be a bit rigid in lein.

we just started using boot-reload because about a year ago but it's still a bit unclear to me how to make it play well with hoplon and have sub-second updates without reloading the whole page.

@micha: What is the reason for this .git check: https://github.com/boot-clj/boot/blob/master/boot/base/src/main/java/boot/App.java#L129-L133

Boot only reads build.boot from workdir so I don't understand what is this projectdir supposed to be

i think it was for if you have subdirectories with their own build.boot files in them

@juhoteperi: ^^

i don't think it's officially documented, so we could remove it

Yeah, I don't know if causes any problems but looks like it might if the project is not git project

And doesn't seem very useful

yeah it was premature

Empperi mentioned some problems on IRC about Boot not reading boot.properties file (on Windows)

that's a case of less-is-more i think

Doesn't look like that is the reason, but we wondered why does it exist

maybe we remove that in 2.6.0

I'm looking forward to 2.6

Hi there, to do multiple builds with boot-cljs, does it matter where in the project you put the *.cljs.edn files? In the wild, I've seen examples with the edn files under src, but I've also seen them under resources/js.

Anything in the input-path of the fileset with a .cljs.edn extension will be available.

Anything in source-paths or resource-paths

Cool. Thanks @taylor.sando

Is the init-fns mandatory?

It seems that boot-cljs does the right thing in the default case (when no builds are specified). How does it know what the main function is?

It doesn't. I will just require all namespaces in the fileset.

@juhoteperi: Ah, I see. Cool. So can I omit the init-fns in the ednfile?

Ah, I suppose not.

Ah I meant the case with no cljs.edn files

But yes, you can usually omit init-fns

If your namespaces have top-level calls to start the app, init-fns is unncessary

OK, very nice. Thanks.