i am concerned boot is building itself, it's not supposed to be a build tool

just kidding, great work @martinklepsch 😉

haha

@micha: the idea was to also run the pretty printed script

ah, it can be very slow

it added like > 100 ms to the time

i firget exactly how much

but it would also depend on how big the script is

i also tried with fipp, it was still slow

man, i'm very close to having boot download the app automatically

extremely close

are you doing some work on boot today?

yep

yesterday i got a lot done

cool, then I won’t feel as bad for asking stuff on sundays 😉

my changes add full support for all boot env vars in boot.properties files

you can have boot.properties in the cwd, in a parent directory if it's a git repo, and in BOOT_HOME/boot.properties

it processes them in order, merging the properties, BOOT_HOME, then git project, then cwd

then it merges env vars

I had this idea yesterday of a higher order task that executes some other tasks with a scoped fileset, e.g. only stuff in boot/worker/

then it merges system properties

also boot --version prints all config that starts with BOOT_

for any of those places

so when someone pastes the output we can see their whole setup

nice

yeah i made a macro for that

(with-env [{:dependencies ... :source-paths ...}] ...)

I got pretty far with an approach only using fileset https://gist.github.com/martinklepsch/6ff00508cc49158f270b

basically it just does what clojure's binding macro does

yeah you can do the same with the fileset

Not sure if that’s a total mess or not but it felt pretty sweet to just derive all from fileset

you'll need a macro

what for?

yeah i like it

what is the "directory in fileset to use as root"?

if I have :source-paths #{“boot”} I could do something like this:

(with-dir
    :dir (-> settings :pod :dir)
    :task (comp (pom) (aot :all true) (jar :file "boot-pod.jar")))

what is settings there?

map of stuff — what’s being passed as :dir is ”pod/src/“

i wonder if the "env' should be a fileset instead of an atom

am i on a different page?

there is no point i don't think

since you can't undo adding something to the classpath

i mean not really

you can remove it from the atom

but that doesn't remove it from the classloader

when you make a pod you pass it an immutable env map

@alandipert: I had that thought too

what about a pod with a fileset in it

tasks need to run in the main thread though

@micha: why?

if you're not running it in the main thread then it's just a function in a pod which we already have

well you can alraedy run functions in pods, the main thread is the place where you can mutate the state of the build jvm environment

which is where tasks are useful

i guess no matter what there needs to be a calling environment

we can add more things to the "world" inside the env/fileset, but there still needs to be something that takes that as an argument

there is the option of using a shebang boot script

and kicking off multiple boots

you can look at the boot.App class

there is the runBoot() method

static method

that's what i used in the boot server experiment

you can run completely separate boot builds that way

and now that toby fixed our leak problem it should actually work

the issue I have with set-env! is that it feels unintuitive how it affects watching etc. tasks that set-env! some paths don’t compose very well with watching

yeah i suspect that shuffling the env around isn't the real solution

yeah

the shebang thing though

that’s why I was looking into scoping while fileset is being passed around

that could be the way to do a multi-build like boot

like actually start multiple builds at once

and let them run in their own threads/runtimes

sounds awesome

you mentioned removing things from classpath isn’t possible, can you think of another approach that would allow temporary scoping as it would be needed in the with dirs task?

i think you'll end up with weird edge cases

when i used it it was just to hide things from tasks that do their work in a pod

like the boot-jetty task, for example, i wanted to hide some dependencies from it

because when you make a pod you use the existing env plus whatever deps the pod has

so i used my macro to temporarily hide some deps from the task, and then replace them

it only needed to huide them during the task construction phase

couldn’t you just have filtered deps before creating the pod?

it wasn't my pod

in general i think it's good how we make pods in tasks

by adding pod deps to the global env deps

becaus that gives the user power to hide things they don't want to be in the pod or whatever

if they need to

that's how you can start a separate boot process in your boot script

that reuses the existing worker pod

> well you can alraedy run functions in pods, the main thread is the place where you can mutate the state of the build jvm environment > which is where tasks are useful can you elaborate on that? what are the useful bits about mutating the state of the build env? for building a jar I wouldn’t need this right?

thta makes a new worker

well like you said with the watcher for instance

there is a lot of state in the main thread

that's why boot.core isn't available in pods

you definitely can't remove classes from the classpath once they're loaded in a classloader

you can remove directories though, but only because of some clever things in the watcher

when you remove a directory from the env it just uses the watcher to remove files from the fixed directories the classloader was created with

it doesn't actually remove directories from the classloader

I’m thinking I could make a pod, give it a dir I created as classpath + required deps + fileset and then it gives me a fileset back

it could

but what's the point?

isolation

you could just run your function in a pod like we already do

and have it write to a tempdir, like normal

the fileset doesn't give you anything

I want to run tasks in the pod pom, jar etc

yeah that's impossible reall i think

except when you use tasks like functions

i mean like the boot cljs compiler functions

that run in the pod

aren’t tasks functions once the factory fn is called?

yes

but they need a reference to the next task, etc

which can't pass between pods of course

I can return the fileset from the pod and call the next handler myself (I think ..)

andif the task references anything in boot.core it can't run in a pod

yeah as long as it doesn't use boot.core stuff it could work

what problem are we solving?

seems like there might be some other ways we could try too

yeah coming to think the same

problem: we have 5 libs. we want to have a boot watch build-all install like thing that builds libs whose files have changed and installs them.

It felt really enabling to program things with values/fileset, I hope we can have more of that and less of set-env!

yeah you should try the runBoot() approach

you can do that from a task i bet

we need the target task thing though

i guess your tasks that you call from the runboot can set-env! :target-ath

so that would be ok

it would just be boot build-all-dev or something

I think I’ll go with the stupid non-watch supporting way for now, wanted to do the fileset patch stuff today too 😄

you could have the watch in the individual tasks you call with runboot

how about this

where project1,2 are profile type tasks

you could then use the :pipelines arg to spin up multiple runBoots

with the given args for each one

i guess the problem is when they depend on each other

then you really only want one pipeline

@micha: do you have the with-env macro handy?

the with-env macro can be simplified, it was just a quick thing for a specific issue

how does boot/boot/project.clj synchronize things if it doesn’t depend on anything?

makefile

the makefile does all the orchestration

it's lame but it mostly works

the boot server concept could actually be useful for the multibuild case

use it with something like make or the other cooler thing

i forget what it's called

micha: cmake 😉

no the crazy one

whew! i thought i lost it

https://github.com/apenwarr/redo

with the boot server in place we can use this to fully orchestrate

i think

it's a whole philosophy

kind of into it

@micha: the barbarywatchservice thing — is that still only for Java 6?

it's still the only way to get fsevents in osx

it works in java 8

that's what we use on osx

otherwise you can only have polling

yeah, was just wondering if it’s for java 6 compat

which is slow and eats cpu like a mofo

no boot doesn't try to support java 6 at all

the nio stuff is crucial

ok cool

regarding boot/boot/project.clj again: I see a jar is made but it doesn’t depend on anything, is this really just supposed to be an empty jar?

yes

i didn't know how to put just apom in maven

*a pom

it's there for synchronization

because clojars isn't transactional

that's the last thing that gets pushed

k

and boot looks for the version of that when it wants to update

that way it doesn't do a partial update

[#"^(?!boot\.repl-server).*$”] — this means all but boot.repl-server right?

yeah

we need that for the aot task 😉

we have a better thing

clojure

like you pass in the namespaces you want to aot

heard about that but is it any good?

😄

like in the project.clj there i have to later remove things that got included that should have been

but then I need to list all the things

you can use a function for that

tools.namespace etc

that's what we do in the tests and stuff

meh, some option like :exclude would be way more user friendly

i mean that regex is not helping anyone understand what's going on

or making it easier

cause when i put it in there i had to research regexes again

and test it

if i had a function i could have used filter in a second

agree about regexes but just a set with ns that shouldn’t get aot'd

yeah, we could

clojure.set/difference is a thing though

:exclude would work there though

can’t follow?

you can use that i mean

:namespaces (set/difference (all-nses) #{'foo.bar})

@micha: true but all that requires me to pull in c.t.n and that seems overkill for saying “all but these"

not saying it’s hard or impossible it’s just not what I consider “user-friendly"

we could add c.t.n to boot.core

a function to give you the namespaces etc

but yeah

we should do both

lol

I’m thinking it would be cool to move more towards things operating on a fileset. Pods could setup isolated envs based on those and some other params and then outside of pods we can combine filesets again and work with them as regular values. There’s not so much stuff in boot.core that would actually be useful if you’re just operating on filesets.

agree

we can almost pass filesets into pods

we can no?

just need to encode File objects

ah, yeah

which we can do, of course

@micha: in that branch I just pushed to boot build-lib fails because nrepl-server gets aot’d but when I just to boot core it works fine

not sure what’s going on there

I assume set-env! doesn’t reset classpath and some other build step has :aot all

@micha: do you see a way around unnecessary fs operations without blob ids of previous file? you think it’s ok to assume if lastmod of blobfile equals lastmod of tmpfile they’re the same?

(commit! [this]
    (util/with-let [{:keys [dirs tree blob]} this]
      (apply file/empty-dir! (map file dirs))
      (doseq [[p tmpf] tree]
        (let [srcf (io/file blob (id tmpf))]
          (file/copy-with-lastmod srcf (file tmpf))))))

Also what you you think about storing blob ids instead of last id. With something like this we could very easily lookup previous versions, potentially we could even store the same ids and have “generations”.

With that we could super easily create a previous fileset from the current. (at least for files that are still present)

i wasn't suggesting storing "last id", but rather storing the current state of the filesystem

which is necessarily a singletom

-ton

lol singletom == onetom?

@micha: I know what you were suggesting but I don’t see much drawbacks with storing that information in the fileset itself. Also where exactly would we store the filesystem state? How would we have access to it when commit! is called?

storing it in the fileset itself would defeat the purpose, because the fileset is immutable, and the underlying filesystem that we want to keep a record of is not

we'd store the filesystem state in an atom in the tmpdir namespace

since it's a singleton

and we update it each time we commit!

to reflect the current state of the filesystem

so when commit! runs it compares the cached state against the fileset, does a diff to get the patch, and then applies the changes

and then it updates the cache, which is the fileset

basically the tmpdir namespace needs to hold onto the fileset that was used in the latest commit! call

then when commit! is called again with a different fileset it can compare the two and make a path

*patch

@micha: it would only store state of the blob store part of the filesystem which could be considered immutable to some extent?

it would store the fileset object

it has all the information it needs there

and you can make a patch with the diff function in that namespace very easile

diff the two filesets and you have your patch

then you update those things that changed, were added, or were removed

and reset the atom to contain the new fileset

It would certainly be a straightforward approach I’m just thinking embedding this information in the fileset keeps more doors open (e.g. parallel tasks)

it would defeat the purpose though

What do you think about my point that blob storage is (mostly) immutable?

it is, but the working set is not

blob storage is append-only, which is good

but if you keep the information about the hard links that are supposed to be on the disk in the fileset it would be useless

because if you call commit! a few times it would no longer reflect anything useful

I’m really only suggesting to make TmpFiles contain ids instead of id — nothing else added

maybe i misunderstand the objective

i don't see the purpose of that

we're trying to optimize the commit! process, right?

yes

by diffing what's on the disk with what's in the fileset object, right?

without scanning the whole disk etc

for one it would be useful in that step, we could just check if last two ids of tmpfile match.

define "last two"

so here's the situation basically

two most recent

task pipeline: A -> B -> C -> D

so A does things and passes fileset to B

and so on

each one adds history to the fileset

via :ids

now suppose that B is the watch task

so C and D have called commit! on the fileset

so they've added files that B never saw in its fileset or the history

now B calls commit! on the fileset it saved, the one given to it by A

now the working set is corrupted

because all the files that C and D added weren't removed

because they don't appear in the history of the fileset that A gave to B

if you want to keep a history you can, because you can hold onto a fileset if you want to, and put them in a vector or whatevee

a vector of filesets

but that won't help the commit! optimization things

for removing stale things we would still use file-seq and delete the ones not in fileset

commit! needs to compare the mutable filesystem to the immutable fileset

that would be more IO

listing files is very fast

so is deleting hard links

which is what we do

i don't see any advantage

but making them is not (cheap). also deleting is probably still more expensive than ls.

if we want to make commit! more efficient by patching then there is one way to do that

save the fileset in tmpdir atom after every commit! and patch against that

the advantage would solely be to have one datastructure that provides a datastructure->filesystem abstraction that could be used in any context and does not rely on global state

but it does rely on global state

the filesystem is a globally shared resource

there is no way to get around that

but it only stores hashes/just strings

the filesystem stores files and directories

i'm talking about the working set, not the blob dir

take react for example

you don't store a history of dom state in the virtual dom

you have a virtual dom and the current real dom

you diff those

then apply patch

same thing here

the key element is that the actual classpath (which will be populated with hard links) is global singleton and mutable

so it makes sense that we rely on global state to mnage it

because the whole point of it is to do global stateful things

in the context of boot the point is to do global stateful things but the general idea of fileset has nothing to do with classpath, that’s just how boot uses it

right, that's where commit! comes in

commit! is classpath

that's where immutable becomes mutable

and glob al

commit! is classpath only because dirs are on classpath, not because it has inherently something to do with classpath?

ok filesystem then

anyways, I’ll try to come up with something more tangible and then we can discuss further

filesystem is globally stateful

excellent!

I think my main point is: fileset is so awesome it should be usable as a library too. global state to make patches makes that harder

thanks for the discussion ✌️

lisp can do it!

i feel like the problem(s) boot (and the fileset, tmpdirs, etc) are trying to solve is not stated clearly and the current implementation details are clouding and skewing our understanding (at least mine, but it seems im not alone).

of course micha you have quite a clear picture in mind and you keep trying to explain that picture from many different angles but somehow it's not permeating into other's minds easily

haha

i will write a better thing about it

@martinklepsch: if the fileset would be a library of its own, how would you describe what problem is it solving? what would be its interface boundary?

i did organize it such that it could stand on its own

boot.core does what you'd need to do to use it as a library

micha: im not suggesting any specific action now. i just made an observation, hoping it inspires u, martin, deraen and other core contributors.

i'm almost done!

boot will be downloading itself soon

including the binary

i was just re-reading the "Development Process" of the C4.1 (Collective Code Construction Contract -- http://rfc.zeromq.org/spec:22) yesterday and i noticed an interesting point in it: > Users SHALL NOT log feature requests, ideas, suggestions, or any solutions to problems that are not explicitly documented and provable. they way you (micha) argue follows this principles already, i just thought it's useful to make it explicit that "formally agreeing on a problem definition" first might help discussions a lot.

yeah totally

and i always learn new things from the process

and i see the way martin & deraen is handling the github issues are very much inline with c4 too

downloading of binaries is working!

@micha: awesome!

@micha: ^ playing around further with the idea I came to the conclusion that really all that’s needed is some “staging area” like concept. added a prototype like thing here: https://github.com/boot-clj/boot/commit/83e6761feb0b47bd7d7be0952dab7203b7703d7c — what do you think?

(haven’t tested this very much yet so may explode)

we need to clearly state the problem i think

staged changes are a type of history

but the fileset is not linear like that

it's declarative

consider the case where we have a ppeline of 4 tasks, A, B, C, and D

A gets an empty fileset and adds the file foo

B gets that fileset and adds the file bar

so now we have ["foo" "bar"] in the fileset

but B holds onto the original fileset it got from A, the one with ["foo"] in it

C gets the fileset with foo and bar and removes foo and adds baz

so D gets a fileset like ["bar" "baz"]

it doesn't do anything to the fileset

now when D and C return, B doesn't return

instead, like the watch task, it calls commit! on the fileset it got from A

notice there is nothing staged in there or anything

we want the filesystem to have only foo in it after commit! is done

this would not happen with your thing

there are the files bar and baz that will need to be fished out via a scan of the filessytem

and then foo would need to be added, again by scanning the filesystem to make sure it's not there already and pointing to the right place

@martinklepsch: ^^

@micha: first of all thanks for your patience :)

i could be confused, of course

What you're saying boils down to: if you commit the fileset with an earlier state it will not properly change the filesystem to match the fileset contents - is that right?

or more precisely

there are many fileset objects "out there" in the program in different places

and commit! can be called on any of them at any time and in any order

therefore we must conclude that what commit! does cannot depend on the history of what was done to any individual fileset object

especially since they're just immutable objects

they can't even hold any state

they just have a value

so this means that some oracle type authority needs to hold the state in an atom or something that is mutable

a place, not a value

currently that place is the filesystem itself

if i understand the goal correctly we want to avoid messing around in the filesystem because IO is slow

this means that we need a different place that can hold the info that describes the filesystem

so we can fish around in there instead

presumably it will be faster to look in a clojure map than to do IO on the filesystem

so

it seems like the only way to do this is to have an atom, say boot.tmpdir/state

that would hold a fileset

or nil as the case may be

perhaps populated initially by a scan of the filesystem, whatever

but when commit! is called, it simply compares the fileset in boot.tmpdir/state with the new fileset

since the filesystem can only be modified via commit!, we know that the fileset in state describes the filesystem fully and accurately

so we can get a diff of state and fs

this will tell us which links need to be removed, moved, or created

I see your point about the stage thing now. :D

after the comparison and after the filesystem links are all correct, commit! updates state, (reset! state fs)

now state contains a fileset that accurately describes the filesystem

and commit! can return

Wondering if it could still be used as an optimization. If no stage is given the filesystem state would be read from the filesystem. Would need to test how fast that would actually be to see if it makes sense at all.

@micha: with the atom approach - what would I do if I want to mess with two file sets simultaneously

why would it be read form the filesystem?

you'd need a state atom for each one

that complicates things, but you can just have a map in state where the keys are the fileset dirs

if you were going to allow multiple filesets simultaneously you'd need to make sure that no fileset could be created with an underlying dir that is shared with any other fileset

that would cause insanity

so you can use the dirs as a key in the state atom and the value is the current filesystem

you would not need to do any IO or look in the filesystem to build the patch

you'd just do a diff on the fileset in state against the one passed to commit!

the only IO would be creating or removing links

which is unavoidable, of course

Yeah, nesting things by dir makes sense, I think that removes my main concern

in boot we don't have that problem because you can only have one set of fileset dirs

@micha sorry it took me so long to accept the global state thing, somehow thought it could be done w/o but you're right it's not possible unless we construct a complete filesystem from scratch by looking at the filesystem

yeah

and we want to manage global state, that's why filesets

Thanks for talking me through it, I hope you didn't gain to much gray hair 😄

hahaha 😄

Now bed time, have a nice evening!

later!