@mobileink: i'm not aware of anything similar but i'd be curious to know

did a superficial websearch. impression: classpaths isolation remains an issues for gradle. don't get me wrong, I'm not anti-gradle, just interested in dispassionate comparison.

what really surprises me about VMs and platforms after the boot experiience is that most things don't even have a classpath really

which to me now is inconceivable

then i periodically read about upgrading rails apps and am able to concieve of the idea vicariously

maybe gradle can learn from boot, just as boot can learn from gradle, e.g. flavors.

i've never used them but when i looked at them i had a sense that flavors could be a library/task

kind of like the way people implement "profiles" in boot

I think classpath skullduggery is a barrier to newcomers. certainly was/is for me.

i think the best remedy is to not hide it from you

because hiding it only gets the beginner out on a ledge of false analogies

I hope to find time to look at a boot equiv. of flavors soon.

so the classpath is key for doing things in the JVM, the best we can do is provide some good tools to help understand and manage it

gradle flavors are heavily exploited in android.

@micha yes. but in my experience we cam do much better at explaining the mysteries to noobs. classpaths, classloaders, still pretty arcane even to me.

anyone using npm dependencies with clojurescript nodejs based project?

please point me to an example

hello everyone! I'm presenting on boot at a local clojure meetup, and would love to gather feedback on my slides. Let me know if there's anything I should add/remove! Thanks! http://blog.jethrokuan.com/talks/boot/

@jethroksy: I would explain Lein problems more thoroughly than just the config file

what about the slide after that?

- No pods -> Plugins fork additional JVMs - Declarative configuration -> Plugins don't compose - No filesets -> User needs to manage temp directories themselve (http://www.slideshare.net/juhoteperi/clojutre-build-tooling-with-boot-42005279)

What I mean is, (as I see it) each of Boot design choices is targeted to fix something, so that might be useful to explain when introducing pods and fileset etc.

ah yes

I try to talk about it when I'm introducing the concepts in later slides

the problem is lein users don't know what pods are

Yeah

so it's hard to explain at the start before the introduction to boot

I'll add in the comparisons in the later slides

Not many Lein users probably know either how Lein works internally

good point

That is, when Lein plugin has some dependencies it forks another JVM and loads separate Clojure env with those dependencies

pods is a super cool idea

you can do the whole presentation without pods

we had this last week in Zurich Clojure meetup

people were still impressed

Yeah, they are mostly implementation detail and mostly interesting to task developers

I have directory/files outside of the classpath (CSSes in a npm module), want to copy some into the fileset, what's the best way to do it? Shelling out to mv sounds like a hack.

But they also lower Boots memory consumption when compared to Lein because Boot doesn't need multiple JVM processes, so that could be something users are interested in

But maybe people aren't running that many Lein plugins parallel today compared to 1.5 years ago, as Cljx is no longer needed

I think dependency hell is a real problem in Lein, and pods is just way too elegant a solution to not show

pods are definitely worth mentioning

I won't go into too much detail

because I really can't

talk is around 30 minutes

@jaen have you seen :asset-path

thanks for all the feedback @juhoteperi

@jethroksy: I don't think it will help me? I have a directory I serve assets from and it's different from where I have to get those files.

So I can't point :assets-path there.

the fileset manages two things: 1. files in classpath 2. files in asset path

I don't think the fileset api has access to anything beyond that

Yeah, but I want to put external files into the fileset in my task.

Shelling out to cp with the target relative to tmp-dir!-created directory works

But shelling out to do that feels… off. This seems like something that would be a common task so I thought there's maybe a helper function I'm missing.

you can use java.nio.file.Files/copy

@dm3: does that work on directories too?

assuming you wouldn't want to do http://docs.oracle.com/javase/tutorial/essential/io/examples/Copy.java

try looking in boot sources - I think they have some utils

@dm3: hah that looks fairly involved, that's why I figured maybe someone had already created some helper for that; I'll stick with shelling out for now, but look through sources for alternative later then.

what was the snippet for modifying boot env when creating a task?

@jaen: there is a nice trick in clojure for recursively copying directories:

(defn copyrec [src-dir dest-dir]
  (doseq [f (->> src-dir io/file file-seq (filter #(.isFile %)))]
    (io/copy f (io/file dest-dir (relative-to src-dir f)))))

i think there are some functions in boot.file that might be useful

that namespace isn't part of the public api but it's available from any pod

@micha: and if I want to filter files out I just modift the predicate in that filter?

Looks neat, thanks!

I'll also take a look at that ns

i think there is a boot.file/sync! function

actually that might be in boot.core too

it might do what you need

@micha: https://github.com/boot-clj/boot/blob/master/doc/boot.core.md#sync?

Would be good, but it says it deletes missing files, I want to keep what was already there though.

But now I know where I should be looking

@micha - is the with-env thing now in boot proper?

@jaen: if you are going to add to the fileset you don't need to keep existing files

you can just make a tmp-dir for that sync proecss

and use add-resource etc later

like with add-resource you don't need to have all the files in the same directory

you can call it multiple times with different directories

and they're all overlayed

Ah, yes

add-resource unions with what's already there.

yep

it's satisfying when options are not needed

@dm3 2.6.0 will have it hopefully

And if I need to filter, I've got your snippet (though it would make sense to probably include something like that in the core).

if you want to filter you can do that when you do add-resource

it accepts :include and / or :exclude kwarg options

Nice, didn't know about that

Also ugh, missclicked on fork D :

also :mergers option if you want to merge files from your tmpdir with files of the same path that are already in the fileset, instead of clobbering them

you can specify a map of path regex to merger function

The html doc doesn't say what the :include option expects; is that a set of paths, set of files, predicate to filter things that gets path, that gets File object?

Or I'm blind and don't see that?

@richiardiandrea fixed the documentation black hole there, hope to merge things this week

but if you do boot uber -h you can see the docs

it's the same for the uber task as for add-resource etc

except there are no default filters or mergers when you call add-resource

the uber task sets some defaults

but it just passes options through to add-resource

@jaen: https://github.com/boot-clj/boot/blob/master/doc/boot.task.built-in.md#uber

Yep, I've just run boot uber -h locally; it seems it expects a set of regexes that match file paths.

Nice, thanks a lot for the help!

sure, np!

this: boot sift -m "(.*clj$):WEB-INF/classes/\$1" does what I want; it moves .clj files from src to WEB-INF/classes, according to show -f. But this: boot sift -m "(.*clj$):WEB-INF/classes/\$1" target discards the WEB-INF stuff. I guess the result of sift is not getting committed. Is there a way to do this on the command line?

@mobileink: are the .clj files in :source-paths or :resource-paths?

i think that might be the issue

I have :source-paths #{"src"} :resource-paths #{"resources/public"}. E.g. src/hello/echo.clj gets moved to WEB-INF/classes/hello/echo.clj.

you can add "src" to :resource-paths instead of :source-paths

sift :move doesn

t change the roles of the files

ok, that works. why? i don't see the connection.

that's the difference between :sourec-paths and :resource-paths, the roles

files in :source-paths don't end up in packaging

oh right, input/output roles

personally i think the whole roles thing is maybe overcomplicating things, but people tell me they like it

is there a way to display roles in the fileset? e.g. show -r

hm no

but show -f could color them or something

@jethroksy: good luck on your presentation, I like your slides

i would favor something like [io] favicon.ico, [i ] echo.clj

in any case, is it correct that each item in the fileset is marked with its roles? stuff in :resource-paths is marked [io], stuff in :source-paths is marked [i ]?

yeah that's true

https://github.com/boot-clj/boot/wiki/Filesets#fileset-components

most jvm build tools have some separation of source and resource

boot just makes it a general attribute

ok, so the rule seems to be that fileset items carry their roles with them as they move thru the pipeline. target will write anything marked [o]. correct? so if echo.clj is in :resource-paths, it gets marked [io] in the initial fileset; if it gets moved around, it will still be [io] when it gets to target, so it gets written.

not tied to any specific model for how to build a project

yeah

the idea is the output files are files that will be included in packaging

the target task is a packaging task

pretty cool. but personally i think the idea of items carrying their roles around is much easier to understand, since i don't think of stuff output to the target dir as involving packaging until I'm ready to package and deploy.

during development, the target dir is just a workspace

it's important to have consistent, simple concepts though

rather than concepts that are specific to building java libraries and concepts that are specific to building clojurescript apps and ...

i guess that's the core principle

boot itself doesn't know how to build anything

i mean anything specific

right, that's why i like the concept of attaching roles to items persistently. i found the build/packaging distinction a little puzzling, since i don't normally think like that, it's all just build and run for me most of the time

during development the target directory is undefined state

it's only in a defined state after boot exits

well, "development" to me means the whole process of building, running, testing, etc. i guess by "during development" you mean "while boot is running"?

yeah

i guess that's how i do "development"

boot's just running in incremental mode

@alandipert: thanks! I'm using some of your libs for my demos

i think you're right though, in hindsight maybe it owuld have been better to use sift explicitly to select only output files etc in the pipeline before the packaging tasks

ok, right, i guess i think of boot tasks like serve or run as different beasts than build tasks that produce target output. in any case, now i think i finally understand what :resource-paths does and how roles function in the build process. very cool.

as time goes on it seems like any build-related concepts in boot have been kind of a mistake and are better left to the user explicitly

like the target task, and possibly this

what counts as a packaging task, for you? i would call jar and war packaging tasks, for example.

for sure

packaging is construction of artifacts that will be used in other environments

basically packaging represents building a classpath for another JVM process

so that might be building a jar or war file

or it may be building a "classpath" for a cljs single page app that is deployed to s3

my cljs applications have a "classpath" oriented architecture, too

basically that's how i approach all applications now

jars in maven provide one thing only: a classpath snapshot

while you're building the jar and stuff you have a merging of the classpath that will be part of thejar and the classpath your build program needs to run

that's why the fileset roles are there

input files are part of the classpath the build program uses

output files are part of the classpath you want to create in your final artifacts

of course files in the fileset can be both at the same time

but i think of my build programs as a transformation of classpaths

no time at the moment to edit the wiki but wanted to register this somewhere

👍

build pgms as classpath transformers, interesting, never thought of it like that before

it's interesting because you might think, why do you need to have a separate environment as the destination for the classpath you build?

like when i deploy to docker, for example, boot is the entry point

it builds the classpath for the application, then runs it

this actually solves the questions i was asking when i first joined this channel - i couldn't for the life of me see how to move stuff from src to WEB-INF/classes. now it's easy peasy

maybe you need a slogan for boot. Howsabout: Boot. ... it will change your life 😉

hahaha

really, it has completely changed the way i think about the practical side of software construction.

that will be added to the testimonials section 😉

fine by me, i got more of 'em 😉

i find myself thinking about build/config stuff in terms of functionals (tasks) in the same way i think about structuring my programs in terms of functions. makes for a much more integrated view of the whole process than you have with "classic" build tools, where you have to think of configuring foreign stuff.

yeah for me the "build" and "application" concepts have merged somewhat

the line dividing them is definitely blurred

@micha - could you please point me to where the with-env is implemented. Was looking in repo but couldn't find

i believe @dominicm made a PR that includes it

doesn't seem to be there

if you look on my github there are some gists about it

@dm3: ^^

and i think the juxt/edge project is where @dominicm was developing this stuff, so it's there too https://github.com/juxt/edge

@dm3: jackpot: https://github.com/SevereOverfl0w/boot-snippets/blob/master/src/io/dominic/boot_snippets.clj

thanks!

i've been heads-down with work stuff lately, but hopefully all the awesome new stuff will be integrated into the 2.6.0-SNAPSHOT branch soon

so we can test it and use it

@micha reading on the yesterday's conversation between you and :richardiandrea regarding the propagation of :source-paths changes to new pods. It seems from the user-facing API it would be nice to propagate all parts of the environment, not only the :directories. E.g. with-env would only work for :dependencies and :directories, not :source-paths which is very confusing. Am I missing something?

does the fact that I want to change :source-paths dynamically indicate that I need to run a new boot core?

well pods don't have any notion of source vs resource

so mocking it in the API i think would make things more complex and confusing

pods just have a classpath, which is expressed in :directories

you can change :source-paths dynamically, no problem

the set-env! machinery correctly adjusts the classpath etc

pods are like the docker of jvms !

indeed!

i wish we can make it more easier to use as a library

something like osgi but without the ugly parts

yeah the issue is that if you load clojure outside a pod then all pods are compromised

so the entry point to the application needs to be the pod java factory

alan's uberboot project is how you can use pods without boot i think

it builds you an uberjar with the correct entrypoint and all clojure code run in pods

i wonder why the clojure dependency outside pods is all about?

boot.sh sets up the jvm for pods before loading clojure

and it loads clojure into a pod

without that you couldn't get isolation

because of the way clojure manipulates the classpath

at least i couldn't make it work

i tried many things before taking the nuclear option of writing a java application as the entrypoint 😄

interesting work!

so everything in boot is being run in a pod?

even when we run a custom task which does not run a pod

how is it that once you are inside a pod clojure being on classpath is not a problem for spawning further pods?

@micha a proposal to make the env for pods less confusing would be to remove the unused keys, I found a bit weird that get-env in the pod was returning source-paths if it is actually just using directories. Given that pod are low level (and they should stay like that) it makes sense to me to restrict their env map. This will also help with discovery at the repl, dunno whether this is possible, just an idea ;)

in my case I'm trying to use with-env and add a source path which is otherwise not present. Task which is modified by with-env creates a pod when instantiated, so the pod captures the proper :source-paths which at that time contain the correct files (boot auto synchronizes the sources when set-env! is called). However, once the with-env has ended boot resynchronizes the :directories with the old :source-paths and the files are gone.

Yes yesterday I was doing the same but creating the pod manually and using add-classpath

@kul: all pods are siblings, whose parent is the java class boot.App

so they can communicate via boot.App

for example your build.boot program runs in clojure in a pod that is automatically created by boot

you can call boot.pod/make-pod from that pod, which calls a static method of the boot.App class which is on the classpath of every pod

that static method creates a new pod via the same mechanism that created the main pod, and returns a reference to it

with that reference in hand you can call methods on the pod instance to evaluate clojure forms in the pod or call functions etc

@dm3: isn't that the expected behavior of with-env?

it's supposed to restore the original state when its body has been evaluated

@richiardiandrea: i think you may be right about removing extra keys

have to think about it though

@juhoteperi: is there anything holding back a 0.4.6 release for boot-reload?

@micha np I have just thrown it there ;)

@pandeiro I don't think ao. I'll check when I get home.

@micha - that's the expected behaviour, yes. But the problem is it's not very useful. Are there more scenarios that could make use of with-env? Currently I can only see that if you have a task that creates the pod in the constructor and if you limit your with-env call to :directories and :dependencies then you can influence the task.

@dm3: yep that's the use case

a niche thing, whch is why it's never been added to boot yet

ok, so it's more like with-pod-env

and it seems worthwhile to differentiate between the Core env and Pod env as @richiardiandrea suggested

agreed

for the new file-roles branch of boot-fails i have a question about roles. it's easy do demonstrate at the command line that a .clj file in :asset-path can be moved but will not be compiled by a subsequent aot -a task. from that I infer that the aot task inspects the fileset and selects only files marked +input. and you can transform roles as well, e.g. boot show -f sift -S assets aot -a target will add +input to the assets and so aot compile them. so i infer that we can treat "role" as a kind of metadata attribute, rather than a rule that boot enforces. question: can i violate this discipline? can i write my own task in order to transform a [+output, -input] file? of course conceptually any such task could be viewed as one that first adds +input, but i'm wondering if there is anything in boot that would prevent me from reading an asset, transforming it and writing the result.

@mobileink: nope, nothing

other than maintaining sanity

ok, good, so boot provides the mechanisms you need but does not play sheriff - other than keeping filesets immutable. just like clojure, in a word.

yeah boot doesn't know or care

it just provides the abstractions

it's up to you to build things with it

have you considered that we can think of boot as a kind of os filesystem-based implementation of Clojure's map api? it doesn't look like the map api, but the concepts are all there. just think of files as key-val pairs, where key is name and val is content. seems to me that the fileset implements this even if it does not directly present a key/value interface.

that makes it easy to explain the fileset pipeline - its a transform pipeline, exactly the same as you would have with ordinary maps

the fileset has the :tree key, whose value is a map of path to TmpFile record type

but it's not that simple

because it's like git

it's not truly pure

it's managing a content-addressed storage area

the fileset object contains immutable data that is useful

sure but that's an implementation detail, imo - you could use a database instead of the os filesystem, for example. main thing is immutability layer over mutable stuff

but it's not sufficient

yeah but that's what makes it incompatible with the immutable collections absrtaction

like you can present it as a mostly immutable thing

and in most ways it works

but that's not sufficient

where's the incompatibilty? clojure's implementations are also built on mutable host stuff, no?

well for example, if you serialize the fileset you don't have enough information to deserialize it

i ask because i'm working on a blog post explaining filesets using standard {:foo "bytestream here..."} notation

they're not really values in the true sense

because they're pointing to things that are mutable and whatnot

ah, but what does serialize the fileset mean? to me, you could serialize a fileset to e.g. a database, and you could in principle deserialize it

i guess i need to think about it more

ok, what i need to add to make the comparison work is the observation that the host java constructs behind IPersistentMap etc. are effectively immutable because no other process can touch them and clojure is designed not to change them - but as host objects they could be mutated. where the host env is the os filesystem, of course you have the possibility of mutation. but boot arranges to hide that at least for the duration of the boot run, using the mechanism of hidden temp files to ensure "boot-time" immutability. any other mechanism that ensures boot-time immutability could work as well. so to the user, the file system rooted at the project root is just as immutable as any clojure map, for the duration of the build process. does that sound correct? i hope so, it's very cool

that sounds legit to me

if so, i think what you really have here extends well past build/config tasks. i imagine it could form the basis of a lib that would make immutable fs operations easily available to any clojure pgm. this is related to the idea i mooted the other day about supporting a standard map api for boot.

yes, we've keep it separate to make it easy to lift out into a library

yeah, i can see the wisdom of that now

there are a few places where it's boot specific for performance reasons

but that could be generalized i imagine

also lifting it out into a library would mean maintaining it in two places, because boot can't have dependencies

which i am hesitant to do

i've got some backing at work for introducing clojure, and i expect some resistance from both pgmers and mgrs - i think boot will be a huge argument in favor of clojure. i don't know of any other system that does this sort of stuff at all

and everybody hates their current build tools

yeah i always have

although i still perversely enjoy gnu make

i'll give up make (and emacs) when they pry my cold dead fingers off it (to paraphrase charlton heston) 😉

haha yes

hey all. i'm getting an error using advanced compilation with externs in deps.cljs

ERROR: JSC_DUPLICATE_EXTERN_INPUT. Duplicate extern input

ERROR: JSC_DUPLICATE_EXTERN_INPUT. Duplicate extern input: /Users/pleasetrythisathome/.boot/cache/tmp/Users/pleasetrythisathome/Code/work/populace/populace/1r0q/fwjp2k/js/booking/booking.ext.js at (unknown source) line (unknown line) : (unknown column) ERROR: JSC_DUPLICATE_EXTERN_INPUT. Duplicate extern input: /Users/pleasetrythisathome/.boot/cache/tmp/Users/pleasetrythisathome/Code/work/populace/populace/1r0q/fwjp2k/js/timekit/timekit.ext.js at (unknown source) line (unknown line) : (unknown column) ERROR: JSC_DUPLICATE_EXTERN_INPUT. Duplicate extern input: /Users/pleasetrythisathome/.boot/cache/tmp/Users/pleasetrythisathome/Code/work/populace/populace/1r0q/ywpinm/js/booking/booking.ext.js at (unknown source) line (unknown line) : (unknown column) ERROR: JSC_DUPLICATE_EXTERN_INPUT. Duplicate extern input: /Users/pleasetrythisathome/.boot/cache/tmp/Users/pleasetrythisathome/Code/work/populace/populace/1r0q/ywpinm/js/timekit/timekit.ext.js at (unknown source) line (unknown line) : (unknown column)

looks like the files are in the classpath twice perhaps? any idea why that'd happen?

you can use show -f to see if they're in local files

only see them once

i mean if you see them at all in the fileset you probably have issues

because they're in jhars too presumably

*jars

not in jars currently...i was just including them in src/

i have a ticket to put them into cljsjs packages...

i suppose that might solve my problem as well

it's seeing them twice the fileset

you can see in the paths

1r0q/ywpinm and 1r0q/fwjp2k

yea i see that