just tried boot on windows 10 for the first time. has anyone experienced an issue where the target dir simply doesn’t appear?

i tried boot aot pom uber jar and it ends with Writing project.jar… but the target dir is nowhere in sight

after doing a search, i found the build files in .cache in my home dir. i’m confused why these files are not in target in my current working dir

@sekao: you need the target task. boot aot pom uber jar target

Recent boot versions don't write to target without using it.

ok thank you

@seancorfield: an idea for boot new can be a "rewriter", that is something that fetches a piece of code that exists already and "patches" it with new one. A use case would be to add require forms to an existing namespace from a template/boot new call

@richiardiandrea: That sounds like a "generator" which is pretty easy (and already supported). I have generators for creating a new file / namespace and for adding a new function to a namespace.

It appears that I'm having a similar problem to https://github.com/boot-clj/boot/issues/397. But apparently this was fixed a while ago. I'm trying to run "boot dev", but every time I save the file, it breaks and says it couldn't delete the old file.

I'm on Windows too.

@mhr: can you try bisecting the tasks in the dev pipeline to see which one is causing the problems?

just dipping a first toe in the water with boot (maybe will build a task to run the real Clojure REPL), but how does one get the user-space classpath? (i.e. the one you’d pass to java -cp) boot show -c/C gives me … not that

@richhickey: can you show an example of the java -cp command line you'd like to be able to do please?

there are different classpaths in different pods simultaneously usually. show -c/C is for the core pod

@micha I’d expect classpath for e.g. boot -s src to have clojure.jar and the src directory (or wherever that gets reflected in the fileset) and that’s it

right now it includes all of the java system libs

if I had deps I’d expect to see them

but not system jars, they are not passed via -cp

so something like what mvn dependency:build-classpath or lein classpath would give you

@richhickey: how should boot reflect the dynamic classpath configuration? Like for example when boot runs it copies the source files into its own managed temporary directories, so it can freely move, delete, or modify the files without touching your project.

maybe show -c is the wrong thing, but how does one plumb the user classpath through tasks?

and also dependencies can be computed at runtime and added dynamically

@micha dynamic is ok, the problem here is system things in classpath

ah ok, I will research this

that path is not ok to pass to -cp

so this should be URLs on the classpath but not the bootclasspath?

right

ok we can fix that

I’d also need to be able (when needed) to have classpath clean of e.g. boot/pod, tcrawley/dynapath etc - i.e. just the production runtime classpath

this is to facilitate your development workflow?

we need to completely understand what we are running, and have that reflect what we are shipping. So dynamic stuff ok sometimes, at others we need to be sure we aren’t getting different behavior due to build/dev-time deps

there has to be a way to run w/o any ‘dev-experience’ stuff

ah i see now (i think)

most typically we will gen classpaths into files (with e.g. mvn dependency:build-classpath) and use that statically for testing/production

I’m ok to have dynamic/dev support for e.g. REPL invocation

so a classpath is a concatenation of system (never supplied to -cp), dev tooling deps, and runtime/prod deps. Being able to talk about them separately is ideal

so to check that i understand: if i'm working on a library, that library might have dependencies. i should be able to do dynamic things when i build the library, but there should also be a way to emit the dependencies in a java -cp compatible way so i can do java -cp <deps>:mylib.jar TestClass or something similar to test that everything works in a clean environment.

yes, and also to plumb that through boot tasks

would it make sense to add a show --cp option, which you can use at any place in the build pipeline?

usually dependencies are only added once, except for tasks, which have no transitive dependencies and are :scope "test"

yes, but as I said, I think there are 3 things, your current -c shows them merged together - system jars, boot’s tooling jars, and application dep jars

the -c/C are options added for interop with IDEs, which seems like a separate concern?

I don’t know why the IDEs would want to see bootclasspath stuff either

but if the env kept the pieces separate you could use them as needed

yes we need to fix that, about the bootclasspath

hmm

perhaps the solution is to emit a project in the target dir

with the classpath in a file

right now env has boot-class-path and fake-class-path, I was not sure what those meant/included

that would decouple the source and other artifacts from your project files

that "real" classpath is the classpath that boot actually uses. boot does not put your project files on the classpath, because boot tasks need to be able to modify them at will, delete them, whatever

but those directories are not named directories that the user will manipulate directly, they're managed by boot

the "fake" classpath was to support IDE tooling, where the user's project files needed to be mapped to things on the classpath for operations that do modify your project, like refactoring

personally i think it's pretty leaky and i don't really use that myself

I was trying to figure out how the classpath was conveyed to the repl, as I’d like to make a task that used Clojure REPL and not NREPL

when boot starts up and you tell it which directories to put on the classpath, boot builds an overlay type filesystem of those directories, and copies the files from them into its own temp dirs

those temp dirs are on the classpath

since those are managed by boot you can do things like remove a directory from the :source-paths, and those files will be removed from boot's internal classpath

which is nice, but the main thing is to decouple your project files which you own from the files on the classpath which boot owns and can modify without interfering with your project

so for your first issue, the one with show -c/C, i propose a task that collects the source files and compiled artifacts etc from the classpath when the task is run, and emit them to a directory, along with a file that is a -cp compatible listing of jars, etc. for java -cp

so you can still use all the dynamic tasks at build time, but you can emit a static version of the project at that point in time, which you can use java -cp to test

does that make sense?

if I understood boot better maybe I could say yes 🙂

would your expectation be that the CI build would continue from there with those files?

i.e. they get jar’ed for production?

it seems like you are leaving your fileset system then

yes, or you could have boot create the jar

and you don't want to test your jar by making a pom that contains your dependencies and using maven to test it?

in any case certainly as part of build we need to end up with the production classpath in a file, that goes into our deliverable and is how we start our app

would the pom.xml contain that information?

i feel like i'm missing something, is there an example i can study of how you do it currently?

we always start our apps with java -cp cat classpath-file``

single backticks above

we know we’ve run CI with that same file

what would the classpath file contain? would those be paths relative to CWD or paths in maven local repo?

we never use dev tooling to launch in production

the uber task has an --as-jars option that will import dependency jar files into the fileset

which can then be enumerated and added to the classpath-files listing

i think this might be the solution

i will make a demo and put it on github, so there is a concrete example

the end result will be your jar file in the target dir with the jars it depends on in a lib subdirectory, and a file containing a java-cp compatible listing of those dep jars

cd target && java -cp `cat classpath.txt` -jar project.jar arg1

or something like that to run it

would that be acceptable?

it would, but the general ‘get the classpath’ stuff we were talking about before still needed for arbitrary tooling

does boot leave jars in ~/.m2 or move them into filesets?

when you do (set-env :dependencies '[...]) boot uses pomegranate to fetch the dependencies from maven and store them in local .m2

and add them to the classpath of the core pod

(that's where the build.boot expressions are evaluated)

uberjars are easier since you can use wildcard in classpath

if you want to build a jar and install it you would do that with the install task, which will install it in local .m2

again using pomegranate

when you build a jar that jar is just in the fileset

the install task, for example, gets it from there

the target task will emit the fileset to a directory, so you can just have the jar in the target dir if you wish

right, ok I think I understand, first thing I would ask for is a variant of -c that didn’t include system and the equivalent for task plumbing so one could convey the classpath-du-moment around

don’t you have this issue with the javac task?

https://github.com/boot-clj/boot/blob/master/boot/core/src/boot/task/built_in.clj#L676

the javac task uses the classpath as it exists at that moment during boot's run

yep

but when boot exits those files etc no longer exist

because they're all temp

that’s fine, still passing those bootclasspath jars to -cp not good ^

so it's difficult to make a show -c that gives you something useful after boot exits

definitely we will get rid of that

that’s ok, I want to play in fileset system for tasks

cool

right, that's why i think the best thing is to emit a "project" at the end of your dynamic pipeline

thanks for the help!

like a directory with your jar and all the dep jars, with the classpath file ready to go

at the end of all your dynamic build jobs

yes, that would be neat

ok great i will make a repo and link here and cc you

thanks for making clojure, it's made my life tangibly better 🙂

you’re welcome!

I ran into this problem when I added Boot support to cfmljure (the crazy CFML/Clojure bridge we use at World Singles). With Leiningen, I could do lein with-profile production do clean, compile, classpath and get a bare bones classpath and then I added those paths to the CFML classpath so it could dynamically load Clojure source code. With Boot, the best I managed was boot some-task aot show -C where some-task added the src folder and user-space dependencies. That still ended up adding a bunch of host Java paths and Boot JAR file paths — but since I only needed to load specific Clojure source namespaces into an already running CFML engine, it wasn’t horrible.

The some-task part could be omitted if build.boot added all the dependencies needed directly (but we have several different tasks with different dependencies so we have a rather complex setup).

I was surprised there wasn’t a direct equivalent to lein classpath.

@micha: Could you checkpoint the classpath as Boot starts up, before it reads build.boot and mark those elements as "not user-space" and then for the new option to show just omit those?

@seancorfield: i'm not sure what the boot equivalent to lein classpath would be, with dynamic classpath?

@seancorfield: it is different from a generator in that you already have a function/ns in place, can I use generators to read and replace with my stuff?

@richiardiandrea: A generator can do anything it wants.

Look at the function one: it appends text to an existing file.

It could easily read a file, modify it, and write it back out.

oh ok, yes, so my idea was to have some sort of automatic patching, but now I think it is too arbitrary and probably better to just do it manually

@micha: I was just suggesting that Boot could use whatever mechanism show -C uses to get the classpath, and run that at startup — after Boot is up and running but before it reads build.boot (and maybe before .boot/profile.boot?) — and just keep a snapshot of that to know that those things should not be included in the output of show —cp (well, the Clojure JAR needs adding back in).

ah i see

yeah or perhaps make them explicit dependencies with :scope "test"

nm that's nonsense

I see boot has BOOT_CLOJURE_VERSION but is there a way I can get boot to use my brand-new local clojure.jar?

@richhickey: if you install it in local .m2 BOOT_CLOJURE_VERSION should work

@micha I was afraid you’d say that 🙂

I can’t be installing every interim build of clojure while I’m working on it.

we could add an env var with a URL

I would love a way to run w/jars (just like Java can), w/o mvn

yes i think developing clojure itself is a unique situation 🙂

right

so would you like to have an env var with a URL that points to clojure.jar?

so I’m usually doing java -server -cp /Users/rich/dev/clojure/clojure.jar…

ah ok

any way I can supply it would be fine

ok we can make that work

thanks!

my pleasure!

@micha, what do you mean? I'm pretty sure it's "watch" that's dysfunctional.

@mhr: how do you mean?

when I run "boot dev" for the hello world app in hoplon, it sets up the jetty server and everything works. if I make save a change to the hello hoplon cljs file, it will note that and make the "speak" sound acknowledging that. But if I navigate to localhost:8000 and save another change, it spits out a bunch of exceptions. about how the temporary file boot created cannot be deleted: "index.html: The process cannot access the file because it is being used by another process."

did you disable the jetty memorymapped files optimization?

that is known to cause issues on windows

https://github.com/tailrecursion/boot-jetty#windows-users

that shows the system property you want to set: org.eclipse.jetty.servlet.Default.useFileMappedBuffer

cool

I'll try that

not 100% sure but i think (System/setProperty "org.eclipse.jetty.servlet.Default.useFileMappedBuffer" "false") might be the thing to do

yep, that fixed it! thanks so much micha!

micha, is there a way to modify watch? I'd like to make it so that it only updates the page when the saved version is diffed and shown to have an actual change.

@mhr: definitely, watch is just a task, so you can look at the source and make your own implementation

boot.core has some functions you might want to use, like fileset-diff for example

given two fileset objects, that function returns a new fileset containing only changed/added files

where is the source located? It seems like it's an uberjar, but I might be wrong about that.

https://github.com/boot-clj/boot/blob/master/boot/core/src/boot/task/built_in.clj#L307-L352

it's in the boot/core jar

the uberjar is just for bootstrapping boot

huh

I mean for windows, it's just boot.exe

yes, boot.exe is just to get enough stuff going to download boot jars from maven and load them into the JVM

it doesn't actually contain boot

that's how it's possible to do boot -u to update, for example

how do I access the actual boot source on my system so I can edit it?

i don't think you need to edit it, you can define your own task in a library

if you want to hack on boot itself you can clone the boot repo

and get the source that way

ah, so I should copy the watch source and substitute a modified version of that to get the functionality I want

👍 exactly

if it downloads boot, can I run boot without access to internet?

boot is cached in your local maven repo

so once you download it it won't need the internet anymore

okay

but of course if you add dependencies to your project that aren't in your local maven cache you'll need the internet for those

same situation basically

I'm not familiar with the java ecosystem really, I don't really know what maven does

it collects dependencies for you

based on coordinates you specify

the useful thing it does is resolve transitive dependencies for you

newbie question, if I pass :exclusions in the pod env (I would like to exclude clojurescript from the backend artifact) is it going to include the transitive dependencies?

@richiardiandrea: setting :exclusions will exclude that transitive dependency plus any of its own transitive dependencies, unless another dependency pulls it in

for example if i have [[foo/bar "1.2.3" :exclusions [foop/barp]] [baz/baf "2.3.4"]] in my dependencies, and baz/baf has a transitive dependency on foop/barp you will get the version of foop/barp from there

yes I just tried and there is no trace of either clojurescript or the closure compiler in my uberjar, thanks for confirming @micha, maybe I should PR https://github.com/boot-clj/boot/wiki/Boot-Environment

done 😉

so it seems (and I may be wrong) that boot relies almost entirely on dynamic deps. What happens when things go wrong, e.g. conflicts? While I agree generally that builds are processes, deps are not. If I don’t trust Aether et al, then what?

@richhickey: boot only forces you to use maven for boots own dependencies, which have no transitive dependencies to interfere (except for dynapath)

you can add jars to the classpath with BOOT_JVM_OPTIONS if you wish

via -cp

conflicts are usually handled by creating pods in which you can load conflicting dependencies

so in general you would add a single set of dependencies to the main pod via :dependencies, which get resolved as a unit

I don’t mind maven as a way to get jars, just aether as a way to runtime resolve/load

then for additional dynamic dependencies you may create pods with their own atomically resolved dependencies that are isolated from the ones you already have

then you can evaluate expressions in the pods

to do work

BOOT_JVM_OPTIONS is outside of boot though, so I’d have to wrap boot with something that did that, possibly per project?

there is an issue to have the launcher shim read that file

the only reason we haven't yet done it is because of windows

ah

I was looking for a way to list deps in a file per project and have boot pick that up on its way up

there is a boot.properties file you may use to specify these things

well not exactly at the moment, because it's read by the java process

so BOOT_JVM_OPTIONS is the one thing you can't do in there

right, I want deps loaded by -cp, not aether

but we could modify boot.sh to read that file before it starts the J

VM

in general, aether great for fiddling around at repl

perhaps a -cp opton to boot itself?

so deps and what ends up on cp are different

deps get chased transitively to produce cp. I’d like the file to be a list of deps

with overrides possible, and failing on conflicts not resolved by overrides

what would be a conflict in that case?

you aren't talking about maven artifacts, just jar URLs right?

so deps file says a/b/c, is they in turn use x/y/z boot ends up with a/b/c/x/y/z classpath. But if a and b call for different versions of x, should fail until I explicitly list a version of x in deps

no, maven artifacts

ah ok

so there are two separate concerns here, right?

maven's dep system, but -cp loading, not aether

you can use aether to gen the classpath

ah interesting

yes

i did this yesterday for a hadoop thing

i used boot just to resolve the transitive deps

have you seen boot show -p?

it's a task to visualize conflicts

currently boot will not throw an exception when aether resolves conflicts for you

we can add that

yes, that kind of thing is great, but the default (unlike mavens) should be fail on conflict rather than magic

ok we can totally do that

so something like boot -depsfile foobar ?

with a default so people can do something common in their projects?

hmm perhaps

there could be an option when you add dependencies, maybe

yes, if the file is there

so you would use a task to generate the depsfile, like boot make-deps-file --out deps perhaps?

then when you use the depsfile you could use the option as above

boot -depsfile deps repl for instance

how does make-deps-file know the deps?

you could specify them as an option to the make-deps-file task

Following along, what about a new strict set-env dispatch kW like :dependencies! Like set-env! :Dependencies but fails on conflict

I presumed I’d just put e.g. [[org.clojure/test.check "0.9.0”]…] in a file in the first place

Btw hi @richhickey , welcome!

sure that would also work

Hi

Maybe you can only call it once too

when people work with maven/lein, in addition to all the fake declarativeness, there’s a simple list of deps. That part is good

so a task that takes a vector of maven coordinates and builds a deps file, throwing an exception if conflicts are not overrided

that input vector could come from a file, or wherever you like

I don’t get the need for a task to gen still

the task is where the logic would be for performing that function

like it would call into the aether machinery in boot and resolve the dependencies, figure out what needs to be written to the file

ah, because shim has no java yet

yes, but there are two separate things here, the generating of the deps file using maven machinery, and the bootstrapping of boot itself

‘generating deps’ still not an idea for me. People start knowing their deps. What needs to be generated is a classpath

for generating the deps file i think a task would be sufficient, but for bootstrapping boot without maven we will add an option to accept -cp style deps from the file

yes sorry, that's what i meant

generating a classpath should fail and report conflicts when present

the task would consume dependencies and emit the classpath

yes

so, love this idea (gen-classpath task and boot shim picking up static classpath from file or having -cp arg)

that’s how we work here

i will wrap this all up in a github repo tonight and see if it looks like the approach might work

fantastic

here’s how I got my REPL btw (no task needed):

excellent!

does boot repl leverage anything about nrepl specifically?

@richhickey: not really, only that REPLy, the frontend, talks to it

i've wanted to use just a socket repl, but i'm not sure how to get the frontend to work that way

REPLy is started in a pod, so it doesn't introduce a dependency to your project

but we couldn't figure out how to not use nrepl for the server

the repl task does suport options like setting the initial namespace, things like that. those are implemented as nrepl middleware

but the repl in boot is kind of a messy special case

the ideal situation i think would be if nrepl could evaluate expressions via a socket repl, so we could start nrepl and all of its dependencies (like middleware for emacs cider and so on) in a separate pod

as it is i believe nrepl needs to be loaded in the same clojure runtime and can't be used in a separate pod

yeah, I don’t ever use nrepl

I’ve got the above working in inferior-lisp and I’m a happy camper 🙂

fast startup, fast execution, no fiddleware

the nrepl dependency is loaded on demand, so that should be good for your use, it won't be there if you don't use repl task

right

@micha, how would I use the fileset-diff function with filesets in the pipeline? I don't understand where the before and after arguments come from.

@mhr: fileset objects are immutable, so you can hold onto one, then diff it against another one later, to see what changed

i will find an example, one sec

this is the way boot tasks avoid doing unnecessary work -- they keep their own private cache of work that was done, and by diffing the fileset they can know which of those cached artifacts are stale