Fork me on GitHub
#boot
<
2016-06-08
>
sekao01:06:39

just tried boot on windows 10 for the first time. has anyone experienced an issue where the target dir simply doesn’t appear?

sekao01:06:05

i tried boot aot pom uber jar and it ends with Writing project.jar… but the target dir is nowhere in sight

sekao01:06:32

after doing a search, i found the build files in .cache in my home dir. i’m confused why these files are not in target in my current working dir

mynomoto02:06:34

@sekao: you need the target task. boot aot pom uber jar target

mynomoto02:06:01

Recent boot versions don't write to target without using it.

sekao02:06:44

ok thank you

richiardiandrea03:06:37

@seancorfield: an idea for boot new can be a "rewriter", that is something that fetches a piece of code that exists already and "patches" it with new one. A use case would be to add require forms to an existing namespace from a template/boot new call

seancorfield05:06:44

@richiardiandrea: That sounds like a "generator" which is pretty easy (and already supported). I have generators for creating a new file / namespace and for adding a new function to a namespace.

mhr08:06:12

It appears that I'm having a similar problem to https://github.com/boot-clj/boot/issues/397. But apparently this was fixed a while ago. I'm trying to run "boot dev", but every time I save the file, it breaks and says it couldn't delete the old file.

mhr08:06:23

I'm on Windows too.

micha12:06:30

@mhr: can you try bisecting the tasks in the dev pipeline to see which one is causing the problems?

richhickey13:06:43

just dipping a first toe in the water with boot (maybe will build a task to run the real Clojure REPL), but how does one get the user-space classpath? (i.e. the one you’d pass to java -cp) boot show -c/C gives me … not that

micha14:06:03

@richhickey: can you show an example of the java -cp command line you'd like to be able to do please?

micha14:06:32

there are different classpaths in different pods simultaneously usually. show -c/C is for the core pod

richhickey14:06:01

@micha I’d expect classpath for e.g. boot -s src to have clojure.jar and the src directory (or wherever that gets reflected in the fileset) and that’s it

richhickey14:06:55

right now it includes all of the java system libs

richhickey14:06:40

if I had deps I’d expect to see them

richhickey14:06:58

but not system jars, they are not passed via -cp

richhickey14:06:25

so something like what mvn dependency:build-classpath or lein classpath would give you

micha14:06:59

@richhickey: how should boot reflect the dynamic classpath configuration? Like for example when boot runs it copies the source files into its own managed temporary directories, so it can freely move, delete, or modify the files without touching your project.

richhickey14:06:15

maybe show -c is the wrong thing, but how does one plumb the user classpath through tasks?

micha14:06:20

and also dependencies can be computed at runtime and added dynamically

richhickey14:06:43

@micha dynamic is ok, the problem here is system things in classpath

micha14:06:01

ah ok, I will research this

richhickey14:06:49

that path is not ok to pass to -cp

micha14:06:50

so this should be URLs on the classpath but not the bootclasspath?

micha14:06:56

ok we can fix that

richhickey14:06:39

I’d also need to be able (when needed) to have classpath clean of e.g. boot/pod, tcrawley/dynapath etc - i.e. just the production runtime classpath

micha14:06:38

this is to facilitate your development workflow?

richhickey14:06:10

we need to completely understand what we are running, and have that reflect what we are shipping. So dynamic stuff ok sometimes, at others we need to be sure we aren’t getting different behavior due to build/dev-time deps

richhickey14:06:39

there has to be a way to run w/o any ‘dev-experience’ stuff

micha14:06:54

ah i see now (i think)

richhickey15:06:20

most typically we will gen classpaths into files (with e.g. mvn dependency:build-classpath) and use that statically for testing/production

richhickey15:06:33

I’m ok to have dynamic/dev support for e.g. REPL invocation

richhickey15:06:35

so a classpath is a concatenation of system (never supplied to -cp), dev tooling deps, and runtime/prod deps. Being able to talk about them separately is ideal

micha15:06:39

so to check that i understand: if i'm working on a library, that library might have dependencies. i should be able to do dynamic things when i build the library, but there should also be a way to emit the dependencies in a java -cp compatible way so i can do java -cp <deps>:mylib.jar TestClass or something similar to test that everything works in a clean environment.

richhickey15:06:10

yes, and also to plumb that through boot tasks

micha15:06:41

would it make sense to add a show --cp option, which you can use at any place in the build pipeline?

micha15:06:25

usually dependencies are only added once, except for tasks, which have no transitive dependencies and are :scope "test"

richhickey15:06:43

yes, but as I said, I think there are 3 things, your current -c shows them merged together - system jars, boot’s tooling jars, and application dep jars

micha15:06:44

the -c/C are options added for interop with IDEs, which seems like a separate concern?

richhickey15:06:14

I don’t know why the IDEs would want to see bootclasspath stuff either

richhickey15:06:38

but if the env kept the pieces separate you could use them as needed

micha15:06:46

yes we need to fix that, about the bootclasspath

micha15:06:01

perhaps the solution is to emit a project in the target dir

micha15:06:10

with the classpath in a file

richhickey15:06:22

right now env has boot-class-path and fake-class-path, I was not sure what those meant/included

micha15:06:28

that would decouple the source and other artifacts from your project files

micha15:06:23

that "real" classpath is the classpath that boot actually uses. boot does not put your project files on the classpath, because boot tasks need to be able to modify them at will, delete them, whatever

micha15:06:52

but those directories are not named directories that the user will manipulate directly, they're managed by boot

micha15:06:34

the "fake" classpath was to support IDE tooling, where the user's project files needed to be mapped to things on the classpath for operations that do modify your project, like refactoring

micha15:06:57

personally i think it's pretty leaky and i don't really use that myself

richhickey15:06:12

I was trying to figure out how the classpath was conveyed to the repl, as I’d like to make a task that used Clojure REPL and not NREPL

micha15:06:15

when boot starts up and you tell it which directories to put on the classpath, boot builds an overlay type filesystem of those directories, and copies the files from them into its own temp dirs

micha15:06:23

those temp dirs are on the classpath

micha15:06:12

since those are managed by boot you can do things like remove a directory from the :source-paths, and those files will be removed from boot's internal classpath

micha15:06:54

which is nice, but the main thing is to decouple your project files which you own from the files on the classpath which boot owns and can modify without interfering with your project

micha15:06:14

so for your first issue, the one with show -c/C, i propose a task that collects the source files and compiled artifacts etc from the classpath when the task is run, and emit them to a directory, along with a file that is a -cp compatible listing of jars, etc. for java -cp

micha15:06:17

so you can still use all the dynamic tasks at build time, but you can emit a static version of the project at that point in time, which you can use java -cp to test

micha15:06:47

does that make sense?

richhickey15:06:06

if I understood boot better maybe I could say yes 🙂

richhickey15:06:39

would your expectation be that the CI build would continue from there with those files?

richhickey15:06:54

i.e. they get jar’ed for production?

richhickey15:06:14

it seems like you are leaving your fileset system then

micha15:06:16

yes, or you could have boot create the jar

micha15:06:14

and you don't want to test your jar by making a pom that contains your dependencies and using maven to test it?

richhickey15:06:42

in any case certainly as part of build we need to end up with the production classpath in a file, that goes into our deliverable and is how we start our app

micha15:06:03

would the pom.xml contain that information?

micha15:06:28

i feel like i'm missing something, is there an example i can study of how you do it currently?

richhickey15:06:40

we always start our apps with java -cp cat classpath-file``

richhickey15:06:03

single backticks above

richhickey15:06:37

we know we’ve run CI with that same file

micha15:06:52

what would the classpath file contain? would those be paths relative to CWD or paths in maven local repo?

richhickey15:06:59

we never use dev tooling to launch in production

micha15:06:40

the uber task has an --as-jars option that will import dependency jar files into the fileset

micha15:06:05

which can then be enumerated and added to the classpath-files listing

micha15:06:17

i think this might be the solution

micha15:06:43

i will make a demo and put it on github, so there is a concrete example

micha15:06:01

the end result will be your jar file in the target dir with the jars it depends on in a lib subdirectory, and a file containing a java-cp compatible listing of those dep jars

micha15:06:16

cd target && java -cp `cat classpath.txt` -jar project.jar arg1

micha15:06:20

or something like that to run it

micha15:06:02

would that be acceptable?

richhickey15:06:32

it would, but the general ‘get the classpath’ stuff we were talking about before still needed for arbitrary tooling

richhickey15:06:57

does boot leave jars in ~/.m2 or move them into filesets?

micha15:06:53

when you do (set-env :dependencies '[...]) boot uses pomegranate to fetch the dependencies from maven and store them in local .m2

micha15:06:03

and add them to the classpath of the core pod

micha15:06:16

(that's where the build.boot expressions are evaluated)

richhickey15:06:33

uberjars are easier since you can use wildcard in classpath

micha15:06:03

if you want to build a jar and install it you would do that with the install task, which will install it in local .m2

micha15:06:09

again using pomegranate

micha15:06:19

when you build a jar that jar is just in the fileset

micha15:06:30

the install task, for example, gets it from there

micha15:06:07

the target task will emit the fileset to a directory, so you can just have the jar in the target dir if you wish

richhickey15:06:49

right, ok I think I understand, first thing I would ask for is a variant of -c that didn’t include system and the equivalent for task plumbing so one could convey the classpath-du-moment around

richhickey15:06:31

don’t you have this issue with the javac task?

micha15:06:08

the javac task uses the classpath as it exists at that moment during boot's run

micha15:06:22

but when boot exits those files etc no longer exist

micha15:06:27

because they're all temp

richhickey15:06:58

that’s fine, still passing those bootclasspath jars to -cp not good ^

micha15:06:01

so it's difficult to make a show -c that gives you something useful after boot exits

micha15:06:12

definitely we will get rid of that

richhickey15:06:22

that’s ok, I want to play in fileset system for tasks

micha15:06:45

right, that's why i think the best thing is to emit a "project" at the end of your dynamic pipeline

richhickey15:06:48

thanks for the help!

micha15:06:00

like a directory with your jar and all the dep jars, with the classpath file ready to go

micha15:06:13

at the end of all your dynamic build jobs

richhickey15:06:29

yes, that would be neat

micha15:06:48

ok great i will make a repo and link here and cc you

micha15:06:30

thanks for making clojure, it's made my life tangibly better 🙂

richhickey15:06:21

you’re welcome!

seancorfield16:06:46

I ran into this problem when I added Boot support to cfmljure (the crazy CFML/Clojure bridge we use at World Singles). With Leiningen, I could do lein with-profile production do clean, compile, classpath and get a bare bones classpath and then I added those paths to the CFML classpath so it could dynamically load Clojure source code. With Boot, the best I managed was boot some-task aot show -C where some-task added the src folder and user-space dependencies. That still ended up adding a bunch of host Java paths and Boot JAR file paths — but since I only needed to load specific Clojure source namespaces into an already running CFML engine, it wasn’t horrible.

seancorfield16:06:43

The some-task part could be omitted if build.boot added all the dependencies needed directly (but we have several different tasks with different dependencies so we have a rather complex setup).

seancorfield16:06:03

I was surprised there wasn’t a direct equivalent to lein classpath.

seancorfield16:06:16

@micha: Could you checkpoint the classpath as Boot starts up, before it reads build.boot and mark those elements as "not user-space" and then for the new option to show just omit those?

micha17:06:34

@seancorfield: i'm not sure what the boot equivalent to lein classpath would be, with dynamic classpath?

richiardiandrea17:06:38

@seancorfield: it is different from a generator in that you already have a function/ns in place, can I use generators to read and replace with my stuff?

seancorfield17:06:14

@richiardiandrea: A generator can do anything it wants.

seancorfield17:06:33

Look at the function one: it appends text to an existing file.

seancorfield17:06:46

It could easily read a file, modify it, and write it back out.

richiardiandrea17:06:45

oh ok, yes, so my idea was to have some sort of automatic patching, but now I think it is too arbitrary and probably better to just do it manually

seancorfield17:06:17

@micha: I was just suggesting that Boot could use whatever mechanism show -C uses to get the classpath, and run that at startup — after Boot is up and running but before it reads build.boot (and maybe before .boot/profile.boot?) — and just keep a snapshot of that to know that those things should not be included in the output of show —cp (well, the Clojure JAR needs adding back in).

micha17:06:33

yeah or perhaps make them explicit dependencies with :scope "test"

micha18:06:06

nm that's nonsense

richhickey18:06:39

I see boot has BOOT_CLOJURE_VERSION but is there a way I can get boot to use my brand-new local clojure.jar?

micha18:06:36

@richhickey: if you install it in local .m2 BOOT_CLOJURE_VERSION should work

richhickey18:06:40

@micha I was afraid you’d say that 🙂

richhickey18:06:19

I can’t be installing every interim build of clojure while I’m working on it.

micha18:06:43

we could add an env var with a URL

richhickey18:06:48

I would love a way to run w/jars (just like Java can), w/o mvn

micha18:06:20

yes i think developing clojure itself is a unique situation 🙂

micha18:06:44

so would you like to have an env var with a URL that points to clojure.jar?

richhickey18:06:44

so I’m usually doing java -server -cp /Users/rich/dev/clojure/clojure.jar…

richhickey18:06:04

any way I can supply it would be fine

micha18:06:27

ok we can make that work

micha18:06:40

my pleasure!

mhr18:06:38

@micha, what do you mean? I'm pretty sure it's "watch" that's dysfunctional.

micha18:06:01

@mhr: how do you mean?

mhr19:06:05

when I run "boot dev" for the hello world app in hoplon, it sets up the jetty server and everything works. if I make save a change to the hello hoplon cljs file, it will note that and make the "speak" sound acknowledging that. But if I navigate to localhost:8000 and save another change, it spits out a bunch of exceptions. about how the temporary file boot created cannot be deleted: "index.html: The process cannot access the file because it is being used by another process."

micha19:06:37

did you disable the jetty memorymapped files optimization?

micha19:06:48

that is known to cause issues on windows

micha19:06:17

that shows the system property you want to set: org.eclipse.jetty.servlet.Default.useFileMappedBuffer

mhr19:06:34

I'll try that

micha19:06:17

not 100% sure but i think (System/setProperty "org.eclipse.jetty.servlet.Default.useFileMappedBuffer" "false") might be the thing to do

mhr19:06:08

yep, that fixed it! thanks so much micha!

mhr19:06:29

micha, is there a way to modify watch? I'd like to make it so that it only updates the page when the saved version is diffed and shown to have an actual change.

micha19:06:37

@mhr: definitely, watch is just a task, so you can look at the source and make your own implementation

micha19:06:02

boot.core has some functions you might want to use, like fileset-diff for example

micha19:06:27

given two fileset objects, that function returns a new fileset containing only changed/added files

mhr19:06:55

where is the source located? It seems like it's an uberjar, but I might be wrong about that.

micha19:06:51

it's in the boot/core jar

micha19:06:22

the uberjar is just for bootstrapping boot

mhr19:06:59

I mean for windows, it's just boot.exe

micha19:06:43

yes, boot.exe is just to get enough stuff going to download boot jars from maven and load them into the JVM

micha19:06:32

it doesn't actually contain boot

micha19:06:50

that's how it's possible to do boot -u to update, for example

mhr19:06:06

how do I access the actual boot source on my system so I can edit it?

micha19:06:33

i don't think you need to edit it, you can define your own task in a library

micha19:06:52

if you want to hack on boot itself you can clone the boot repo

micha19:06:57

and get the source that way

mhr19:06:51

ah, so I should copy the watch source and substitute a modified version of that to get the functionality I want

micha19:06:02

:+1: exactly

mhr19:06:12

if it downloads boot, can I run boot without access to internet?

micha19:06:29

boot is cached in your local maven repo

micha19:06:49

so once you download it it won't need the internet anymore

micha19:06:13

but of course if you add dependencies to your project that aren't in your local maven cache you'll need the internet for those

micha19:06:24

same situation basically

mhr19:06:39

I'm not familiar with the java ecosystem really, I don't really know what maven does

micha19:06:47

it collects dependencies for you

micha19:06:57

based on coordinates you specify

micha19:06:17

the useful thing it does is resolve transitive dependencies for you

richiardiandrea20:06:02

newbie question, if I pass :exclusions in the pod env (I would like to exclude clojurescript from the backend artifact) is it going to include the transitive dependencies?

micha20:06:27

@richiardiandrea: setting :exclusions will exclude that transitive dependency plus any of its own transitive dependencies, unless another dependency pulls it in

micha20:06:37

for example if i have [[foo/bar "1.2.3" :exclusions [foop/barp]] [baz/baf "2.3.4"]] in my dependencies, and baz/baf has a transitive dependency on foop/barp you will get the version of foop/barp from there

richiardiandrea20:06:44

yes I just tried and there is no trace of either clojurescript or the closure compiler in my uberjar, thanks for confirming @micha, maybe I should PR https://github.com/boot-clj/boot/wiki/Boot-Environment

richhickey21:06:22

so it seems (and I may be wrong) that boot relies almost entirely on dynamic deps. What happens when things go wrong, e.g. conflicts? While I agree generally that builds are processes, deps are not. If I don’t trust Aether et al, then what?

micha21:06:46

@richhickey: boot only forces you to use maven for boots own dependencies, which have no transitive dependencies to interfere (except for dynapath)

micha21:06:12

you can add jars to the classpath with BOOT_JVM_OPTIONS if you wish

micha21:06:14

conflicts are usually handled by creating pods in which you can load conflicting dependencies

micha21:06:43

so in general you would add a single set of dependencies to the main pod via :dependencies, which get resolved as a unit

richhickey21:06:54

I don’t mind maven as a way to get jars, just aether as a way to runtime resolve/load

micha21:06:15

then for additional dynamic dependencies you may create pods with their own atomically resolved dependencies that are isolated from the ones you already have

micha21:06:25

then you can evaluate expressions in the pods

micha21:06:28

to do work

richhickey21:06:42

BOOT_JVM_OPTIONS is outside of boot though, so I’d have to wrap boot with something that did that, possibly per project?

micha21:06:21

there is an issue to have the launcher shim read that file

micha21:06:34

the only reason we haven't yet done it is because of windows

richhickey21:06:11

I was looking for a way to list deps in a file per project and have boot pick that up on its way up

micha21:06:15

there is a boot.properties file you may use to specify these things

micha21:06:28

well not exactly at the moment, because it's read by the java process

micha21:06:37

so BOOT_JVM_OPTIONS is the one thing you can't do in there

richhickey21:06:56

right, I want deps loaded by -cp, not aether

micha21:06:06

but we could modify boot.sh to read that file before it starts the J

richhickey21:06:19

in general, aether great for fiddling around at repl

micha21:06:57

perhaps a -cp opton to boot itself?

richhickey21:06:33

so deps and what ends up on cp are different

richhickey21:06:07

deps get chased transitively to produce cp. I’d like the file to be a list of deps

richhickey21:06:29

with overrides possible, and failing on conflicts not resolved by overrides

micha21:06:45

what would be a conflict in that case?

micha21:06:05

you aren't talking about maven artifacts, just jar URLs right?

richhickey21:06:31

so deps file says a/b/c, is they in turn use x/y/z boot ends up with a/b/c/x/y/z classpath. But if a and b call for different versions of x, should fail until I explicitly list a version of x in deps

richhickey21:06:50

no, maven artifacts

micha21:06:26

so there are two separate concerns here, right?

richhickey21:06:28

maven's dep system, but -cp loading, not aether

richhickey21:06:50

you can use aether to gen the classpath

micha21:06:51

ah interesting

micha21:06:02

i did this yesterday for a hadoop thing

micha21:06:15

i used boot just to resolve the transitive deps

micha21:06:27

have you seen boot show -p?

micha21:06:40

it's a task to visualize conflicts

micha21:06:17

currently boot will not throw an exception when aether resolves conflicts for you

micha21:06:30

we can add that

richhickey21:06:30

yes, that kind of thing is great, but the default (unlike mavens) should be fail on conflict rather than magic

micha21:06:14

ok we can totally do that

richhickey21:06:47

so something like boot -depsfile foobar ?

richhickey21:06:08

with a default so people can do something common in their projects?

micha21:06:18

hmm perhaps

micha21:06:32

there could be an option when you add dependencies, maybe

richhickey21:06:54

yes, if the file is there

micha21:06:32

so you would use a task to generate the depsfile, like boot make-deps-file --out deps perhaps?

micha21:06:13

then when you use the depsfile you could use the option as above

micha21:06:34

boot -depsfile deps repl for instance

richhickey21:06:57

how does make-deps-file know the deps?

micha21:06:42

you could specify them as an option to the make-deps-file task

alandipert21:06:55

Following along, what about a new strict set-env dispatch kW like :dependencies! Like set-env! :Dependencies but fails on conflict

richhickey21:06:03

I presumed I’d just put e.g. [[org.clojure/test.check "0.9.0”]…] in a file in the first place

alandipert21:06:11

Btw hi @richhickey , welcome!

micha21:06:15

sure that would also work

alandipert21:06:48

Maybe you can only call it once too

richhickey21:06:07

when people work with maven/lein, in addition to all the fake declarativeness, there’s a simple list of deps. That part is good

micha21:06:15

so a task that takes a vector of maven coordinates and builds a deps file, throwing an exception if conflicts are not overrided

micha21:06:35

that input vector could come from a file, or wherever you like

richhickey21:06:39

I don’t get the need for a task to gen still

micha21:06:56

the task is where the logic would be for performing that function

micha21:06:23

like it would call into the aether machinery in boot and resolve the dependencies, figure out what needs to be written to the file

richhickey21:06:53

ah, because shim has no java yet

micha21:06:28

yes, but there are two separate things here, the generating of the deps file using maven machinery, and the bootstrapping of boot itself

richhickey21:06:03

‘generating deps’ still not an idea for me. People start knowing their deps. What needs to be generated is a classpath

micha21:06:21

for generating the deps file i think a task would be sufficient, but for bootstrapping boot without maven we will add an option to accept -cp style deps from the file

micha21:06:26

yes sorry, that's what i meant

richhickey21:06:30

generating a classpath should fail and report conflicts when present

micha21:06:42

the task would consume dependencies and emit the classpath

richhickey21:06:13

so, love this idea (gen-classpath task and boot shim picking up static classpath from file or having -cp arg)

richhickey21:06:26

that’s how we work here

micha21:06:56

i will wrap this all up in a github repo tonight and see if it looks like the approach might work

richhickey21:06:35

here’s how I got my REPL btw (no task needed):

micha21:06:44

excellent!

richhickey21:06:25

does boot repl leverage anything about nrepl specifically?

micha21:06:46

@richhickey: not really, only that REPLy, the frontend, talks to it

micha21:06:12

i've wanted to use just a socket repl, but i'm not sure how to get the frontend to work that way

micha21:06:35

REPLy is started in a pod, so it doesn't introduce a dependency to your project

micha21:06:56

but we couldn't figure out how to not use nrepl for the server

micha21:06:08

the repl task does suport options like setting the initial namespace, things like that. those are implemented as nrepl middleware

micha21:06:47

but the repl in boot is kind of a messy special case

micha22:06:47

the ideal situation i think would be if nrepl could evaluate expressions via a socket repl, so we could start nrepl and all of its dependencies (like middleware for emacs cider and so on) in a separate pod

micha22:06:34

as it is i believe nrepl needs to be loaded in the same clojure runtime and can't be used in a separate pod

richhickey22:06:55

yeah, I don’t ever use nrepl

richhickey22:06:58

I’ve got the above working in inferior-lisp and I’m a happy camper 🙂

richhickey22:06:30

fast startup, fast execution, no fiddleware

micha22:06:52

the nrepl dependency is loaded on demand, so that should be good for your use, it won't be there if you don't use repl task

mhr23:06:50

@micha, how would I use the fileset-diff function with filesets in the pipeline? I don't understand where the before and after arguments come from.

micha23:06:56

@mhr: fileset objects are immutable, so you can hold onto one, then diff it against another one later, to see what changed

micha23:06:07

i will find an example, one sec

micha23:06:54

this is the way boot tasks avoid doing unnecessary work -- they keep their own private cache of work that was done, and by diffing the fileset they can know which of those cached artifacts are stale