@richiardiandrea: Is it possible to close postgres after boot is killed? No idea if this is a feature boot can support. A sort of "end of stream" call.

@dominicm: boot.core/cleanup is a thing, not sure if that helps with killing postgres though

@martinklepsch: Looks to be on the mark

@dominicm: cleanup is also useful for stopping things (cleaning up side effects from tasks) when the pipeline ends, like in the REPL

this basically replaces Component et al in my workflow

so like if you do (def f (future (boot (dev)))) and then later do @f and C-c, it will cleanup after itself so you can run (boot (dev)) again without starting more servers on the same ports or whatever

@dominicm: it is a good idea and I'd add it in order to kill the docker instance in this case

I have a really weird problem, an exception with tools.reader, which seems old clojure.lang.ExceptionInfo: Unable to resolve var: reader/*alias-map* in this context

but the version in show -d is 1.0.0-alpha1

@richiardiandrea: is this in the repl?

when I launch boot-cljs-repl

not with the compiler

which is super weird

yeah the repl evals things in a separate context where you need to bind dynamic vars

so maybe the dep in boot-cljs-repl is old? does it download it itself?

i'll check

that dynamic var was added back in the latest versions

actually no, it was from 0.10.0-alpha2

mmm no boot-cljs-repl does not fetch tools.reader

Did someone try calling "sub boot projects" from boot? I'm wondering if I could use Boot to build all Cljsjs packages from from single root project

@juhoteperi: the only thing is the CWD

you can't change that in a single JVM

Does it have to start new Java for each cwd?

yes

I don't think that's going to be feasible 😞

There is 235 projects currently

i think you could do it, as long as the build.boot things in each subproject don't assume that . is the subproject dir

you can run boot-in-boots in parallel

yey!

and your task could eval the subproject build.boot

haha yeah @richiardiandrea is the expert here

lol

I could probably eval all the build.boots for version info

sorry the "haha" came out wrong

And then start separate JVMs for each changed subproject

i don't think the CWD is very important for the subprojects, right?

like you can add source/resource paths relative to the cljsjs root

they'd all be running with their own tmpdirs so that's not a problem i don't think

Any examples of how to eval sub build.boots?

perhaps rather than eval you could make a namespace that defines tasks in each subproject

then in the main build.boot in each subboot you could just (require 'subproject.build)

and you'd have ./subproject/build.clj

or something like that

Perhaps, and that might save some boilerplate in each subproject

But currently it is easy for contributors to build a single subproject from the subproject dir

And simple (similar to normal boot projects)

yeah with this scheme you'd need ot build the subprojects from the main one like boot build -p jquery perhaps

weird, for the problem above I recloned the repo and everything worked

@micha so the problem was something else, basically a dep in profile.boot was importing tools.reader so I needed to add an :exclusions in it, the :exclusions clause does not work in build.boot...did you know that (of course you did 😸) ?

@richiardiandrea: how do you mean doesn't work?

oh because profile.boot is evaluated first

yeah

yeah that's true

the thing in profile.boot should be doing its work in a pod probably

just noting, because maybe we should write it somewhere, i don't know if I would change the behavior

you want to avoid transitive dependencies introduced by profile.boot

exactly

the solution is to package whatever you need in profile.boot as functions or tasks in a library that has no transitive deps

that uses pods to do the work that need the deps

right, so basically deps in profile.boot adds to the global classpath and I was not expecting that, but now it makes sense

executing in a pod looks a bit cumbersome but at least no deps leak in the main environment: https://gist.github.com/arichiardi/9072fdddd0fe3ac855550718dc7eb59e

@richiardiandrea: you could use cleanup to shut the container down, too, or add a shutdown hook

if that's a problem

Yeah next step it will be that...no well, the problem was the deps leak from profile.boot only..solved with a pod

I seem to be inconsistently getting this error when writing the target dirs after a reload. It goes away if I restart my boot process. Is this a known problem?

Writing main.cljs.edn...
Compiling ClojureScript...
• main.js
Writing target dir(s)...
java.util.concurrent.ExecutionException: java.nio.file.NoSuchFileException: target/some/project/myns.cljs___jb_tmp___
      java.nio.file.NoSuchFileException: target/some/project/myns.cljs___jb_tmp___
    file: "target/some/project/myns.cljs___jb_tmp___"
sun.nio.fs.UnixException.translateToIOException               UnixException.java:   86
  sun.nio.fs.UnixException.rethrowAsIOException               UnixException.java:  102
  sun.nio.fs.UnixException.rethrowAsIOException               UnixException.java:  107
   sun.nio.fs.UnixFileSystemProvider.implDelete      UnixFileSystemProvider.java:  244
   sun.nio.fs.AbstractFileSystemProvider.delete  AbstractFileSystemProvider.java:  103
                                            ...                                       
                        boot.filesystem/delete!                   filesystem.clj:  163
                         boot.filesystem/patch!                   filesystem.clj:  178
                                            ...                                       
                clojure.core/apply/invokeStatic                         core.clj:  652
                        clojure.core/partial/fn                         core.clj: 2534
                                            ...                                       
      boot.core/fileset-syncer/fn/iter/fn/fn/fn                         core.clj:  930
            clojure.core/binding-conveyor-fn/fn                         core.clj: 1938
                                            ...                                       
Elapsed time: 0.162 sec

is something else doing things in that directory?

Not that I know of

Here is the task:

(deftask web-dev
         []
         (comp
           (serve :dir "target/")
           (watch)
           (cljs-repl)
           (speak)
           (reload)
           (cljs)
           (target)))

strange

It happens very infrequently

is that an editor file?

like a tempfile created by your text editor/ide?

Not sure. I use Cursive.

does cursive know about the target dir?

it seems like something is deleting files in there

something other than boot

which boot really can't intelligently deal with

there is no way to make an atomic compare and swap of a directory's contents

which OS are you on?

Ubuntu 15.10

ah so perhaps lsof could help

to see if another program is doing things in there

is it always the same file

?

as an experiment you can try switching to vi for a minute and see if you can make it happen

Hmm. Not sure on that one either. I should've been logging each error when it happened but I haven't been. I know it happens in different projects.

or nano or emacs etc

Well the thing is I have no idea what causes the error 😛

yeah this would eliminate cursive as a possible cause

it's either an obscure bug in the boot filesystem patching code, or another process is deleting files in the target dir

It does seem to happen only in cljs projects with a clj file

a lot of editors expect to be able to create and delete stuff

so eliminating cursive as a cause would be helpful

@cfleming: Does Cursive create a file that looks like target/some/project/myns.cljs___jb_tmp___?

good question

and in the target dir, also a good question

lsof on some of the files in there, see if cursive knows about them

I am no expert at vim/emacs so temporarily switching would be too painful 😝

i mean lsof on cursive process

lsof -p <cursive_pid> |grep target

I can do that on the IntelliJ process. I don't think Cursive has a process itself.

sure, does it show anything?

Nope

the some/project/ business is also very curious

I renamed the directories

ok another approach 🙂

When I pasted..

why are you writing to the target dir while in live reload mode?

is it to interop with other programs?

So that serve has the latest files

ah try without the target task, and without any options to serve

that will just serve from the classpath

Oh really? Didn't know that

the __jb_tmp___ thing is strange

@kenny: Cursive doesn’t, but that ‘jb’ is suspicious, it might be a JetBrains thing which might mean IntelliJ is doing it.

Cursive doesn’t have a separate process, the PID of IntelliJ is what you want for lsof

I’d be surprised if that were an IntelliJ file though, since IntelliJ doesn’t do anything with cljs

Actually, I take it back: private static final String EXTENSION_TMP = "_jb_tmp___";

and also how would it know about the target dir

From the IntelliJ project

@micha: I set that for the project from the lein config

IntelliJ has a concept of the output dir

ah, so it could be trying to do atomic updates

That’s something called SafeFileOutputStream - right

craete temp file, then move

private static final String EXTENSION_TMP = "___jb_tmp___";
private static final String EXTENSION_OLD = "___jb_old___”;

that would confuse boot, because boot diffs that against the fileset

and it's impossible to do that atomically unless you use locks

@kenny: can you please open an issue? i can investigate

Sure

seems like other programs should be fine creating files that boot doesn't know about

so something in boot is doing the wrong thing there, perhaps

@cfleming: any idea what intellij could be doing there?

@micha: No, sorry - as you suspected, it’s doing a semi-atomic file creation.

But without a stacktrace I don’t know where that’s being called from.

ah thanks

it might be possible to have it craete those temp files in a different place, perhaps

like the temp file doesn't need to be in the target dir, just on the same filesystem

Why does it happen inconsistently? Seems like it should always happen. Is it a race condition?

yes it's a race i think

https://upsource.jetbrains.com/idea-ce/file/idea-ce-97c79e88f838fed73f4ea1c3aca8a37ae68e8253/platform/util/src/com/intellij/util/io/SafeFileOutputStream.java?nav=0:0:focused

between boot checking to see what it needs to delete, and intellij deleting it first

i mean moving it

ah bummer

not configurable

it's still an issue if other processes are writing to files that boot puts in the target dir

if that's the case you need to use the -L,--no-link option to the target task

as boot will use hard links where it can to be more efficient, but if something writes to them while boot is still running that will corrupt the fileset

Oh, BTW I get a 404 using serve with no options and no target

I am attempting to access a HTML file that is not index.html

so you should use (target :no-link true) in your task there

I'll try it

that's a different issue than the one you reported just now

The serve one?

the --no-link one is to prevent a different issue

lol is that the above?

if boot makes a hard link to myns.cljs, which it will by default, and something writes to it, that will change the contents of the file in the fileset storage

but that would corrupt the fileset storage if the contents change, because those internal files are organized by their contents checksum

yep emacs lockfiles too

when you use --no-link boot will make copies of the files and put those in the target dir, rather than hard links

the links are done for performance

Will using --no-link cause a noticeable perf change?

can you diff the cljs file in the target dir against the one in your project resource paths

if you have a lot of files or large files yes

is it actually writing something in those files?

@richiardiandrea: actually no, the editor lockfiles are a different issue

there isn't any way for boot to know the difference between one of your files and an editor temp file

so we can't really do anything about the emacs files

Not sure how to view those files. They are not currently in target dir.

presumably the myns.cljs file is in there, no?

Yes it is

No __jb_tmp__ file though

yeah so what's happening there is that something in intellij wants to write to myns.cljs in the target dir

it's using the safefileoutputstream class to do it atomically

so what it does is it writes to a temp file, the _jb_tmp___ one

then it renames that to myns.cljs

renaming can be done atomically, so it ensures that you don't ever see myns.cljs while it's beig written to

so what you want to know is what is being written to that file

so diffing myns.cljs in the target dir against the one in your project would provide some insight i think

seems like it's probably some kind of intellij feature you can disable

I have continued working so I don't have the file set from when the error occurred anymore

did you try diffing?

Not while I had the file set

I can now if it helps

yes please

and when you see the error again

Sure

it'll succeed most of the time

you'll only see the error occasionally, when intellij creates the file at exactly the right time

No difference between the files right now

The target dir has changed since the error though

Will post the result of the diff once the error occurs again

excellent, thanks

Will also save the file set so I can run additional tests if needed. Need any other info now?

no that's great

@kenny: can you try using a .bootignore file?

.bootignore

___jb_tmp___$

Placed at root directory?

yes in your project

Shouldn't I do that after the error occurs, otherwise we won't really know if the bootignore solved the problem or not?

sure

you need to restart boot for the ignore file to take effect

BTW, not sure if this is apparent or not but the error occurs after using boot for a while. It doesn't occur at boot start up.

maybe intellij is doing stuff periodically

so maybe it takes a while before it does the thing

Got it

It happens if I rapidly change the file

how are you configuring cursive?

like how are you connecting cursive and boot

specifically, does cursive know about the internal boot temp directories?

Not sure what extent Cursive knows about boot temp dirs. My project is configured as a lein project. The project.clj is generated using boot lein-generate. boot web-dev is being ran in IntelliJ external tools.

There is no difference in the files. Though not sure if diff in linux will show the new lines.

The whole boot process freezes after the error

yeah we could just catch those exceptions, but that could cause other problems

does the issue still occur with the .bootignore?

Well it seems this is an exceptional case so makes more sense to not catch the error

I'll try that now