Fork me on GitHub
#boot
<
2016-06-14
>
dominicm08:06:48

@richiardiandrea: Is it possible to close postgres after boot is killed? No idea if this is a feature boot can support. A sort of "end of stream" call.

martinklepsch09:06:51

@dominicm: boot.core/cleanup is a thing, not sure if that helps with killing postgres though

dominicm10:06:19

@martinklepsch: Looks to be on the mark

micha16:06:07

@dominicm: cleanup is also useful for stopping things (cleaning up side effects from tasks) when the pipeline ends, like in the REPL

micha16:06:25

this basically replaces Component et al in my workflow

micha16:06:43

so like if you do (def f (future (boot (dev)))) and then later do @f and C-c, it will cleanup after itself so you can run (boot (dev)) again without starting more servers on the same ports or whatever

richiardiandrea17:06:47

@dominicm: it is a good idea and I'd add it in order to kill the docker instance in this case

richiardiandrea17:06:03

I have a really weird problem, an exception with tools.reader, which seems old clojure.lang.ExceptionInfo: Unable to resolve var: reader/*alias-map* in this context

richiardiandrea17:06:24

but the version in show -d is 1.0.0-alpha1

micha17:06:28

@richiardiandrea: is this in the repl?

richiardiandrea17:06:39

when I launch boot-cljs-repl

richiardiandrea17:06:51

not with the compiler

richiardiandrea17:06:56

which is super weird

micha17:06:22

yeah the repl evals things in a separate context where you need to bind dynamic vars

richiardiandrea17:06:13

so maybe the dep in boot-cljs-repl is old? does it download it itself?

richiardiandrea17:06:09

that dynamic var was added back in the latest versions

richiardiandrea17:06:39

actually no, it was from 0.10.0-alpha2

richiardiandrea17:06:48

mmm no boot-cljs-repl does not fetch tools.reader

juhoteperi17:06:27

Did someone try calling "sub boot projects" from boot? I'm wondering if I could use Boot to build all Cljsjs packages from from single root project

micha17:06:02

@juhoteperi: the only thing is the CWD

micha17:06:16

you can't change that in a single JVM

juhoteperi17:06:18

Does it have to start new Java for each cwd?

juhoteperi17:06:35

I don't think that's going to be feasible 😞

juhoteperi17:06:50

There is 235 projects currently

micha17:06:27

i think you could do it, as long as the build.boot things in each subproject don't assume that . is the subproject dir

micha17:06:44

you can run boot-in-boots in parallel

micha17:06:00

and your task could eval the subproject build.boot

micha17:06:19

haha yeah @richiardiandrea is the expert here

juhoteperi17:06:38

I could probably eval all the build.boots for version info

micha17:06:42

sorry the "haha" came out wrong

juhoteperi17:06:53

And then start separate JVMs for each changed subproject

micha17:06:42

i don't think the CWD is very important for the subprojects, right?

micha17:06:59

like you can add source/resource paths relative to the cljsjs root

micha17:06:22

they'd all be running with their own tmpdirs so that's not a problem i don't think

juhoteperi18:06:24

Any examples of how to eval sub build.boots?

micha18:06:24

perhaps rather than eval you could make a namespace that defines tasks in each subproject

micha18:06:47

then in the main build.boot in each subboot you could just (require 'subproject.build)

micha18:06:04

and you'd have ./subproject/build.clj

micha18:06:07

or something like that

juhoteperi18:06:30

Perhaps, and that might save some boilerplate in each subproject

juhoteperi18:06:44

But currently it is easy for contributors to build a single subproject from the subproject dir

juhoteperi18:06:55

And simple (similar to normal boot projects)

micha18:06:03

yeah with this scheme you'd need ot build the subprojects from the main one like boot build -p jquery perhaps

richiardiandrea18:06:26

weird, for the problem above I recloned the repo and everything worked

richiardiandrea18:06:44

@micha so the problem was something else, basically a dep in profile.boot was importing tools.reader so I needed to add an :exclusions in it, the :exclusions clause does not work in build.boot...did you know that (of course you did 😸) ?

micha18:06:24

@richiardiandrea: how do you mean doesn't work?

micha18:06:44

oh because profile.boot is evaluated first

micha18:06:52

yeah that's true

micha18:06:20

the thing in profile.boot should be doing its work in a pod probably

richiardiandrea18:06:31

just noting, because maybe we should write it somewhere, i don't know if I would change the behavior

micha18:06:36

you want to avoid transitive dependencies introduced by profile.boot

micha18:06:13

the solution is to package whatever you need in profile.boot as functions or tasks in a library that has no transitive deps

micha18:06:24

that uses pods to do the work that need the deps

richiardiandrea19:06:10

right, so basically deps in profile.boot adds to the global classpath and I was not expecting that, but now it makes sense

richiardiandrea20:06:16

executing in a pod looks a bit cumbersome but at least no deps leak in the main environment: https://gist.github.com/arichiardi/9072fdddd0fe3ac855550718dc7eb59e

micha20:06:22

@richiardiandrea: you could use cleanup to shut the container down, too, or add a shutdown hook

micha20:06:52

if that's a problem

richiardiandrea20:06:27

Yeah next step it will be that...no well, the problem was the deps leak from profile.boot only..solved with a pod

kenny22:06:59

I seem to be inconsistently getting this error when writing the target dirs after a reload. It goes away if I restart my boot process. Is this a known problem?

Writing main.cljs.edn...
Compiling ClojureScript...
• main.js
Writing target dir(s)...
java.util.concurrent.ExecutionException: java.nio.file.NoSuchFileException: target/some/project/myns.cljs___jb_tmp___
      java.nio.file.NoSuchFileException: target/some/project/myns.cljs___jb_tmp___
    file: "target/some/project/myns.cljs___jb_tmp___"
sun.nio.fs.UnixException.translateToIOException               UnixException.java:   86
  sun.nio.fs.UnixException.rethrowAsIOException               UnixException.java:  102
  sun.nio.fs.UnixException.rethrowAsIOException               UnixException.java:  107
   sun.nio.fs.UnixFileSystemProvider.implDelete      UnixFileSystemProvider.java:  244
   sun.nio.fs.AbstractFileSystemProvider.delete  AbstractFileSystemProvider.java:  103
                                            ...                                       
                        boot.filesystem/delete!                   filesystem.clj:  163
                         boot.filesystem/patch!                   filesystem.clj:  178
                                            ...                                       
                clojure.core/apply/invokeStatic                         core.clj:  652
                        clojure.core/partial/fn                         core.clj: 2534
                                            ...                                       
      boot.core/fileset-syncer/fn/iter/fn/fn/fn                         core.clj:  930
            clojure.core/binding-conveyor-fn/fn                         core.clj: 1938
                                            ...                                       
Elapsed time: 0.162 sec

micha22:06:19

is something else doing things in that directory?

kenny22:06:28

Not that I know of

kenny22:06:49

Here is the task:

(deftask web-dev
         []
         (comp
           (serve :dir "target/")
           (watch)
           (cljs-repl)
           (speak)
           (reload)
           (cljs)
           (target)))

kenny22:06:11

It happens very infrequently

micha22:06:12

is that an editor file?

micha22:06:24

like a tempfile created by your text editor/ide?

kenny22:06:54

Not sure. I use Cursive.

micha22:06:11

does cursive know about the target dir?

micha22:06:33

it seems like something is deleting files in there

micha22:06:38

something other than boot

micha22:06:49

which boot really can't intelligently deal with

micha22:06:11

there is no way to make an atomic compare and swap of a directory's contents

micha22:06:31

which OS are you on?

kenny22:06:37

Ubuntu 15.10

micha22:06:47

ah so perhaps lsof could help

micha22:06:59

to see if another program is doing things in there

micha22:06:09

is it always the same file

micha22:06:01

as an experiment you can try switching to vi for a minute and see if you can make it happen

kenny22:06:03

Hmm. Not sure on that one either. I should've been logging each error when it happened but I haven't been. I know it happens in different projects.

micha22:06:07

or nano or emacs etc

kenny22:06:31

Well the thing is I have no idea what causes the error 😛

micha22:06:43

yeah this would eliminate cursive as a possible cause

micha22:06:11

it's either an obscure bug in the boot filesystem patching code, or another process is deleting files in the target dir

kenny22:06:28

It does seem to happen only in cljs projects with a clj file

micha22:06:33

a lot of editors expect to be able to create and delete stuff

micha22:06:58

so eliminating cursive as a cause would be helpful

kenny22:06:18

@cfleming: Does Cursive create a file that looks like target/some/project/myns.cljs___jb_tmp___?

micha22:06:35

good question

micha22:06:45

and in the target dir, also a good question

micha22:06:58

lsof on some of the files in there, see if cursive knows about them

kenny22:06:04

I am no expert at vim/emacs so temporarily switching would be too painful 😝

micha22:06:10

i mean lsof on cursive process

micha22:06:07

lsof -p <cursive_pid> |grep target

kenny22:06:17

I can do that on the IntelliJ process. I don't think Cursive has a process itself.

micha22:06:29

sure, does it show anything?

micha23:06:09

the some/project/ business is also very curious

kenny23:06:24

I renamed the directories

micha23:06:26

ok another approach 🙂

kenny23:06:31

When I pasted..

micha23:06:40

why are you writing to the target dir while in live reload mode?

micha23:06:47

is it to interop with other programs?

kenny23:06:04

So that serve has the latest files

micha23:06:24

ah try without the target task, and without any options to serve

micha23:06:31

that will just serve from the classpath

kenny23:06:40

Oh really? Didn't know that

micha23:06:04

the __jb_tmp___ thing is strange

cfleming23:06:15

@kenny: Cursive doesn’t, but that ‘jb’ is suspicious, it might be a JetBrains thing which might mean IntelliJ is doing it.

cfleming23:06:33

Cursive doesn’t have a separate process, the PID of IntelliJ is what you want for lsof

cfleming23:06:12

I’d be surprised if that were an IntelliJ file though, since IntelliJ doesn’t do anything with cljs

cfleming23:06:48

Actually, I take it back: private static final String EXTENSION_TMP = "_jb_tmp___";

micha23:06:50

and also how would it know about the target dir

cfleming23:06:58

From the IntelliJ project

cfleming23:06:15

@micha: I set that for the project from the lein config

cfleming23:06:25

IntelliJ has a concept of the output dir

micha23:06:43

ah, so it could be trying to do atomic updates

cfleming23:06:47

That’s something called SafeFileOutputStream - right

micha23:06:48

craete temp file, then move

cfleming23:06:57

private static final String EXTENSION_TMP = "___jb_tmp___";
private static final String EXTENSION_OLD = "___jb_old___”;

micha23:06:08

that would confuse boot, because boot diffs that against the fileset

micha23:06:27

and it's impossible to do that atomically unless you use locks

micha23:06:54

@kenny: can you please open an issue? i can investigate

micha23:06:07

seems like other programs should be fine creating files that boot doesn't know about

micha23:06:25

so something in boot is doing the wrong thing there, perhaps

micha23:06:07

@cfleming: any idea what intellij could be doing there?

cfleming23:06:51

@micha: No, sorry - as you suspected, it’s doing a semi-atomic file creation.

cfleming23:06:03

But without a stacktrace I don’t know where that’s being called from.

micha23:06:09

ah thanks

micha23:06:27

it might be possible to have it craete those temp files in a different place, perhaps

micha23:06:54

like the temp file doesn't need to be in the target dir, just on the same filesystem

kenny23:06:13

Why does it happen inconsistently? Seems like it should always happen. Is it a race condition?

micha23:06:55

yes it's a race i think

micha23:06:15

between boot checking to see what it needs to delete, and intellij deleting it first

micha23:06:27

i mean moving it

micha23:06:09

ah bummer

micha23:06:25

not configurable

micha23:06:42

it's still an issue if other processes are writing to files that boot puts in the target dir

micha23:06:06

if that's the case you need to use the -L,--no-link option to the target task

micha23:06:38

as boot will use hard links where it can to be more efficient, but if something writes to them while boot is still running that will corrupt the fileset

kenny23:06:45

Oh, BTW I get a 404 using serve with no options and no target

kenny23:06:08

I am attempting to access a HTML file that is not index.html

micha23:06:50

so you should use (target :no-link true) in your task there

kenny23:06:59

I'll try it

micha23:06:12

that's a different issue than the one you reported just now

kenny23:06:49

The serve one?

micha23:06:31

the --no-link one is to prevent a different issue

richiardiandrea23:06:47

lol is that the above?

micha23:06:58

if boot makes a hard link to myns.cljs, which it will by default, and something writes to it, that will change the contents of the file in the fileset storage

micha23:06:47

but that would corrupt the fileset storage if the contents change, because those internal files are organized by their contents checksum

micha23:06:21

yep emacs lockfiles too

micha23:06:52

when you use --no-link boot will make copies of the files and put those in the target dir, rather than hard links

micha23:06:05

the links are done for performance

kenny23:06:00

Will using --no-link cause a noticeable perf change?

micha23:06:15

can you diff the cljs file in the target dir against the one in your project resource paths

micha23:06:31

if you have a lot of files or large files yes

micha23:06:51

is it actually writing something in those files?

micha23:06:43

@richiardiandrea: actually no, the editor lockfiles are a different issue

micha23:06:16

there isn't any way for boot to know the difference between one of your files and an editor temp file

micha23:06:34

so we can't really do anything about the emacs files

kenny23:06:58

Not sure how to view those files. They are not currently in target dir.

micha23:06:20

presumably the myns.cljs file is in there, no?

kenny23:06:26

Yes it is

kenny23:06:50

No __jb_tmp__ file though

micha23:06:20

yeah so what's happening there is that something in intellij wants to write to myns.cljs in the target dir

micha23:06:36

it's using the safefileoutputstream class to do it atomically

micha23:06:51

so what it does is it writes to a temp file, the _jb_tmp___ one

micha23:06:05

then it renames that to myns.cljs

micha23:06:32

renaming can be done atomically, so it ensures that you don't ever see myns.cljs while it's beig written to

micha23:06:46

so what you want to know is what is being written to that file

micha23:06:10

so diffing myns.cljs in the target dir against the one in your project would provide some insight i think

micha23:06:29

seems like it's probably some kind of intellij feature you can disable

kenny23:06:23

I have continued working so I don't have the file set from when the error occurred anymore

micha23:06:34

did you try diffing?

kenny23:06:42

Not while I had the file set

kenny23:06:48

I can now if it helps

micha23:06:50

yes please

micha23:06:57

and when you see the error again

micha23:06:13

it'll succeed most of the time

micha23:06:56

you'll only see the error occasionally, when intellij creates the file at exactly the right time

kenny23:06:38

No difference between the files right now

kenny23:06:05

The target dir has changed since the error though

kenny23:06:31

Will post the result of the diff once the error occurs again

micha23:06:38

excellent, thanks

kenny23:06:21

Will also save the file set so I can run additional tests if needed. Need any other info now?

micha23:06:48

no that's great

micha23:06:31

@kenny: can you try using a .bootignore file?

micha23:06:05

.bootignore

___jb_tmp___$

kenny23:06:31

Placed at root directory?

micha23:06:37

yes in your project

kenny23:06:34

Shouldn't I do that after the error occurs, otherwise we won't really know if the bootignore solved the problem or not?

micha23:06:04

you need to restart boot for the ignore file to take effect

kenny23:06:11

BTW, not sure if this is apparent or not but the error occurs after using boot for a while. It doesn't occur at boot start up.

micha23:06:46

maybe intellij is doing stuff periodically

micha23:06:57

so maybe it takes a while before it does the thing

kenny23:06:05

It happens if I rapidly change the file

micha23:06:44

how are you configuring cursive?

micha23:06:53

like how are you connecting cursive and boot

micha23:06:12

specifically, does cursive know about the internal boot temp directories?

kenny23:06:52

Not sure what extent Cursive knows about boot temp dirs. My project is configured as a lein project. The project.clj is generated using boot lein-generate. boot web-dev is being ran in IntelliJ external tools.

kenny23:06:11

There is no difference in the files. Though not sure if diff in linux will show the new lines.

kenny23:06:10

The whole boot process freezes after the error

micha23:06:40

yeah we could just catch those exceptions, but that could cause other problems

micha23:06:09

does the issue still occur with the .bootignore?

kenny23:06:11

Well it seems this is an exceptional case so makes more sense to not catch the error

kenny23:06:17

I'll try that now