Fork me on GitHub
#hoplon
<
2015-10-10
>
micha00:10:27

it's other code that's not explicitly closing streams

micha00:10:36

the clojure compiler does this, for instance

micha00:10:07

but i think it's pretty likely that there are other places in libraries where this happens

symbit01:10:42

Yes, Issue 117 is the same root cause that I'm seeing and it looks like it's a FileDescriptor leak in Clojure 1.7, which will be fixed for Clojure 1.8. I upgraded to 1.8-alpha5 at least for dev, but ClojureScript has the same issue as Alex pointed out.

symbit01:10:56

clojure.lang.Compiler$CompilerException: java.lang.VerifyError: (class: cljs/util$last_modified, method: invokeStatic signature: (Ljava/lang/Object;)Ljava/lang/Object;) Can only throw Throwable objects, compiling:(util.cljc:142:1) java.lang.VerifyError: (class: cljs/util$last_modified, method: invokeStatic signature: (Ljava/lang/Object;)Ljava/lang/Object;) Can only throw Throwable objects

alandipert01:10:24

ah, that's another clj bug

symbit01:10:48

837 was closed around the 1.3 era.

symbit01:10:53

cljs/util$last_modified

symbit01:10:45

It looks like the same smoking gun as the Clojure bug that was fixed. Alex Miller noted that it exists in CS as well.

symbit02:10:42

I updated the boot.build to the latest of clojure and clojure script, but am getting the exception above when running boot dev on the address-book example. Also tried updating boot.properties to use 1.8.0-alpha5 as the boot version, still no dice. I noticed that the boot.properties in the ~/.boot/cache/ says 1.7.0 despite removing the cache dir. Any Ideas?

alandipert13:10:39

@symbit: i think 1.8-alpha5 has the file system fix, but has the verifyerror bug

alandipert13:10:54

@symbit: you may consider building and deploying your own 1.7 with the last modifiied patch applied

symbit14:10:53

@alandipert: Thank Alan. I'll go the custom 1.7 route. Interesting to see 1809 is yet another unrelated bug. Yikes.

micha15:10:17

the latest versions of boot have the option to set your own clojure dep

micha15:10:26

via env var

micha15:10:53

@symbit: eg. BOOT_CLOJURE_NAME=me.micha/clojure BOOT_CLOJURE_VERSION=1.8.0-micha1 boot ...

micha15:10:15

in case you need to make more extensive modifications simple_smile

micha15:10:29

@symbit: one thing you could try is spinning when you get the can't delete error

micha15:10:53

if it works okay we can do that on windows

micha15:10:09

like keep trying to delete the file until it succeeds

micha15:10:25

gc will eventually run and the streams will be closed

micha15:10:26

symbit: are you pretty good with windows scripting? we'd like to have a script that calls the boot jar instead of the launch4j thing, i think

symbit16:10:15

I'm using my patched 1.7.0 clojure version, but on change of the src file I get the same: java.io.IOException: Couldn't delete C:\Users\me\.boot\cache\tmp\somedir\dev\clj\address-book\4aw\gg42n1\index.html.out for every file in the cache being moved/deleted.

micha16:10:00

yeah i think the odds of fixing every open stream in every dependency you might have is not really feasible

micha16:10:22

can you try adding a loop to keep trying to delete if it gets that exception?

symbit16:10:23

The last line of the stack trace is different: java.nio.file.FileAlreadyExistsException: C:\Users\me\.boot\cache\tmp\somedir\dev\clj\address-book\4aw\gg42n1\index.html -> C:\Users\me\.boot\cache\tmp\somedir\dev\clj\address-book\4aw\-7kd6cy\a3e4acfecc74eee2a20111610f7c9624

micha16:10:26

in the boot code

symbit16:10:08

where abouts?

symbit16:10:28

deftask watch?

micha16:10:33

no, deeper

micha16:10:05

boot should only ever delete things it owns

micha16:10:10

so it should be safe to do this

micha16:10:22

you'll never have a reference to any files in the fileset

symbit16:10:32

Is this an example of a file in the fileset? java.nio.file.FileAlreadyExistsException: C:\Users\me\.boot\cache\tmp\somedir\dev\clj\address-book\4aw\gg42n1\index.html

micha16:10:54

anything in $BOOT_HOME

micha16:10:04

i.e. ~/.boot

micha16:10:09

(by default)

micha16:10:40

you can modify the boot/pod project and install the jar in your local maven repo

micha16:10:57

like from the boot-clj/boot repo dir do:

micha16:10:31

cd boot/pod
... make your changes to src/boot/file.clj ...
lein install

micha16:10:41

i think that should work

symbit16:10:53

You're thinking put a retry (say, 10 times) loop there if exception?

micha16:10:18

yeah something like that

micha16:10:36

maybe throw a System.gc() and a small delay in the loop, too

micha16:10:20

calling gc() multiple times seems to help, i've experimented with that for other purposes before

micha16:10:41

iirc that's what visualvm does, too

micha16:10:52

for their "force gc" button

micha16:10:28

this java ti tools thing seems like something we might look into though

micha16:10:17

nm, tool interface doesn't seem to have anything we need

micha16:10:40

the force gc thing in there doesn't run finalization code, so it wouldn't help here

symbit16:10:12

How do you make boot.exe?

micha16:10:54

the makefile does it, using launch4j

micha16:10:06

you shouldn't need to do that though

micha16:10:24

since you're not updating the boot.App java class

micha16:10:39

the code you're modifying is jsut a maven dependency

symbit16:10:13

ok, which dep should I be putting into .m2 for boot.exe to use?

symbit16:10:02

I converted the makefile to run in windows. ran make deps and make install

micha16:10:59

ok yeah that would work too

micha16:10:09

you're good to go then

micha16:10:18

you don't need to mess with the boot exe at all

symbit16:10:33

did the 'make install' update the needed loop fix dep in the local maven repo?

symbit16:10:53

I think I'm catching up.. We're just running the project.clj for pod.

micha17:10:45

boot needs 2-3 different pods with different dependencies when it starts up and prepares to run your build.boot

micha17:10:59

that's why there are a number of jars there

micha17:10:08

the boot/pod jar being one of those

micha17:10:22

different pods have different dependency jars

micha17:10:56

boot loads its own code from maven

micha17:10:01

via these jars

micha17:10:32

the boot.exe thing is just a shim that gets maven working and craetes the pods boot needs to bootstrap itself

symbit18:10:23

Ok, will try this later tonight. Thanks Micha.

micha18:10:35

sure, let us know how it turns out

micha18:10:48

if the looping works for you we could incorporate that into boot

symbit20:10:57

No Dice. I also added a sleep 50 ms between reties and a call to (System/gc).

micha20:10:27

how many times did you let it loop for?

symbit20:10:44

For each file

micha20:10:39

maybe 10 isn't enough

symbit20:10:31

I'll try 100 per file.

micha20:10:29

after the first one passes i'd expect them all to pass

symbit20:10:03

Same thing.

micha20:10:50

is there a way to trick windows into allowing you to delete things?

micha20:10:13

like "relax bro we got this"

micha21:10:18

maybe a fuse filesystem or something

micha21:10:29

i wonder if there is such a thing already

micha21:10:51

like a fuse filesystem that makes things sane

symbit21:10:57

Ha! I'm guessing that something is holding on to the file handles. Perhaps the Browser? Numerous files in the cache are trying to be deleted without success.

micha21:10:18

nothing should be looking in those temp dirs though

micha21:10:29

boot copies your source files into the tempdirs

micha21:10:52

and at the end copies the artifacts to the target dir

micha21:10:44

there is a tool you can use to see what's holding the references though

micha21:10:51

it's in the issue i linked to

micha21:10:01

that's how the clj bug was found

symbit21:10:35

I'm using ProcessExplorer to see which process is holding on to the cache dir. It's the Java process.

micha21:10:21

what does it mean to hold onto a directory?

micha21:10:29

boot only ever cares about files

micha21:10:41

it doesn't really even see directories, like git

symbit21:10:16

In windows if an app has a file open, no one can remove the directory the file is in or any parent directory the file is in, unlike in Unix.

micha21:10:42

boot doesn't remove directories though

symbit21:10:49

I'm not a windows guy though.

micha21:10:03

it will create them if it needs to create them in order to create a file

micha21:10:14

but it doesn't pay attention to them otherwise

micha21:10:38

no temp dirs should be deleted while boot runs though

symbit21:10:40

ok, just an example. Same goes for files. If you are editing a file in windows you can't delete the file from another process.

micha21:10:22

there is a tool that was used to see which java code was opening input streams and not closing them

micha21:10:34

there is a link to it in the issue

micha21:10:52

if boot is the culprit we can fix that

micha21:10:36

what about running boot in docker or vmware or something?

micha21:10:55

i'm not sure how docker works on windows

micha21:10:24

especially when you map volumes and whatnot

symbit21:10:58

So boot is creating files in dir X then moving them dir Y. When I edit the source file what is trying to do? From the output on windows at least it looks like it's trying to delete all the files in the cache.

micha21:10:42

yeah it is

micha21:10:53

some of them maybe unnecessarily

symbit21:10:00

I could try to mimic what boot is doing to see if I can recreate the issue or prove that the file can be closed, which would mean boot has to close in another way or something.

micha21:10:06

but it will need to delete things in there

micha21:10:35

here is what's happening

micha21:10:03

suppose you have :source-paths #{"src"}

micha21:10:12

and the file src/index.cljs.hl

micha21:10:47

when you run boot it copies that file into a tempdir managed by boot

symbit21:10:09

tempdir is .boot/cache/... ?

micha21:10:45

this happens constantly, whether you have the watch task or not

micha21:10:09

a background thread monitors your source paths and copies files into the tempdir whenever anything is added or changed

micha21:10:23

this has nothing to do with the watch task though, it's an internal boot thing

micha21:10:41

so now there is the latest version of your file in the tempdir

micha21:10:51

when you build something it creates a fileset

micha21:10:22

when it does this it takes all the files in that tempdir and copies them to the blob storage dir if necessary

micha21:10:34

they're named by their md5 checksum

micha21:10:42

so it doesn't copy things unnecessarily

micha21:10:58

the files in this blob storage dir are set to not be writeable by anyone

micha21:10:08

they're read-only

micha21:10:16

the fileset maps paths to blob files

symbit21:10:17

what uses the files that go into the directories like /4t8/gg42n1/ ?

micha21:10:43

when you call commit! on a fileset object, it creates hard links from other tempdirs that are in the classpath to blob files

micha21:10:15

well actually it deletes all the existing hard links and then rebuilds them according to the fileset object map

symbit21:10:16

what API creates the hard link?

micha21:10:26

it's in the boot.file namespace

micha21:10:57

so when you change index.cljs.hl, a new version of the file is copied to the tempdir

micha21:10:17

and when the fileset is created to build the project it gets copied into the blob storage

micha21:10:23

(because it has a new md5 checksum)

micha21:10:40

and when commit! is called the hardlink is moved to point to the new blob file

symbit21:10:44

... to a new temp dir or the old existing one?

micha21:10:13

no tempdirs are created or destroyed

micha21:10:41

boot creates all the tempdirs it needs when it starts up

symbit21:10:25

BTW, the retry code is still chugging with 100 retries per file and 5ms + GC call.

symbit21:10:46

I set hard-link to false. The first change/save to index.cljs.hl, compiled fine, but the browser didn't refresh.

symbit21:10:08

However subsequent changes to the source file throws: java.io.FileNotFoundException: C:\Users\me\.boot\cache\tmp\somedir\dev\clj\address-book\5jc\-yrrdlm\index.html.js (The requested operation cannot be performed on a file with a user-mapped section open)

symbit21:10:12

I'm guessing windows doesn't have hard links as we know them in unix.

micha21:10:33

why do you say that?

micha21:10:10

you turned hard links off, no?

symbit21:10:01

Just a guess. Would have to debug into the createLink method introduced in Java 1.7

symbit21:10:40

I changed to do sym-link. ha! instead of rendering the source file, it shows the directory. Clicking on index.html gives a 404. Yeah, I don't think windows can handle the 'advanced' file management API's being called.

micha21:10:17

sym links are ignored specifically by the fileset stuff

micha21:10:43

because they could result in deleting your entire home directory or something

micha21:10:00

boot needs to be able to completely own the things that are in the tempdirs

micha21:10:10

so it can freely delete them or move them around etc

micha21:10:20

so it refuses to recognize symlinks

symbit21:10:44

ok, Just for fun I'm calling the unused method sym-link