Fork me on GitHub
#boot
<
2016-12-27
>
richiardiandrea01:12:30

@nonrecursive about tests, I have noticed that the pod in boot-test really slows things down probably because it is running in a pod

richiardiandrea01:12:08

it looks like the pod is cold/new every time but it is weird, as checking the code shows a simple call to :refresh: https://github.com/adzerk-oss/boot-test/blob/fa0dcc54e846f24d17356b0a11bb156cd07f0771/src/adzerk/boot_test.clj#L90

flyboarder02:12:33

@micha so im kinda just lost now, still trying to make a restartable server, dont know why this is being so difficult, clearly missing something

micha02:12:42

@flyboarder what's up?

flyboarder02:12:09

well, I removed the pod stuff as it was distracting

micha02:12:42

with fileset-diff you obtain a new fileset with only the changed files in it

micha02:12:59

you can then inspect that to see if the file you're interested in has changed

micha02:12:03

and do something then

micha02:12:21

you can store the previous fileset for the diff in an atom

micha02:12:56

(fileset-diff nil some-fileset) returns some-fileset so you're good for the first iteration when the previous fileset atom will be nil

micha02:12:11

i think you want to just wrap the stop part in (when (seq cljses) ... perhaps?

micha02:12:28

that way it will only happen when there are changed files of the type you're interested in

micha02:12:22

and the start part you probably want to wrap in (when-not @server @(future (start)))

micha02:12:35

or something like that

micha02:12:57

that way it will only start when it hasn't started yet or if it's been stopped

flyboarder02:12:03

@micha edited, how can I make it non-blocking?

micha02:12:33

i don't see the edit

micha02:12:43

but if you don't deref the future it will not block

micha02:12:57

derefing the future is telling it to block

flyboarder02:12:51

ok, hmm it starts the server but throws some random error when a file is changed

micha02:12:17

what's the sync! part for there?

flyboarder02:12:33

for the files which are served

flyboarder02:12:25

java.lang.IllegalArgumentException: No value supplied for key: nodejs.out/app/users.cljs

flyboarder03:12:49

hmm seems it doesnt like to rebuild files

flyboarder03:12:49

hmm so rerunning the future causes that

micha03:12:54

looks like a kwargs mismatch

lambdahands16:12:09

So I switched from Leiningen to Boot in a project, and it's been working great so far. My team is now planning to start working on a ClojureScript UI, so I added Figwheel to our project. Everything works smoothly, except for this error, which from all of the archeology I've done about it has to do with an OS X library:

2016-12-27 10:33 java[61220] (FSEvents.framework) FSEventStreamStart: register_with_server:ERROR: f2d_register_rpc() => (null) (-21)

micha16:12:45

are you working with a network filesystem?

micha16:12:59

like your home dir on nfs or something

lambdahands16:12:23

No, I'm just on my laptop at the moment

micha16:12:53

do you have a large number of files?

micha16:12:58

in your project

lambdahands16:12:34

Hmm, I don't believe I do - let me check the file count in the directories I'm pulling in

lambdahands16:12:46

So the src/ directory contains 123 files

lambdahands16:12:13

Which I don't think is a lot, considering that contains a lot of DB logic, an events system, etc

lambdahands16:12:12

But the resources directory contains 390 files, most of it from an out/ directory that ClojureScript compiled to. I believe it contains all of the CLJS and Google Closure modules

micha16:12:17

yeah that's not a lot

micha16:12:26

i was thinking more like 20k files

micha16:12:07

which version of OSX/java?

lambdahands16:12:49

I'm on OSX 10.12 and Java 1.8.0_77

micha16:12:08

is 10.12 the latest one?

lambdahands16:12:53

Yeah, that's macOS Sierra - I'm wondering if the latest OS has something to do with it

micha16:12:02

is it actually breaking your build?

micha16:12:23

sometimes it's just garbage printed to the screen but not actually interfering with the build

lambdahands16:12:25

So it's not breaking my build, and I was okay with just ignoring the errors (a lot of them would pop up at once), but I then began to realize that Figwheel would intermittently fail to load new code after save

lambdahands16:12:35

But I'm curios that you said ~20K files might cause stuff like this - is there a way for me to inspect the fileset and get a count of files that are being watched?

micha16:12:38

sure, you can insert show -f anywhere in the pipeline to see the files at that point

micha16:12:58

like boot pom show -f jar target

micha16:12:14

or boot pom jar show -f target

micha16:12:54

also you can use the -vv boot option

micha16:12:02

like boot -vv watch foo bar

micha16:12:17

that will log a message when it registers a watch on any directory

micha16:12:26

maybe grep that

lambdahands16:12:27

Whoa, way cool!

micha16:12:03

-vv logs a lot of stuff so grep is probably helpful

micha16:12:49

can you paste your set-env! configuration?

lambdahands17:12:42

Sure! I set it conditionally, but my development environment looks like:

:source-paths #{"src" "test" "spec" "dev" "ui/admin"}
:resource-paths #{"resources"}
:dependencies dev-deps

micha17:12:28

ah ok, that looks fine

micha17:12:49

i was thinking maybe if something like "." was in there or something

micha17:12:57

weird stuff could happen

lambdahands17:12:00

There is also another possibility: after some searching I just found that Figwheel also uses FSEvents to trigger updates. If two processes are watching the same files, would that produce errors like the ones I'm getting?

micha17:12:28

yeah i think that could be it, i think there is a global throughput limit for fs events

micha17:12:32

and maybe for watches

micha17:12:01

that error you pasted might be caused by multiple watches on the same directory

micha17:12:19

are you putting figwheel on the target dir?

micha17:12:29

or are you configuring it to watch your project dirs

micha17:12:35

cause boot will put watches on those

micha17:12:50

but boot doesn't ever do anything with the target dir

micha17:12:07

it's write-only as far as boot is concerned

micha17:12:36

also have you tried the boot-reload/oot-cljs-repl tasks?

micha17:12:45

s/oot/boot/

lambdahands17:12:30

Ohh, I see - figwheel is compiling to the resources directory so my server can pick them up

micha17:12:54

with boot you really want unidirectional data flow

micha17:12:21

or things change out from under boot and it can get confused

micha17:12:45

there is no way to eliminate race conditions when you're dealing with the filesystem

micha17:12:13

that's why the fileset does copy on write

micha17:12:18

so files are immutable

lambdahands17:12:52

So in a way boot is a lot like a virtual DOM for the filesystem?

micha17:12:05

yeah pretty much sorta

micha17:12:21

it's like git really

micha17:12:34

where you have a working tree that changes when you checkout

micha17:12:49

but the actual content is stored in blob storage

micha17:12:54

content-addressed

micha17:12:57

and immutable

micha17:12:16

the files in the filesystem from the fileset are actually just hard links into the immutable blob storage

micha17:12:38

so when you change a file's contents and replace a file in the fileset with your new modified version

micha17:12:45

boot adds the new version to blob storage

micha17:12:54

and updates the hard link to point to the new version

micha17:12:06

atomically

micha17:12:39

well not atomically because it needs to delete the link and make a new link that points to the new thing

micha17:12:54

but the main thing is that the old version is still there

micha17:12:08

so you can "rewind" back to the old version

lambdahands17:12:32

Very interesting - so if many events are fired at once, and some result in a change in a file's contents, boot has to react to each change and possibly hit a threshold?

micha17:12:03

boot does some debouncing there

micha17:12:15

it's set to 10ms by default but you can configure it

micha17:12:39

because of the way fsevents work they tend to send lots of events

micha17:12:55

most of which are sort of redundant

micha17:12:03

i mean for boot's needs

micha17:12:13

so it does some batching

micha17:12:33

and because of race conditions it doesn't use the events themselves

micha17:12:49

it just uses the events as a notification that something changed

micha17:12:58

it still diffs the filesystem itself

micha17:12:29

that doesn't completely eliminate race conditions of course

micha17:12:33

but it helps

micha17:12:55

like if you're writing and deleting files before boot can process them things can get weird

micha17:12:35

like if you have a process that makes thousands of temp files in the project dirs and deletes them quickly

micha17:12:04

boot just ignores them when they're deleted before it can handle them

micha17:12:10

the more i work with inotify/fsevents the more i realize i don't know what i'm doing

lambdahands17:12:26

Gotcha - I'm messing around with the debounce time and also configuring figwheel to use polling, but I don't think it's going to solve the issue you laid out of stuff happening underneath boot

lambdahands17:12:43

s/inotify/computers

micha17:12:49

lol right

micha17:12:01

making a thing like inotifywait today in C

micha17:12:40

want to do like inotifywait /some/dir | inotifytail | zmqpub 0.0.0.0 66666

micha17:12:12

yeah i don't want to complect the watching with doing work

micha17:12:54

like i pipe the stream of change messages to a tool that tails the files as they change and prints the lines to sdtout, and pipe that to a 0mq pubsub publisher

micha17:12:59

for isntance

lambdahands17:12:31

Ohh very interesting

micha17:12:38

but the existing inotifywait doesn't do exactly what i need

micha17:12:03

it doesn't have an option to initilize by sending a modification event for all files in the dir

micha17:12:05

when it starts

micha17:12:16

which i would need for the tailing process

micha17:12:29

cause it's stateful, needs to keep track of the files in the dir as they change

lambdahands17:12:10

Ahh, I see - could you use another process to mock those events after scanning the directory and send to inotifytail?

micha17:12:57

how would that process know which dir to scan?

lambdahands17:12:21

good point - I was thinking whichever one was being passed to inotifywait. I'm not super well read on unix/pipes - but there may be a way to call a sequence of processes and pipe them both to another?

micha17:12:30

the inotify C api is pretty hairy, so i was thinking to have one program that just does that and exposes a stream of changes on stdout, but maybe i need to just make a nice c library and roll that into the tail program?

micha17:12:02

seemed nice and unixlike to have a separate program tht just does that

lambdahands17:12:06

Yeah that does seem to be more unixy

pesterhazy18:12:30

@micha, have you looked at watchman? It does exactly that

pesterhazy18:12:58

The react native folks use it in the way you suggest, with success

pesterhazy18:12:37

It was already suggested

micha18:12:28

it's a whole thing

micha18:12:44

i just want a minimal unix tool

pesterhazy18:12:51

I guess it does add a pretty big Dependency

micha18:12:41

whenever i use any facebook stuff i end up ditching it for something simple

vikeri18:12:48

A question, when developing a boot library I would like the updates to the library to get reflected without having to restart the boot process in the example project. If I start my usual boot chain of commands in the example project with boot -c boot-react-native/boot-react-native it understands when changes happens to the library and reruns the task in the example project. The invoked task itself does not seem to be updated with the new code though.

pesterhazy18:12:53

@vikeri I thought of that, I suppose with simpler libraries you could add the other project's src dir to your src paths?

micha18:12:17

@vikeri the new code should be available to the task, maybe you need to rerun some code in there

micha18:12:29

like it's holding references to objects created earlier

micha18:12:41

like require :reload needs to be done in the task

pesterhazy18:12:59

does boot re-load jars if they've changed?

micha18:12:18

checkout deps are exploded into a directory on the classpath

micha18:12:32

but jars are immutable in the classloader

micha18:12:40

so you can't reload them on the jvm as far as i know

micha18:12:04

you can do shenanigas with making new classloaders and whatnot

micha18:12:34

the thing you're reloading is the task itself?

vikeri18:12:56

Yeah, I wan’t to update the task here: https://github.com/mjmeintjes/boot-react-native/blob/master/src/mattsum/boot_react_native.clj#L235 and see the changed debug output in the next build in the example project.

vikeri18:12:32

So is that even possible? Or do I have to restart the JVM?

micha18:12:22

you don't need to restart the jvm, but you do need to restart the boot pipeline

micha18:12:28

you can do this in the repl for instance

micha18:12:40

because the task returns an anonymous function (the middleware)

micha18:12:07

so updating the function that returns the middleware (the task function) doesn't automatically create a new middleware and inject it into the pipeline

vikeri18:12:44

I see, so how do I restart the boot pipeline in the repl #noob

micha18:12:09

you can do like this

micha18:12:26

boot.user=> (boot (foo) (bar) (baz))

micha18:12:28

in the repl

micha18:12:36

then ctrl-c will kill that pipeline

micha18:12:39

and you can do it again

micha18:12:42

or you can do this

micha18:12:02

boot.user=> (def p (future (boot (foo) (bar) (baz))))

micha18:12:16

and @p to bring it back to the foreground

micha18:12:23

and then ctrl-c to kill it

vikeri18:12:34

Ah, I see, so running it in the repl instead of the terminal

micha18:12:46

yeah that will give you a persistent jvm session

vikeri18:12:30

Great, usually I just do boot repl -c

vikeri18:12:59

Hmm, something is wrong with boot-react-native though, getting the error java.net.BindException: Address already in use when I ctrl-c then rerun the command.

hlship19:12:05

Does anyone else do this?

(defn b
  "A convienence; accepts any number of values, converts them to strings, splits each
  atring at whitespace, and passes the results to `boot`.  This is usually easier and better
  than making each argument an individual string."
  [& args]
  (apply boot (->> args
                   (map str)
                   (map str/trim)
                   (mapcat #(str/split % #”\s+”)))))

hlship19:12:16

I’m building tools where I want to stay in the REPL due to startup time, so I want it easy to run commands.

hlship19:12:13

e.g., (b “read-config -e production hosts -m phone write”) rather than (boot “read-config” “-e” “production” “hosts” “-m” “phone” “write”).

vikeri19:12:36

Can I reload the build.boot file while I’m in a boot repl? E.g. adding a new task

micha19:12:56

@vikeri (load-file "build.boot")

micha19:12:18

clojure.core/load-file

vikeri19:12:53

Of course… Should’ve tried that.

pesterhazy19:12:23

You can also connect through nrepl and reeval the buffer

vikeri19:12:06

Now I got the (boot (dev)) from the repl working, but it still doesn’t seem to reload the external library even though I use boot -c boot-react-native/boot-react-native:0.3-rc3 repl to start the repl.

micha20:12:07

didi you do (require 'namespace.containing.dev :reload-all)?

juhoteperi20:12:29

@micha How do cleanup-fns work with several Boot "pipelines"? I tried adding cleanup-hook to repl task, and it seems that when I close task started in boot repl, the cleanup hook is called even for the main repl

micha20:12:06

oh interesting

micha20:12:20

yeah the repl is a special case

juhoteperi20:12:13

I think it would generally make sense that running (boot ...) inside Boot would create "new cleanup-fns scope"

juhoteperi20:12:28

I guess currently cleanup-fns is global

micha20:12:08

yeah that's the problem

micha20:12:16

we need cleanup scopes

micha20:12:56

should be a safe modification though

juhoteperi21:12:00

The current implementation can cause problems with Boot-reload and other tasks using cleanup hook: boot dev and running another reload task from the repl will close one from dev task

micha21:12:01

none of that is directly used

micha21:12:17

there is also another issue with the cleanup things

micha21:12:23

when a thread is killed before it can run

micha21:12:42

like when you do ctrl-c in the repl

micha21:12:51

it can kill threads before the cleanup can run

micha21:12:53

@juhoteperi perhaps the boot function should establish a dynamic scope for cleanup handlers

micha21:12:07

each boot invocation establishes its own cleanup scope

micha21:12:32

i guess that assumes that they terminate from the inside out

juhoteperi21:12:40

@micha Yeah, that sounds good

micha21:12:56

you could have futures though

micha21:12:14

maybe that's the user's responsibiltu

micha21:12:49

if you make a future or run boot in another thread you can always add a cleanup handler to terminate it when your cleanup happens?

richiardiandrea21:12:23

I am very close to a first stable version of boot-reload + figwheel !

mynomoto21:12:17

@richiardiandrea how is that going to work? In my experience boot-reload is more reliable than figwheel on lein. Does the reload part stays the same?

richiardiandrea22:12:46

The client and some internals of boot-reload and boot-cljs (minor) will change

richiardiandrea22:12:47

The reload code will stay the same and everything will still be dumped on the fileset only (no external mutable dirs)

mynomoto22:12:09

@richiardiandrea pretty cool! Looking forward to it boot-clj