Fork me on GitHub
#boot
<
2017-01-06
>
gtrak03:01:11

Is there a way to disable inotify watches in general? would prefer a system property?

gtrak03:01:48

I've hit a limit on circleCI+boot with a relatively small clojure project

alandipert03:01:01

so, there isn't, that i'm aware of

alandipert03:01:20

but it occured to me it might be possible to no-op the inotify calls with LD_PRELOAD

gtrak03:01:29

I was wondering about something like that!

gtrak03:01:48

was very hard to google for

gtrak03:01:13

grepping through the boot code leads me to believe disabling watches there would be invasive

alandipert03:01:20

oh boy, it appears to work

alandipert03:01:24

~/g/a/lbbasic➜ LD_PRELOAD=/home/alan/Desktop/libwrapinotify.so INOTIFY_DEBUG=true boot dev
DEBUG:inotify_init:157: inotify enabled
DEBUG:inotify_add_watch:165: inotify enabled
DEBUG:inotify_add_watch:165: inotify enabled
DEBUG:inotify_add_watch:165: inotify enabled
DEBUG:inotify_add_watch:165: inotify enabled
DEBUG:inotify_rm_watch:174: inotify enabled
DEBUG:inotify_rm_watch:174: inotify enabled
DEBUG:inotify_rm_watch:174: inotify enabled
DEBUG:inotify_rm_watch:174: inotify enabled
DEBUG:inotify_init:157: inotify enabled
DEBUG:inotify_add_watch:165: inotify enabled
DEBUG:inotify_add_watch:165: inotify enabled
DEBUG:inotify_add_watch:165: inotify enabled
DEBUG:inotify_add_watch:165: inotify enabled
DEBUG:inotify_add_watch:165: inotify enabled

gtrak03:01:32

how'd you compile it?

alandipert03:01:25

gcc -c -fPIC wrap_inotify.c -o wrap_inotify.o
gcc wrap_inotify.o -shared -o libwrapinotify.so

alandipert03:01:41

note that it proxies the calls by default

alandipert03:01:52

so you wanna comment out the real_inotify_* calls to do the thing

gtrak03:01:17

there's that INOTIFY_DISABLE env var

alandipert03:01:31

oh sweet, yes

alandipert03:01:23

altho - when you do that, boot fails

gtrak03:01:37

right, was just thinking that it would just cause the same failure mode

alandipert03:01:37

since it simulates running out of watches

gtrak03:01:53

so I need to modify it to noop

gtrak04:01:33

so I managed to get boot to 'build' by returning an auto-incrementing counter! It spins CPU for a while, then core dumps. unfortunately, I think I have to do something fancier to get stdout/err to work again. If I just return a constant every time, it hangs.

alandipert04:01:05

yup - at the same place

alandipert04:01:28

i'm trying hashing *pathname in add_watch, so that i return the same watch id

gtrak04:01:41

ah, brilliant

alandipert04:01:41

thinking the JNI maybe relies on getting the same watch id for the same path

gtrak04:01:24

I'm confused why the debug shows the low-number stdout ints as being watched

gtrak04:01:59

rather, I added more debugging, and I'm seeing that.

sonelliot10:01:19

This is a newbie question: How would I go about writing a task that executes some external program (say cc) on a group of files with a particular extension (say .c) and then transfers the compiled files (say .o) onto the next handler?

micha12:01:37

@sonelliot you can use the show -f task to see the tree of files in the fileset before and after, like this

micha12:01:46

boot show -f demo show -f

micha12:01:05

sometimes helpful for debugging

sonelliot13:01:10

Thanks @micha I'm just looking over your snippet now. Very impressive! 🙂

micha13:01:27

it has some things you don't strictly need, like the fileset diffing

micha13:01:57

if you don't want to do the diffing you can simply empty the temp dir before compiling and recompile everything

micha13:01:10

with the diffing the say i did it it will only compile changed files

micha13:01:30

but it also is not correctly handling removed files, which you may want if you go the diffing route

micha13:01:24

there is a function that can give you the removed files:

(fileset-removed previous-fileset current-fileset)

micha13:01:37

that gives you a fileset containing only the removed files

micha13:01:06

you can then doseq over the .c files from that fileset and delete the .o files from the temp dir

micha13:01:09

if you want to

sonelliot13:01:54

I think the diffing is a nice touch. Given a large enough code base you would probably want to do incremental compilation like make does.

micha13:01:10

and a lot of times you don't really carea bout removed files

micha13:01:36

but it's simple to make it correct

micha13:01:44

you'd just need two doseq loops

micha13:01:01

and two diff sequences

sonelliot13:01:05

So the cc function. Could you pass all of the .c files to it at once? Like (apply sh (concat ["cc"] paths)).

sonelliot13:01:45

Otherwise, I guess cc would need to be invoked for every file at a time.

micha13:01:09

yeah i just made a doseq because it ws the simplest example

micha13:01:55

but you're just programming in clojure now, you can do anything in your task

micha13:01:28

you could even have a dependency graph that figures out which object files need to be recompiled based on changes to header or source files etc

sonelliot13:01:52

I see, so it's better to fully leverage the boot architecture in a task.

micha13:01:14

it's up to you really

micha13:01:21

whatever gets the job done 🙂

micha13:01:51

i generally start with the simplest most naive thing

sonelliot13:01:56

Would the cc function then take the file contents and via stdin? Or can it read the .c file in the temp dir?

micha13:01:01

and then shave it when necessary

micha13:01:19

it can do either of thoe things

micha13:01:49

like gcc generally wants paths to files

micha13:01:16

but you could also use io/copy to stream the contents to stdin of the process

sonelliot13:01:58

Cool. I'll have a play around with it. Thanks again!

micha13:01:05

sure, have fun!

sonelliot13:01:01

Another question. How can I exclude a set of extensions from the initial fileset? For example, I occasionally use Emacs .dir-locals.el to customize my editor settings for certain projects. It's annoying that they end up target.

micha14:01:14

@sonelliot there are a few ways, one of which is to set up a .bootignore file

micha14:01:20

one regex per line

micha14:01:38

or you can use the sift task to remove them

sonelliot14:01:54

Cool. .bootignore does the trick!

pesterhazy14:01:12

didn't know about .bootignore -- that should help with inotify issues on circleci as well I think

sonelliot14:01:50

Also, that boot show -f <task> show -f trick is gold!

sonelliot14:01:18

Does boot support mixed source projects like Leiningen? Something like javac command?

micha14:01:42

sure, there is a built-in javac task

micha14:01:53

boot -h will show all available tasks

micha14:01:19

or (boot (help)) in the repl

sonelliot14:01:55

Excellent. Thanks. 🙂

gtrak15:01:58

would you guys accept a patch to disable registering inotify watches via env key? I'm thinking about just a one-liner if surgery https://github.com/boot-clj/boot/blob/master/boot/worker/src/boot/watcher.clj#L62

alandipert15:01:46

no luck w/ the ld preload?

alandipert15:01:22

and probably, re: patch

alandipert15:01:27

but a release would be a ways out

gtrak15:01:46

LD_PRELOAD is feeling like a timesink, I'm not sure.

alandipert15:01:11

i got close last night, i'll try a little longer

alandipert15:01:13

felt pretty close

alandipert15:01:20

in the meantime a PR would be great

gtrak15:01:27

cool! I'll try down that path.

alandipert15:01:29

we are gonna try to do a release every 4-6 mos

alandipert15:01:17

maybe we could do another small release for this, we can discuss on pr

alandipert15:01:58

hehe i think i got it computing

martinklepsch16:01:12

@alandipert seeing stubwatch, why again can’t we stop using boot from adding watches willy nilly? 😄

alandipert16:01:48

because... you haven't made a PR? 💖

alandipert16:01:15

in srsns if gtrak's change actually works i feel good about a little release

alandipert16:01:27

i poked around and it doesn't seem like it would break anything

alandipert16:01:14

what little hesitation i have about making changes like this comes from the fact that: why should we change, circleci is the one that sucks

martinklepsch16:01:15

@alandipert 😄 ah, I somehow thought there would be a fundamental issue somewhere

alandipert16:01:23

obviously the more constrained your environment is, the less boot will work

alandipert16:01:53

but it seems like this problem is common enough, and probably easy enough to address, that adding a flag to turn of watches is a good move

martinklepsch16:01:56

@alandipert that’s cool, never ran into the issue myself just found your stubwatch workaround interesting and somehow thought it’s the last option 😄

alandipert16:01:22

it seems crazy, but i think it's crazier that circleci runs builds with so few resources and in such an apparently weird environment

alandipert16:01:28

no way i would use them

martinklepsch16:01:49

@alandipert re release: I made another PR https://github.com/boot-clj/boot/pull/561 and https://github.com/boot-clj/boot/pull/551 should probably also get merged before release

alandipert17:01:13

an interesting thing about dosh

alandipert17:01:23

it makes me wonder if functions that take arguments lists for other functions or commands, as arguments

alandipert17:01:32

should receive them as a seq instead of varargs

alandipert17:01:41

if dosh worked that way, we could have just added a timeout-ms argument to dosh

martinklepsch17:01:48

@alandipert right the varargs stuff made this a bit more convoluted that in had to be

martinklepsch17:01:39

Sorry, forgot about changelog 🙂

alandipert17:01:56

np, it's good

alandipert17:01:59

forces me to understand lol

martinklepsch17:01:27

@geoffs if you’re up for it you could rebase your notify fixes on master now and use dosh-timed

martinklepsch17:01:38

maybe we can convince Alan to merge that as well 🙂

alandipert17:01:56

as long as you guys are sure it really works on mac now 😉

martinklepsch17:01:37

I’ll give it test-run once the PR is updated

martinklepsch17:01:44

but I’m optimistic 😄

borkdude17:01:21

I have two tasks that are very similar and I want to re-use the option list:

(def opts [c foo bool “some docstring”])
(deftask task1 “docstring” ~opts …)
(deftask task2 “docstring” ~opts …)
Is this possible somehow?

borkdude17:01:48

It seems I wanted to do something similar like in defproject, but this is custom behavior: https://crossclj.info/fun/leiningen.core.project/unquote-project.html

alandipert17:01:15

(defn opts [] '[c foo bool "some docstring"])
(deftask task1 "docstring" #=(opts) …)

alandipert17:01:21

since macros compose outside-in, vs inside-out like functions, you have to use either read-time eval or wrap them with another macro, to syntactically extend them

alandipert17:01:50

#= is doing it with read-time eval, splicing in the form you want at read time, before macro expansion time

borkdude17:01:44

I want to use read-time eval then, I get unmatched delimiter though (the first defn was changed to def)

alandipert17:01:01

sorry, had a typo - forgot arg list for opts

alandipert17:01:41

boot.user=> (defn args [] '[x y])
#'boot.user/args
boot.user=> (defn add2 #=(args) (+ x y))
#'boot.user/add2
boot.user=> (add2 1 2)
3
complete example

borkdude17:01:10

Ah, why does it have to be a function though?

alandipert17:01:02

i think #= only takes op forms like (f ...), and it doesn't evaluate the arguments

alandipert17:01:34

not sure the exact limitations, just know it does funcalls OK... so the last time i used it, i made a function to return what i wanted

borkdude17:01:26

thanks, it works 🙂

geoffs17:01:50

@martinklepsch @alandipert I will rebase and update my PR to use dosh-timed. And I will definitely try it out on both Mac and Linux platforms to make sure it works 😄

geoffs17:01:01

also, once I figure out the details for how to invoke a random exe on Windows, I'll add support for Windows visual notifications!

alandipert17:01:16

cool stuff, thanks!

gtrak18:01:29

@alandipert I'll check it out!

gtrak19:01:46

seems like stubwatch works locally, but it doesn't reliably fail on circle without it either. I'll try using it and see if anything happens.

alandipert19:01:24

i chatted w/ @micha a little about adding an env var or option, and consensus is env var and/or long-only option to boot

alandipert19:01:46

so if you wanna PR that at your leisure, i'm down to try it out

gtrak19:01:44

great! Yea, I think that's a bit cleaner. Circle has declined to resolve this on a rails-related issue, so I imagine there's some infra implications.

gtrak19:01:08

what's 'long-only' ? you mean log?

alandipert19:01:18

na like, --long-option vs. -l

alandipert19:01:29

the short ones are precious

alandipert19:01:36

seems best to keep them open for more common things

gtrak20:01:43

re --no-inotify in general, it seems like cli flags hit boot.util dynamic vars, eg boot.util/*colorize*, and env/boot.properties are looked up by boot.App/config? Do those streams ever cross?

gtrak20:01:41

ah, answered my own question, disregard, colorize?-system-default looks it up via boot.App/config, I'll follow that pattern.

gtrak21:01:20

glad we went through the LD_PRELOAD experiment because it makes it easy to see if the boot-level stuff is working.

alandipert22:01:45

@gtrak looks pretty sweet! i can take a deeper look next week

alandipert22:01:50

you don't have a mac to test on by any chance do you?

gtrak22:01:24

i could dig one up, nothing looks mac-specific

alandipert22:01:46

agree, im just always more comfortable if we can get mac and windows ppl to at least try it

gtrak22:01:11

is putting 'inotify' in the property/flag too concrete? it's called something else on windows/mac

alandipert22:01:32

good point. yeah, probly

gtrak22:01:10

BOOT_WATCHERS_DISABLE and --no-watchers?

alandipert22:01:14

perhaps --disable-watchers?

alandipert22:01:19

so as to line them up kinda