Is there a way to disable inotify watches in general? would prefer a system property?

I've hit a limit on circleCI+boot with a relatively small clojure project

so, there isn't, that i'm aware of

but it occured to me it might be possible to no-op the inotify calls with LD_PRELOAD

and i found https://readmes.numm.org/init/upstart/init/tests/wrap_inotify.c

I was wondering about something like that!

was very hard to google for

grepping through the boot code leads me to believe disabling watches there would be invasive

oh boy, it appears to work

~/g/a/lbbasic➜ LD_PRELOAD=/home/alan/Desktop/libwrapinotify.so INOTIFY_DEBUG=true boot dev
DEBUG:inotify_init:157: inotify enabled
DEBUG:inotify_add_watch:165: inotify enabled
DEBUG:inotify_add_watch:165: inotify enabled
DEBUG:inotify_add_watch:165: inotify enabled
DEBUG:inotify_add_watch:165: inotify enabled
DEBUG:inotify_rm_watch:174: inotify enabled
DEBUG:inotify_rm_watch:174: inotify enabled
DEBUG:inotify_rm_watch:174: inotify enabled
DEBUG:inotify_rm_watch:174: inotify enabled
DEBUG:inotify_init:157: inotify enabled
DEBUG:inotify_add_watch:165: inotify enabled
DEBUG:inotify_add_watch:165: inotify enabled
DEBUG:inotify_add_watch:165: inotify enabled
DEBUG:inotify_add_watch:165: inotify enabled
DEBUG:inotify_add_watch:165: inotify enabled

how'd you compile it?

I'm trying to figure it out from https://code.launchpad.net/~jamesodhunt/upstart/bug-1157713-inotify-test-failures/+merge/159334

gcc -c -fPIC wrap_inotify.c -o wrap_inotify.o
gcc wrap_inotify.o -shared -o libwrapinotify.so

WORD

note that it proxies the calls by default

so you wanna comment out the real_inotify_* calls to do the thing

there's that INOTIFY_DISABLE env var

oh sweet, yes

altho - when you do that, boot fails

right, was just thinking that it would just cause the same failure mode

since it simulates running out of watches

so I need to modify it to noop

so I managed to get boot to 'build' by returning an auto-incrementing counter! It spins CPU for a while, then core dumps. unfortunately, I think I have to do something fancier to get stdout/err to work again. If I just return a constant every time, it hangs.

yup - at the same place

i'm trying hashing *pathname in add_watch, so that i return the same watch id

ah, brilliant

thinking the JNI maybe relies on getting the same watch id for the same path

I'm confused why the debug shows the low-number stdout ints as being watched

rather, I added more debugging, and I'm seeing that.

This is a newbie question: How would I go about writing a task that executes some external program (say cc) on a group of files with a particular extension (say .c) and then transfers the compiled files (say .o) onto the next handler?

@sonelliot https://gist.github.com/micha/f8e15235a2e447259b3384eeb4b37427

@sonelliot you can use the show -f task to see the tree of files in the fileset before and after, like this

boot show -f demo show -f

sometimes helpful for debugging

Thanks @micha I'm just looking over your snippet now. Very impressive! 🙂

it has some things you don't strictly need, like the fileset diffing

if you don't want to do the diffing you can simply empty the temp dir before compiling and recompile everything

with the diffing the say i did it it will only compile changed files

but it also is not correctly handling removed files, which you may want if you go the diffing route

there is a function that can give you the removed files:

(fileset-removed previous-fileset current-fileset)

that gives you a fileset containing only the removed files

you can then doseq over the .c files from that fileset and delete the .o files from the temp dir

if you want to

I think the diffing is a nice touch. Given a large enough code base you would probably want to do incremental compilation like make does.

right

and a lot of times you don't really carea bout removed files

but it's simple to make it correct

you'd just need two doseq loops

and two diff sequences

So the cc function. Could you pass all of the .c files to it at once? Like (apply sh (concat ["cc"] paths)).

sure

Otherwise, I guess cc would need to be invoked for every file at a time.

yeah i just made a doseq because it ws the simplest example

but you're just programming in clojure now, you can do anything in your task

you could even have a dependency graph that figures out which object files need to be recompiled based on changes to header or source files etc

I see, so it's better to fully leverage the boot architecture in a task.

it's up to you really

whatever gets the job done 🙂

i generally start with the simplest most naive thing

Would the cc function then take the file contents and via stdin? Or can it read the .c file in the temp dir?

and then shave it when necessary

it can do either of thoe things

*those

like gcc generally wants paths to files

but you could also use io/copy to stream the contents to stdin of the process

Cool. I'll have a play around with it. Thanks again!

sure, have fun!

Another question. How can I exclude a set of extensions from the initial fileset? For example, I occasionally use Emacs .dir-locals.el to customize my editor settings for certain projects. It's annoying that they end up target.

@sonelliot there are a few ways, one of which is to set up a .bootignore file

one regex per line

or you can use the sift task to remove them

Cool. .bootignore does the trick!

didn't know about .bootignore -- that should help with inotify issues on circleci as well I think

Also, that boot show -f <task> show -f trick is gold!

👍

Does boot support mixed source projects like Leiningen? Something like javac command?

sure, there is a built-in javac task

boot -h will show all available tasks

or (boot (help)) in the repl

Excellent. Thanks. 🙂

would you guys accept a patch to disable registering inotify watches via env key? I'm thinking about just a one-liner if surgery https://github.com/boot-clj/boot/blob/master/boot/worker/src/boot/watcher.clj#L62

no luck w/ the ld preload?

and probably, re: patch

but a release would be a ways out

LD_PRELOAD is feeling like a timesink, I'm not sure.

i got close last night, i'll try a little longer

felt pretty close

in the meantime a PR would be great

cool! I'll try down that path.

we are gonna try to do a release every 4-6 mos

maybe we could do another small release for this, we can discuss on pr

hehe i think i got it computing

i poosh

@gtrak can you give this a whirl? https://github.com/alandipert/stubwatch

@alandipert seeing stubwatch, why again can’t we stop using boot from adding watches willy nilly? 😄

because... you haven't made a PR? 💖

in srsns if gtrak's change actually works i feel good about a little release

2.7.2

i poked around and it doesn't seem like it would break anything

what little hesitation i have about making changes like this comes from the fact that: why should we change, circleci is the one that sucks

@alandipert 😄 ah, I somehow thought there would be a fundamental issue somewhere

obviously the more constrained your environment is, the less boot will work

but it seems like this problem is common enough, and probably easy enough to address, that adding a flag to turn of watches is a good move

@alandipert that’s cool, never ran into the issue myself just found your stubwatch workaround interesting and somehow thought it’s the last option 😄

it seems crazy, but i think it's crazier that circleci runs builds with so few resources and in such an apparently weird environment

no way i would use them

@alandipert re release: I made another PR https://github.com/boot-clj/boot/pull/561 and https://github.com/boot-clj/boot/pull/551 should probably also get merged before release

an interesting thing about dosh

it makes me wonder if functions that take arguments lists for other functions or commands, as arguments

should receive them as a seq instead of varargs

if dosh worked that way, we could have just added a timeout-ms argument to dosh

@alandipert right the varargs stuff made this a bit more convoluted that in had to be

Sorry, forgot about changelog 🙂

np, it's good

forces me to understand lol

@geoffs if you’re up for it you could rebase your notify fixes on master now and use dosh-timed

maybe we can convince Alan to merge that as well 🙂

as long as you guys are sure it really works on mac now 😉

I’ll give it test-run once the PR is updated

but I’m optimistic 😄

cool

I have two tasks that are very similar and I want to re-use the option list:

(def opts [c foo bool “some docstring”])
(deftask task1 “docstring” ~opts …)
(deftask task2 “docstring” ~opts …)

It seems I wanted to do something similar like in defproject, but this is custom behavior: https://crossclj.info/fun/leiningen.core.project/unquote-project.html

(defn opts [] '[c foo bool "some docstring"])
(deftask task1 "docstring" #=(opts) …)

since macros compose outside-in, vs inside-out like functions, you have to use either read-time eval or wrap them with another macro, to syntactically extend them

#= is doing it with read-time eval, splicing in the form you want at read time, before macro expansion time

I want to use read-time eval then, I get unmatched delimiter though (the first defn was changed to def)

sorry, had a typo - forgot arg list for opts

boot.user=> (defn args [] '[x y])
#'boot.user/args
boot.user=> (defn add2 #=(args) (+ x y))
#'boot.user/add2
boot.user=> (add2 1 2)
3

Ah, why does it have to be a function though?

i think #= only takes op forms like (f ...), and it doesn't evaluate the arguments

not sure the exact limitations, just know it does funcalls OK... so the last time i used it, i made a function to return what i wanted

cool

thanks, it works 🙂

:thumbsup:

@martinklepsch @alandipert I will rebase and update my PR to use dosh-timed. And I will definitely try it out on both Mac and Linux platforms to make sure it works 😄

also, once I figure out the details for how to invoke a random exe on Windows, I'll add support for Windows visual notifications!

cool stuff, thanks!

@alandipert I'll check it out!

seems like stubwatch works locally, but it doesn't reliably fail on circle without it either. I'll try using it and see if anything happens.

cool stuff

i chatted w/ @micha a little about adding an env var or option, and consensus is env var and/or long-only option to boot

so if you wanna PR that at your leisure, i'm down to try it out

great! Yea, I think that's a bit cleaner. Circle has declined to resolve this on a rails-related issue, so I imagine there's some infra implications.

what's 'long-only' ? you mean log?

na like, --long-option vs. -l

ah

the short ones are precious

seems best to keep them open for more common things

re --no-inotify in general, it seems like cli flags hit boot.util dynamic vars, eg boot.util/*colorize*, and env/boot.properties are looked up by boot.App/config? Do those streams ever cross?

ah, answered my own question, disregard, colorize?-system-default looks it up via boot.App/config, I'll follow that pattern.

glad we went through the LD_PRELOAD experiment because it makes it easy to see if the boot-level stuff is working.

@gtrak looks pretty sweet! i can take a deeper look next week

you don't have a mac to test on by any chance do you?

i could dig one up, nothing looks mac-specific

agree, im just always more comfortable if we can get mac and windows ppl to at least try it

is putting 'inotify' in the property/flag too concrete? it's called something else on windows/mac

good point. yeah, probly

BOOT_WATCHERS_DISABLE and --no-watchers?

perhaps --disable-watchers?

so as to line them up kinda

sure

cool