Fork me on GitHub
#boot
<
2016-09-05
>
dominicm13:09:15

Is there any reason to use (get-env :directories) instead of (get-env :source-paths) when setting the refresh dirs?

onetom13:09:42

@dominicm you might have some .edn files in assets for example?

martinklepsch13:09:59

@dominicm directories is also tmp dirs and not what you provide in source/resource etc

dominicm13:09:40

Sorry. When I say refresh directories, I mean for REPL reloading.

martinklepsch13:09:27

source-paths is mirrored in a dir in (get-env :directories)

martinklepsch13:09:44

hypothetically you could generate clojure files as part of your build — they would not show up in source-paths

dominicm13:09:42

@martinklepsch Hmm, that does pose some potential problem.

dominicm13:09:26

Although, other than cljx (which is deprecated), I doubt that's a more common practice than using cljc files, which is the bug I'm trying to fix.

dominicm14:09:21

Although, I've just seen https://github.com/Deraen/boot-ctn/blob/master/src/deraen/boot_ctn.clj has a remove-fn, which could be used to remove problematic paths.

dominicm14:09:45

But it seems that a whitelist is better than a blacklist.

juhoteperi14:09:22

I've been thinking about this for some days

juhoteperi14:09:28

And there doesn't seem to be easy solution

juhoteperi14:09:05

Boot-cljs will write all input files to fileset and they will be part of classpath, tools.namespace will reload those

juhoteperi14:09:20

This will sometimes cause problems if those cljc namespaces define protocols

dominicm14:09:15

@juhoteperi Out of interest, did you run into something similar to https://github.com/stuartsierra/component/issues/45

dominicm14:09:37

Yeah, looks like it. It's highlighted itself very brightly when you use bidi with cljs and clj.

juhoteperi14:09:22

Using source-paths (the dirs in working dir) is (or at least was) very problematic because then tools.namespace file tracker would be looking at different files than those that are available in classpath (temp dirs managed by boot)

juhoteperi14:09:00

That is how Boot works with c.t.n 0.2 and without boot-ctn (or manual set-refresh-dirs)

juhoteperi14:09:26

At list that's why I think boot-ctn exists... 😄

juhoteperi14:09:51

c.t.n 0.3 uses clojure.java.classpath and automatically uses the temp dirs

juhoteperi14:09:29

And re: why it is very problematic, if file watcher is looking and working dir it might decide to reload some namespace but the changes might not be yet reflected on the temp dirs

juhoteperi14:09:50

c.t.n reloads ns -> old ns is reloaded -> when new file is in temp dir, it is not reloaded again

dominicm14:09:12

Ah, I see. So really it would be better to only use the non-temp directories?

juhoteperi14:09:29

No, just the opposite! 😄

juhoteperi14:09:52

If you use non-temp dirs (working directory) that is going to happen

dominicm14:09:16

Oh, for 0.3, gotcha. Not 0.2.

juhoteperi14:09:32

But when c.t.n looks at the temp dirs, it will try to reload namespaces only when the files are really available in classpath

dominicm14:09:34

Yeah, makes sense.

dominicm14:09:23

How do you remove the cljs/cljc files from the classpath then, by blacklisting the folder?

juhoteperi14:09:47

Won't be very easy to blacklist the folder as it is created by Boot-cljs

dominicm14:09:15

Ah, I thought you'd already solved this problem on a project.

juhoteperi14:09:51

Nope, I'm not currently using problematic libraries with Boot projects

juhoteperi14:09:12

I fixed one Lein project where it was easy to just blacklist the cljs output dir

dominicm14:09:27

You could "catch" the files that were added by the cljs task (by comparing the fileset?) and remove those I guess. The temp dirs only is a problem if you're using 0.3 of c.t.n (which is alpha)

martinklepsch14:09:53

I'm wondering, is it important that the output directory is on the classpath?

dominicm14:09:27

@juhoteperi If you use (set-refresh-dirs "src" "dev" "test") on 0.3 of c.t.n, then it would only read the input files in the first place? It doesn't matter if it's copied or not?

dominicm14:09:38

@martinklepsch Yeah, to serve the assets

juhoteperi14:09:36

@dominicm C.t.n doesn't reload files from refresh-dirs, those dirs are only to check which namespaces are reloaded using (require ... :reload) which will check the classpath

martinklepsch14:09:18

@dominicm ah right

dominicm14:09:44

@juhoteperi Oh! I understand now. That's a pain.

dominicm14:09:29

There's a private function in boot.core that might be useful.

juhoteperi14:09:44

Also the temp dir watched by c.t.n is not the same temp dir as is created by Boot-cljs

juhoteperi14:09:13

Boot manages a few directories with different roles, one for resource files in fileset IIRC, and files from Boot-cljs temp dir are synced into that dir

juhoteperi14:09:29

It also contains files from other tasks in pipeline, like CSS from Less

dominicm14:09:34

@juhoteperi So if c.t.n watched the temp dir that boot created for source files (not resource files), that might work?

juhoteperi14:09:06

It needs to watch both as some clj code might (and probably will) be on resource paths

juhoteperi14:09:18

But! Watching user source + resource dirs could work!

juhoteperi14:09:22

Looks like user dirs are those from the project dir, i.e. not created by tasks

juhoteperi14:09:27

like boot-cljs

dominicm14:09:45

Yeah, they are.

juhoteperi14:09:40

boot.user=> (apply clojure.tools.namespace.repl/set-refresh-dirs (map (memfn getPath) (concat (#'boot.core/user-source-dirs) (#'boot.core/user-asset-dirs)))) ("/home/juho/.boot/cache/tmp/home/juho/Source/saapas/lxf/aixfad" "/home/juho/.boot/cache/tmp/home/juho/Source/saapas/lxf/coz8kz")

juhoteperi14:09:45

this could work as a workaround

dominicm14:09:11

Worked for me without the getPath, but that might mean mine doesn't work at all.

dominicm14:09:21

But my reloads are working

dominicm14:09:31

Does require happen from the temp paths? My testing seems to indicate it doesn't.

juhoteperi14:09:26

Hmm, what temp paths did you try? These from user-* functions?

dominicm14:09:01

(apply clojure.tools.namespace.repl/set-refresh-dirs (#'boot.core/user-source-dirs)) worked for me, obviously needs asset dirs attaching

juhoteperi14:09:16

(System/getProperty "fake.class.path") should show the files in classpath

dominicm14:09:49

/home/dominic/src/juxt/edge/src is at the front.

dominicm14:09:15

So maybe the copying isn't a problem? If that's consistent & searches are sequential.

dominicm15:09:49

@juhoteperi Not entirely sure how I'd test whether or not we need to do this. Do you know?

juhoteperi15:09:28

Nope. Maybe it works nowadays.

dominicm15:09:22

¯\_(ツ)_/¯

dominicm15:09:48

I'll try and test it out. It'll make it into edge if it works 🙂

borkdude15:09:17

if I want to use parallel builds in clojurescript, do I write (cljs :compiler-options {:parallel-build true})?

dominicm15:09:11

@juhoteperi I've never seen code in the asset-path, when does that happen?

borkdude15:09:46

Why is :optimizations not under :compiler-options is what made my ask that question. Convenience?

juhoteperi15:09:12

@borkdude Yes, top-level optimizations is for convenience

juhoteperi15:09:29

top-level :source-map option has also some additional logic to properly set source-map filepath when using other optimizations settings than :none

juhoteperi15:09:34

Ah nvm, that's not right, the logic is used even if :source-map is only set under :compiler-options

micha16:09:24

hi @dominicm @juhoteperi , catching up on the ctn conversation

micha16:09:42

anything i can help with?

dominicm16:09:30

@micha You might have an interesting input, I'll try give you a summary: Building cljs, copies .cljc files onto the classpath as part of the build process. These cljc files then get picked up by c.t.n, and refreshed. So you end up refreshing things like bidi. This causes protocols to get wiped, which causes errors.

dominicm16:09:18

My solution was to specify only to refresh src, dev and test (traditional source-paths, excluding resource-paths), but @juhoteperi mentioned it might conflict with temp paths a little.

micha16:09:25

ah, in the :output-dir?

micha16:09:35

the cljc files that are problematic?

juhoteperi16:09:45

Yes, c.t.n doesn't look at cljs files

micha16:09:14

right but i mean the problem is not the cljc files themselves, but the copies that are in the output dir for source maps, right?

dominicm16:09:59

I'm not sure what the clojurescript compiler does with them (they might need to be on the filesystem for google closure for example -- probably not)

micha16:09:45

if you have a cljc file that is compiled by cljs compiler, you will have a copy of that cljc file that is present in the "output dir" that will be in your web root

micha16:09:50

this is for source maps

dominicm16:09:01

Ah, interesting.

micha16:09:12

so then you have like foo/core.cljc whch is the real source file

micha16:09:23

and also main.out/foo/core.cljc which is for source maps

juhoteperi16:09:39

Hmm, interesting, I wonder why c.t.n doesn't ignore files in "wrong" paths

juhoteperi16:09:59

I guess it will just take any changed files and read ns from those

micha16:09:59

we could adjust boot-cljs to use add-asset for this

micha16:09:06

instead of add-resource

micha16:09:26

there isn't really a good use case for subsequent tasks to have any of that on the classpath i guess

juhoteperi16:09:38

Hmmm. Asset files are not available on classpath and thus it won't work with ring resource handlers?

dominicm16:09:41

Would that break the ability to serve the cljs files from the resource path (e.g. an uberjar)

juhoteperi16:09:57

Asset files are added to uberjar

juhoteperi16:09:13

One problem is that Ring resources handler doesn't know about our file roles

micha16:09:22

yeah and those files aren't used for computing, they're just assets used to link to in the browser

micha16:09:25

for source maps

micha16:09:34

nothing will use them to actually do any computing

dominicm16:09:44

Ah, so maybe it's okay then.

micha16:09:14

@juhoteperi yes this is why i like to use boot-jetty, because it does what the uberjar would do

micha16:09:24

it only looks at files with the output role

micha16:09:46

it makes a pod with the stuff that will be on the uberjar classpath for its classpath, i should say

micha16:09:12

so the resource middleware works the same in dev with boot-jetty as it works in the production uberjar

micha16:09:52

also boot-static, which simplifies serving static resources

micha16:09:04

i've been investigating the gradle daemon

micha16:09:07

looks pretty good

juhoteperi16:09:28

Boot-jetty runs the app web handler logic inside a pod? So REPL won't work?

micha16:09:53

yeah, it makes a new pod each iteration

micha16:09:30

no reason why it can't start a repl server in there though

micha16:09:37

like a repl server

juhoteperi16:09:43

Sounds very slow

micha16:09:06

yeah i guess it could be

micha16:09:10

i don't use it that way though

micha16:09:37

i generally use the castra rpc stuff on the backend

micha16:09:45

so i don't need jetty to develop it

micha16:09:53

they're just functions

micha16:09:08

i imagine that's what i'd do with a regular ring handler too

micha16:09:54

so the repl i use is not in the actual jetty context

micha16:09:03

it's in the regular boot context

micha16:09:03

it's not really necessary to make a new pod each iteration either

micha16:09:43

the reason why it does was to make actual jetty apps more correct in dev

micha16:09:03

like if it works with boot-jetty it will definitely work in tomcat

micha16:09:18

that means eliminating dangling state

micha16:09:46

but i can see the use case for a persistent server environment too

dominicm16:09:09

So using the asset path won't work for most people then.

dominicm17:09:34

If the require will always run from the local directories first, I see no problem with a whitelist still.

flyboarder19:09:43

Does boot have a way to suppress output?

micha19:09:02

there is a -q option, like boot -q ...

micha19:09:11

also the watch task has its own -q option

flyboarder19:09:45

hmm can I use that within a task to suppress the tasks im using?

flyboarder19:09:57

like a global var?

flyboarder19:09:05

id like to do something like that

micha19:09:52

it's a global setting

micha19:09:55

corresponds to like (reset! boot.util/*verbosity* 0)

micha19:09:54

dealing wtih IO like that is a pain though, because any thread can write to stdout

micha20:09:20

so the best thing is to make it quiet as early as possible

micha20:09:33

so nothing is created with access to stdout/stderr

micha20:09:47

like you call some java library that does printf

flyboarder20:09:55

so if im just calling tasks that use util/info could I not swap the verbosity level between tasks?

micha20:09:18

for that case i believe you could, yeah

flyboarder20:09:26

cool i will try

micha20:09:34

actually maybe not

micha20:09:39

one sec let's see

flyboarder20:09:58

print* seems to check verbosity each time

micha20:09:08

yeah but it's running in different pods

micha20:09:18

so you'd have to set verbosity in each pod

micha20:09:31

they all have their own instances of the boot.util namespace

flyboarder20:09:54

does each task get it’s own pod?

micha20:09:05

they make pods

micha20:09:15

and call boot.util/info etc in the pod sometimes

micha20:09:12

there might be something we can do

micha20:09:04

so we could make all the info/dbug/fail stuff happen in the worker pod

micha20:09:16

it would be a pretty simple change

micha20:09:00

just wrap all the print* exprs in (with-eval-worker (print* ...))

flyboarder20:09:40

so how do I go setting things in a pod?

flyboarder20:09:53

right now im just comping tasks

micha20:09:11

what are you wanting to do actually?

flyboarder20:09:20

basically my snippet above

flyboarder20:09:30

one sec ill post

micha20:09:49

i mean like what is the high level goal?

micha20:09:01

what's the issue with the output

flyboarder20:09:41

So i’d like to provide my own output

micha20:09:04

ok yeah so the way to do that would be to do like the watch task does

flyboarder20:09:11

it works really well just ugly as heck in console, really underlying task errors dont help much i’d like to provide it in my wrapper task

micha20:09:43

binding *out* and *err* to a string writer

micha20:09:30

your case is a little more complicated because you want to re-enable printing after your task runs

micha20:09:52

so what you'd do is save *out* and *err* i guess

micha20:09:06

and have one task that sets them to the string writer before calling the next task

micha20:09:12

you'd put that in the front of the comp

micha20:09:30

then another task that sets them back to the saved *out* and *err*

micha20:09:34

and calls the next task

micha20:09:43

you'd put that one at the end of the comp

flyboarder20:09:41

interesting ill give that a go

micha20:09:09

(let [*out*' *out* *err*' *err*]
  (comp
    (eat-output)
    (semgit/...)
    ...
    (restore-output *out*' *err*')))

micha20:09:00

if restore-output needs to have access to the output that was eaten you could have the StringWriters in lexical scope too and pass them in to both

flyboarder20:09:59

awesome thanks!

micha20:09:05

i guess really you want one task, redirect-output

flyboarder20:09:47

yeah i can make a task that has a pre post tho right

micha20:09:48

(let [out1 *out*
      err1 *err*
      out2 (new StringWriter)
      err2 (new StringWriter)]
  (comp
    (redirect-output out2 err2)
    (semgit/...)
    (semgit/...)
    (redirect-output out1 err1)))

micha20:09:30

i guess once you have that task you can make a little macro to wrap it all up

micha20:09:23

(with-quiet
  (comp
    (semgit/...)
    ...

flyboarder20:09:22

@micha like this? not sure if this would work with the macro you were thinking

flyboarder21:09:14

^note above does not work… yet

flyboarder21:09:30

I updated the snippet with a working silencer task.

flyboarder21:09:37

I dont yet know how to re-enable output other than a task which does the opposite.