Fork me on GitHub
#boot
<
2017-04-10
>
anmonteiro04:04:34

is there a way to install all dependencies in a build.boot file but ignore the set-env! call to source paths?

martinklepsch13:04:03

@anmonteiro don’t think that’s possible. Workaround could be storing these things in an edn file.

thheller14:04:10

so I'm trying to work around some boot issues, boot seems to copy a lot of files

thheller14:04:38

is there any documentation on how boot constructs the classpath?

thheller15:04:27

how do I even tell what those directories once were?

thheller15:04:35

any way to stop this behaviour and just give me the original paths?

thheller15:04:47

(those are from the classpath btw)

thheller15:04:38

that doesn't help me. I'm not in a boot task, I'm in clojure code

martinklepsch15:04:07

basically boot creates a bunch of directories with different “roles” https://github.com/boot-clj/boot/wiki/Filesets#fileset-components

martinklepsch15:04:56

@thheller you never explained what you want to do so not sure what specifically to tell you? 😄

thheller15:04:46

ah right hehe. this is for shadow-devtools (which means shadow-build)

thheller15:04:36

got it almost working so far just the paths are causing trouble

thheller15:04:42

I filter all paths names resources but they are not named resources anymore in boot 😛

thheller15:04:41

also the classpath ordering is very different in boot

martinklepsch15:04:38

right because they’re on the classpath and the contents of resources are added to the classpath, as you sure know 🙂

martinklepsch15:04:04

so you want to get all files that originate from your resources folder?

thheller15:04:15

no I do NOT want them, that is the point 🙂

thheller15:04:37

in shadow-build I search the classpath for potential CLJS resources

thheller15:04:04

so if you have compiled CLJS sources in resources they are treated as sources

thheller15:04:24

which is why I filter all paths names resources

martinklepsch15:04:35

In boot adding compiled stuff to “resources” is sort of an anti-pattern

martinklepsch15:04:51

you said you don’t have access to boot.core or you do have access?

thheller15:04:52

in shadow-build as well

thheller15:04:09

I'm just looking at random open source repos that use boot

thheller15:04:24

and try to port them to shadow-devtools for the CLJS work

thheller15:04:37

so I can figure out how the boot part works 😉

micha15:04:41

what do you want to know about how boot constructs the classpath?

thheller15:04:06

@micha in lein the source paths of the user always come first

thheller15:04:29

that does not seem to be true in boot

micha15:04:32

the classpath is a key value store though

micha15:04:03

there isn't really a first there

thheller15:04:09

I'm aware .. and that is the problem.

thheller15:04:28

yes there is a first

thheller15:04:46

say some library has a foo.txt in a jar

thheller15:04:00

(io/resource "foo.txt") gives me that file

thheller15:04:16

support I want to overwrite that, I put a foo.txt into my source path

thheller15:04:23

source loading order now uses mine

thheller15:04:46

that is not true if the classpath has something listed in front of my paths

thheller15:04:26

should not be a problem since you'd rarely want to overwrite files from those jars

thheller15:04:40

still trying to make sense of all this

micha15:04:39

if you add the source paths after you add the jar then they will override the jar's things

thheller15:04:05

I don't add any source path

micha15:04:24

then how would a file overwrite something from the jar?

micha15:04:44

if you don't add the directory containing the file to the classpath?

micha15:04:05

interestingly the directories do override things from jars in either case

thheller15:04:19

don't worry about it .. I'm just used to lein classpaths and noticed that boot looks different. doesn't mean it is wrong, just different

micha15:04:43

i don't think the order even matters

thheller15:04:45

but I suppose there is no way to tell which path something like /Users/zilence/.boot/cache/tmp/Users/zilence/code/boot-test/n7q/mh5540 used to be?

micha15:04:45

no, because any task might be manipulating files there

micha15:04:08

with lein et al you are working with the actual files in the user's project

micha15:04:28

which is a problem because then you need to keep track of which things the user owns and which things the build tool can mess with

micha15:04:54

boot copies the user's files into anonymous temp dirs

thheller15:04:57

well the file the build tool (`shadow-build`) messes with are in target

micha15:04:22

the target dir is in an undefined state basically as far as boot is concerned

micha15:04:27

while boot is running

micha15:04:35

it's write-only access basically

micha15:04:55

it's a place to dump the final state of the build

micha15:04:00

before boot exits

thheller15:04:25

no it is for temporary files 😉

thheller15:04:37

the final output is chosen by the user

micha15:04:38

there are anonymous temp dirs for that

thheller15:04:18

the temp dirs do not survive a jvm reboot?

micha15:04:48

you can save things to a cache but it's rarely useful (cache invalidation is hard)

thheller15:04:02

cache invalidation is solved in shadow-build

micha15:04:06

like for instance boot does cache exploded jars in the uber task

thheller15:04:10

actually pretty easy

micha15:04:16

with the cache key the md5 hash of the jar

micha15:04:35

you can see the uber task for an example of the caching in action

thheller15:04:47

but the target part actually works just fine so that doesn't worry me

thheller15:04:56

boot doesn't seem to touch it and that is fine

thheller15:04:49

it doesn't have to be target, could be any directory really. lein has a target anyways so that is what I use

micha15:04:22

yeah now that the target task is a separate thing it's not as bad

thheller15:04:27

actually except for the resources part I have everything working now

micha15:04:41

what is the resources issue?

thheller15:04:58

I'll just put a warning somewhere if you try to output CLJS into resources

thheller15:04:07

@micha shadow-build/`shadow-devtools` compile CLJS and use the java classpath to locate CLJS resources (so anything compilation relevant)

thheller15:04:27

if it find a .js file with a goog.provide it will treat that as a resource

thheller15:04:24

so if there is a cljs/core.js in resources that will goog.provide cljs.core 😉

thheller15:04:05

the workaround for this is ignoring the resources folder .. which I can't do in boot since the name is different

thheller15:04:08

but if you don't generate into resources everything is fine

micha15:04:29

seems like a warning would be fine

micha15:04:35

just don't do that (tm)

thheller15:04:19

yeah its a bad habit anyways but every other CLJS tutorial seems to put everything into resources 😞

juhoteperi15:04:26

With Boot writing (with target task) to a path that is in resource-paths, would cause infinite loop with watch task

juhoteperi15:04:39

So I don't think it is very common with Boot

thheller15:04:31

that example repo does 😉

thheller15:04:08

well technically it doesn't I guess

thheller15:04:31

dunno where it actually all ends up

juhoteperi15:04:36

Oh right, to classpath, not to a folder in resource-paths

juhoteperi15:04:57

Yes, it is common that Cljs results are added to classpath and it does indeed cause problems..

thheller15:04:58

the index.html includes ./main.js but it is never written there

thheller15:04:01

so no idea 🙂

thheller15:04:14

I like actually looking at generated sources but couldn't find where boots put stuff

juhoteperi15:04:46

~/.boot/cache/tmp/{path-to-project}/{boot-process}/{temp-dir-id}/path

thheller15:04:48

doesn't seem to work without the serve part

micha15:04:27

also the show -f task might be handy for you to see where things are in the fileset

micha15:04:06

example: boot -s src show -f

thheller15:04:04

doubt I can use the filesets, this is all intended to be build-tool agnostic so it works everywhere

thheller15:04:11

works fine now except for the resources part

thheller15:04:05

need to read up on how this works in boot-cljs, that index.html accesses main.js in the same directory confused me

juhoteperi15:04:13

@thheller serve serves files fromm classpath, both index and main.js are in classpath

thheller15:04:16

yeah but how does main.js get there. it isn't there when I stop boot

juhoteperi15:04:49

boot-cljs adds the results to classpath

thheller15:04:44

so the tmp dirs are removed when I quit boot?

thheller15:04:18

or where does boot-cljs put the analyzer cache files?

micha16:04:00

tmp dirs are garbage collected the next time you run boot in that project

micha16:04:09

so you can still access the files after boot exits

micha16:04:13

like to debug things

juhoteperi16:04:22

boot-cljs doesn't currently handle cache between processes

thheller16:04:58

ah so everything is compiled from scratch after restarts?

thheller16:04:22

but that is soo slow?

juhoteperi16:04:00

hasn't been problem for me as I don't restart boot

thheller16:04:31

hehe yeah thats one option

anmonteiro17:04:04

@martinklepsch thanks. looks like I didn’t need it anyway after all

anmonteiro17:04:21

I thought I did but that was just a symptom I was doing something wrong 🙂

juhoteperi17:04:29

@thheller Do you know if there is a good value to use for cache key for analysis cache? Perhaps the Cljs version? Do compiler-options matter, I think Cljs compiler should take those into account itself?

juhoteperi17:04:06

Boot does have cache-dirs, but using them requires a cache key: https://github.com/boot-clj/boot/blob/master/boot/core/src/boot/core.clj#L359

anmonteiro17:04:30

@juhoteperi the analysis cache depends on compiler options I believe

thheller17:04:36

@juhoteperi it is more complicated than that .. in shadow-build I use the last modified timestamp of everything that was required to compile the file

thheller17:04:26

so the timestamp of cljs.core, goog, etc

thheller17:04:40

then also some of the compiler-options that where used as some affect the output

thheller17:04:04

(def cache-affecting-options
  [:static-fns
   :elide-asserts
   :source-map])

thheller17:04:16

for cljs.core there is also :optimize-constants (shadow-build uses something else to automate that)

thheller17:04:52

if you combine all those things you can make a pretty reliable cache key

thheller17:04:13

the problem really is accounting for everything that went into compiling a file

thheller17:04:35

since it must account for macros as well

thheller17:04:38

I currently do not account for any of the new stuff yet, so no externs inference etc

thheller17:04:43

those will affect the cache as well

thheller17:04:35

but at the moment they are always generated so it works

dominicm17:04:28

I've been thinking about cljs caching too. It's a hard problem to solve. Lein doesn't get it right. But it's a complaint I hear a lot, I'd love to have cljs caching on boot

dominicm17:04:39

Oh really? Then it may make it's way into edge

thheller17:04:04

yeah .. only worked on that today

thheller17:04:13

needed to figure out boot first 😛

dominicm17:04:35

Ah, I'd be happier if it used the file set api

thheller17:04:36

but it is just clojure code, so nothing ties it to lein or boot

thheller17:04:48

yeah that won't happen sry

thheller17:04:52

well it might happen, I don't really understand what they are yet

thheller17:04:02

time will tell 😉

dominicm17:04:59

It's mostly just about inputs and outputs. So your devtools.edn should come from the source file set. You should create a temp folder and write to that. (This can be managed by the boot task)

thheller17:04:35

hehe shadow-build actually always write everything to a cache folder, then does a flush to write to the actual output dir

thheller17:04:52

so pretty similar to boot I guess

thheller17:04:49

if it wasn't for the cache it wouldn't write anything until flush .. its all kept in memory

thheller17:04:51

boot -d org.clojure/clojure:1.9.0-alpha15 repl
nREPL server started on port 56759 on host 127.0.0.1 - 
REPL-y 0.3.7, nREPL 0.2.8
Clojure 1.7.0

thheller17:04:20

ah suppose i need BOOT_CLOJURE_VERSION

pesterhazy17:04:24

BOOT_CLOJURE_VERSION=1.9.0-alpha15

dominicm17:04:43

I see no reason that you couldn't hook this up to boot easily then!

thheller17:04:11

I run this from the REPL most of the time but I guess I could write some boot tasks that set the required config parts

thheller18:04:35

quick question on with-pass-thru .. not sure I actually need it

thheller18:04:04

shadow-devtools needs access to all dependencies since it must be able to require macro namespaces

thheller18:04:19

boot can do that without with-pass-thru right?

thheller18:04:04

or can that hurt boot in same way or other? (if I require random CLJ code?)

micha18:04:28

with-pass-thru is a macro that is used to eliminate some middleware boilerplate

micha18:04:33

it doesn't do anything else

micha18:04:01

(with-pass-thru fs (doit))
;; same as
(fn [next-task]
  (fn [fileset]
    (doit)
    (next-task fileset)))

thheller18:04:17

so how do you correctly write a deftask that never exits? ie. launches a CLJS repl

micha18:04:41

if doit blocks it will never get to the next expresion

thheller18:04:07

(require '[shadow.cljs.devtools.api :as cljs])

(deftask cljs
  [m mode VAL kw "mode (dev, once, release)"
   b build VAL kw "build id (defined in shadow-cljs.edn)"]
   
  (with-pass-thru fs
    (let [{:keys [mode build] :as opts} *opts*]
      (case mode
        :dev
        (cljs/dev build)
        :once
        (cljs/once build)
        :release
        (cljs/release build)
        ;; else
        (throw (ex-info "invalid mode" opts))
        ))))

thheller18:04:05

I guess deftask is expected to return rather than actually calling the code directly? or is it fine to leave out the with-pass-thru?

thheller18:04:37

(cljs/dev build) actually starts a CLJS REPL as well so that basically never finishes until the user exits

thheller18:04:57

and does code reloading in the background, no idea how I supposed to hook that in the fileset stuff yet

micha18:04:24

your task must return middelware

micha18:04:46

that middleware will be composed with other task middleware to form a pipeline

thheller18:04:15

right but a pipeline won't work for :dev as that never finishes?

micha18:04:17

the watch task will call its next task whenever files change

micha18:04:31

yeah you don't want to do that probably

thheller18:04:42

so a different task for dev

micha18:04:25

the body of the with-pass-thru will be evaluated each time the pipeline is run

micha18:04:39

so with the watch task it could be evaluated multiple times

thheller18:04:08

hmm ok, I guess I need to write this as a proper boot plugin then

thheller18:04:17

its all really built to be standalone

micha18:04:01

have you tried boot repl and just use it that way?

thheller18:04:31

yeah that works fine

micha18:04:35

not sure where boot comes in if it's set up to be used in a standalone fashion

thheller18:04:37

just wanted something like that, sometimes you just want a :release built without running REPL

micha18:04:09

yeah you can do that in boot, too

micha18:04:23

pass an expression to the repl task or something like that

thheller18:04:47

hmm? last time I checked there was no lein run?

micha18:04:23

because it's trivial to do with boot

micha18:04:49

(deftask runit []
  (with-pass-thru _
    (require 'my.namespace)
    ((resolve 'my.namespace/-main) "arg1" :arg2)))

micha18:04:00

then you can do boot runit

thheller18:04:19

hmm but that still requires writing something into build.boot 😉

micha18:04:34

sure, like what you need to write in the project.clj

thheller18:04:00

well that only the :dependencies entry

thheller18:04:01

nothing more

micha18:04:51

you could make tasks in your jar

thheller18:04:08

yeah .. I need to research boot way more to do that correctly

micha18:04:25

the runit task is all you need

micha18:04:33

just put your own function in there

thheller18:04:50

I'll just use BOOT_CLOJURE_VERSION=1.9.0-alpha15 boot -r src/main -d thheller/shadow-devtools:0.1.20170410 repl as an example for now

thheller18:04:04

repl is fine for now

thheller18:04:21

proper deftask if someone actually wants to use this

micha18:04:59

if you add dependencies with -d boot will look for tasks and automatically require the namespaces

pesterhazy18:04:43

this PR would remove the need for a deftask for this kind of thing, if I understand you correctly @thheller: https://github.com/boot-clj/boot/pull/597

pesterhazy18:04:19

or at least make it a bit more convenient

thheller18:04:53

yes, thats what I need 😉

thheller19:04:27

I might make a boot-shadow-devtools at some point

dominicm19:04:30

@thheller I think you don't want a task that blocks forever: - the repl / boot cljs repl tasks do not. - check out boot-sass and boot-cljs for examples of tasks that know how to be called repeatedly

thheller19:04:26

yeah that is my impression as well

thheller19:04:45

also don't need to watch files if boot can tell me

kenny19:04:59

Anyone have any idea why I might be getting this rather opaque error message when trying to push a jar to clojars?

gpg: signing failed: No such file or directory
That’s it. That is the entire error message 😞

kenny19:04:02

This is the output from running boot build-jar push-release

Writing pom.xml and pom.properties...
Writing boot-tasks-1.4.jar...
Installing boot-tasks-1.4.jar...
Signing boot-tasks-1.4.jar...
Could not sign /var/folders/09/rhhm7hw51j396mljlvx22str0000gn/T/pom987689368313024651.xml
gpg: signing failed: No such file or directory
gpg: signing failed: No such file or directory


If you don't expect people to need to verify the authorship of your jar, don't set :gpg-sign option of push task to true.
   clojure.lang.ExceptionInfo: /var/folders/09/rhhm7hw51j396mljlvx22str0000gn/T/pom987689368313024651.xml.asc (No such file or directory)
    data: {:file
           "/var/folders/09/rhhm7hw51j396mljlvx22str0000gn/T/boot.user190151716003145822.clj",
           :line 29}
java.io.FileNotFoundException: /var/folders/09/rhhm7hw51j396mljlvx22str0000gn/T/pom987689368313024651.xml.asc (No such file or directory)

kenny19:04:29

Not sure what that pom987689368313024651.xml file is anyway. It must be auto-generated by boot.

micha19:04:02

@kenny have you tried with the -v options like boot -v build-jar ...?

kenny19:04:47

Just did. Doesn’t provide any additional insight into the error

micha19:04:06

the task is calling the command line gpg program

micha19:04:21

the error message is originating there i think

kenny19:04:29

I have no idea what file it is talking about though

kenny19:04:45

Maybe the weird pom939484453697768650.xml file?

micha19:04:59

yeah that seems to be the one

kenny19:04:08

Is that generated by boot?

micha19:04:34

generated by the pom task

kenny19:04:37

It’s not part of the fileset after build-jar though

juhoteperi19:04:38

Could be internal temp file from push task, I think it writes some temp files for gpg

micha19:04:46

try doing just boot pom show -f

kenny19:04:02

Not there

micha19:04:04

oh right, yeah it dies

micha19:04:07

i mean it does

kenny19:04:26

So then why is gpg trying to encrypt that file?

juhoteperi19:04:53

@kenny try something like gpg --sign anyfile.txt

juhoteperi19:04:28

perhaps the No such file or directory is about missing gpg executable?

kenny19:04:45

Ah, indeed you are correct.

kenny19:04:12

Pretty bad error message

juhoteperi19:04:47

Yeah, it is probably just the exception message from shell calling the command

kenny19:04:50

Any idea how to correctly install gpg on a mac? I assumed brew install gnupg would do it but apparently not

juhoteperi20:04:05

oh well, it does link to homebrew also

juhoteperi20:04:16

though it does mention gnupg2

kenny20:04:28

I also install gnupg2 via homebrew

juhoteperi20:04:38

is gpg2 available?

juhoteperi20:04:58

If so, you need to se BOOT_GPG_COMMAND to gpg2

kenny20:04:16

gpg2 is available

kenny20:04:24

I’ll try that I suppose

kenny20:04:39

This was much easier to setup on Linux..

thheller20:04:46

I set it up once .. scared to touch it since then 😛

kenny20:04:28

YES! That magically made it work

richiardiandrea20:04:18

Same here, for some reason you have to set it to gpg2

dominicm21:04:16

There's two versions of gpg. Instead of breaking backwards compat, they created gpg2. Some people use gpg1 without realising.

richiardiandrea21:04:16

@dominicm this means though that we are only gpg2 compatible, and I could not find it anywhere in the doc (maybe it's just me)

dominicm21:04:28

@richiardiandrea probably! I think there's confusion in this area.

dominicm21:04:01

I think some platforms may symlink also. But I have no evidence, just hazy memories

dominicm21:04:55

Some tools have an ordered check

dominicm21:04:05

https://superuser.com/a/655250 interesting, so the differences aren't in API, oops! I thought the default key type changed

dominicm21:04:55

I suppose there's no clear answer as of 2.1, as the keys can't see each other.

dominicm21:04:32

I'm kinda leaning towards gpg2 being the expected binary here. Especially since gpg 2.1