Fork me on GitHub
#boot
<
2017-12-20
>
aj taylor04:12:07

I'd like to test some changes that I made to a dependency locally, and was wondering how I'd link it to my project

bostonaholic05:12:48

anyone know how to get https://github.com/hashobject/boot-s3 to work with boot-environ?

bostonaholic05:12:07

I want to grab my aws keys from env vars during a task

aj taylor05:12:45

Fair enough! I swear I looked for it šŸ™‚

seancorfield05:12:08

@bostonaholic boot-environ is deprecated -- did you read the underlying environ README showing how to use it with Boot? (I haven't tried it -- just asking)

seancorfield05:12:30

@aj I asked Bing clojure boot checkout dependencies šŸ™‚

bostonaholic05:12:18

which seems to be the replacement

seancorfield05:12:44

@bostonaholic Cool, yeah, looks like it got folded in.

bostonaholic05:12:13

I canā€™t figure out how to get enviro to pick up .boot-env and allow me to (env :aws-access-key)for instance

bostonaholic05:12:27

so I can pass that as an arg to s3-sync task

seancorfield05:12:02

Are you sure .boot-env is on your classpath?

bostonaholic05:12:12

that might just be it

seancorfield05:12:40

Hmm, according to the README, it sounds like .boot-env is generated by the environ task...?

bostonaholic05:12:01

yeah, that doesnā€™t make sense to me

bostonaholic05:12:06

it shouldnā€™t be

bostonaholic05:12:47

that wouldnā€™t make any sense

seancorfield05:12:45

Ah... because tasks in the pipeline might want to read it from the fileset perhaps? And you might want to pass the values via the command line (the -e argument to the environ task) or via actual code (the environ task with an :env argument)? It does seem a bit odd.

bostonaholic05:12:10

Iā€™ve used environ plenty of times with lein but I canā€™t get it to work with boot

seancorfield05:12:57

OK, I just tried it at the command line -- not clear from the README but the boot-environ dependency is purely for the Boot task that sets environment variables into .boot-env. The environ dependency is for the environment variable reading part it seems.

seancorfield05:12:24

sean@sean-xps:~/clojure$ boot -d environ:1.1.0 -d boot-environ:1.1.0 environ -e foo=bar repl
...
boot.user=> (require '[environ.core :refer [env]])
nil
boot.user=> (env :foo)
"bar"
boot.user=>

bostonaholic05:12:12

that makes sense

bostonaholic05:12:33

I think I have to figure out if/how to put .boot-env in the classpath

seancorfield05:12:48

That's generated by the boot-environ part.

seancorfield05:12:05

It sounds like you want to invoke the s3-sync task, passing environment variable values as arguments?

seancorfield05:12:37

So you only need environ, not boot-environ, since you only want to read from the environment...

bostonaholic05:12:52

when I use lein-environ, I have a .lein-env file that looks like {:database-url ...} so I can (environ.core/env :database-url) in my code

seancorfield05:12:27

Let me try creating a .boot-env file then just to test that...

bostonaholic05:12:45

I canā€™t figure out how to get the .boot-env in the classpath so environ can read from it

seancorfield05:12:01

Got it working ...

sean@sean-xps:~/clojure$ echo '{:foo "bar"}' > .boot-env
sean@sean-xps:~/clojure$ boot -d environ -d boot-environ -s . call -p -e "(require '[environ.core :refer [env]])" -e '(env :foo)'
Classpath conflict: org.clojure/clojure version 1.9.0-RC1 already loaded, NOT loading version 1.5.1
nil
bar
sean@sean-xps:~/clojure$ FOO=WAT boot -d environ -d boot-environ -s . call -p -e "(require '[environ.core :refer [env]])" -e '(env :foo)'
Classpath conflict: org.clojure/clojure version 1.9.0-RC1 already loaded, NOT loading version 1.5.1
nil
WAT

seancorfield05:12:53

I could probably use -r for :resource-paths instead of -s for :source-paths but actual environment variables override the contents of .boot-env it seems (good!).

seancorfield05:12:20

The .boot-env file is only picked up if you add the boot-environ dependency. Nope, just tried without boot-environ and it does find it...

seancorfield05:12:45

So I think that confirms that boot-environ is only needed to write to .boot-env I think, so you can manipulate the environment via the task.

bostonaholic05:12:31

hmmm, I must be missing something

bostonaholic05:12:44

(deftask try-env []
  (prn (-> env keys sort))
  (prn "============" (env :aws-access-key))
  identity)

bostonaholic05:12:14

that is my testing task that (env :aws-access-key) returns nil

bostonaholic05:12:28

when I boot try-env from cli

seancorfield05:12:04

env is a function so (-> env keys sort) won't work

bostonaholic05:12:16

env is a hashmap

bostonaholic05:12:01

it prints out all the rest of my env vars

bostonaholic05:12:15

but doesnā€™t include the ones Iā€™ve added in .boot-env

seancorfield05:12:25

Oops, yeah, sorry... my command line was wrong...

seancorfield05:12:36

Well, I put {:foo "BAR"} in .boot-env and with -r . it found it just fine.

bostonaholic05:12:48

any idea how to tell build.boot to add .boot-env to the classpath?

seancorfield05:12:20

-r . or (merge-env! :resource-paths #{"."}) assuming it's in the top-level of your project

bostonaholic05:12:53

getting closer

seancorfield05:12:47

And the environ task combines with whatever's in .boot-env:

sean@sean-xps:~/clojure$ echo '{:foo "bar"}' > .boot-env
sean@sean-xps:~/clojure$ boot -d environ -d boot-environ -r . environ -e bar=cmd-line call -p -e "(require '[environ.core :refer [env]])" -e '(-> env keys sort)' -e '(env :foo)' -e '(env :bar)'
Classpath conflict: org.clojure/clojure version 1.9.0-RC1 already loaded, NOT loading version 1.5.1
nil
(:awt-toolkit :bar :boot-app-path :boot-class-path :boot-file :fake-class-path :file-encoding :file-encoding-pkg :file-separator :foo :home ...)
bar
cmd-line

bostonaholic05:12:29

I put .boot-env into resource paths and it worked

bostonaholic05:12:41

but now I have to figure out a better way

bostonaholic05:12:13

(merge-env! :resource-paths #{"."}) results in ā€œAssert failed: The :source-paths, :resource-paths, and :asset-paths must not overlap.ā€

seancorfield05:12:41

Right, that was just an example given that I'm using just the command line and have no build.boot file.

seancorfield05:12:22

The .boot-env file can either be in the :source-paths locations or the :resource-paths locations to be on the classpath.

seancorfield05:12:53

You don't need .boot-env at all if you just want to read from actual environment variables...

bostonaholic06:12:32

yeah, I know. thatā€™s how I have ci set up

bostonaholic06:12:49

but this is for local so I can save creds

seancorfield06:12:24

So .boot-env can be anywhere -- you just need to merge its path into either resource paths or source paths.

bostonaholic06:12:14

I just put it in resources/ because if itā€™s in . I keep getting Assert failed: The :source-paths, :resource-paths, and :asset-paths must not overlap.

bostonaholic06:12:27

:source-paths #{"src"}
 :asset-paths #{"html"}
 :resource-paths #{"."}

seancorfield06:12:34

Yes, you would get that assertion -- as I noted above.

seancorfield06:12:04

But it can also be outside your project and use a path like "../ci-env" so it would read ../ci-env/.boot-env

bostonaholic06:12:28

itā€™s weird that it behaves slightly different than lein-environ

seancorfield06:12:04

Boot is very different to Leiningen šŸ™‚

seancorfield06:12:37

We switched almost two years ago...

bostonaholic06:12:18

Iā€™ve only used boot for very small projects, not large apps

seancorfield06:12:51

We have a mono-repo with over a dozen subprojects all managed with Boot...

seancorfield06:12:34

About 75,000 lines of Clojure, including tests.

seancorfield06:12:51

Our build.boot is well over 1,000 lines šŸ™‚

bostonaholic06:12:53

I work with Phil so I doubt weā€™ll be switching away from leiningen anytime soon šŸ˜œ

bostonaholic06:12:17

(although Iā€™m sure heā€™d be for it if it made sense)

seancorfield06:12:28

We just needed to do stuff that was hard to do in Leiningen plugins. We needed things more dynamic.

seancorfield06:12:14

For example, we do source code analysis with tools.namespace to determine inter-project dependencies and automatically build the :source-paths on the fly.

bostonaholic06:12:24

ugh, I canā€™t put .boot-env in :resource-paths otherwise it ends up in my distribution

bostonaholic06:12:32

glad I checked that before syncing to s3

seancorfield06:12:57

That means we can refactor code across subprojects without needing to change anything in the build.

seancorfield06:12:54

Do you mean s3-sync sucks up everything in your :resource-paths? That seems unfriendly. Such things wouldn't go into the JAR file...

bostonaholic06:12:11

it sucks up from target-path

seancorfield06:12:02

@bostonaholic The s3-sync task accepts a :source argument and should only upload that directory?

bostonaholic06:12:30

I think the bug is in the second line

bostonaholic06:12:48

for lein-env it uses io/file but boot-env using io/resource

bostonaholic06:12:58

so boot-env has to be in resources to be picked up

bostonaholic06:12:09

but lein-env can be at the root of the fs

seancorfield06:12:56

It's not a bug -- it's doing exactly what the README says it should.

bostonaholic06:12:37

youā€™re right, it does say ā€œon the classpathā€ in the readme

bostonaholic06:12:42

wonder why itā€™s differentā€¦

seancorfield06:12:30

Because that's in line with how Boot works.

seancorfield06:12:38

If the S3 task syncs a folder in the target path, you can probably use sift to move everything except .boot-env to that folder.

seancorfield06:12:04

A Boot task to move stuff around in the fileset.

bostonaholic06:12:58

do you have a link? Iā€™m not finding it anywhere

bostonaholic06:12:59

I was looking in the wiki

seancorfield06:12:00

I would imagine you can exclude .boot-env with sift -i .boot-env -v

seancorfield06:12:00

@bostonaholic Let me know how it works out in the end. And thank you for the chance to learn about environ and boot-environ šŸ™‚

bostonaholic20:12:56

Thanks for all of your help the other night. Hereā€™s how it ended.

bostonaholic20:12:01

(set-env!
 :source-paths #{"src" "dev"}
 :dependencies '[...
                 [environ "1.1.0" :scope "test"]
                 ...])

(require
 ...
 '[environ.core :refer [env]]
 ...)

(deftask build
  "Build distribution."
  []
  (comp
   (cljs :optimizations :advanced
         :source-map true)
   (sift :include #{#".boot-env"}
         :invert true)
   (target :dir #{"target"})))

(deftask deploy
  "Deploy distribution"
  []
  (s3-sync :source ""
           :bucket (env :aws-bucket)
           :access-key (env :aws-access-key)
           :secret-key (env :aws-secret-key)))

bostonaholic20:12:21

ended up putting .boot-env in ./dev

bostonaholic06:12:22

thanks A TON for all of your help

bostonaholic06:12:39

still some fiddling Iā€™ve got to do, but itā€™s getting very close

bostonaholic06:12:00

granted, deploys work on ci and Iā€™m just trying to get them to work locally

seancorfield06:12:03

If you're interested in reading more about our experiences with Boot http://seancorfield.github.io/blog/categories/boot/

bostonaholic06:12:46

(conj reading-list *1)

seancorfield06:12:57

We've had Clojure in production since 2011. We have 1.9.0 in production (as of today -- we were on prerelease builds before that). We use clojure.spec fairly heavily šŸ™‚

bostonaholic06:12:39

Iā€™m at CircleCI and use all of those as well, just not boot

seancorfield06:12:56

For most of our processes, we use Boot tasks to build uberjars, but we also still use Boot directly in production for some stuff...

seancorfield06:12:09

Ah, CircleCI is a very cool company!