Fork me on GitHub
#boot
<
2016-09-22
>
domkm00:09:27

I'm getting an error boot.user/stop is not resolvable as a var from cider-refresh. I swear I used to be able to do this in an old project. Has anyone run into this?

kenbier17:09:35

is there a more succint way to write this, or some built in boot task?

kenbier17:09:45

(deftask wait-at-end
  "A task that runs all successive tasks and then waits."
  []
  (fn middleware [next-task]
    (fn handler [fileset]
      (next-task fileset)
      (wait))))

martinklepsch17:09:16

(comp (next-task) (wait))

martinklepsch17:09:49

This might not be what you want though

kenbier17:09:06

yeah i dont think it is 😕

kenbier17:09:26

im using this task and it also tries to put itself at the end

kenbier17:09:53

so my s3-sync task never gets run, as its waiting on wait to finish

kenbier17:09:49

@martinklepsch i see you contributed to this task, you didnt have this issue 😉 ?

martinklepsch17:09:35

@kenbier I eventually ended up writing an S3 sync task as well. (not because this one didn't work but because I had some different ideas)

kenbier17:09:19

ah nice, is it on github?

martinklepsch17:09:57

@kenbier I see the issue, I'd suggest that the task actually should be ran after all other tasks but I guess that's outside of your control

martinklepsch17:09:20

not sure if there's a way to make it work with wait

richiardiandrea17:09:07

can confirm that confetti has been successfully used for http://clojurescript.io's assets 😄

martinklepsch17:09:57

@richiardiandrea are you still using it?

richiardiandrea17:09:52

well unfortunately once I changed company I could not deploy anymore there, but it was linked to Jenkins so I guess the new maintainer (if any) is still using it yes

richiardiandrea17:09:13

I also wanted to create a task for glacier

richiardiandrea17:09:39

for backup purposes, and it might become handy again for that

alandipert17:09:43

@richiardiandrea wow http://clojurescript.io is super cool

richiardiandrea17:09:07

it has been fun and made replumb possible

richiardiandrea17:09:23

Klipse is for instance using replumb under the hood 😄

richiardiandrea17:09:41

but then, well, I joined a product-based project, still maintaining replumb but got kind of abandoned...the code is still on github: https://github.com/Lambda-X/cljs-repl-web

martinklepsch17:09:53

Still think it's probably one of the better things I've made, but I'm apparently more excited about deploying static web-apps than most others 😄

richiardiandrea17:09:38

yes Martin it was is working very smoothly indeed, my current project manager does not want to deploy on S3, this is the only reason why I am not using it anymore...but during the interview, six months+ ago, I showed them how I was deploying http://clojurescript.io and they were all impressed 😄

richiardiandrea18:09:36

it is so cool to deploy with a simple boot -e type=dev build deploy-s3

kenbier18:09:06

@richiardiandrea what do you deploy to then?

richiardiandrea18:09:01

for our first MVP we are on Digital Ocean, nothing fancy, we just need to show things and get feedback/customers

donaldball20:09:20

Hi boot friends. I’m running into an issue with leiningen that’s finally sending me to seriously investigate boot. Hooray. Unfortunately, I’m not having much better luck with boot yet.

donaldball20:09:02

The underlying problem I’m trying to solve is to be able to cheaply and simply manage a private maven repository. S3 seems to fit the bill nicely.

micha20:09:08

hello @donaldball, welcome

micha20:09:57

there are two options described in this wiki page: https://github.com/boot-clj/boot/wiki/S3-Repositories

donaldball20:09:06

Unfortunately! We’re using aws-vault to manage our aws credentials, which uses Amazon STS to issue temporary credentials, so we don’t need to trust our tool chain very much or store credentials in configuration files. It’s pretty swank.

donaldball20:09:24

Unfortunately, it requires the java sdk to be at least 1.9.x because I don’t know reasons

donaldball20:09:55

Unfortunately everyone - leiningen and boot included, seem to depend on 1.7 indirectly because reasons

donaldball20:09:56

Neither option on that page work, alas. lein-s3-wagon only likes to use hard-coded credentials, which is a show stopper for us, and the spring boot maven depends on an older version of the amazon sdk.

donaldball20:09:39

Some folks have forked the spring boot maven to use a newer version which would be fine, maybe? So I’m trying to use it, but it itself is only hosted in a non-standard public maven repo.

micha20:09:45

ok i have two possible workarounds for you 🙂

donaldball20:09:52

All ears 🙂

donaldball20:09:00

I should note that I’m trying this atm:

donaldball20:09:02

:wagons (fn [wagons]
           (conj wagons
                 '[fi.yle.tools/aws-maven "1.4.0"
                   :schemes {"s3" #(fi.yle.tools.aws.maven.SimpleStorageServiceWagon.)}]))
 :repositories (fn [repos]
                 (conj repos
                       '["fi.yle.tools" {:url ""}]
                       '["sparkfund-snapshots" {:url ""}]))

micha20:09:08

the first is build boot with java 1.9

micha20:09:27

is that a real thing now?

micha20:09:41

i haven't tried doing AOT with java 1.9

donaldball20:09:45

I don’t think it is yet, just guessing

micha20:09:55

but you could just eliminate AOT in the boot project.clj files

micha20:09:03

that only helps startup time

micha20:09:09

which you may not care about so much

donaldball20:09:37

not so much, though having devs have to use a non-standard boot seems fragile in the long run

micha20:09:44

but there are other ways to safely handle credentials, if that's the problem you want to ultimately solve

donaldball20:09:14

That’s all I truly care about, though we’re moderately invested in the aws-vault tool atm

micha20:09:15

like for instance my workflow uses environment variables in my shell, that's where i have my AWS creds

micha20:09:29

and i use System/getenv in the build.boot

micha20:09:38

then you can decouple the two

micha20:09:51

like how you get the creds into the shell environment is separate and up to you

micha20:09:16

personally i have a gpg encrypted file like

micha20:09:38

export AWS_ACCESS_KEY_ID=xxxx
export AWS_SECRET_ACCESS_KEY=yyyy

micha20:09:05

and i do eval "$(gpg -d ~/.creds.asc)" in my bash shell

micha20:09:24

but you could make a little java program that uses the AWS tools or whatever

micha20:09:37

or they probably have a cli tool for it already

micha20:09:51

sorry, didn't mean to go off on a tangent there... what is the issue you're seeing with boot in java 1.9?

donaldball20:09:07

No worries, I appreciate alternate angles

donaldball20:09:36

I haven’t yet tried boot with java 1.9; what would that change that might help?

micha20:09:57

then you could use the version of the aws sdk that supports the vault stuff, right?

donaldball20:09:20

If I can, that would solve my problem

donaldball20:09:34

Where would I go to learn more about trying this?

micha20:09:04

i think boot Should Work™ with java 1.9

donaldball20:09:25

Oh, sorry, I think I may have not been very clear

donaldball20:09:50

It’s not the Java version that needs to be 1.9, it’s the amazon java library that needs to be 1.9+

donaldball20:09:42

So the essential problem as I understand is that the boot dependency resolver is running in a jvm classloader that has the old and sad amazon library

micha20:09:07

not the dependency resolver, but yes

micha20:09:35

one thing you can try is when you add the :wagons, you can override the aws sdk version

micha20:09:49

the :wagons key is a normal dependency vector

micha20:09:02

so you can use :exclusions there and so on

micha20:09:29

but i doubt the wagon libraries would just work

donaldball20:09:32

I’m afraid I don’t know where to find documentation on the syntax of the dependency vectors

micha20:09:39

are you able to use the AWS SDK (of the correct version) to fetch temporary AWS_ACCESS_KEY_ID etc?

donaldball20:09:37

yeah. I even wrote my own lein plugin that works with the right amazon library version, but I ran headlong into essentially this problem with leiningen also 😛

micha20:09:57

in boot you can make a "pod" in which you load your version of the AWS SDK

micha20:09:06

isolated from all other AWS SDKs

micha20:09:19

i think this is maybe what you're missing in your current setup

donaldball20:09:30

Yeah, that’s the big reason I’m excited about this option

micha20:09:42

if you want to pop open a REPL you can try it

donaldball20:09:50

But it’s not at all clear to me how I would do the dependency resolution in that pod

micha20:09:57

you wouldn't

micha20:09:06

you'd use that pod to obtain the temporary credentials

micha20:09:18

then you can feed those to the crusty old wagon things

donaldball20:09:49

ah, but the crusty old wagon things can’t use the temporary credentials… they just flat out don’t work in older versions of the amazon sdk because… i… can’t even really imagine a good reason, frankly

micha20:09:16

you don't get strings that are used like AWS_ACCESS_KEY_ID?

donaldball20:09:41

you do, they just don’t work with old versions of the amazon lib. It’s nuts.

micha20:09:14

is the vault the hashicorp one?

micha20:09:03

well you can try this, maybe it'll work:

micha20:09:28

(set-env!
  :wagons
  '[[com.amazonaws/aws-java-sdk "1.9"]
    [org.springframework.build/aws-maven "5.0.0.RELEASE"
       :schemes {"s3" (fn [] (org.springframework.build.aws.maven.SimpleStorageServiceWagon.))}
       :exclusions [com.amazonaws/aws-java-sdk]])

micha20:09:18

something like that, that's the dependency vector stuff i mentioned before

donaldball20:09:13

yeah, alas, it seems not to have an effect; the boot process still yields an error that suggests it itself has an old amazon lib

micha20:09:49

that's odd

micha20:09:00

i don't think it should have any AWS stuff unless you add it

micha20:09:23

ok it's possible that you need to exclude specific parts of aws

micha20:09:32

like the aws-java-sdk-s3 or whatever

micha20:09:51

you can see the deps tree for the pod that's doing the deps like this:

micha20:09:19

boot -BPd boot/aether show -d

donaldball20:09:28

:tears of joy:

donaldball20:09:48

But alas, com.amazon isn’t in that list

donaldball20:09:11

Which says that you were totally right about boot itself not providing amazon, but the exclusions isn’t having the effect I’d like

micha20:09:30

i have a new idea

donaldball20:09:43

(set-env!
 :source-paths #{"src"}
 :wagons (fn [wagons]
           (conj wagons
                 '[org.springframework.build/aws-maven "5.0.0.RELEASE"
                   :schemes {"s3" #(org.springframework.build.aws.maven.SimpleStorageServiceWagon.)}
                   :exclusions [com.amazonaws/aws-java-sdk-core
                                com.amazonaws/aws-java-sdk-s3]]))
 :repositories (fn [repos]
                 (conj repos
                       '["sparkfund-snapshots" {:url ""}]))
 :dependencies '[[clj-ssh "0.5.14"]
                 [com.amazonaws/aws-java-sdk-core "1.11.29"]
                 [com.amazonaws/aws-java-sdk-s3 "1.11.29"]

donaldball20:09:48

is the current stanza fwiw

micha20:09:28

barp $ boot -BPd boot/aether -d org.springframework.build/aws-maven:5.0.0.RELEASE show -d           

micha20:09:49

that shows [com.amazonaws/aws-java-sdk "1.7.1"]

donaldball20:09:11

yeah, that’s the problematic version

donaldball20:09:48

it’s implied that it’s a direct dependency of boot/aether, which is surprising, I don’t see that in https://github.com/boot-clj/boot/tree/master/boot/aether anywhere

donaldball20:09:37

er, no, I’m misreading the output, it’s from org.springframework.build/aws-maven as expected

donaldball20:09:07

I gotta run to a school thing now, I’ll keep fiddling. Thanks @micha, you’ve been more than helpful.

kenbier20:09:19

@martinklepsch i want to use sync-bucket from confetti, and all i want to sync is some dir in target. would calling (sync-bucket “target/foo”) be sufficient?

kenbier21:09:03

that post-task thing is really annoying me from boot-s3 and i had to copy it over to get it to work

kenbier21:09:30

im hoping to be able to do (comp (some-stuff) (sync-bucket) (run) (wait))

kenbier21:09:15

I omitted the bucket and keys for brevity

micha21:09:19

good luck @donaldball !

micha21:09:47

@donaldball there was a bug in my paste above, an extra single quote before the :exclusions

micha21:09:54

i edited the snippet

martinklepsch21:09:16

@kenbier have you had a look at boot sync-bucket --help?

martinklepsch21:09:14

the sync-bucket task operates on the fileset (by default it just syncs the entire fileset) so you have all regular boot utilities (mostly sift) to filter and move around files in the fileset

domkm21:09:42

Can modifying dependencies multiple times before a build begins negatively affect dependency resolution? I'd like to have a base list of dependencies and then dynamically add dev/prod deps inside of their respective tasks.

juhoteperi21:09:31

@domkm You can add dev deps as :scope "provided" and use uber :exclude-scope to drop them from uberjar. No idea about prod deps, I have never hit a case where I would only use some dep on prod.

domkm21:09:47

Thanks @juhoteperi

juhoteperi21:09:35

You could also use set-env! on tasks, but I avoid that as it adds unnecessary mutation and I don't see reason to not use Maven features for this

domkm21:09:25

"By default, entries from dependencies with the following scopes will be copied to the fileset: compile, runtime, and provided. The --include-scope and --exclude-scope options may be used to add or remove scope(s) from this set."

domkm21:09:48

Looks like test is excluded already so this probably isn't necessary

domkm21:09:52

Good to know though 🙂

juhoteperi21:09:07

Yeah that's what I use for Boot tasks and deps used on tests

juhoteperi21:09:24

But I use provided scope for Cljs deps (compiler and libs)

juhoteperi21:09:47

Could use test but that feels wrong 😄

domkm21:09:22

@juhoteperi Any issue with dynamically setting resource-paths and source-paths?

juhoteperi21:09:13

Should work. I try to avoid it but I guess it can't be helped sometimes.

domkm21:09:24

Okay, thanks

juhoteperi21:09:44

Usually you can use :source-paths for tests and test-resources

juhoteperi21:09:54

As they aren't included in jar

dominicm21:09:03

It's worth noting that the Clojure AOT will load in everything on the classpath, even if the scope is provided.

micha22:09:06

yeah everything it needs to resolve to compile the namespace you tell it to AOT

micha22:09:28

this is a good reason to avoid AOT 🙂

juhoteperi22:09:50

aot loads namespaces that your main ns depends (or more specifically, namespaces there the ns you tell aot to complete depend)

dominicm22:09:42

@juhoteperi yep. problematic if it's expected that a different dependency version to load in though.

kenbier22:09:36

@martinklepsch no ill check it out thanks