Fork me on GitHub
#boot
<
2015-10-28
>
nberger00:10:21

@alandipert cool. Decomposing into smaller tasks sounds good. One thing about getting a failed build in ci: I want the build to fail, but I also want the junit output, so I still can't see how to get the two things at the same time

nberger00:10:55

To be clear: the only way to get an exit status not= 0 in current boot is to throw? And then output to target-path is not done?

alandipert00:10:20

or (System/exit 1)

alandipert00:10:48

i can imagine a task that prints the output and then exits 1 if it's failing

alandipert00:10:02

there was some discussion around the best way to transmit this kind of information in a pipeline earlier this year, http://hoplon.discoursehosting.net/t/best-practices-for-creating-extensible-tasks/358, that might interest you

nberger00:10:14

I'll check that out, thanks

alandipert00:10:33

sorry this is turning into a bit of a goose chase

alandipert01:10:21

for my own purposes i'm happy to fail the build and catch upstream, but i'm open to any of these other ways, as long as running tests the simple way doesn't get any more complicated

martinklepsch12:10:07

I just built boot with boot built by boot. 🎉

micha12:10:23

OMG ITS REAL

esp112:10:12

so, we’re trying to fix this reload issue in boot-http: https://github.com/pandeiro/boot-http/pull/27

esp112:10:09

basically, we’d like to get the input dirs so that reload can watch them, but the point at which we need them is outside of the context where we have a fileset

esp112:10:21

is it reasonable to use the :directories in the boot env for this? or is there a better way to access the synchronized input dirs?

martinklepsch12:10:45

@esp1: I think (:dirs fileset) might also be good?

micha12:10:59

boot.core/input-dirs is a thing i think

micha12:10:31

(input-dirs fileset)

micha12:10:01

if you don't have a fileset then (:directories (get-env)) is ok too

micha12:10:31

those are both source-paths and resource-paths

micha12:10:40

but not asset-paths

micha12:10:53

but i think boot-http doesn't know about asset-paths anyway

esp112:10:55

ok perfect

micha12:10:57

so that's the one you want

micha12:10:33

i think to be correct you'd want the server to serve output dirs

micha12:10:52

that's the stuff that would end up in a war file or uberjar

esp112:10:03

yes, but this is for watching changes to namespaces

esp112:10:13

so we want to watch the source files

micha12:10:23

those would be resource dirs then

micha12:10:37

which are on the classpath but also emitted to jars / wars

micha12:10:07

ideally the local server would serve the same files as what would be served if you used the war task to make a warfile from your app

micha12:10:11

or an uberjar

micha12:10:54

boot-jetty shows one method of achieving this

esp112:10:27

yes. technically tho what we’re doing is telling wrap-reload what dirs to watch in order to reload namespace changes so we don’t have to restart stuff when we’re doing development

micha12:10:51

but it would be nice if your development environment matched your production one

micha12:10:11

so everything isn't different in production and you have to debug and refactor

esp112:10:13

does boot-jetty support hot reloading in dev?

martinklepsch12:10:20

@micha: in CI I’m downloading 2.4.0 but then it downloads 2.3.0 for some reason — any ideas why that could be? there are no properties or env vars as far as I can see: https://travis-ci.org/boot-clj/boot/builds/87880803#L94

esp112:10:47

hm. i couldn’t get it to work for me

micha12:10:53

are you on windows?

micha12:10:08

yeah it should work, you need to use ring-reload though

martinklepsch12:10:21

(link updated)

esp112:10:37

that’s actually why i got involved w/this bug. i switched to boot-http and then discovered hot reloading didn’t work unless your source was in a dir path starting with “src"

esp112:10:54

i haven’t gone back yet to see if that’s the same issue w/boot-jetty

micha12:10:55

and to get the directories the best way is (:directories boot.pod/env)

micha12:10:21

that is available in any pod, including where you don't have access to boot.core

esp113:10:02

and just to reiterate, (:directories boot.pod/env) always returns the input directories? (i.e. source-dirs + resource-dirs)

esp113:10:16

and just those dirs

micha13:10:28

@martinklepsch: it downloads 2.3.0 to bootstrap itself, you need to have a boot.properties file if you want to avoid that

micha13:10:50

because the loader is stupid, it doesn't know what the latest version of boot is

micha13:10:55

so i hardcoded 2.3.0 in there

micha13:10:15

so it gets 2.3.0 and then runs boot -u

micha13:10:26

the boot.jar contains the logic to find the latest version

micha13:10:43

i wanted to keep boot loader as dumb as possible

martinklepsch13:10:45

we could ship the loader with the release version whenever we do a new one though?

martinklepsch13:10:09

i.e. with 2.4.3 we could tell it to download 2.4.3?

micha13:10:10

we won't be updating the loader after we work out the bugs

micha13:10:17

what's the point?

micha13:10:24

stability is key with the loader

micha13:10:43

if you set BOOT_VERSION when you call boot for the first time it will get that version

martinklepsch13:10:47

Agree, it’s mostly that it’s confusing to show App Version 2.3.0 when I just downloaded 2.4.0

micha13:10:50

instead of 2.3.0

micha13:10:28

yeah maybe update to 2.4.2

micha13:10:47

versions of boot.jar after 2.3.0 don't show the app version anymore

micha13:10:52

so that would be less confusing

micha13:10:09

ok i'll tack it onto my PR

pandeiro13:10:34

Is there anything in the wiki about this (:directories boot.pod/env) or is this implementation detail stuff? I'd it safe to use in libs?

alandipert13:10:54

@pandeiro: it's safe to use

onetom14:10:31

> I just built boot with boot built by boot. "The Singularity Is Near" simple_smile

alandipert14:10:18

hey @nberger i was chatting with micha about the test task and he had a really cool idea about how it could work

alandipert14:10:45

that might be the key to doing notification in boot generally, like we may want to start using the pattern for other notifiers also

micha14:10:13

@martinklepsch: how do we want to handle the boot.sh for homebrew?

micha14:10:21

i saw in the recipe there is a shasum

micha14:10:52

the boot.sh thing isn't really versioned

micha14:10:14

i guess we can make tags for it in github, which will allow us to upload binaries to gh releases

micha14:10:47

we can name the tag as the shasum of boot.sh actually

martinklepsch14:10:50

@micha: Why not just do it like before?

onetom14:10:21

why is it not versioned?

onetom14:10:20

because it's in one source repo with the boot.core lib and what boot.core version gets bumped the boot.sh might not necessarily be changed, so it's confusing to label the "exact same" binary with a different version? (or something like that?)

micha14:10:08

yeah, the loader doesn't have anything to do with boot versions

micha14:10:17

it has its own totally separate version

micha14:10:33

we don't want to release a new version of boot just to update the loader

micha14:10:55

that's why i like the shasum

micha14:10:06

it doesn't imply any relationship to boot version

micha14:10:33

also @martinklepsch your boot-builds-boot build.boot file is incredible

micha14:10:49

many :thumbsup:

onetom14:10:06

so older loaders would still be compatible more recent boot apps/libs?

micha14:10:45

it doesn't know anything about boot other than that there is a boot.App class with a main() method

micha14:10:52

it has no idea what any of that does

micha14:10:06

so any version of the loader is compatible with any version of boot

micha14:10:13

but some versions of the loader have bugs

onetom14:10:46

(i havent realized until now that the loader is another component in the game and it's different from the boot app itself... but then again, im not fluent at all in java to understand such internals. yet.)

nberger14:10:47

@alandipert: I'm interested, how would that work?

micha14:10:08

@onetom: basically the only reason to change the loader is to fix a bug, never to add functionality

alandipert14:10:29

@nberger: the thing to consider is the fileset returned by the next task

onetom14:10:39

@micha: forever 1.0 simple_smile

alandipert14:10:18

imagine the test task adds a test report to the fileset, and passes it to the next task. now, imagine it also retains the fileset returned by this call. then it looks inside the returned fileset - if the report is there, it does something with it, like throw or print

martinklepsch14:10:44

@micha: what do you think about a separate repo for the loader then different versioning schemes would clearer separated

alandipert14:10:58

@nberger: now, downstream tasks can choose what ot do with the report, and whether or not to "bubble it back" to the task that added it

micha14:10:01

martinklepsch: yes, that's the solution

micha14:10:21

i don't see any reason why that wouldn't work

micha14:10:33

and you could then have incrementing versions for homebrew

martinklepsch14:10:35

@micha: thanks for the praise, I’m happy about any suggestions (style, idiomatic, …) simple_smile

micha14:10:52

dude it looks amazing

micha14:10:59

i have nothing to add to that

nberger15:10:05

@alandipert: I think I'm following you, but I'm not sure how it helps to solve the clojure.test + junit scenario. One important thing to note on that scenario is that the tasks needs to fail (not (zero? exit-status)) but also target needs to be synced, because we want the junit reports to be in the fileset but at the same time we are especially interested in getting the reports when the build fails

martinklepsch15:10:15

@micha: I’m working on a change that checks if the current commit already has the given tag and does not try to create it if so

martinklepsch15:10:25

@micha: does that make sense to you?

micha15:10:50

i think so

nberger15:10:04

@alandipert: but yes, I like the two-stages task idea, it helps to make the code more clear and extensible

martinklepsch15:10:03

@micha: maybe we wait with the split until after we merge boot building?

nberger15:10:23

It's still that I can't see how to fix the problem of putting a report into the fileset and expect it to be materialized into target-path, even if the pipeline fails. I was thinking of throwing a specific exception that would be recognized by boot and in that case the target would be synced

alandipert15:10:24

@nberger: ah, i see what you mean now - we need the test report no matter what

alandipert15:10:01

boot test; [[ -e "target/failures.xml" ]] && exit 1 maybe?

nberger15:10:03

That's right. The junit report is of value when the tests fail

alandipert15:10:26

i'm not sure we can support both failing and not failing

alandipert15:10:38

and would need t orely on an external check of failed-ness in this case

nberger15:10:08

Checking the existence of failures.xml might be brittle I think... and it makes so that we have to be careful to not generate the report when there are no failures, which is not always desired

alandipert15:10:57

another unsavory option that comes to mind that @micha will hate is osme kind of after-sync hook... where we could exit1

nberger15:10:31

Both fail but no fail... I see the issue there... I was thinking of relaxing the rule of "when we fail, we don't sync the target". It would be opt-in of course, the default would always be to not sync it

alandipert15:10:51

it's kind of like we want a "finally" clause for boot itself that syncs the fs

nberger15:10:39

yes... one pesky detail is that the task that throws must send the resulting fileset as part of the exception?

alandipert15:10:51

although... soon, there will be an explicit target task

alandipert15:10:15

which would sync before returning the fileset

alandipert15:10:35

or at least, it could

nberger15:10:50

I'll keep thinking on this task. In the meantime I'll use the simplified version that outputs right to the filesystem instead of the fileset. Let's see if that target task helps, or we'll arrive to a solution one way or another simple_smile

jgdavey15:10:27

Credit due to @juhoteperi for the saapas example. I started with that build.boot and modified as necessary.

chrisn15:10:22

Have you guys ever seen this:

clojure.lang.ExceptionInfo: java.lang.NoClassDefFoundError: org/apache/maven/wagon/Wagon, compiling:(NO_SOURCE_PATH:0:0)
    data: {:file "/tmp/boot.user6112313111468063070.clj", :line 15}
clojure.lang.Compiler$CompilerException: java.lang.NoClassDefFoundError: org/apache/maven/wagon/Wagon, compiling:(NO_SOURCE_PATH:0:0)
         java.lang.NoClassDefFoundError: org/apache/maven/wagon/Wagon
       java.lang.ClassNotFoundException: org.apache.maven.wagon.Wagon
                                  ...
                    clojure.core/eval                          core.clj: 3081
boot.aether/load-wagon-mappings/fn/fn                        aether.clj:  193

chrisn15:10:41

only happens on build machine of course. Not on my machine, not on the sysops' team machines, etc. Only on one machine.

alandipert15:10:28

@nberger: sounds good. thanks again for helping think it through

nberger15:10:08

@alandipert: thank you for the help

chrisn16:10:55

what jvm do you guys run, out of curiosity?

chrisn16:10:39

Is there a way to get boot do dump the entire classpath it is providing to any given pod?

chrisn16:10:44

-vv doesn't seem to do this.

martinklepsch16:10:47

@chrisn: running 1.8. Don’t remember seeing your issue. Do you always get this?

chrisn16:10:29

On exactly one machine yes.

chrisn16:10:49

I deleted m2, I have apt-get removed packages, and switched to the oracle jvm 1.8.

martinklepsch16:10:28

what’s boot -V? or does that also not work?

chrisn16:10:28

I wonder if I can alias java to java -verbose:class

chrisn16:10:17

[email protected]:~/workspace/pi-build-catalog$ boot -V docker-target
#
#Wed Oct 28 16:14:38 GMT 2015
BOOT_CLOJURE_NAME=org.clojure/clojure
BOOT_VERSION=2.4.2
BOOT_CLOJURE_VERSION=1.7.0
#App version: 2.4.2

martinklepsch16:10:42

@chrisn: tried running built in tasks I assume?

chrisn16:10:02

nothing will work, like show -U

martinklepsch16:10:39

hrm. you could try rm ~/.boot/cache but I don’t see how that would change something

chrisn16:10:30

The jvm isn't lying, if you set the boot_jvm_options to be:

-verbose:class
then grep -i for wagon you only get aether stuff.

chrisn16:10:45

So for some reason on this exact machine the classpath isn't getting setup correctly on the pod.

chrisn16:10:18

While it is getting setup correctly on every other machine. Also, lein deploy will work on this machine for this project meaning it is using our private s3 repo from leiningen.

jfntn17:10:55

I’m setting up a project that mainly consists in generating clojure source files from json files. I have a direct mapping of json-file -> processing-pipeline -> output clj. I’d like to define a task for this whole process, but I’m not sure if I really need to work with filesets for this, since I have this pre-determined wiring of inputs to outputs?

alandipert17:10:06

@jfntn: sounds like a perfect case for writing a function that's fileset in, fileset out - a task

martinklepsch17:10:37

BAM build passes!

jfntn17:10:53

@alandipert: so I’d just write the files to a hard-wired path in my src dir, and then commit! the fileset?

alandipert17:10:59

more like, add the directory containing your json as a source-path, and then run your task that does the transformation

alandipert17:10:23

@jfntn: oh wait... you want to feed the files back into your project code?

martinklepsch17:10:00

jfntn: do you want them on the classpath or what is the ultimate thing you want to do with these clojure files?

jfntn17:10:02

the json dir is added in the resource-paths already

jfntn17:10:39

I could do without adding them to the class path, ultimately there will be another task that will jar up the generated code

martinklepsch17:10:52

perfect boot scenario! 👍

jfntn17:10:14

Glad I figured this much 😉

martinklepsch17:10:46

@jfntn: basically you write those generated clojure files to some tmp-dir! (see above), add this dir as resource via add-resource and then call the jar task afterwards

martinklepsch17:10:51

@jfntn: out of curiosity - what kind of project is this? json -> clojure sounds interesting 😄

jfntn18:10:00

A bunch of HTTP helpers generated from https://github.com/for-GET/know-your-http-well and some RFCs too if I can some clean xml files 😛

alandipert18:10:27

@jfntn: https://github.com/boot-clj/boot/wiki/Filesets#examples sounds very close to what you want, just need to bring your own compile-lc!

jfntn18:10:46

ok great, that should be enough to keep things moving, thanks @alandipert, @martinklepsch !

raywillig18:10:50

haha @alandipert were you born yet when that email was written? btw is CL finished yet?

alandipert18:10:09

no & yes, but they didn't listen to stallman

chrisn19:10:31

Still working on the classloader issue I have on one random machine. What I would like to do is log clearly every time add-classpath is called in pod.clj. I guess other relevant information would be which classloader it is getting from the current thread or some identifying information about the classloader stack. Is there some conceivable reason that either the correct dependency information for the s3 plugin isn't getting added to the pod or the classloader is ignoring the add url command?

chrisn19:10:50

Or any other scenario we can think of where the wagon dependency magically isn't getting found?

chrisn19:10:33

For reference, when it works we get:

[Loaded org.apache.maven.wagon.Wagon from file:/home/chrisn/.m2/repository/org/apache/maven/wagon/wagon-provider-api/2.2/wagon-provider-api-2.2.jar]

chrisn19:10:32

maybe someone installed the jboss classloader and now things are just utterly jacked?

alandipert19:10:07

@chrisn: i wonder if the maven repo on the build machine is corrupt somehow

micha19:10:20

chrisn: can you paste the md5sum of your boot.sh please?

alandipert19:10:21

@chrisn: you could test by setting BOOT_LOCAL_REPO to something different

micha19:10:23

just to make sure

chrisn19:10:04

I got locked out of the machine so my testing for the day is over 😞.

chrisn19:10:42

I will try those things. I had completely deleted the ~/.m2 directory before though.

chrisn19:10:51

That was one of the first things I tried.

pandeiro20:10:21

very cool discovering :(directories boot.pod/env) thanks @esp1 @alandipert

pandeiro20:10:41

i'm amazed i never looked at that before but it's a testament to how well boot manages this that i never had to

martinklepsch21:10:06

I’m excited! I want to merge this now! Who wants to test? simple_smile simple_smile

micha21:10:45

i'm in the middle of some stuff here at work, but i can test later

micha21:10:06

don't let that stop anyone else from testing now though simple_smile

micha21:10:26

super excited about it though

micha21:10:49

how is the workflow with --watch enabled?

micha21:10:57

does it iterate pretty fast?

micha21:10:36

also now that we have a sweet boot build we can make tests and stuff

micha21:10:41

integration tests

pandeiro21:10:09

wow congrats

martinklepsch21:10:19

Workflow with —watch works pretty well mostly. Sometimes I get things like java.lang.Exception: can't resolve symbol (boot.jar/spit-jar!) during the first uberjar build, don’t know what’s going on there.

pandeiro22:10:51

when invoking boot tasks with edn opts, it's necessary to do :some-edn-opt (pr-str {:a true :b [1 2 3]}) even inside e.g. build.boot?

pandeiro22:10:37

ok apparently not, cool. must be something else simple_smile

pandeiro22:10:48

it's illegal to do (apply some-task {:some "a" :args "b"}) ? is there another way i can call a task with an indefinite map of opts?

martinklepsch22:10:14

👍 good question, wanted to do that too the other day

alandipert22:10:23

@pandeiro: try (apply some-task (mapcat identity {...}))

alandipert22:10:34

the problem is the map seqs into a seq of pairs

martinklepsch22:10:53

oh that makes sense!

alandipert22:10:55

(apply some-task {:x y}) == (apply some-task '([:x y]))

pandeiro22:10:50

ah wow thank you guys 👍

pandeiro22:10:57

totally baffled by that

jfntn23:10:01

Circling back to my earlier code generation questions, is there a way I can commit new files to the project /src/ tree too?