Fork me on GitHub
#boot
<
2016-03-17
>
georgek04:03:19

Hi everybody (and specifically whomever said hi earlier in the thread) simple_smile

onetom11:03:34

@seancorfield: is it normal that the output of boot-expectations includes this exception info?

Ran 1 tests containing 1 assertions in 20 msecs
1 failures, 0 errors.
clojure.lang.ExceptionInfo: Some tests failed or errored
    data: {:test 1,
           :pass 0,
           :fail 1,
           :error 0,
           :run-time 20,
           :ignored-expectations 0,
           :type :summary}
              clojure.core/ex-info/invokeStatic               core.clj: 4617
                           clojure.core/ex-info               core.clj: 4617
seancorfield.boot-expectations/eval151/fn/fn/fn  boot_expectations.clj:   65
...

onetom11:03:06

also it takes ~3s to rerun the tests when i only have 1 sample check (expect nil? nil)

onetom11:03:32

adding :size 4 option to the pod-pool decreased subsequent runs to ~1.8s, which is still too high for a fluid TDD flow, plus doesn't make sense. what takes so much time when i have a warmed up pod?

onetom12:03:12

hmm... 2 pods seem to be initialized after every run of the expectation task (i can see the requires happening twice if i put a (println "requiring into test pod:" r) before (pod/require-in fresh-pod r) in seancorfield.boot-expectations/init)

chrisn13:03:46

Hi everyone, we have been using boot to help build our docker files with some success but I would like to know a better way and use boot more correctly. The goal is to build docker files where the rapidly changing jars are on one layer and the rest of the dependencies are on another layer. The boot file lists the target jar as the only dependencies and then I use the uber command with as jars. I finally (in another command) do some magic to go through the jars that the uber command found and copy different sets into a Dockerfile so when we run docker build it puts things on the correct layers. This approach has some serious potential because I can potentially make docker images out of any jar in maven just by pointing to it and knowing the launch command I need to use....

chrisn13:03:54

I feel like this is a good direction but very hacky and I get warnings and such with latest boot.

chrisn13:03:23

Is there a better approach? In this case the filesystem abstraction works against me because I need the files on disk else I need to do something else; this is the primary reason I split it into two different commands.

chrisn13:03:57

Really I would like everything including building the container (and finding native depenencies and such done in one place in a library we can just include and use.

chrisn13:03:09

But in order to build that I think I need to understand boot a lot better.

dm313:03:11

why do you need the files outside of the fileset? (they are still on disk, even if inside the fileset)

onetom13:03:04

@chrisn: what's the point of separating the rapidly changing jars from the rest of the deps? docker handles updates very efficiently anyway, no? it does a binary diff and all that, iirc.

chrisn13:03:54

No it does not do a binary diff.

chrisn13:03:31

If, for instance you have an uberjar you can look forward to a very large upload every time you build a docker image.

chrisn13:03:36

@dm3: I specifically need to do things like unzip the files into other files and such for native dependencies.

chrisn13:03:05

They may exist on disk but they aren't committed to the target directory.

onetom13:03:09

wait, u r just building the uber jar to collect all the dependencies?

chrisn13:03:33

Yes, that was the only way I could figure out to get boot to place all the dependencies into one directory.

chrisn13:03:53

Another option would be to pull them from m2 all the way into the docker image.

onetom13:03:59

have a look inside the uber task maybe

onetom13:03:21

and also there is a BOOT_HOME env var where u can specify where should the local maven repo be

chrisn13:03:40

what advantage would that have?

onetom13:03:56

sorry, this one: > BOOT_LOCAL_REPO The local Maven repo path (~/.m2/repository).

onetom13:03:24

u wouldn't need to create an uber jar and peek into it

chrisn13:03:42

I am not peeking into it, I use a command :as jars that just copies the jars locally.

onetom13:03:09

u just run something which pulls down all the dependencies and u will have it under the folder u specified in BOOT_LOCAL_REPO

chrisn13:03:44

Not flattened into one directory. This would make the java command harder correct?

onetom13:03:50

the point is to gather only the jars which are necessary, right?

chrisn13:03:56

has a different complete path for each jar.

chrisn13:03:06

.m2 has a different complete path.

chrisn13:03:35

currently my java command in docker is java -cp "/srv/*" main-class-name.

chrisn13:03:51

Yes that is the point.

onetom13:03:23

what's the reason for not using boot as your docker command?

onetom13:03:51

u would just need to prime the local maven repo and it wouldn't go out to the network then

onetom13:03:06

at least that's how adzerk is doing it if i understood well

chrisn13:03:18

correct that may have some side effects given attempting to run generic jars.

chrisn13:03:47

but that definitely an option; the docker build would be to get boot to gather the dependencies into the docker image and then off you go.

chrisn13:03:00

Although then everything would be on one layer and you have the original problem again.

onetom14:03:02

okay, i still don't understand the original problem then. you said your problem is that u would need to upload a full uber jar every time u change something small in it

onetom14:03:17

because docker wouldn't do the diffing

chrisn14:03:34

Do you understand docker layers?

chrisn14:03:40

Each docker command generates a new layer.

chrisn14:03:50

docker is capable of caching layers.

chrisn14:03:01

alone. An entire layer is either same or different.

chrisn14:03:10

we good so far?

onetom14:03:39

though u can specify a pipeline of commands which would produce a layer, right?

chrisn14:03:47

K. Now, lets say I have one 75K jar I am working on and 100MB of dependency jars which isn't unlikely.

chrisn14:03:41

What I want is the dependencies on one layer and the target jar (or set of jars) on another layer. Furthermore I want native dependencies placed into a particular directory so that an LD_PRELOAD command can work.

chrisn14:03:31

So first you need to, given the set of jars, separate rapidly changing jars from the rest and pull out native dependencies.

chrisn14:03:53

Then build your layers and such.

onetom14:03:58

so @juhoteperi was doing something around this problem domain. iirc, he was running a boot show -d for example to get all the deps (which unfortunately hasn't included the pod-only deps). that could be your dependency layer.

onetom14:03:41

and your "app layer" would be just boot prod

onetom14:03:08

and boot will handle constructing the classpath for u

onetom14:03:48

but im not the expert here as u can tell simple_smile

chrisn14:03:10

Haha, thanks for taking some time to figure this out with me.

onetom14:03:11

i just hope i can spark your imagination until some real experts pop up

chrisn14:03:29

Sure, and I do appreciate that.

onetom14:03:41

im just trying to give back, since i've learnt so much here

chrisn14:03:41

I feel like the main problem is that I am not using the fileset after the uber task but exiting and restarting boot.

chrisn14:03:11

It seems like if I would do the processing on the fileset produced after the uber task then I could do everything on one task except build the docker image.

chrisn14:03:15

This would help.

chrisn14:03:01

I currently use java.file.listFiles or something like that and I could do an analogous operation on the current fileset.

onetom14:03:07

@chrisn: but have you looked into the uber task yet? it's just effectively 32 LoC: https://github.com/boot-clj/boot/blob/master/boot/core/src/boot/task/built_in.clj#L424-L512

onetom14:03:55

try to run boot pom uber show -f to see what does it do in the fileset

onetom14:03:17

that way you can just use the sift :move task to move the jar files into the output fileset, which you can then dump into the (set-env! :target-paths #{"somewhere"}) using the (target) task

onetom14:03:28

@seancorfield: my main question above has been solved; the expectations task is fast now. see my PR https://github.com/seancorfield/boot-expectations/pull/11

onetom15:03:17

@micha: would it make sense to have boot.notify accept a msg instead of just a number? that way for example boot-expectation could say "2 failures and 1 error", or something like that.

seancorfield17:03:58

@onetom: Yeah, we’ve had several discussions about the error handling / exiting strategy in Boot. I identified four types of exit that might be needed and we only have two right now (success and exception).

seancorfield17:03:19

I saw the PR notice, haven’t yet looked at it … will try to get to that today …

juhoteperi17:03:35

@onetom: I'm not longer running show -d to retrieve the deps, instead I just run boot run-tests or whatever in CircleCI depdencies step, this way even the dynamic deps from tasks will be cached ny CircleCI

dominicm18:03:35

I just got a basic asciidoc boot task working

dominicm18:03:27

For now it just lets me do a form of literate programming. But I hope to get a ton of options in there eventually.

dominicm18:03:34

Perhaps I'll wrap it first.

dominicm18:03:44

But still, might be fun to use with perun

martinklepsch18:03:14

Yeah, there was some work in that direction but was never finished/merged

dominicm18:03:59

I saw that, calling out to jruby directly, but asciidoctor has a pretty good java library (that uses jruby for you!)

kenny19:03:10

Hi. Is it possible to programmatically load an arbitrary build.boot file and run get-env in that file?

micha19:03:40

you can use clojure.core/load-file to evaluate the forms in the build.boot file

kenny19:03:30

Right but I need to get a key off the boot env through get-env. How would I run that in the context of the arbitrary build.boot?

micha19:03:53

what do you mean by "the context" there?

micha19:03:06

build.boot is just a file that contains forms to be evaluated

micha19:03:15

it's a program

kenny19:03:41

Yes but how does get-env know which build.boot I am referencing? I am already running a boot project. It's like I need to switch the ns but this needs to be done programmatically, not in the repl.

micha19:03:02

there is only one env, though

micha19:03:22

the build.boot file is just a program

micha19:03:33

it's not a build specification or anything like that

micha19:03:49

it's just a sequence of clojure forms to be evaluated

micha19:03:33

when boot starts it does (load-file "build.boot")

micha19:03:00

it's like eval, but for forms contained in a file instead of in memory

kenny19:03:45

Hmm.. This idea may not work then. Here is some context on what I am trying to do. I am writing a local build server (similar to lambdacd but using boot tasks). Any time a change is detected in a monitored git repo the build server will clone the project, and execute a boot task in that project (e.g. By running boot build in the command line in that directory). Rather than having the task be hard coded in my builder I would like it to be optionally specified in the boot env for that project. But I need a way to get that value from that project's boot env.

kenny19:03:20

So then if I require boot.core in my code, run load-file on a arbitrary build.boot, and call get-env, how will it know which build.boot file to use, assuming that multiple previous projects' build.boot files have already been loaded.

kenny19:03:36

So it's almost like I need to run load-file in some sort of temporary clojure ns

micha19:03:46

wait so why not have a edn file in the repo that says which task to use?

micha19:03:53

or something like that

micha19:03:03

or a convention

micha19:03:22

you could make your own boot task you deploy to maven

micha19:03:28

and people could just use that

micha19:03:41

instead of having a yml file like travis uses

kenny19:03:45

I thought about that but I'm not sure if every project should drag around another file

kenny19:03:01

It seems more convenient to just write it in the boot env. What would you prefer?

micha19:03:14

i wouldn't use the boot env for that, really

micha19:03:31

the boot env is a data bag of unmanaged shared state

micha19:03:08

why not have your thing just calla specially named task that people would implement in their build.boot?

micha19:03:27

then they could do anything

micha19:03:39

you just provide the name of the task that they will need to implement

kenny19:03:16

I thought about that too but then you get the problem of overriding. For example at my company we have a project that has a whole bunch of default boot tasks that are nice to have in every project. In that project we have a boot task called build which just does boot pom build-jar, if I remember right. Now if I have a new project that requires that boot task project in its build.boot, I can't make another task named build without using replace-task!. Just a little bit of annoying boilerplate.

micha19:03:41

you'd name it something distinctive

micha19:03:39

you have the same problem with the env anyway, since you need to have a specially named key

micha19:03:56

you can just name the special task whatever you would have make the name of the env key be

kenny19:03:42

Right but then in that distinctive task you would need to call the build task in every project. Seems more effective to just have a specific task set if needed.

kenny19:03:23

So if the build-task is not set in the boot env for the cloned project then the builder will just run the default boot build

kenny19:03:48

Otherwise it will run the specified task

kenny19:03:02

Maybe it is better to explicitly force the user to specify how he/she wants that project to be built rather than making any assumptions about the project

micha19:03:27

i don't understand what you mean there

micha19:03:37

i'm a little disconnected today

micha19:03:12

you're going to be building projects that people will be adding a marker for you, to help you know how to build it, right?

micha19:03:23

like they put your special key somewhere with a hint

micha19:03:29

is that right?

micha19:03:56

so instead of putting the special key, why not put a specially named task?

micha19:03:04

cut out the middleman

micha19:03:23

the boot env is not really meant to be a databag

micha19:03:39

it's where the JVM state is kept

kenny20:03:31

So then maybe having an edn file would be better. The reason why the specially named task wouldn't work the best is because you may need to override the task. For example, say the specially named task was called auto-build. Also say that you are in a similar situation where you have a project with a collection of frequently used boot tasks. In that collection of frequently used boot tasks is one called build. 80% of projects are built by simply using the build task. So in order for the specially named task to work I would need to call the build task in the auto-build task which would need to be done in 80% of my projects. That seems like a lot of boilerplate. I was thinking why not have the default be to run boot build, however, if the project has specified a different task to build the project then use that task instead.

micha20:03:02

i would call it, for example, my-service-build

micha20:03:18

where my-service is the name of your service, which is presumably relatively unique

micha20:03:48

like if you were travis CI you could use travisci-build

micha20:03:06

you can put a map of metadata at the top of the build.boot file

micha20:03:24

and do (clojure.core/read-string "build.boot")

micha20:03:31

that would give you the first form

micha20:03:37

if it's a map then you could look in there

micha20:03:18

basically the edn file approach, but without a second file

kenny20:03:36

At that point might as well do the edn file rather than pollute the build.boot

micha20:03:16

definitely seems preferable to loading the build.boot in a pod or whatever

micha20:03:32

personally i would perfer to have the CI service just run a shell script

micha20:03:38

i don't know why they don't do that

kenny20:03:20

But why would you want to write your build steps in shell 😬

micha20:03:42

that would be the simplest thing for me

micha20:03:52

install java, download boot, whatever

micha20:03:05

run the boot script

micha20:03:28

that kind of thing is most easily done in the shell, no?

kenny20:03:12

I guess it depends on your env. All that stuff is already done for me because we use docker.

micha20:03:33

then your shell script would be very simple simple_smile

kenny20:03:41

Just pull the docker image and wholah!

micha20:03:53

but would still support more complex things without inventing a whole new paradigm

micha20:03:31

i am not a fan of like .travis.yml at all

kenny20:03:50

It wouldn't be hard to add a shell script to build the project though

micha20:03:10

it's the common denominator of UNIX systems

kenny20:03:20

It seems like your use case for shell scripts is more for a deploy than building though, true?

micha20:03:35

for anything that needs to be an abstract interface on a unix system really

micha20:03:47

just have the thing call a named program

micha20:03:55

that program can be a shell script or a c program whatever

micha20:03:00

it will figure out what to do

micha20:03:21

and the system should put all the context my program will need in the environment or command line arguments

micha20:03:31

and it should log stdout/err

micha20:03:49

that's the lowest common denominator on Unix

micha20:03:09

and it supports even the most complex requirements, because the shell script can do anything

micha20:03:20

as little or as much as you want

kenny20:03:07

That is true but I think the use case for my project may be different than the one you are thinking. Though I can see it heading toward that in the future. My project is essentially trying to solve the problem of missing dependencies when pulling a new project from github. For example, my company has lots of private repos that we don't want on clojars/maven/etc and we dont want to maintain a build server. So instead we want a local continuous delivery pipeline to be always running on each of our computers. This will pull and build all the deps from our private repos so that we you actually open a project that uses 20 different private libraries you wont spend 20 mins pulling all the projects from github and building each one individually.

micha20:03:59

have you considered pushing artifacts to S3 for example?

micha20:03:28

we've had good results with that so far

kenny20:03:53

We did that at my previous company and it sort of worked. I distinctly remember consistently getting a stacktrace printed when a project did not exist in S3 which was annoying. Anyways, I did think of doing that but there are a couple reasons I wanted to avoid it. One is that the developer still has to remember to deploy it to S3. By using my project we eliminate that problem because the developer only has to remember to push his code. Secondly we are very protective of IP so the less places IP exists, the better.

kenny20:03:12

So essentially my project is eliminating the developer from needing to run a deploy task every time he wants the lib available to other developers. In fact, if a dev forgot to run the deploy task I imagine you would find yourself in a very similar situation where you would need to pull and manually build possibly several projects.

micha20:03:56

interesting

micha20:03:16

we don't really need to update multiple things when we update a dependency

micha20:03:34

the things that are already working will continue to work, usually

micha20:03:21

if we're making a non-compatible change we have some more involved procedures we use anyway, which makes the issue of forgetting something a moot point

kenny20:03:20

Ah. Yeah that makes sense for a mature product but we are still developing

kenny20:03:47

May I ask how big is your team? >5?

micha20:03:29

yes we have 5 plus 1 part time

micha20:03:20

sorry, 6 plus 1

micha20:03:55

i'm working remote today so i had to think about it

kenny20:03:44

😛 Thanks for the help!

micha20:03:03

if you still want to put data in the boot env you can use boot show -e to have boot print that edn

micha20:03:23

like have your process run that in the repo

micha20:03:32

and slurp and read the output

kenny20:03:03

Ohh that's very interesting.. Though now that you've mentioned it I'm not sure that it is the right place for it