Fork me on GitHub
#boot
<
2016-04-01
>
richiardiandrea01:04:15

boot.parallel has come to life, that patch is becoming huge 😄

dave14:04:05

this is amazeballs

jamesleonis16:04:47

It’s also not GIFs or other images. It’s straight text animation with javascript!

richiardiandrea18:04:53

I created boot.parallel and moved the primitives there

richiardiandrea18:04:19

with boot.test/runtests as entry point for parallel testing, there is a video above

richiardiandrea18:04:00

of course it should be considered an alpha because...well...it is not been battle tested

martinklepsch18:04:01

@richiardiandrea: nice stuff! I wonder — is this also meant for stuff like compiling cljs & sass in parallel? i.e. does it have fileset-merging capabilities?

richiardiandrea18:04:29

boot.parallel has all you need for that yes, it should correctly pass the files through

richiardiandrea18:04:52

I kind of pass the fileset around in my tests

richiardiandrea18:04:19

but not actually tested it very carefully 😱

richiardiandrea18:04:14

a test framework not tested is like a cake without icing, but I will probably dedicate more time on it once the warning on deftask is fixed

richiardiandrea18:04:29

which really bothers me (see video)

richiardiandrea18:04:39

ok @martinklepsch actually scratch that

richiardiandrea18:04:07

the fileset is not passed around at the moment

richiardiandrea18:04:20

as boot in boot is executed in a future and a with-pass-thru

richiardiandrea18:04:00

there is the concept of sync-map a HashMap of shared data that is passed down pods

richiardiandrea18:04:33

so if you add your fileset there (it should contain Java data structures only)

richiardiandrea18:04:54

it will be passed around and read at the end of the parallel computation

richiardiandrea18:04:04

from boot.pod/data

richiardiandrea18:04:12

why all this? Because we are executing boot in boot in pods and pods' only certainty is that they have the JDK loaded

nberger18:04:21

even if it doesn't have fileset-merge semantics, this sound like a great start!

richiardiandrea18:04:45

it is achievable with the above workaround yes 😄

nberger18:04:04

oh wow even better!

richiardiandrea18:04:10

because tasks are executed serially

richiardiandrea18:04:25

we are kind of breaking the chain and execute in parallel

richiardiandrea18:04:49

it can be improved of course

richiardiandrea19:04:19

but I this was what I came up with together with Micha so I am pretty happy for now

alandipert19:04:10

watched the vid, can see why the override warning is annoying now

richiardiandrea19:04:50

yeah, I am adding an env var as we speak

richiardiandrea19:04:16

I did investigate why it happens

richiardiandrea19:04:57

and it looks that boot in boot calls boot.main-main and that requires the namespaces

richiardiandrea19:04:57

every time, not on clean slate

richiardiandrea19:04:21

while the core pod should be always a fresh one

richiardiandrea19:04:36

Actually @alandipert there is a thing that might be the cause

richiardiandrea19:04:09

the call to load-file on the script here

richiardiandrea19:04:30

because I am should have everything loaded the first time, script (and its requires in it)

richiardiandrea19:04:44

I will open a PR so that we don't lose this and go to lunch 😄

micha19:04:06

@richiardiandrea: the problem with running things like sass and cljs in parallel is that you'd have to do them in separate boot instances

micha19:04:22

because there can be only one classpath per pod

micha19:04:01

so if you have one boot and two tasks are manipulating filesets at the same time, what happens when they both call commit!, which one wins?

richiardiandrea19:04:37

yes things need to be changed in order to address the parallel execution

micha19:04:46

not calling commit, or not writing the fileset to disk results in a corrupted fileset

micha19:04:33

because paths in the fileset are pointing to placees where a file should be but doesn't exist

micha19:04:54

and the classpath would be wrong too

richiardiandrea19:04:12

so fileset are immutable, which is really great, but commit! is not thread safe

micha19:04:30

right, because the classpath isn't thread-local

richiardiandrea19:04:02

so the next step is to make commit! thread safe simple_smile

micha19:04:23

i'm not sure that makes sense

micha19:04:31

or rather, i can't think of how it would work simple_smile

richiardiandrea19:04:51

yeah, it was just a sentence thrown there

richiardiandrea19:04:00

maybe, attentatively, in a parallel task you can keep track of fileset diffs as they come and at the end return the result

micha19:04:24

still you have the classpath issue

micha19:04:44

like if you have 6 tasks

micha19:04:08

A -> [ (B -> C) (D -> E) ] -> F

micha19:04:15

where the parens are done in parallel

micha19:04:32

B adds things to the classpath via the fileset and calls commit!

micha19:04:49

D also calls commit after modifying the classpath

micha19:04:55

who wins?

richiardiandrea19:04:59

yes that wouldn't work

richiardiandrea19:04:24

I thought pods had a completely separated classpath

alandipert19:04:58

i guess it's the classpath mergers problem like one has with uberjars

richiardiandrea19:04:24

another thing, I don't know very well in which way sass modifies the classpath atm

richiardiandrea19:04:56

but in general the problem is clear, if only we could reify these changes, then you can pass them among pods and down the computation...for then collecting the results in boot.pod/data

alandipert19:04:44

by reify changes you mean maintain a log of classpath additions?

richiardiandrea19:04:27

maybe, just brainstorming here 😄

micha19:04:51

hm yeah if we could make a pod and initialize its classpath with the incoming fileset

micha19:04:55

then do work

richiardiandrea19:04:56

but yes it is not straightforward

micha19:04:04

then merge the outgoing filesets

micha19:04:15

yeah there are tricky things too, like the repl

richiardiandrea19:04:44

but executing a repl in parallel would be odd 😉

richiardiandrea19:04:33

some hammock time is needed...

alandipert19:04:36

i can imagine playing the log for 2 divergent branches over a clean pod, consulting a user-supplied merge-fn as necessary

richiardiandrea19:04:00

yes that's btw where all the tools are converging, see onyx

richiardiandrea19:04:17

the log model is super powerful

richiardiandrea19:04:27

but ok, I also opened an issue for the deftask warning and that's really a thing I have to solve or parallel tests will be very noisy

micha20:04:02

you know, i think the parallel tasks thing could work

micha20:04:24

probably some of the issues are just due to inelegant code

micha20:04:30

in boot.core

richiardiandrea20:04:32

Well now at least we can try it out ;)

micha20:04:39

did you see the with-pod macro in 2.6.0-snapshot?

micha20:04:49

that might be useful for the parallel task thing

micha20:04:01

because you can pass tasks into the pod with that

micha20:04:14

it's like with-eval-in, but without a lot of the restrictions

micha20:04:18

like you can do this:

micha20:04:40

(defn doit [x] (+ x 1))

(with-pod p
  (def y 100)
  (~doit y))

micha20:04:45

and that will produce 101

richiardiandrea20:04:41

Oh wow great but I think the restrictions i have are mostly in using boot.pod/data

richiardiandrea20:04:53

Where no Clojure is allowed

richiardiandrea20:04:57

And the second thing is that runBoot passes through a main that was not really thought for non-cli calls

richiardiandrea20:04:30

But I think these are just rough edges, the functionality is there ;)

micha20:04:32

we wouldn't be using runboot for the parallel tasks thing though

micha20:04:36

just a pod

micha20:04:30

it would involve some fancy footwork with directories on the filesystem for the classpath, but i think we could probably do it

micha20:04:51

if we could get the parallel tasks working then there would really be no advantage the dependency graph architecture has over the pipeline

richiardiandrea20:04:09

Ah, yeah if we can isolate a pure pod exexution is better, now the user script is loaded for every runboot for instance, which is not good (see last issue)

micha20:04:38

yeah agree

kenny21:04:28

Hi. I am making a docker with boot pre-installed. Is it possible to download all the deps for a task (e.g. push requires s3-wagon-private which has all its internal deps)?

alandipert21:04:44

@kenny: the best way is to run the task in your docker build

alandipert21:04:03

well, that's what i do

alandipert21:04:07

there might be better ways

kenny21:04:04

@alandipert: But you can't run the push task without a repo

kenny21:04:24

It will throw repo not found exception if I put a dummy repo in

micha21:04:53

@kenny: just adding wagon-s3-private to the :wagons in set-env! should be enough to fetch the dependency

kenny21:04:10

It does not fetch the dep until push is ran

micha21:04:51

that's confusing, because when you do (set-env! :wagons '[...]) it's loading the maven dep into a pod

micha21:04:09

it doesn't try to do any laziness there or anything

micha21:04:17

it doesn't know about the push task

kenny21:04:31

Actually, now that I think about it, it doesn't know about the wagon until later. Hmm.. Can you specify a wagon in the command line? boot -d?

micha21:04:52

you can definitely add the dep with -d for your purposes

micha21:04:07

but you still need the :wagons key in the env to load the dep into the right pod

micha21:04:19

it needs to go into the worker pod

micha21:04:33

basically :wagons allows you to load arbitrary deps into boot's worker pod

micha21:04:04

but with :wagons it also sniffs in the jar for the leiningen file that tells it how to configure pomegranate to load the wagon into aether

micha21:04:19

clear as mud, right?

kenny21:04:10

Haha. boot -d did the trick

micha21:04:24

sweet, yeah, you just need to populate the maven cache

esp123:04:35

i’m trying to figure out how to organize a web project that has both client (cljs) and server (clj) components w/boot. i originally stuffed everything into the same build.boot and had both client/server dependencies, etc in the global env, but that’s not such a great idea. i’m guessing i should be using pods? or should i split the project into separate build.boot files? and if the latter is the recommendation, is there a nice way to have a master build.boot call another build.boot in a pod?

martinklepsch23:04:35

@esp1: I think edge might be interesting: https://github.com/juxt/edge

esp123:04:47

@martinklepsch: ooh, thanks i’ll take a look

martinklepsch23:04:48

@juhoteperi: nice, this + protected branches sounds ideal