This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2016-04-01
Channels
- # aatree (3)
- # admin-announcements (2)
- # beginners (42)
- # boot (142)
- # cider (12)
- # cljsrn (11)
- # clojure (126)
- # clojure-greece (2)
- # clojure-poland (7)
- # clojure-russia (81)
- # clojure-uk (10)
- # clojurescript (81)
- # component (27)
- # core-typed (2)
- # cursive (18)
- # euroclojure (1)
- # gorilla (1)
- # hoplon (85)
- # immutant (2)
- # jobs (3)
- # leiningen (2)
- # off-topic (49)
- # om (151)
- # onyx (19)
- # parinfer (3)
- # re-frame (36)
- # reagent (2)
- # spacemacs (5)
- # untangled (32)
- # yada (9)
boot.parallel
has come to life, that patch is becoming huge 😄
@richiardiandrea: A good tool for console demos: https://github.com/tweekmonster/tmux2html
whoa!
It’s also not GIFs or other images. It’s straight text animation with javascript!
wow very cool
I created boot.parallel
and moved the primitives there
with boot.test/runtests
as entry point for parallel testing, there is a video above
of course it should be considered an alpha because...well...it is not been battle tested
@richiardiandrea: nice stuff! I wonder — is this also meant for stuff like compiling cljs & sass in parallel? i.e. does it have fileset-merging capabilities?
boot.parallel
has all you need for that yes, it should correctly pass the files through
I kind of pass the fileset around in my tests
but not actually tested it very carefully 😱
a test framework not tested is like a cake without icing, but I will probably dedicate more time on it once the warning on deftask is fixed
which really bothers me (see video)
ok @martinklepsch actually scratch that
the fileset is not passed around at the moment
as boot in boot is executed in a future
and a with-pass-thru
https://github.com/boot-clj/boot/pull/401/files#diff-846bb999f0287f56727d66b894457bb2R59
there is the concept of sync-map
a HashMap of shared data that is passed down pods
https://github.com/boot-clj/boot/pull/401/files#diff-846bb999f0287f56727d66b894457bb2R18
so if you add your fileset there (it should contain Java data structures only)
it will be passed around and read at the end of the parallel computation
from boot.pod/data
why all this? Because we are executing boot in boot in pods and pods' only certainty is that they have the JDK loaded
it is achievable with the above workaround yes 😄
because tasks are executed serially
we are kind of breaking the chain and execute in parallel
it can be improved of course
but I this was what I came up with together with Micha so I am pretty happy for now
watched the vid, can see why the override warning is annoying now
yeah, I am adding an env var as we speak
I did investigate why it happens
and it looks that boot in boot calls boot.main-main
and that requires the namespaces
every time, not on clean slate
while the core pod should be always a fresh one
https://github.com/arichiardi/boot/blob/sift-fix-test/boot/core/src/boot/parallel.clj#L54
Actually @alandipert there is a thing that might be the cause
https://github.com/arichiardi/boot/blob/sift-fix-test/boot/core/src/boot/main.clj#L200
the call to load-file on the script here
because I am should have everything loaded the first time, script (and its requires in it)
I will open a PR so that we don't lose this and go to lunch 😄
@richiardiandrea: the problem with running things like sass and cljs in parallel is that you'd have to do them in separate boot instances
so if you have one boot and two tasks are manipulating filesets at the same time, what happens when they both call commit!, which one wins?
yes things need to be changed in order to address the parallel execution
because paths in the fileset are pointing to placees where a file should be but doesn't exist
so fileset are immutable, which is really great, but commit!
is not thread safe
understand
so the next step is to make commit!
thread safe
yeah, it was just a sentence thrown there
maybe, attentatively, in a parallel task you can keep track of fileset diffs as they come and at the end return the result
yes that wouldn't work
I thought pods had a completely separated classpath
i guess it's the classpath mergers problem like one has with uberjars
another thing, I don't know very well in which way sass
modifies the classpath atm
but in general the problem is clear, if only we could reify these changes, then you can pass them among pods and down the computation...for then collecting the results in boot.pod/data
by reify changes you mean maintain a log of classpath additions?
maybe, just brainstorming here 😄
but yes it is not straightforward
true that
but executing a repl in parallel would be odd 😉
(or maybe not?)
some hammock time is needed...
i can imagine playing the log for 2 divergent branches over a clean pod, consulting a user-supplied merge-fn as necessary
yes that's btw where all the tools are converging, see onyx
the log model is super powerful
but ok, I also opened an issue for the deftask warning and that's really a thing I have to solve or parallel tests will be very noisy
Well now at least we can try it out ;)
Oh wow great but I think the restrictions i have are mostly in using boot.pod/data
Where no Clojure is allowed
And the second thing is that runBoot passes through a main that was not really thought for non-cli calls
But I think these are just rough edges, the functionality is there ;)
it would involve some fancy footwork with directories on the filesystem for the classpath, but i think we could probably do it
if we could get the parallel tasks working then there would really be no advantage the dependency graph architecture has over the pipeline
Ah, yeah if we can isolate a pure pod exexution is better, now the user script is loaded for every runboot for instance, which is not good (see last issue)
Hi. I am making a docker with boot pre-installed. Is it possible to download all the deps for a task (e.g. push requires s3-wagon-private which has all its internal deps)?
@kenny: the best way is to run the task in your docker build
well, that's what i do
there might be better ways
@alandipert: But you can't run the push
task without a repo
@kenny: just adding wagon-s3-private to the :wagons
in set-env!
should be enough to fetch the dependency
that's confusing, because when you do (set-env! :wagons '[...])
it's loading the maven dep into a pod
Actually, now that I think about it, it doesn't know about the wagon until later. Hmm.. Can you specify a wagon in the command line? boot -d
?
but with :wagons
it also sniffs in the jar for the leiningen file that tells it how to configure pomegranate to load the wagon into aether
i’m trying to figure out how to organize a web project that has both client (cljs) and server (clj) components w/boot. i originally stuffed everything into the same build.boot and had both client/server dependencies, etc in the global env, but that’s not such a great idea. i’m guessing i should be using pods? or should i split the project into separate build.boot files? and if the latter is the recommendation, is there a nice way to have a master build.boot call another build.boot in a pod?
@esp1: I think edge
might be interesting: https://github.com/juxt/edge
Woah, this will be useful: https://github.com/blog/2141-squash-your-commits
@martinklepsch: ooh, thanks i’ll take a look
@juhoteperi: nice, this + protected branches sounds ideal