Our big ol’ Ant script no longer uses Leiningen anywhere… just Boot… which has simplified a lot of things (although simulating our homegrown "with-browser" plugin for WebDriver testing was… interesting!).

@seancorfield: are you running on windows?

Our CFML applications now use Boot instead of Leiningen to figure out their classpath. The only thing left now is our test suite which still relies on the old Leiningen classpath logic in the CFML/Clojure bridge.

Mac OS X for DEV, Linux for CI onward.

ah ok, cool

I only use Windows for casual use in the evenings (I have a Dell XPS 12 Convertible — tablet / ultrabook — which is great for sitting in front of the TV and dabbling or sitting in bed with).

Did something specific change on Windows? Or do you need something tested there?

i'm just trying to get an idea of where boot is at with windows support

windows 10?

Yes. Fast Ring Insider builds

Been on Windows 10 since February...

Boot has been behaving well on that laptop, to be honest.

i think the issues with boot on windows were mostly fixed by changes in windwos 10

The only weirdness is that boot repl hangs in Git Bash (but works just fine in the regular Command prompt).

All the other Boot stuff seems to work fine in Git Bash, except for the REPL.

whew! good news

i'll be able to look into the repl thing now

Any more thoughts on the lein template stuff?

Do you think a boot new task would be worth having?

we had a few ideas bouncing around

one of them would be to include the template with the project

like for boot-expectations for example

you could make a task in that namespace that generates stub tests

then a user could do like

Seems like leveraging all the existing lein template ecosystem would be better?

boot -d boot-expectations new --arg something

the new task in your project could use any underlying template machinery

or like for hoplon for example

we have a boot task that runs the compiler

hoplon/boot-hoplon in maven

we could make a task named new in that namespace

that scaffolds up a hoplon app

Next week I’ll take a look at a couple of lein-templates I’ve written and see what those might look like in the Boot world… Being able to run existing lein-templates or something similar (boot-templates) seems worth exploring…

boot -d hoplon/boot-hoplon new --name my-project

you wouldn't need a build.boot or anything to do that

lein encodes the info about how to find the template in the group-id

and the artifact-id

I know.

but with boot you can just specify the dependency directly

which i think is nicer

But I bet if Boot could run existing lein-templates, it would encourage template maintainers to add build.boot files to those templates...

ah i see

i bet that can be done

generate a project map

and call the entry point

which is well defined for templates anyway, because that's what the lein task does

yeah i like that idea

so boot new something myproject would download and run the something/lein-template project and the result could easily have both project.clj and build.boot

yep i believe so

(how much of build.boot could be autogenerated from project.clj?)

i guess it would be polite to ask what they think about this in #C0AB48493

i can imagine maybe it would be uncool if people made lein-templates that don't work with leiningen, or araen't maintained for lein

confusing leiningen users

Right, I would expect boot new something myproject to look for something/boot-template first, and fall back to the Leiningen template if that can’t be found.

So it would know how to run existing lein templates but would primarily know about new Boot templates (so we’d have to figure out what that code should look like).

I’m curious enough to have a go at a prototype regardless

sweet, i can migrate the hoplon lein-template to the new scheme

Oh, and while I think about it, I found a case where boot-test doesn’t compose nicely because it sets up the pod dependencies outside the with… middleware call. I’ll send a PR for it once I have a simple repro test.

great, thanks

I ran into that today and had to modify how I was setting up my testing context. TL;DR: if you do (comp (with-pass-thru fs (set-env! … testing dependencies …)) (test)) then the tests don’t see the dependencies all the time.

one thing you might want to try there, is doing the set-env! before the with-pass-thru

in the body of the task definition

Right, but I shouldn’t have to do that.

that's part of the "lifecycle"

If I’m composing "anonymous" tasks that isn’t so easy.

why?

a task like that is similar to a lien profile

the general pattern for those is:

Because (comp (sometask) (someothertask) (with-pass-thru fs (set-env! …)) (test)) should "Just Work" without worrying about lifting the set-env! call.

(deftask with-foo []
  (set-env! ...)
  identity)

you can just wrap in (do ...

it's useful to have a more nuanced lifecycle

like it's useful to be able to make the pods before the pipeline starts

generally i think it's better to mutate global state outside the pipeline

like set-env! especially

Then perhaps what you want is a with-… macro that supports that better so anonymous task composition is cleaner?

(comp (task1) (task2) (do (set-env! ...) (task3)))

I think the (do … identity) thing is really ugly and non-intuitive (remember: you work with Boot all the time — I’m coming from the p.o.v. of a relatively new user).

you can do like this if you prefer:

(comp (task1) (task2) (do (set-env! ...) (with-pass-thru _)))

that should work, too

That’s still ugly.

what would you suggest for macro?

Don’t know yet. Will give it some thought.

i guess the clearest thing would be to lift the set-env! out of the comp entirely

I got caught out because the comp pipeline I used for boot-expectations didn’t work for boot-test and it took me a while to figure out why...

Well, then you can’t compose tasks where you don’t what the set-env! to affect earlier parts of the composed pipeline...

one of the reasons it's good to keep set-env! out of the pipeline is that any step of the pipeline shouldn't see the effects of the steps after it

Hmm… OK… I can see that being a valid reason…

that gives you the benefit of all the functional immutability stuff

like a task can aggressively cache things, because it can be sure that the state can only change if the fileset is different

i.e. given two filesets, if they're the same it can use cached artifacts

but if the state includes the mutable portion from tasks after it, it can no longer safely do that

I still think this is a big gotcha for folks new to Boot. It held me up for a couple of hours this morning because I couldn’t see why (comp (testing) (expectations)) worked but (comp (testing) (test)) did not — essentially — and it was due to boot-expectations creating the pod inside the with-… macro and boot-test creating it outside.

And there’s nothing in the docs to explain this subtlety… at least not that I could find.

hm yeah

a diagram would be helpful perhaps

it should be noted here maybe: https://github.com/boot-clj/boot/wiki/Tasks#task-anatomy

Well, that does mention "Local state" but it doesn’t call out where set-env! should or should not be called.

And you get subtly different behavior trying to compose tasks depending on when set-env! is called (unfortunately).

yeah i'll add a note there

yeah the pipeline is built in two stages

it's first composed, and then it's run

very similar to stateful transducers really

Essentially you’re saying set-env! should never be called inside a with-… macro I think?

same structure and whatnot

well you might do it, but it might not be seen by other tasks in their constructor

which is where they'll make pods etc

Well, in (comp (task-a) (task-b) (task-c)) you actually run "[4] Local state" first, then you compose, then you run.

so three stages.

right

yep

same as transducers, no?

For users who are used to writing their own transducers, perhaps that is an appropriate analogy 😉

i haven't actually used them for real yet

https://raw.githubusercontent.com/matthiasn/talk-transcripts/master/Hickey_Rich/Transducers/00.36.36.jpg

This is important for the docs, since it determines whether a task later in the pipeline might succeed or fail: https://clojurians.slack.com/archives/boot/p1451012348006025

the top defn could be changed to deftask and you have a task

Actually, no. Look again at where the state is created...

…inside the fn

well in that slide yes

Bottom line: drawing a parallel with transducers is not going to be very helpful for a lot of Clojure users.

that would work with tasks, too

it would just run them in reverse

i mean reverse order

I think the docs should at least be clear that (with-… (set-env! …) …) is a bad idea

yeah

that task anatomy section can have an example

and going back to boot-expectations, do you think this code should change? https://github.com/seancorfield/boot-expectations/blob/master/src/seancorfield/boot_expectations.clj#L35-L44

(to move the with-pass-thru below the creation of the pods

ah yes

And then boot-expectations would have the same "restriction" as boot-test that any preceding set-env! would need to occur in "stage 1" of the pipeline (before composition)...

i think so

arent you creating a new pod pool each iteration if the watch task for instance is running the pipeline multiple time?

times?

Apparently so. That wasn’t clear from the docs (or from our discussions when I was writing the task and getting feedback here)

https://github.com/seancorfield/boot-expectations/issues/8 <— I’ll fix it, and then update my build.boot at work to "do the right thing" with set-env!…

that's some pretty lean code, looks very nice

It’s heavily borrowed from boot-test TBH.

oh i ran into an issue with clojure.jdbc on friday, can i ask a quick question?

Sure. Turnabout is fair play!

awesome thanks!

And then I must go feed the cats!

i'm using the jtds driver with windows sql server, and in sql server date types are stored in UTC always (there's no timezone info whatsoever in the database)

when i perform a query that has a column of type TIMESTAMP i get a java.sql.Timestamp object

but it's not GMT

if i do (.getTime the-timestamp) i get the number of millisec since epoch

Sounds like this section of the docs might help? http://clojure-doc.org/articles/ecosystem/java_jdbc/using_sql.html#protocol-extensions-for-transforming-values

i could transform the values, yes

java.jdbc itself doesn’t do anything with the values that come back from the driver (as far as I know)

but it seems scary to me, because the driver should know

since this is part of the definition of the datatype in sql server

i have bad feels about adding my own transformations to something as important as dates

We use MySQL at work and have all our servers ntp sync’d and set to UTC so everything — app, db, O/S — all know it’s UTC time.

As far as I know, the only way to reliably use UTC end-to-end is to have everything running in UTC… ?

like java.util.Date is by definition UTC

and all the things that subclass it

there is the deprecated timezone offset stuff

I don’t think you’re correct. Joda Time is UTC, java.util.Date is local time by definition.

hm

it's a very strange thing, because i'm in EST

That’s been my experience over the years, working with dates in Java and databases.

if i have a date in the database of 1970-01-01 00:00:00

Well, I’m PST and java.util.Date locally matches my own time. On servers on the East Coast, I get EST, unless the server is running on UTC, in which case java.util.Date is UTC.

it will come back as "1969-12-31 19:00:00 GMT"

it applies the transformation to GMT on a date that's already GMT

What TZ is your client? What TZ is your DB? What TZ is the server on which the DB is running?

it bothers me that this should matter

MySQL, for example, can operate in a different TZ to the server its own which makes life very complicated.

when i store a date in the db it's just data

an offset in nanosec from an arbitrary absolute coordinate

the epoch

local time zones etc. shouldn't have anything to do with that

like "how far is it from ny to la?"

"well where are you now?"

But your client is timezone aware.

sure, but the data in the database is absolute, not relative to my client in any way

so if i make a query from any client i should get the same result

i should get some object representing the same absolute coordinate in time as what i inserted in the database

Google shows a lot of questions like yours about timezones so this is clearly a common confusion people have.

What data type are you actually using in the database?

TIMESTAMP

which is defined as nanosec since epoch

http://stackoverflow.com/questions/409286/should-i-use-field-datetime-or-timestamp/602038#602038 for example

GMT

"In MySQL 5 and above, TIMESTAMP values are converted from the current time zone to UTC for storage, and converted back from UTC to the current time zone for retrieval"

oh this is the ms sql server

I know. I think you’ll find it’s the same.

When I Google this I see similar comments for Oracle, Postgres, etc

the type has no timezone info associated with it

All I can say is: this is not an issue with clojure.java.jdbc itself.

right

This is what everyone has to deal with using date/time in Java with JDBC

yeah i think the jtds driver is bonkers here

Have you tried the MS SQL Server JDBC driver?

i haven't

java.jdbc supports it (I test with both drivers)

oh really?

i had a hard time getting it to work

You just have to manually d/l it and localrepo install it.

is an example in the tests for java.jdbc?

yes

awesome

i bet the MS driver will do the right thing

i hope, anyway

See https://github.com/clojure/java.jdbc/blob/master/src/test/clojure/clojure/java/test_jdbc.clj#L83-L91 and https://github.com/clojure/java.jdbc/blob/master/src/test/clojure/clojure/java/test_jdbc.clj#L37-L56 for setup.

amazing, thank you

I have a Windows XP VM running SQL Server Express and hit it via both drivers from my host Mac when testing java.jdbc.

Note that I don’t specifically test a timestamp column in any DB in those tests (such a test would only verify that putting a particularly value into a column and retrieving it just yielded the same value).

Anyways, must go feed cats.

thanks for the tips

boot-expectations 1.0.2 released with that pod caching issue fixed.

Now I’m updating our build.boot at work to account for your feedback Micha

sweet!

does it seem any snappier?

when doing incremental build/test cycles

Haven’t tried that part yet. I’ve shaved over a minute off our local build/test script just by using Boot instead of Leiningen (reducing the number of JVM startups!).

🚀

Our Ant script is already 100 lines shorter too

removing lines of ant script is satisfying

Not having to deal with building, installing, and calling two custom Leiningen plugins helps there.

Last part of the first pass done: Leiningen is no longer needed for any aspect of our build process (it’s still invoked in a few shell scripts — they’re tomorrow’s priority).

One thing the discussion about templates made me think about is - how about considering rails-like generators in scope for that? It could be a great boon for a rails-killer we may hopefully get someday.

Also another question I have - I've made a task to sync dependencies with an EDN file when it changes. This of course has the downside of not working when existing dependencies conflict with newly introduced ones (mainly for transitive ones). As far as I understand pods could be a solution to that, since they introduce classpath separation. Any pointers how should I approach writing a task which would run the rest of the pipeline inside a pod with appropriate dependencies which is torn down and rebuilt if they have changed? I'm not sure how to approach that so I don't break things like watch and whatnot.

I mean, I think I sorta see how to do it - do a control task that evaluates the next handler inside the pod, but that would let me only use the task only once in the pipeline and not for example have different dependencies for parts of the pipeline.

Now, I'm not sure if that's something that would be ever needed, but would that be possible to implement somehow?

So doing it like this doesn't seem to work:

(deftask with-dependencies
  []

  (let [dependencies (atom (read-dependencies!))
        current-pod  (atom (make-pod-with-dependencies @dependencies))]
    (fn [next-handler]
      (fn [fileset]
        (let [new-dependencies (read-dependencies!)]
          (when (not= @dependencies new-dependencies)
            (util/info "Dependencies changed, updating...\n")
            (reset! dependencies new-dependencies)
            (pod/destroy-pod @current-pod)
            (reset! current-pod (make-pod-with-dependencies new-dependencies))))
        (pod/with-eval-in @current-pod
          (~next-handler ~fileset))))))

Hmm, I think I'm probab;y hitting this restriction here > The expr and result must be able to be printed by pr-str and then read by read-string. So what would be a way to pass the fileset into the pod? Or at least evaluate a boot pipeline inside a pod?

@jaen: can you explain the dependency EDN file please?

like the high level goal?

@micha: well, the goal is for dependencies to be updated automatically - the dependencies.edn is just a vector of project dependencies and each time it changes I want to updated the dependencies without restarting boot.

So for I'm just using set-env to update the dependencies and it mostly work, until something conflicts.

And I was wondering if there would be a way to isolate that inside a pod I can tear down and create again with new dependencies.

So conflicts would be avoided.

you can run boot in boot

that might be what you want

i don't fully understand the issue with conflicts

the way you have it set up you add dependencies to the project by adding them to a dependencies.edn file instead of calling set-env! right?

Yes.

But sometimes it can so happen that a dependency will pull a transitive dependency

That conflicts with a depdendency already pulled by something else

And I thought pods would avoid that.

yeah there is no way to remove jars from a classloader

also you don't have access to boot.core namespace in pods

Yes, I remember you saying that. But I thought pods are isolated so that issue could be sidestepped that way?

yeah that's true

if you load the dependencies into a pod you can destroy the pod later

that destroys the classloader and everything

Yeah, so that's what I'm trying to achieve.

So I can build a pod from the dependencies in the file

Run tasks in that

And if dependencies change, tear it down and rebuild

So no conflicts arise.

so to paraphrase: you want to be able to snapshot the classloader state and restore to the snapshot state later if bad dependencies were added etc

you can use pods if you don't need anything from boot.core namespace

but boot.core contains singleton stateful things

so it can't be easily imported into a pod

Oh, I see.

So that basically makes what I want to do impossible?

Because from what I can tell

You can't send things over to a pod

i don't want to say impossible

If it's not serialisable

And you seem to say you can't really run boot inside a pod

Because if something needs boot.core then you're toast.

well you can run the whole boot in a pod

haha this is confusing

https://github.com/micha/multi-module-build/blob/master/build.boot

Yeah, a bit : D

this is an example of running boot inside boot

https://github.com/micha/multi-module-build/blob/master/build.boot#L18-L30

and https://github.com/micha/multi-module-build/blob/master/build.boot#L70-L71

Basically I'm just trying to figure out how to run boot, if I would want to be able to isolate the project dependencies so I can update everything without killing boot.

so maybe that's a good solution, the runboot thing

And the most sensible way seems to be able to run all the tasks in a boot instantiated inside the boot.

Ok, let me look at that.

the goal of that demo was slightly different from whta you want to do

the goal of that was to demonstrate how to build multiple separate modules in a project when the modules depend on each other

the bravo subproject there depends on the alpha project

with that build.boot you can build them both in parallel

@jaen the nature of dependency conflicts makes it hard to handle them in a reversible way

usually the problem is when you get an incompatible version of some transitive dep

the bad jar contains class definitions for some classes, but you need a different version of those classes

but it's transitive, so usually the instances of those classes are in other code you don't necessarily have visibility into

so even if you could remove the jar from the classpath and replace it with the correct version

there would still be instances of the old classes out there

so you'd still see the broken behavior

does that make sense?

to make sure you fix the dependency problem you need to rebuild any objects that refer to the bad objects

Yeah, hence why I want to run all the tasks inside a pod

And tear it down

So the conflict remains contained to the classloader inside the pod.

I understand the problem this poses if I do it outside of a pod for example - I can contaminate the boot's main classloader

And the conflict could only be resolved by restarting it.

how do you do your development, via repl?

Well, I use component with system.boot, so this all gets reloaded on each code change.

So I don't mind if that makes global state go away

Since by application structure I avoid it.

if you have a repl you'd need to kill it and start a new repl in the new pod, no?

and then connect to that from your repl client

It would appear so, yes.

Okay, I can see how that could be problematic.

boot has things for that

the launch-nrepl function can be used to launch nREPL server in any pod

so that's fine

you'd need to somehow have the nREPL client connect to the new server though

But I would have to reconnect, right? I assume that's what you were getting at?

Yeah

seems doable though

But dependencies being updated should be rare enough to not be a hassle.

Hmm

Is there any way for boot

maybe nREPL middleware that routes expressions to the right pod

To tell me whether a dependency conflict occured or not?

not really

i mean there are dependency conflicts all the time

if you do boot -p in any project you'll see them

Well, I mean those I get yellow warnings for when using set-env to inject new dependencies.

I could avoid tearing down the pod

Unless I know it's necessary.

yeah so the reason for those warnings is because traditionally you'd declare all dependencies in an xml file or project.clj

a static map of deps

the maven machinery can work out dependency conflicts only if it computes them all at once

like if it can see all the dependencies it can work out which transitive deps to choose

but if you resolve dependencies and then later add a new dependency

that new dependency could have a transitive dep that would have changed which version of another already loaded transitive dep was chosen originally

Then you can't do anything, because you can't unload the original choice, right.

yeah

So there's no callback for that or anything?

however if you create a new pod with those deps it does the right thing

because you provide all deps in the pod constructor

so when you add a dependency it would make sense to make a new pod

and you'd avoid that issue

but then you wouldn't be able to have a repl in that pod easily

i mean you'd have the reconnecting issue

Yeah, that's the reason I started looking into pods. I'm just wondering if there's a way to tell if adding a new dependency produced a conflict that would require unloading a jar

And hence a new pod

Or not

Because not all set-env calls updating the deps result in conflicts, right?

At least, it doesn't seem to give me yellow warnings each time I try this, just sometimes.

Anyway, I'll look into those links you gave me and see how it works out

i think if you want to do it you'd wnt to make a new pod whenever you add a dependency

i'm working ona solution to the pod serialization problem today

Oh, that would be cool if possible

but i don't know if that's really going to help in your case

Well, if it would mean I could pass a fileset object into a pod, it just might.

you can make your own serialization in the meantime

Unless I'm misunderstanding pods.

the only issue with the fileset is the java.io.File objects, which you can serialize and deserialize yourself

i think you can use boot-shim.clj to install data readers for the tmpdir record types

Because if you look at that example code I've given I basically just want to call next-handler inside the pod I construct. Not sure if your fix would make that possible or not though.

then pr-str/read-string would work on them

https://github.com/boot-clj/boot/wiki/Configuring-Boot#boot-shimclj-here-be-dragons

you can add to the clojure default data readers there perhaps

and they'll be present in all pods

just for testing could be a fast way to try it

you can do anything in there, including add things to clojure core, whatever

next-handler might be callable from the pod, but you'd need to use the java runnable interface

because if you pass a clojure function to another pod the receiver won't see it as a clojure function

it will see it as a java runnable (i thnk)

you can also use the low-level pod methods

(.invoke mypod "some.ns/f" arg arg)

where arg can be any java object

the some.ns/f function will see the args as java objects

no pr-str/read-string will be done there

this is pretty much the most complicated part of boot lol

although i discovered that arg can be a clojure object.. if it was previously returned from the same pod

yep

I guess that's due to it being created by the same runtime instance?

right

the problem with passing clojure objects is that protocol interfaces are constructed dynamically in each clojure runtime

so like IFn is different in pod a and pod b

And it's not enough to be the same runtime jar, has to be the same instance of runtime?

Ah

You've just answered that, gotcha.

but i think you can use the Runnable interface

all functions in clojure implement that

and that's loaded from the system classloader

so that's the same in all pods

i guess that only works with 0-arg functions though

and it doesn't return anything

so pretty mcuh useless

Is that how boot-shim.clj should look?

(defmethod print-method java.io.File [file ^java.io.Writer w]
  (.write w (str "#test/file " (pr-str (.getAbsolutePath file)))))

(alter-var-root #'clojure.core/default-data-readers
  (fn [current-readers]
    (merge current-readers {'test/file (fn [path] ( path))})))

looks good to me

Then it doesn't seem to help, I must be doing something else wrong. Let's see...

Ah, you need to define both print-method and print-dup it seems.

I can now shuttle the fileset into a pod, but I can't do the same for the next-handler. Let's figure something out for this then.

Hah, I think I've got it.

It's hacky but I created a namespace that just does

(ns boot-hack)

(defn invoke-hack [fun]
  (boot.util/info "INVOKE HACK\n")
  (.run fun))

It kinda works

I just need to figure out why dependencies change doesn't trigger, but it's a step forward for sure.

Welp, was editing the file in target not in the resources '

But it seems I'm not taking something into account still, since I'm getting

java.util.concurrent.ExecutionException: java.nio.file.NoSuchFileException: target/public/assets/javascripts/application.out/goog/deps.js
      java.nio.file.NoSuchFileException: target/public/assets/javascripts/application.out/goog/deps.js
    file: "target/public/assets/javascripts/application.out/goog/deps.js"

But all in all, that's some progress.

the fileset shouldn't contain references to files in the target dir

so that's weird

Well, I've never gotten that error without my pod shenanigans, so it's probably somewhat related; I get that when compiling cljs files.

interesting yeah

@micha: Btw. I had an idea related to separating files to multiple artifacts. It might be useful to have the path of original directory as a field of TmpFile?

original?

Like in less4clj I have three resource-paths and files from each are put into separate jars

ah right

Currently I use regex against path inside fileset

basically tagging files in the fileset

But would be even simpler to say that files from this resource-source should be put into this jar

Yeah something like that

i was thinking about this too

i wonder if we want some kind of stack

I'm not still sure what would be best for task created files?

like git stash

like if sift had a mode where instead of just removing files, it would stash them on a stack that you could pop

I think it would be useful for these kind of operations to provide functions which wrap "middlewares"

like the with-files I wrote

(add-tag (cljs) :cljs)

that would add :cljs to the meta on all the files produced by the cljs task, which it returns?

Yeah hmm, wouldn't bee too useful to tag all files in fileset after cljs. I think it should tag files created by cljs task.

I guess that's not as trivial to implement.

it can be done already, by composing with sift

sift can do all the set operations by meta

so you can tag all the files before the cljs task with one tag

then after with another

and sift the ones that have onle the after

i forget exactly, but it can be done on the command line

but tagging is maybe not the ultimate solution?

seems like we really want to stash the files somewhere and handle each stash separately?

Yeah I guess this needs more thought

i think stashes can be like registers where you can put a fileset

With-files I wrote works quite well. Each artifact can be created inside one with-file context and doesn't see files from others.

can i see an example build.boot with this?

https://github.com/Deraen/less4clj/blob/master/build.boot#L60

ah i see yes

In my case I'm not even interested in filtering task created files, only files from resource-paths

right, but i can see wanting to do the other too

True

we can really do anything with the filesets because they're immutable

so there is a lot of flexibility to design something that works well

lots of ways we can partition and organize them

what about registers instead of stacks?

like (sift :include #{#"^foo"} :save :a)

then later you could do (sift :load :a)

or instead of :a maybe a var or atom

that was very un-lcdi of me lol

Maybe. I don't yet see what a complete build task would look with this.

i got to thinking in stacks because the problem is very similar to forth

I think some "utility" tasks wrapping other tasks (and creating closures or "fileset context") would be useful instead of adding more and more tasks to "base level" with comp

we want to make a tree from a pipeline

like we have this pipeline, but we want to fork and branch

and emit multiple things

at the end

Yeah I think I'm trying to describe a tree

this is exactly the problem faced by forth

which they solve with stacks

then you can process the tree sequentially

by rearranging the things on the stacks

but i think registers might be better and easier to use

well maybe not as good, but easier to use

like this for example:

boot stash -i sadf -- proc1 proc2 target -- pop -- proc3 proc4 target

that would create a fileset and stash it

then run a pipeline on the stuff remaining in the non-stashed fileset

and emit to target

then you pop the stashed fileset and run another pipeline on that

and emit to target

the target task has a --no-clean option that tells it not to delete all the things in the target before emitting

the -- in the example is just to visually separate the pipelines

ignored by boot basically

maybe a stack is the best thing actually

you just order your subpipelines according to the stack order

i could also imagine stack operations, like union, difference, intersection, etc

that would operate on the top two items on the stack

to merge filesets, etc

so you could have total control over how you partition the filesets with simple operations

then you could build any of the more specialized things on top fo that

you would also probably want to run independent operations in parallel

not sure how well that works with stacks

i don't thnk you'd want to do that

because the classpath in the jvm is a singleton

so you can't really run things that affect the classpath in parallel

nondeterministic

I mean if you have independent resource dirs that you can process in parallel to get independent outputs - why do we care about the classpath?

(I didn't read the fileset code :))

well the jvm has only one classpath

so if you have independent resource dirs and you add them in parallel they'll all be on the classpath for all processes

unless you do some things with pods

you can run pipelines in parallel using the multi-module pattern, like this: https://github.com/micha/multi-module-build/blob/master/build.boot#L70-L71

the problem with processing sections of a single pipeline in parallel is that when you call commit! on a fileset it replaces the current things on the classpath with the things in the fileset

so concurrent commits are possible, but you'd get nondeterministic results

the code itself is reentrant, there are semaphores and locks to prevent corrupting the filesystem

one thread would be seeing the results of commit! in the other thread

which would be bad

because they'd be clobbering each other

and the underlying filesystem wouldn't correspond to the fileset object being passed to the next task if another thread called commit! on its own different fileset

that would cause exceptions when trying to read form files etc

*from

if the files no longer are in the filesystem

ok

i think a lot of the need for paralleization in builds is mostly eliminated by caching in tasks

with the immutable fileset tasks can compute diffs easily and not do unnecessary work

I was looking at https://github.com/ndmitchell/shake previously

which parallelizes automatically

was trying to see if that would work for boot

shake relies on building a dependency graph, right?

it builds the dependency graph dynamically though

yeah

but you need to guide it

like with annotations

yeah, but then I was thinking you could annotate boot tasks

if you wanted to

the reason why we didn't go down that road was simplicity

and infer taks results from the fileset changes

yeah, that's the usual build tool approach

but i think there is value in allowing the user to order the tasks

instead of devising a dependency graph via inversion of control

because usually the order in which tasks should run is fairly obvious to the programmer

for the specific task they're doing

but making a dependency graph that will be generally correct is hard

like in any specific instance you can see it, the order is obvious

but if you need to make a system that can do it in the general case it gets really hard

shake builds are also ordered explicitly

and you need to introduce coupling between tasks so you can annotate

ah so it performs a sort of JIT optimization?

learning as it goes?

hm

wait, maybe I'm wrong. The build steps are monadic

i wonder about boot tasks

i imagine there is a monad for what they are

with a name

(defn what-am-i
  (fn [continuation]
    (fn [arg]
      (continuation (f arg))))

that guy

knowing the name of it might unearth something interesting in the literature

is the clojure compiler reentrant?

like can you compile namespaces for AOT in parallel?

i guess in pods you can for sure

oh, maybe not actually

pods wouldn't produce compatible bytecode

aot is really the only thing in boot that could be slow

i mean in a build

boot could be used to run gcc and other stuff which is slow

you could make your gcc task compile things in parallel

that would be pretty straightforward, using send-off or future

of course

but with shake that's out of the box

I'm not solving a problem that I have right now

i suspect that shake doesn't provide the simplicity and flexibility of boot though

like with boot you can noodle around in the repl all day

of course, I'm just thinking if we could add that to boot in some way

JIT type optimiztions might be a possible approach

like what we do for the uber task

but externally

detecting immutability basically and leveraging that to avoid doing work

if you know that a task doesn't mutate the environment and is just a pure function that takes fileset -> fileset

perhaps something can be done there to parallelize

@micha tasks which do that may not want to be run in parallel tho

speak for example should only run when a build finishes, even tho it’s actions can be run in parallel to others

yeah

interesting

i think it would make more sense to create a parallelizing task

which takes the following tasks and runs them in parallel pods

or something like that

or if tasks could somehow be more context aware

like a pod being told which other pods were started in parallel and the task can sort out blocking? I havnt read much pod code so im not sure how it currently works

i like the explicit approach

the multi-module-build example has something like that

it uses semaphores

yeah i was poking around in there

semaphores was an interesting read

the semaphore isn't the ideal concurrency thing there, exactly

but it's close enough to work

i wanted something more like a priority queue of semaphores or something

like named checkpoints

where you could wait on a specific checkpoint to be reached

i couldn't find anything that did that exactly

interesting

maybe phasers, but i can't understand them at all

im a bit confused as how checkout is adding deps to the fileset

the checkout task?

yeah

the checkout task takes a dependency maven coord as argument

and when it runs in the pipeline it finds the jar in your local maven repo

actually before it runs in the pipeline, at construct time

it finds the jar in local maven

and adds the directory that jar is in to the :source-paths

so now it's being watched for changes

now, when the checkout task runs in the pipeline it unzips that jar into a temp dir

and it adds that tempdir to the fileset

so now the files in the jar are in your build classpath

as if you had them in your project source paths

that's why you want to put the checkout task after the watch task

it needs to unzip each time and add to fileset

could it maybe be the checkout task causing hoplon files to not be seen?

hm

i can take a look at that right now if you want

is there a repo handy?

im just poking a round, but the hoplon boot task looks fine

yeah it might be that hoplon only extracts the deps once

ah the repo is kinda large and broken 😛

it assumes the jar is immutable

i think that's the problem

hm no

that shouldn't matter

because the .hl files should be on the classpath now

directly as files

i forgot, did you see the .hl files with show -f?

i think they were in there ok?

well when the project builds the first time it finds all the .hl in my deps, but after the checkout task it doesnt extract or compile them, show -f has the .hl fines in the fileset before and after checkout

are the new deps extracted to the same location as the previous jar?

yeah, a temp dir

maybe the .hl files aren't being updated in the jar?

if the .hl files are the same then they won't be recompiled

but stopping the build and restarting it builds with the new files the first go, before ever needing to do a chekout

that would be a useful addition to the show task, an option where it can print like -f but show only changed files

yeah

what order are you calling your tasks in?

maybe there is an issue there?

https://github.com/flyboarder/theLounge/blob/master/build.boot

i just removed the checkout task

im using the dev-osx

where was the checkout task before?

right after watch?

directly after watch

and which dependency was the checkout dep?

I had the lounge.* shapshot’s in it

ah i think i know what the issue is

try removing those from your :dependencies

and add the checkout back

like dont have them in both deps and checkout?

yeah

i think maybe hoplon is overwriting the checkout files with ones from the jar

i mean cached ones

that it got from the jar originally

i think this can be fixed in boot-hoplon

ah ok, trying

is there already a task that reads deps from an edn that gets watched?

Just pushed a new implementation for the junit reporter PR using the new target task: https://github.com/adzerk-oss/boot-test/pull/11. target is used to sync the junit reports before throwing an exception in case of test errors/failures. Includes a refactoring in the test task so the test summary is saved as fileset metadata, so it can be potentially used by downstream tasks (someone suggested this idea here, I don't remember who but thanks!)

@nberger: awesome! while you're here i'll have a look

yeah thats working! thanks micha! i should have tried that sooner 😕

cool @micha, thanks! I'll be here for a while

@flyboarder: one thing you can try is making your own version of the checkout task that removes the checkout dep from the env :dependencies in its constructor phase

like before the with-pre-wrap or whatever, do a set-env! to remove the dep

that might fix everything so you don't need to remove the dep