Fork me on GitHub
#babashka
<
2021-04-13
>
maxp03:04:32

About parallel :depends . what about

:depends ["sequential" ... ]
or
:depends #{"parallel" ..}
?

borkdude07:04:55

@maxp Interesting, I like the idea. What if you want to execute p1 sequentially and then p2 and p3 in parallel? [p1 #{p2 p3}]] ?

borkdude07:04:05

maybe that case is getting too complex and one should just make another step in between

pithyless07:04:33

:depends #{"parent" ...} implies that the task understands the internal details of parent and whether it can be parallelized. Having a task return future or value leaves that decision in the task. The latter kind of orchestration looks similar to the approach in pedestal interceptors and pathom resolvers: leave the details of sync/async to the the task itself, describe a tree of dependencies separately, and then a higher-level orchestrator is just responsible for calling each of them when it is appropriate (and blocking when necessary).

pithyless07:04:14

I figure, if it the orchestration gets any more complicated than that, maybe it's time to revisit a configuration ala onyx.

borkdude08:04:39

@pithyless I think for something like make / bb.edn it's the other way around, at least in some cases. At work I now have this:

{:tasks {dev-cljs {:doc "Runs front-end compilation"
                   :task (clojure "-M:frontend:cljs/dev")}
         dev-less {:doc "Compiles less"
                   :task (clojure "-M:frontend:less/dev")}
         dev-backend {:task (clojure (str "-A:backend:backend/dev:" platform-alias)
                                     "-X" "dre.standalone/start")}
         -dev {:depends [dev-cljs dev-less dev-backend]}
         dev {:doc "Runs app in dev mode. Compiles cljs, less and runs JVM app in parallel."
              :task (shell "bb run --parallel -dev")}}}
I want dev-cljs, dev-less and dev-backend to run in parallel, but those tasks don't need to know that I want to do that, because they can also run standalone. I make the decision in the dev task (by invoking bb with the --parallel option) but if the proposal of maxp was implemented, I could leave out the dummy -dev task.

borkdude08:04:22

I think make runs anything in parallel if you use -j4 (at most 4 jobs in parallel).

jeroenvandijk08:04:08

Wild idea and not properly thought through; if tasks are properly isolated, parallel could be the default, “sequential” could be a debugging option?

borkdude08:04:56

I think sequential has its place too perhaps. When you run everything in parallel, the output may look more garbled, etc. So having some way to control it might be nice

👍 1
borkdude08:04:25

Some people may depend on side effects happening in p1 first, etc

borkdude08:04:58

Although then p2 should depend on p1

maxp08:04:16

And good question - what means 'parallel' in this case. some special task does fork/join?

maxp08:04:29

btw, for my mind 'depends' does not mean 'execute in strict particular order'

borkdude08:04:32

A nice example provided by @grazfather is this one: https://gist.github.com/borkdude/356669d9b829fe5a891e44f920105f26 When you run it with bb coffeep it takes 11 seconds. When you run it with bb run --parallel coffeep it takes 6 seconds, because bb runs some of the dependencies in parallel.

borkdude08:04:22

@maxp I think that's reasonable: if you say #{p1 p2} you don't care about the order, so bb can (and will by default?) optimize this by running them side by side. And wait for all of them to finish before continuing.

pithyless08:04:40

What about just :depends [p1 p2] and the rule is always: all these tasks are dependencies that may by run in any-order and in parallel. Then you can supply a -j0 (sequential, any order) to -j4 to (in parallel, any order). If semantics are important (either ordering or using return values from one task in another) then those semantics will be explicit in the code - ie. don't use depends but write out the code in the :task ?

borkdude08:04:28

@pithyless I think the nice thing about #{p1 p2} is that you have finer control over which things you want to run in parallel. E.g. for my work tasks I have:

clean (do (shell "bash -c 'rm -rf frontend/public/js/app.js'")
                   (shell "bash -c 'rm -rf frontend/public/js/app.out'")
                   (shell "bash -c 'rm -rf frontend/public/css/main*'"))
         dev:clean {:depends [clean dev]}
and I want clean to finish before dev is started when I run dev:clean, but for my dev task here (https://clojurians.slack.com/archives/CLX41ASCS/p1618301439082200) I do want to run the deps in parallel

borkdude08:04:43

and I think you will use #{p1 p2} more sparingly, only for deps that take a few seconds or more probably

pithyless08:04:16

yeah, I understand; I actually do like the #{p1 p2} and [p1 p2] distinction; I'm just worried we may go down a sugar-syntax rabbit hole. (1) Do we support something like [p1 #{p2 p3} p4]? (2) Will the semantics be obvious to me when I look at a script in a month, without looking up the babashka docs? :)

borkdude08:04:30

@pithyless for [p1 #{p2 p3} p4] I intend not to support it. if you want this, then make it so: p0: depends [p1 p2 p3] p2: depends #{p4 p5}

pithyless08:04:45

@borkdude BTW, how does your dev example work in practice? are those 3 processes that startup and block? Do they garble the output or do they not print to stdout?

borkdude08:04:20

@pithyless those three processes just run in the background and the dev task waits for them to finish, which is never, or until I hit ctrl-c. I have a script for this:

#!/usr/bin/env bb

(ns dev)

(require '[babashka.process :refer [destroy-tree process]])

(def opts {:out :inherit
           :err :inherit
           :shutdown destroy-tree})

(defn cljs []
  (process ["clojure" "-Sforce" "-M:frontend:cljs/dev"] opts))

(defn less []
  (process ["clojure" "-Sforce" "-M:frontend:less/dev"] opts))

(defn clojure []
  (process ["clojure" "-Sforce"
            "-A:backend:backend/dev"
            "-X" "dre.standalone/start"]
           (assoc opts :in :inherit)))

(cljs)
(less)
(-> @(clojure) :exit (System/exit))
but now I can do all of this with bb tasks if I want

borkdude08:04:45

and yes, the output may become garbled, but in practice this doesn't bother me that much, doesn't happen that often

borkdude08:04:14

it's similar to starting processes in bash with &

borkdude09:04:07

Btw, I think it might be nice to release a stable version of bb tasks at ClojureD in June perhaps. You can vote for the talk topic here: https://twitter.com/borkdude/status/1381897272024305665

borkdude12:04:12

Click the link to see info about an informal meetup after the talk organized by Nikolay. It will be a kind of Russian Clojure meetup on Zoom. Join the Telegram channel for more info.