Fork me on GitHub

About parallel :depends . what about

:depends ["sequential" ... ]
:depends #{"parallel" ..}


@maxp Interesting, I like the idea. What if you want to execute p1 sequentially and then p2 and p3 in parallel? [p1 #{p2 p3}]] ?


maybe that case is getting too complex and one should just make another step in between


:depends #{"parent" ...} implies that the task understands the internal details of parent and whether it can be parallelized. Having a task return future or value leaves that decision in the task. The latter kind of orchestration looks similar to the approach in pedestal interceptors and pathom resolvers: leave the details of sync/async to the the task itself, describe a tree of dependencies separately, and then a higher-level orchestrator is just responsible for calling each of them when it is appropriate (and blocking when necessary).


I figure, if it the orchestration gets any more complicated than that, maybe it's time to revisit a configuration ala onyx.


@pithyless I think for something like make / bb.edn it's the other way around, at least in some cases. At work I now have this:

{:tasks {dev-cljs {:doc "Runs front-end compilation"
                   :task (clojure "-M:frontend:cljs/dev")}
         dev-less {:doc "Compiles less"
                   :task (clojure "-M:frontend:less/dev")}
         dev-backend {:task (clojure (str "-A:backend:backend/dev:" platform-alias)
                                     "-X" "dre.standalone/start")}
         -dev {:depends [dev-cljs dev-less dev-backend]}
         dev {:doc "Runs app in dev mode. Compiles cljs, less and runs JVM app in parallel."
              :task (shell "bb run --parallel -dev")}}}
I want dev-cljs, dev-less and dev-backend to run in parallel, but those tasks don't need to know that I want to do that, because they can also run standalone. I make the decision in the dev task (by invoking bb with the --parallel option) but if the proposal of maxp was implemented, I could leave out the dummy -dev task.


I think make runs anything in parallel if you use -j4 (at most 4 jobs in parallel).


Wild idea and not properly thought through; if tasks are properly isolated, parallel could be the default, “sequential” could be a debugging option?


I think sequential has its place too perhaps. When you run everything in parallel, the output may look more garbled, etc. So having some way to control it might be nice

👍 3

Some people may depend on side effects happening in p1 first, etc


Although then p2 should depend on p1


And good question - what means 'parallel' in this case. some special task does fork/join?


btw, for my mind 'depends' does not mean 'execute in strict particular order'


A nice example provided by @grazfather is this one: When you run it with bb coffeep it takes 11 seconds. When you run it with bb run --parallel coffeep it takes 6 seconds, because bb runs some of the dependencies in parallel.


@maxp I think that's reasonable: if you say #{p1 p2} you don't care about the order, so bb can (and will by default?) optimize this by running them side by side. And wait for all of them to finish before continuing.


What about just :depends [p1 p2] and the rule is always: all these tasks are dependencies that may by run in any-order and in parallel. Then you can supply a -j0 (sequential, any order) to -j4 to (in parallel, any order). If semantics are important (either ordering or using return values from one task in another) then those semantics will be explicit in the code - ie. don't use depends but write out the code in the :task ?


@pithyless I think the nice thing about #{p1 p2} is that you have finer control over which things you want to run in parallel. E.g. for my work tasks I have:

clean (do (shell "bash -c 'rm -rf frontend/public/js/app.js'")
                   (shell "bash -c 'rm -rf frontend/public/js/app.out'")
                   (shell "bash -c 'rm -rf frontend/public/css/main*'"))
         dev:clean {:depends [clean dev]}
and I want clean to finish before dev is started when I run dev:clean, but for my dev task here ( I do want to run the deps in parallel


and I think you will use #{p1 p2} more sparingly, only for deps that take a few seconds or more probably


yeah, I understand; I actually do like the #{p1 p2} and [p1 p2] distinction; I'm just worried we may go down a sugar-syntax rabbit hole. (1) Do we support something like [p1 #{p2 p3} p4]? (2) Will the semantics be obvious to me when I look at a script in a month, without looking up the babashka docs? :)


@pithyless for [p1 #{p2 p3} p4] I intend not to support it. if you want this, then make it so: p0: depends [p1 p2 p3] p2: depends #{p4 p5}


@borkdude BTW, how does your dev example work in practice? are those 3 processes that startup and block? Do they garble the output or do they not print to stdout?


@pithyless those three processes just run in the background and the dev task waits for them to finish, which is never, or until I hit ctrl-c. I have a script for this:

#!/usr/bin/env bb

(ns dev)

(require '[babashka.process :refer [destroy-tree process]])

(def opts {:out :inherit
           :err :inherit
           :shutdown destroy-tree})

(defn cljs []
  (process ["clojure" "-Sforce" "-M:frontend:cljs/dev"] opts))

(defn less []
  (process ["clojure" "-Sforce" "-M:frontend:less/dev"] opts))

(defn clojure []
  (process ["clojure" "-Sforce"
            "-X" "dre.standalone/start"]
           (assoc opts :in :inherit)))

(-> @(clojure) :exit (System/exit))
but now I can do all of this with bb tasks if I want


and yes, the output may become garbled, but in practice this doesn't bother me that much, doesn't happen that often


it's similar to starting processes in bash with &


Btw, I think it might be nice to release a stable version of bb tasks at ClojureD in June perhaps. You can vote for the talk topic here:


Click the link to see info about an informal meetup after the talk organized by Nikolay. It will be a kind of Russian Clojure meetup on Zoom. Join the Telegram channel for more info.


does it make more sense to have an :env block or something outside of :tasks, even, that is used to set vars?


e.g. for`platform-alias`


and i imagine that soon people will want to override these vars


@grazfather I thought about this, e.g :vars ... but what benefit would this have over :init?


an invoker could provide these


it would only be more restricting I thing, while introducing semantic problems, like ordering, etc


e.g. bb run dev-backend platform-alias=Solaris would avoid running the thing that normally sets platform-alias


I would say, just use an env var for that. BACKEND=foo bb dev and (def backend (or (System/getenv "BACKEND" ...))

👍 2

Again, I am just taking make semantics… I don’t think it’s something we really need to worry about


ok, yeah I agree


i think for tasks you should write a ‘recipe book’ about how to get things from make in BB. you showed the perfect example of one


did you change your mind about :init vs just a ‘recipe’` for init being a dependency?


@wilkerlucio convinced me that we should leave :init in it ;)

bananadance 2

can I read the reasoning somewhere?


the main reason is that it can get more verbose


sure, but you lose the ability to put ‘expensive’ init, since it runs for all tasks


you can put an expensive init still in its own task


is 1 more word in each task really more verbose?


it is when you have to change from

(my-task "foo")
{:depends [init]
 :task (my-task "foo")}


you can have both options


yeah I think I agree


btw, I also introduced :requires so you can require expensive namespaces in tasks, so not for every task


i saw that in your work example


and all implicit aliases go away, e.g. no automatic fs


I think I like that too. I prefer less magic


more explicit


so if you want to use fs in multiple tasks you can do {:tasks {:requires ([babashka.fs :as fs])}}


or just for one:

{:tasks {foo {:requires ([babashka.fs :as fs])}}}


(built-in namespaces aren't expensive to require tho, interpreted ones can be)


I think I'll also include the idea from @maxp about :depends #{p1 p2}, it comes in quite handy for my work bb.edn


to automatically parallelize?


I think it’s better if that were not supported. Tasks should be parallelizable by default by virtue that they should list all their dependencies, anything not listed, then, must be able to run at the same time. if you need them to run in a certain order, then you define the tasks that way If you want dev to clean and then build you don’t say dev depends on clean and build, you say that dev depends on build which depends on clean


but build doesn't always depend on clean


I also want to be able to run build without clean


so having :depends [clean dev] with order semantics does cover that use while :depends #{p1 p2} means: don't care about the order


but can’t you just do bb run clean dev?


that doesn't work since clean can have command line args


how do you provide args to a task?


bb foo arg1 arg2


ok, so you can’t kick off multiple tasks at all


bb clean && bb dev


haha i mean within one invocation obviously


leiningen has this:

bb do clean foo bar, dev foo bar


so it uses the comma as a special delimiter in do


I kinda like that: bb do clean, dev


ah interesting


we could also have:

:pre [clean]
:post [foobar]


but yeah, it’s worth trying out the set dependencies thing


but :depends [clean] also kind of means :pre


yeah, supporting both seems it would get confusing quickly


:depends [foo x], if foo depends on x then x is executed first anyway, so it can't always guarantee the order, but it will try to do it if possible


feel free to convince me otherwise, but for me personally this is convenient. if it's really a bad idea I'd also like to know


you can also do:

clean:dev (do (shell "bb clean") (shell "bb dev"))
but maybe it's a bit ugly to have to start bb twice within bb

Duck Nebuchadnezzar14:04:20

What would be your recommended approach for something like this?

{:task (do (future (shell "bb watch-cljs")) (shell "bb run"))}

Duck Nebuchadnezzar14:04:20

That's one I use inside a docker container that starts shadowcljs and the main server.


@duck This would be solved if we implemented the :depends #{watch-cljs run} but now you can do bb run --parallel the-task, this will run the deps in parallel


not sure if this will stay in if we also support #{foo bar}

Duck Nebuchadnezzar14:04:30

yeah, I still need to grab the parallel updates


hello, just figured a cool snippet to copy things to clipboard (macos only) using babashka.process and wanted to share with you here 🙂

(proc/process "pbcopy" {:in "any string to put on clipboard"})

💯 2
Tomas Brejla10:04:05

on linux, this seems to work.

;; I have copied this text to my clipboard..

(-> (pipeline
     (pb '[timeout 0.05 xclip -selection clipboard -o])
     (pb ["cat"]))
;; => "I have copied this text to my clipboard.."
It could definitely be simplified. I'm just not experienced with babashka/process

Tomas Brejla10:04:17

probably something like this:

(ns core
  (:require [babashka.process :as proc :refer [$]]))

;; I have copied this text to my clipboard..
(-> ($ timeout 0.05 xclip -selection clipboard -o) deref :out slurp)
;; => "I have copied this text to my clipboard.."

;; store to clipboard
(do (proc/process '[xclip -sel clip] {:in "text to put to " :out :inherit}) nil)

;; read again from clipboard
(-> ($ timeout 0.05 xclip -selection clipboard -o) deref :out slurp)
;; => "text to put to "