Fork me on GitHub
#babashka
<
2021-04-13
>
maxp03:04:32

About parallel :depends . what about

:depends ["sequential" ... ]
or
:depends #{"parallel" ..}
?

borkdude07:04:55

@maxp Interesting, I like the idea. What if you want to execute p1 sequentially and then p2 and p3 in parallel? [p1 #{p2 p3}]] ?

borkdude07:04:05

maybe that case is getting too complex and one should just make another step in between

pithyless07:04:33

:depends #{"parent" ...} implies that the task understands the internal details of parent and whether it can be parallelized. Having a task return future or value leaves that decision in the task. The latter kind of orchestration looks similar to the approach in pedestal interceptors and pathom resolvers: leave the details of sync/async to the the task itself, describe a tree of dependencies separately, and then a higher-level orchestrator is just responsible for calling each of them when it is appropriate (and blocking when necessary).

pithyless07:04:14

I figure, if it the orchestration gets any more complicated than that, maybe it's time to revisit a configuration ala onyx.

borkdude08:04:39

@pithyless I think for something like make / bb.edn it's the other way around, at least in some cases. At work I now have this:

{:tasks {dev-cljs {:doc "Runs front-end compilation"
                   :task (clojure "-M:frontend:cljs/dev")}
         dev-less {:doc "Compiles less"
                   :task (clojure "-M:frontend:less/dev")}
         dev-backend {:task (clojure (str "-A:backend:backend/dev:" platform-alias)
                                     "-X" "dre.standalone/start")}
         -dev {:depends [dev-cljs dev-less dev-backend]}
         dev {:doc "Runs app in dev mode. Compiles cljs, less and runs JVM app in parallel."
              :task (shell "bb run --parallel -dev")}}}
I want dev-cljs, dev-less and dev-backend to run in parallel, but those tasks don't need to know that I want to do that, because they can also run standalone. I make the decision in the dev task (by invoking bb with the --parallel option) but if the proposal of maxp was implemented, I could leave out the dummy -dev task.

borkdude08:04:22

I think make runs anything in parallel if you use -j4 (at most 4 jobs in parallel).

jeroenvandijk08:04:08

Wild idea and not properly thought through; if tasks are properly isolated, parallel could be the default, “sequential” could be a debugging option?

borkdude08:04:56

I think sequential has its place too perhaps. When you run everything in parallel, the output may look more garbled, etc. So having some way to control it might be nice

👍 3
borkdude08:04:25

Some people may depend on side effects happening in p1 first, etc

borkdude08:04:58

Although then p2 should depend on p1

maxp08:04:16

And good question - what means 'parallel' in this case. some special task does fork/join?

maxp08:04:29

btw, for my mind 'depends' does not mean 'execute in strict particular order'

borkdude08:04:32

A nice example provided by @grazfather is this one: https://gist.github.com/borkdude/356669d9b829fe5a891e44f920105f26 When you run it with bb coffeep it takes 11 seconds. When you run it with bb run --parallel coffeep it takes 6 seconds, because bb runs some of the dependencies in parallel.

borkdude08:04:22

@maxp I think that's reasonable: if you say #{p1 p2} you don't care about the order, so bb can (and will by default?) optimize this by running them side by side. And wait for all of them to finish before continuing.

pithyless08:04:40

What about just :depends [p1 p2] and the rule is always: all these tasks are dependencies that may by run in any-order and in parallel. Then you can supply a -j0 (sequential, any order) to -j4 to (in parallel, any order). If semantics are important (either ordering or using return values from one task in another) then those semantics will be explicit in the code - ie. don't use depends but write out the code in the :task ?

borkdude08:04:28

@pithyless I think the nice thing about #{p1 p2} is that you have finer control over which things you want to run in parallel. E.g. for my work tasks I have:

clean (do (shell "bash -c 'rm -rf frontend/public/js/app.js'")
                   (shell "bash -c 'rm -rf frontend/public/js/app.out'")
                   (shell "bash -c 'rm -rf frontend/public/css/main*'"))
         dev:clean {:depends [clean dev]}
and I want clean to finish before dev is started when I run dev:clean, but for my dev task here (https://clojurians.slack.com/archives/CLX41ASCS/p1618301439082200) I do want to run the deps in parallel

borkdude08:04:43

and I think you will use #{p1 p2} more sparingly, only for deps that take a few seconds or more probably

pithyless08:04:16

yeah, I understand; I actually do like the #{p1 p2} and [p1 p2] distinction; I'm just worried we may go down a sugar-syntax rabbit hole. (1) Do we support something like [p1 #{p2 p3} p4]? (2) Will the semantics be obvious to me when I look at a script in a month, without looking up the babashka docs? :)

borkdude08:04:30

@pithyless for [p1 #{p2 p3} p4] I intend not to support it. if you want this, then make it so: p0: depends [p1 p2 p3] p2: depends #{p4 p5}

pithyless08:04:45

@borkdude BTW, how does your dev example work in practice? are those 3 processes that startup and block? Do they garble the output or do they not print to stdout?

borkdude08:04:20

@pithyless those three processes just run in the background and the dev task waits for them to finish, which is never, or until I hit ctrl-c. I have a script for this:

#!/usr/bin/env bb

(ns dev)

(require '[babashka.process :refer [destroy-tree process]])

(def opts {:out :inherit
           :err :inherit
           :shutdown destroy-tree})

(defn cljs []
  (process ["clojure" "-Sforce" "-M:frontend:cljs/dev"] opts))

(defn less []
  (process ["clojure" "-Sforce" "-M:frontend:less/dev"] opts))

(defn clojure []
  (process ["clojure" "-Sforce"
            "-A:backend:backend/dev"
            "-X" "dre.standalone/start"]
           (assoc opts :in :inherit)))

(cljs)
(less)
(-> @(clojure) :exit (System/exit))
but now I can do all of this with bb tasks if I want

borkdude08:04:45

and yes, the output may become garbled, but in practice this doesn't bother me that much, doesn't happen that often

borkdude08:04:14

it's similar to starting processes in bash with &

borkdude09:04:07

Btw, I think it might be nice to release a stable version of bb tasks at ClojureD in June perhaps. You can vote for the talk topic here: https://twitter.com/borkdude/status/1381897272024305665

borkdude12:04:12

Click the link to see info about an informal meetup after the talk organized by Nikolay. It will be a kind of Russian Clojure meetup on Zoom. Join the Telegram channel for more info.

grazfather13:04:52

does it make more sense to have an :env block or something outside of :tasks, even, that is used to set vars?

grazfather13:04:09

e.g. for`platform-alias`

grazfather13:04:10

and i imagine that soon people will want to override these vars

borkdude13:04:54

@grazfather I thought about this, e.g :vars ... but what benefit would this have over :init?

grazfather13:04:13

an invoker could provide these

borkdude13:04:13

it would only be more restricting I thing, while introducing semantic problems, like ordering, etc

grazfather13:04:48

e.g. bb run dev-backend platform-alias=Solaris would avoid running the thing that normally sets platform-alias

borkdude13:04:32

I would say, just use an env var for that. BACKEND=foo bb dev and (def backend (or (System/getenv "BACKEND" ...))

👍 2
grazfather13:04:42

Again, I am just taking make semantics… I don’t think it’s something we really need to worry about

grazfather13:04:07

ok, yeah I agree

grazfather13:04:30

i think for tasks you should write a ‘recipe book’ about how to get things from make in BB. you showed the perfect example of one

grazfather13:04:58

did you change your mind about :init vs just a ‘recipe’` for init being a dependency?

borkdude13:04:15

@wilkerlucio convinced me that we should leave :init in it ;)

bananadance 2
grazfather13:04:05

can I read the reasoning somewhere?

borkdude13:04:32

the main reason is that it can get more verbose

grazfather13:04:08

sure, but you lose the ability to put ‘expensive’ init, since it runs for all tasks

borkdude13:04:22

you can put an expensive init still in its own task

grazfather13:04:24

is 1 more word in each task really more verbose?

wilkerlucio13:04:03

it is when you have to change from

(my-task "foo")
to:
{:depends [init]
 :task (my-task "foo")}

borkdude13:04:27

you can have both options

grazfather13:04:34

yeah I think I agree

borkdude13:04:46

btw, I also introduced :requires so you can require expensive namespaces in tasks, so not for every task

grazfather13:04:58

i saw that in your work example

borkdude13:04:09

and all implicit aliases go away, e.g. no automatic fs

grazfather13:04:22

I think I like that too. I prefer less magic

grazfather13:04:28

more explicit

borkdude13:04:55

so if you want to use fs in multiple tasks you can do {:tasks {:requires ([babashka.fs :as fs])}}

borkdude13:04:17

or just for one:

{:tasks {foo {:requires ([babashka.fs :as fs])}}}

borkdude13:04:40

(built-in namespaces aren't expensive to require tho, interpreted ones can be)

borkdude13:04:26

I think I'll also include the idea from @maxp about :depends #{p1 p2}, it comes in quite handy for my work bb.edn

grazfather13:04:33

to automatically parallelize?

grazfather14:04:32

I think it’s better if that were not supported. Tasks should be parallelizable by default by virtue that they should list all their dependencies, anything not listed, then, must be able to run at the same time. if you need them to run in a certain order, then you define the tasks that way If you want dev to clean and then build you don’t say dev depends on clean and build, you say that dev depends on build which depends on clean

borkdude14:04:31

but build doesn't always depend on clean

borkdude14:04:42

I also want to be able to run build without clean

borkdude14:04:02

so having :depends [clean dev] with order semantics does cover that use while :depends #{p1 p2} means: don't care about the order

grazfather14:04:02

but can’t you just do bb run clean dev?

borkdude14:04:27

that doesn't work since clean can have command line args

grazfather14:04:52

how do you provide args to a task?

borkdude14:04:01

bb foo arg1 arg2

grazfather14:04:21

ok, so you can’t kick off multiple tasks at all

borkdude14:04:32

bb clean && bb dev

grazfather14:04:49

haha i mean within one invocation obviously

borkdude14:04:50

leiningen has this:

bb do clean foo bar, dev foo bar

borkdude14:04:23

so it uses the comma as a special delimiter in do

borkdude14:04:50

I kinda like that: bb do clean, dev

grazfather14:04:59

ah interesting

borkdude14:04:13

we could also have:

:pre [clean]
:post [foobar]

grazfather14:04:14

but yeah, it’s worth trying out the set dependencies thing

borkdude14:04:57

but :depends [clean] also kind of means :pre

grazfather14:04:26

yeah, supporting both seems it would get confusing quickly

borkdude14:04:43

:depends [foo x], if foo depends on x then x is executed first anyway, so it can't always guarantee the order, but it will try to do it if possible

borkdude14:04:39

feel free to convince me otherwise, but for me personally this is convenient. if it's really a bad idea I'd also like to know

borkdude14:04:57

you can also do:

clean:dev (do (shell "bb clean") (shell "bb dev"))
but maybe it's a bit ugly to have to start bb twice within bb

Duck Nebuchadnezzar14:04:20

What would be your recommended approach for something like this?

{:task (do (future (shell "bb watch-cljs")) (shell "bb run"))}

Duck Nebuchadnezzar14:04:20

That's one I use inside a docker container that starts shadowcljs and the main server.

borkdude14:04:40

@duck This would be solved if we implemented the :depends #{watch-cljs run} but now you can do bb run --parallel the-task, this will run the deps in parallel

borkdude14:04:06

not sure if this will stay in if we also support #{foo bar}

Duck Nebuchadnezzar14:04:30

yeah, I still need to grab the parallel updates

wilkerlucio20:04:36

hello, just figured a cool snippet to copy things to clipboard (macos only) using babashka.process and wanted to share with you here 🙂

(proc/process "pbcopy" {:in "any string to put on clipboard"})

9
💯 2
Tomas Brejla10:04:05

on linux, this seems to work.

;; I have copied this text to my clipboard..

(-> (pipeline
     (pb '[timeout 0.05 xclip -selection clipboard -o])
     (pb ["cat"]))
    last
    :out
    slurp)
;; => "I have copied this text to my clipboard.."
It could definitely be simplified. I'm just not experienced with babashka/process

Tomas Brejla10:04:17

probably something like this:

(ns core
  (:require [babashka.process :as proc :refer [$]]))

;; I have copied this text to my clipboard..
(-> ($ timeout 0.05 xclip -selection clipboard -o) deref :out slurp)
;; => "I have copied this text to my clipboard.."

;; store to clipboard
(do (proc/process '[xclip -sel clip] {:in "text to put to " :out :inherit}) nil)

;; read again from clipboard
(-> ($ timeout 0.05 xclip -selection clipboard -o) deref :out slurp)
;; => "text to put to "