Fork me on GitHub
Ben Sless08:05:40

Couple of use cases I've been thinking about, wondering if tasks can be used to implement them: Specifying which tasks can be ran in parallel with each other and which can't Something similar to github actions strategy matrix Does it make sense or should I provide more details?


@ben.sless How it currently works: you can either provide --parallel on the command line or (run 'task {:parallel true}) programmatically. What this means: all children will be ran in parallel (in so far the dependency tree allows it).

Ben Sless08:05:10

So there's no fine-grained support in that resolution

Ben Sless08:05:30

Regarding an expanding test matrix, is it possible to model something like that?

Ben Sless08:05:01

including overrides and and exclusions


@ben.sless I've modeled it after use cases that I deemed common and useful. I wasn't aware of such a test matrix thingie, but feel free to tell more.


It's basically the same as make -j4 where you run steps in parallel with max threads = 4

Ben Sless08:05:20

I created a project for stress testing different servers under different configurations. Some servers aren't compatible with Java 8 Different Java versions can have different GC algorithms Different servers can handle different loads so it makes no sense to stress them all at the same rate This leads to a very large set of options and special cases and I'm looking for the best way to model it

Ben Sless08:05:51

Right now I just have some terrible imperative script


Imperative scripts aren't necessarily bad, it's usually what shell scripting is for. As for executing such a thing with tasks, you could probably get a long way with just iteration and using run and binding *command-line-args* to pass command line args down to tasks. Or you could start new instances of bb using shell by providing env vars with :extra-env

Ben Sless08:05:52

Imperative scripts aren't bad, but I would prefer to describe it as much as I can as data and not code. Might end up reaching for dynamic variables instead, they're the in-process equivalent of an environment anyway


When I want to run such a thing in CircleCI or whatever, I usually generate the yaml using a script


and don't bother learning their matrix DSL at all :)


but if the use case is commons enough, we could consider it later on

Ben Sless08:05:57

It might be covered by (doseq [option options] (binding ,,, (run ,,)) However, even an alias for that would be extremely nice because the loses the whole mechanism of specifying dependencies


@ben.sless > because the loses the whole mechanism of specifying dependencies Not sure if I understood what you were saying there

Ben Sless09:05:45

I would have preferred specifying the dependency order of the parameters which need to be scanned for example: • jdk version • GC algorithms • server library as task dependencies i.e. profile -> (depends on) server -> gc -> jdk

Ben Sless09:05:01

where each level of traversal up the tree will expand to the possible options


that still works if you invoke run and bind to different values of those options

Ben Sless09:05:35

I'll try playing around with it, see if it's sufficient and if I can provide overrides

Ben Sless09:05:22

Although theoretically the order doesn't matter in this part so it could be tasks aren't the best model

Ben Sless09:05:47

I'm sorry, I'm still confused by this Let's say I have this rough layout for tasks' dependencies and their components

{:init (do (defn print-args []
              (prn (:name (current-task))
  :enter (print-args)
  init {:task {}}
  jdk {:depends [init]
       :init (def opts [8 11 15])}
  server {:depends [jdk]
          :init (def opts '[httpkit aleph])}
  profile {:depends [server]
           :task (println "profiling" *command-line-args*)}
how can I take advantage of the dependencies resolution order and not just have to resort to run everythere?

Ben Sless09:05:37

I can always do this:

(doseq [opt opts]
  (binding [*command-line-args* (assoc *command-line-args* k opt)]
    (run task)))
But doesn't it defeat the purpose?


I think in jdk you would use *jdk* so only one. And then in some parent task you would bind *jdk* to the row of values while invoking the dependency tree


same for server


let me try to make an example

Ben Sless09:05:51

Please do, It's not clicking for me at all

Ben Sless09:05:20

Very possible I'm trying to square the circle with babashka here


@ben.sless I found a couple of edge cases (nicer word for bugs ;)), these should be fixed, but here's the idea with workaround for those issues:

{:tasks {:init (do
                 (ns my-ns)
                 (def ^:dynamic *jdk* nil)
                 ;; workaround
                 (alter-meta! (var *jdk*) assoc :dynamic true)
                 (def ^:dynamic *server* nil)
                 (alter-meta! (var *server*) assoc :dynamic true))
         :enter (println "Task:" (:name (babashka.tasks/current-task)))
         jdk (println "JDK:" *jdk*)
         server {:depends [jdk]
                 :task (println "Server:" *server*)}
         run-all (doseq [jdk [8 11 15]
                         server [:foo :bar]]
                   (binding [*jdk* jdk
                             *server* server]
                     (babashka.tasks/run 'server)))}}
bb run-all
Task: run-all
Task: jdk
JDK: 8
Task: server
Server: :foo
Task: jdk
JDK: 8
Task: server
Server: :bar
Task: jdk
JDK: 11


The issues: - dynamic vars aren't dynamic (due to loss of metadata during processing). Workaround: alter-meta!. - each run runs in a random namespace, but all tasks should run in the same namespace. Workaround: set namespace manually and use fully qualified symbols for task built-ins. Both issues should be fixed.


but does the idea make sense now @ben.sless (aside from the inconvenient issues)?


So run-all sets the environment and the rest of the tasks just work as if there is one value at a time

Ben Sless11:05:44

I'll get some food in me and write a coherent response

Ben Sless12:05:15

ok, this makes sense, it's just the implementation I was hoping to avoid Ideally, I wouldn't want to spell out the iteration and binding manually


although (aside from the issues) it doesn't seem too much boilerplate

Ben Sless12:05:36

It isn't, it just expands the more options are involved. I'm trying to figure out the correct idiom


you could write a small macro in :init which expands these options ;)


although I'm not sure if the .edn syntax can withstand the macro syntax. you could put it in a file on your classpath instead and then require it

Ben Sless12:05:41

There are two pieces which are orthogonal - one is the ask ordering which is handled correctly by tasks, one is creating the combination of options


it could probably be just a function too, since with-bindings* is a normal function

Ben Sless12:05:34

This is an idea, and I'm not saying it should be integrated into the task running, but what do you think about the following enhancement to the syntax: Tasks take an additional key, :matrix. Besides *command-line-arguments* add another global variable, *matrix* which starts off as an empty map Every task which has a :matrix will run in the context of a doseq over the parameters where *matrix* is bound to (assoc *matrix* task-name param)

Ben Sless12:05:57


{server {:depends [jdk]
         :matrix [:a :b :c]}
 jdk {:matrix [8 11 15]
      :task (setup-jdk (*matrix* jdk))}
 run {:depends [server]}}


This needs more hammock time, so I would prefer if this can be done in "user" space first. I was trying this:

(def matrix [{:jdk 11 :server "foo"} {:jdk 8 :server "bar"}])

(defn var-name [k]
  (symbol (str "*" (name k) "*")))

(doseq [k (keys (first matrix))]
  (intern *ns* (with-meta (var-name k)
                 {:dynamic true})))

(defn run-matrix []
  (doseq [row matrix]
    (let [ks (keys row)
          vars (map (fn [k]
                      (resolve (var-name k))) ks)
          vals (vals row)]
      (with-bindings* (zipmap vars vals)
        (fn []
          (println (map deref vars)))))))



but somehow I get: Can't dynamically bind non-dynamic var: user/*jdk* (in normal Clojure)


user=> (intern *ns* (with-meta '*foo* {:dynamic true}))
user=> (binding [*foo* 2])
Execution error (IllegalStateException) at user/eval233 (REPL:1).
Can't dynamically bind non-dynamic var: user/*foo*

Ben Sless13:05:49

run-matrix loses the notion of task dependency order 😞 Also, why not just use a single dynamic variable *matrix* then work in its context?


run-matrix was just an out of context demo, not related to tasks. you can use run within the body of run-matrix.


"Why not just" assumes that this is a trivial thing to quickly add, which in my opinion it isn't, I have to think about it more for a while and see how often this comes up

Ben Sless13:05:42

I meant in the demo, not in babashka's guts, sorry if that wasn't clear

Ben Sless13:05:48

seems easier than interning vars on the fly

Ben Sless13:05:05

not trying to push 🙂


yeah, that can work too :)


that probably simplifies things a bit


so in user space I guess you could read from the *matrix* your task's entries and run n times


that won't work either I think


you need some kind of doseq like construction on the top level

Ben Sless13:05:14

yes. My hope was for a means to make that doseq implicit and still maintain the task ordering without having to run them "manually"


well, you only have to run the top-level one manually, but it's the bindings you would have to manage

Ben Sless15:05:05

I may have figured out a way to hack it, just need to break free of a recursive loop

Ben Sless15:05:03

Alright, got it, I'll send it over when I get WiFi on my machine

Ben Sless15:05:13

This sort-of works:

{:min-bb-version "0.4.0"
 {:init (do
          (ns user
             [babashka.tasks :refer :all]))
          (def ^:dynamic *matrix* {})
          (alter-meta! (var *matrix*) assoc :dynamic true)
          (defn $ [k] (get *matrix* k))
          (defn on-enter []
            (prn (:name (current-task)) *matrix*))
          (defn enter-the-matrix
            (let [{:keys [name matrix]} (current-task)]
              (when matrix
                (when-not ($ name)
                  (doseq [e matrix]
                    (binding [*matrix* (assoc *matrix* name e)]
                      (run name))))))))
  :enter (do (on-enter) (enter-the-matrix))
  a {:depends [b]
     :matrix [+ -]}
  b {:depends [c x]
     :matrix [:a :b]}
  x {:matrix ["foo" "bar"]}
  c {:matrix [1 2 3]}}}


that's pretty cool :)

Ben Sless15:05:25

Yeah, but it's incorrect 🙃

Ben Sless15:05:46

Which is evident by the printed output


I need a little help on my workflow. I'm using babashka with vim-iced. I open my script and connect with :IcedInstantConnect babashka using nrepl. In my main script I want to leverage some util fns I have in utils.clj which looks like this:

#!/usr/bin/env bb
(ns utils)

(def bar "barrrr")
Now in my main script I require it like so:
(ns my-script
  (:require [babashka.fs :as fs]
            [ :as shell :refer [sh]]
            [clojure.string :as str]
            [utils :as ut]))
So, immediately after connect, in my my-script buffer, I use :IcedRequire and my stdout buffer shows me:
;; Iced Buffer
clojure.lang.ExceptionInfo: Could not find namespace: utils.
Now, I can switch to the utils.clj buffer and use :IcedRequire there, switch back to the main script buffer, and then use :IcedRequire there successfully. I'm just not sure this is the correct workflow. Could anyone offer any pointers as to how to do the above properly? As a follow-up, how do I then deal with changes to either the main script or the utils file? Do I have to reconnect vim-iced to a new babashka buffer and repeat the dance in order for it to see my latest changes?


@randumbo Do you use bb.edn perhaps? Which version are you on?


You need to set the classpath so babashka knows where it can find utils.clj


And the easiest way to do this now is using bb.edn :paths


@borkdude I have a ~/bb.edn but it only has:

 {foo (shell "echo foo!!!")}}
I'm on babashka v0.4.0.


and where is your utils.clj file?


Same directory as my script that I am calling.


ok, then add to bb.edn: paths ["."]


it's a bit unusual to do this, normally you use a subdirectory like src


And that particular bb.edn should be in the same directory as the script and utils file?


similar to project.clj and deps.edn


if you don't like that, you can also set the classpath manually using babashka.classpath/add-classpath


or load the file using load-file instead of require


The reason I don't have a src/ or something for this is because I'm just writing a small script in my repos folder to do a bit of housekeeping. I suppose I could move it to a repos/bb_scripts/ subfolder or to a different parent altogether.


the takeaway: if you want to use require with your own scripts, make sure the classpath is set


Yep, got that. Given that I do that, are changes to those files then automatically picked up or do I have to do another step to re-eval their contents?


if you want to re-eval you can use (require 'foo :reload)


Thanks very much. That should cover it.