Fork me on GitHub
#babashka
<
2021-05-24
>
Ben Sless08:05:40

Couple of use cases I've been thinking about, wondering if tasks can be used to implement them: Specifying which tasks can be ran in parallel with each other and which can't Something similar to github actions strategy matrix Does it make sense or should I provide more details?

borkdude08:05:52

@ben.sless How it currently works: you can either provide --parallel on the command line or (run 'task {:parallel true}) programmatically. What this means: all children will be ran in parallel (in so far the dependency tree allows it).

Ben Sless08:05:10

So there's no fine-grained support in that resolution

Ben Sless08:05:30

Regarding an expanding test matrix, is it possible to model something like that?

Ben Sless08:05:01

including overrides and and exclusions

borkdude08:05:48

@ben.sless I've modeled it after use cases that I deemed common and useful. I wasn't aware of such a test matrix thingie, but feel free to tell more.

borkdude08:05:20

It's basically the same as make -j4 where you run steps in parallel with max threads = 4

Ben Sless08:05:20

I created a project for stress testing different servers under different configurations. Some servers aren't compatible with Java 8 Different Java versions can have different GC algorithms Different servers can handle different loads so it makes no sense to stress them all at the same rate This leads to a very large set of options and special cases and I'm looking for the best way to model it

Ben Sless08:05:51

Right now I just have some terrible imperative script https://github.com/bsless/stress-server/blob/master/run.clj

borkdude08:05:09

Imperative scripts aren't necessarily bad, it's usually what shell scripting is for. As for executing such a thing with tasks, you could probably get a long way with just iteration and using run and binding *command-line-args* to pass command line args down to tasks. Or you could start new instances of bb using shell by providing env vars with :extra-env

Ben Sless08:05:52

Imperative scripts aren't bad, but I would prefer to describe it as much as I can as data and not code. Might end up reaching for dynamic variables instead, they're the in-process equivalent of an environment anyway

borkdude08:05:39

When I want to run such a thing in CircleCI or whatever, I usually generate the yaml using a script

borkdude08:05:50

and don't bother learning their matrix DSL at all :)

borkdude08:05:07

but if the use case is commons enough, we could consider it later on

Ben Sless08:05:57

It might be covered by (doseq [option options] (binding ,,, (run ,,)) However, even an alias for that would be extremely nice because the loses the whole mechanism of specifying dependencies

borkdude09:05:18

@ben.sless > because the loses the whole mechanism of specifying dependencies Not sure if I understood what you were saying there

Ben Sless09:05:45

I would have preferred specifying the dependency order of the parameters which need to be scanned for example: • jdk version • GC algorithms • server library as task dependencies i.e. profile -> (depends on) server -> gc -> jdk

Ben Sless09:05:01

where each level of traversal up the tree will expand to the possible options

borkdude09:05:22

that still works if you invoke run and bind to different values of those options

Ben Sless09:05:35

I'll try playing around with it, see if it's sufficient and if I can provide overrides

Ben Sless09:05:22

Although theoretically the order doesn't matter in this part so it could be tasks aren't the best model

Ben Sless09:05:47

I'm sorry, I'm still confused by this Let's say I have this rough layout for tasks' dependencies and their components

{:init (do (defn print-args []
              (prn (:name (current-task))
                   *command-line-args*)))
  :enter (print-args)
  init {:task {}}
  jdk {:depends [init]
       :init (def opts [8 11 15])}
  server {:depends [jdk]
          :init (def opts '[httpkit aleph])}
  profile {:depends [server]
           :task (println "profiling" *command-line-args*)}
  }
how can I take advantage of the dependencies resolution order and not just have to resort to run everythere?

Ben Sless09:05:37

I can always do this:

(doseq [opt opts]
  (binding [*command-line-args* (assoc *command-line-args* k opt)]
    (run task)))
But doesn't it defeat the purpose?

borkdude09:05:47

I think in jdk you would use *jdk* so only one. And then in some parent task you would bind *jdk* to the row of values while invoking the dependency tree

borkdude09:05:13

same for server

borkdude09:05:30

let me try to make an example

Ben Sless09:05:51

Please do, It's not clicking for me at all

Ben Sless09:05:20

Very possible I'm trying to square the circle with babashka here

borkdude10:05:38

@ben.sless I found a couple of edge cases (nicer word for bugs ;)), these should be fixed, but here's the idea with workaround for those issues:

{:tasks {:init (do
                 (ns my-ns)
                 (def ^:dynamic *jdk* nil)
                 ;; workaround
                 (alter-meta! (var *jdk*) assoc :dynamic true)
                 (def ^:dynamic *server* nil)
                 (alter-meta! (var *server*) assoc :dynamic true))
         :enter (println "Task:" (:name (babashka.tasks/current-task)))
         jdk (println "JDK:" *jdk*)
         server {:depends [jdk]
                 :task (println "Server:" *server*)}
         run-all (doseq [jdk [8 11 15]
                         server [:foo :bar]]
                   (binding [*jdk* jdk
                             *server* server]
                     (babashka.tasks/run 'server)))}}
bb run-all
Task: run-all
Task: jdk
JDK: 8
Task: server
Server: :foo
Task: jdk
JDK: 8
Task: server
Server: :bar
Task: jdk
JDK: 11
...

borkdude10:05:27

The issues: - dynamic vars aren't dynamic (due to loss of metadata during processing). Workaround: alter-meta!. - each run runs in a random namespace, but all tasks should run in the same namespace. Workaround: set namespace manually and use fully qualified symbols for task built-ins. Both issues should be fixed.

borkdude10:05:22

but does the idea make sense now @ben.sless (aside from the inconvenient issues)?

borkdude10:05:38

So run-all sets the environment and the rest of the tasks just work as if there is one value at a time

Ben Sless11:05:44

I'll get some food in me and write a coherent response

Ben Sless12:05:15

ok, this makes sense, it's just the implementation I was hoping to avoid Ideally, I wouldn't want to spell out the iteration and binding manually

borkdude12:05:14

although (aside from the issues) it doesn't seem too much boilerplate

Ben Sless12:05:36

It isn't, it just expands the more options are involved. I'm trying to figure out the correct idiom

borkdude12:05:14

you could write a small macro in :init which expands these options ;)

borkdude12:05:50

although I'm not sure if the .edn syntax can withstand the macro syntax. you could put it in a file on your classpath instead and then require it

Ben Sless12:05:41

There are two pieces which are orthogonal - one is the ask ordering which is handled correctly by tasks, one is creating the combination of options

borkdude12:05:43

it could probably be just a function too, since with-bindings* is a normal function

Ben Sless12:05:34

This is an idea, and I'm not saying it should be integrated into the task running, but what do you think about the following enhancement to the syntax: Tasks take an additional key, :matrix. Besides *command-line-arguments* add another global variable, *matrix* which starts off as an empty map Every task which has a :matrix will run in the context of a doseq over the parameters where *matrix* is bound to (assoc *matrix* task-name param)

Ben Sless12:05:57

Roughly

{server {:depends [jdk]
         :matrix [:a :b :c]}
 jdk {:matrix [8 11 15]
      :task (setup-jdk (*matrix* jdk))}
 run {:depends [server]}}

borkdude12:05:46

This needs more hammock time, so I would prefer if this can be done in "user" space first. I was trying this:

(def matrix [{:jdk 11 :server "foo"} {:jdk 8 :server "bar"}])

(defn var-name [k]
  (symbol (str "*" (name k) "*")))

(doseq [k (keys (first matrix))]
  (intern *ns* (with-meta (var-name k)
                 {:dynamic true})))

(defn run-matrix []
  (doseq [row matrix]
    (let [ks (keys row)
          vars (map (fn [k]
                      (resolve (var-name k))) ks)
          vals (vals row)]
      (with-bindings* (zipmap vars vals)
        (fn []
          (println (map deref vars)))))))

(run-matrix)

borkdude12:05:08

but somehow I get: Can't dynamically bind non-dynamic var: user/*jdk* (in normal Clojure)

borkdude12:05:12

user=> (intern *ns* (with-meta '*foo* {:dynamic true}))
#'user/*foo*
user=> (binding [*foo* 2])
Execution error (IllegalStateException) at user/eval233 (REPL:1).
Can't dynamically bind non-dynamic var: user/*foo*
:thinking_face:

Ben Sless13:05:49

run-matrix loses the notion of task dependency order 😞 Also, why not just use a single dynamic variable *matrix* then work in its context?

borkdude13:05:35

run-matrix was just an out of context demo, not related to tasks. you can use run within the body of run-matrix.

borkdude13:05:10

"Why not just" assumes that this is a trivial thing to quickly add, which in my opinion it isn't, I have to think about it more for a while and see how often this comes up

Ben Sless13:05:42

I meant in the demo, not in babashka's guts, sorry if that wasn't clear

Ben Sless13:05:48

seems easier than interning vars on the fly

Ben Sless13:05:05

not trying to push 🙂

borkdude13:05:38

yeah, that can work too :)

borkdude13:05:01

that probably simplifies things a bit

borkdude13:05:46

so in user space I guess you could read from the *matrix* your task's entries and run n times

borkdude13:05:04

that won't work either I think

borkdude13:05:33

you need some kind of doseq like construction on the top level

Ben Sless13:05:14

yes. My hope was for a means to make that doseq implicit and still maintain the task ordering without having to run them "manually"

borkdude13:05:09

well, you only have to run the top-level one manually, but it's the bindings you would have to manage

Ben Sless15:05:05

I may have figured out a way to hack it, just need to break free of a recursive loop

Ben Sless15:05:03

Alright, got it, I'll send it over when I get WiFi on my machine

Ben Sless15:05:13

This sort-of works:

{:min-bb-version "0.4.0"
 :tasks
 {:init (do
          (ns user
            (:require
             [babashka.tasks :refer :all]))
          (def ^:dynamic *matrix* {})
          (alter-meta! (var *matrix*) assoc :dynamic true)
          (defn $ [k] (get *matrix* k))
          (defn on-enter []
            (prn (:name (current-task)) *matrix*))
          (defn enter-the-matrix
            []
            (let [{:keys [name matrix]} (current-task)]
              (when matrix
                (when-not ($ name)
                  (doseq [e matrix]
                    (binding [*matrix* (assoc *matrix* name e)]
                      (run name))))))))
  :enter (do (on-enter) (enter-the-matrix))
  a {:depends [b]
     :matrix [+ -]}
  b {:depends [c x]
     :matrix [:a :b]}
  x {:matrix ["foo" "bar"]}
  c {:matrix [1 2 3]}}}

borkdude15:05:56

that's pretty cool :)

Ben Sless15:05:25

Yeah, but it's incorrect 🙃

Ben Sless15:05:46

Which is evident by the printed output

Buidler16:05:47

I need a little help on my workflow. I'm using babashka with vim-iced. I open my script and connect with :IcedInstantConnect babashka using nrepl. In my main script I want to leverage some util fns I have in utils.clj which looks like this:

#!/usr/bin/env bb
(ns utils)

(def bar "barrrr")
Now in my main script I require it like so:
(ns my-script
  (:require [babashka.fs :as fs]
            [clojure.java.shell :as shell :refer [sh]]
            [clojure.string :as str]
            [utils :as ut]))
So, immediately after connect, in my my-script buffer, I use :IcedRequire and my stdout buffer shows me:
;;
;; Iced Buffer
;;
clojure.lang.ExceptionInfo: Could not find namespace: utils.
Now, I can switch to the utils.clj buffer and use :IcedRequire there, switch back to the main script buffer, and then use :IcedRequire there successfully. I'm just not sure this is the correct workflow. Could anyone offer any pointers as to how to do the above properly? As a follow-up, how do I then deal with changes to either the main script or the utils file? Do I have to reconnect vim-iced to a new babashka buffer and repeat the dance in order for it to see my latest changes?

borkdude16:05:38

@randumbo Do you use bb.edn perhaps? Which version are you on?

borkdude16:05:57

You need to set the classpath so babashka knows where it can find utils.clj

borkdude16:05:07

And the easiest way to do this now is using bb.edn :paths

Buidler16:05:54

@borkdude I have a ~/bb.edn but it only has:

{:tasks
 {foo (shell "echo foo!!!")}}
I'm on babashka v0.4.0.

borkdude16:05:09

and where is your utils.clj file?

Buidler16:05:20

Same directory as my script that I am calling.

borkdude16:05:38

ok, then add to bb.edn: paths ["."]

borkdude16:05:13

it's a bit unusual to do this, normally you use a subdirectory like src

Buidler16:05:06

And that particular bb.edn should be in the same directory as the script and utils file?

borkdude16:05:52

similar to project.clj and deps.edn

borkdude16:05:45

if you don't like that, you can also set the classpath manually using babashka.classpath/add-classpath

borkdude16:05:58

or load the file using load-file instead of require

Buidler16:05:07

The reason I don't have a src/ or something for this is because I'm just writing a small script in my repos folder to do a bit of housekeeping. I suppose I could move it to a repos/bb_scripts/ subfolder or to a different parent altogether.

borkdude16:05:54

the takeaway: if you want to use require with your own scripts, make sure the classpath is set

Buidler16:05:44

Yep, got that. Given that I do that, are changes to those files then automatically picked up or do I have to do another step to re-eval their contents?

borkdude16:05:42

if you want to re-eval you can use (require 'foo :reload)

Buidler16:05:23

Thanks very much. That should cover it.