This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2021-05-24
Channels
- # announcements (1)
- # babashka (86)
- # beginners (75)
- # boot-dev (1)
- # cljdoc (18)
- # cljs-dev (1)
- # cljsrn (67)
- # clojure (127)
- # clojure-australia (1)
- # clojure-dev (2)
- # clojure-europe (9)
- # clojure-nl (2)
- # clojure-serbia (2)
- # clojure-spec (11)
- # clojure-uk (14)
- # clojurescript (17)
- # code-reviews (4)
- # conjure (37)
- # core-async (11)
- # datomic (21)
- # emacs (1)
- # helix (36)
- # jobs (6)
- # malli (1)
- # meander (20)
- # re-frame (13)
- # reagent (49)
- # remote-jobs (11)
- # rum (1)
- # sci (1)
- # shadow-cljs (29)
- # sql (17)
- # vim (2)
Couple of use cases I've been thinking about, wondering if tasks can be used to implement them: Specifying which tasks can be ran in parallel with each other and which can't Something similar to github actions strategy matrix Does it make sense or should I provide more details?
@ben.sless How it currently works: you can either provide --parallel
on the command line or (run 'task {:parallel true})
programmatically. What this means: all children will be ran in parallel (in so far the dependency tree allows it).
@ben.sless I've modeled it after use cases that I deemed common and useful. I wasn't aware of such a test matrix thingie, but feel free to tell more.
It's basically the same as make -j4
where you run steps in parallel with max threads = 4
I created a project for stress testing different servers under different configurations. Some servers aren't compatible with Java 8 Different Java versions can have different GC algorithms Different servers can handle different loads so it makes no sense to stress them all at the same rate This leads to a very large set of options and special cases and I'm looking for the best way to model it
Right now I just have some terrible imperative script https://github.com/bsless/stress-server/blob/master/run.clj
Imperative scripts aren't necessarily bad, it's usually what shell scripting is for.
As for executing such a thing with tasks, you could probably get a long way with just iteration and using run
and binding *command-line-args*
to pass command line args down to tasks. Or you could start new instances of bb
using shell
by providing env vars with :extra-env
Imperative scripts aren't bad, but I would prefer to describe it as much as I can as data and not code. Might end up reaching for dynamic variables instead, they're the in-process equivalent of an environment anyway
When I want to run such a thing in CircleCI or whatever, I usually generate the yaml using a script
It might be covered by (doseq [option options] (binding ,,, (run ,,))
However, even an alias for that would be extremely nice because the loses the whole mechanism of specifying dependencies
@ben.sless > because the loses the whole mechanism of specifying dependencies Not sure if I understood what you were saying there
I would have preferred specifying the dependency order of the parameters which need to be scanned for example: • jdk version • GC algorithms • server library as task dependencies i.e. profile -> (depends on) server -> gc -> jdk
I'll try playing around with it, see if it's sufficient and if I can provide overrides
Although theoretically the order doesn't matter in this part so it could be tasks aren't the best model
I'm sorry, I'm still confused by this Let's say I have this rough layout for tasks' dependencies and their components
{:init (do (defn print-args []
(prn (:name (current-task))
*command-line-args*)))
:enter (print-args)
init {:task {}}
jdk {:depends [init]
:init (def opts [8 11 15])}
server {:depends [jdk]
:init (def opts '[httpkit aleph])}
profile {:depends [server]
:task (println "profiling" *command-line-args*)}
}
how can I take advantage of the dependencies resolution order and not just have to resort to run
everythere?I can always do this:
(doseq [opt opts]
(binding [*command-line-args* (assoc *command-line-args* k opt)]
(run task)))
But doesn't it defeat the purpose?I think in jdk
you would use *jdk*
so only one. And then in some parent task you would bind *jdk*
to the row of values while invoking the dependency tree
@ben.sless I found a couple of edge cases (nicer word for bugs ;)), these should be fixed, but here's the idea with workaround for those issues:
{:tasks {:init (do
(ns my-ns)
(def ^:dynamic *jdk* nil)
;; workaround
(alter-meta! (var *jdk*) assoc :dynamic true)
(def ^:dynamic *server* nil)
(alter-meta! (var *server*) assoc :dynamic true))
:enter (println "Task:" (:name (babashka.tasks/current-task)))
jdk (println "JDK:" *jdk*)
server {:depends [jdk]
:task (println "Server:" *server*)}
run-all (doseq [jdk [8 11 15]
server [:foo :bar]]
(binding [*jdk* jdk
*server* server]
(babashka.tasks/run 'server)))}}
bb run-all
Task: run-all
Task: jdk
JDK: 8
Task: server
Server: :foo
Task: jdk
JDK: 8
Task: server
Server: :bar
Task: jdk
JDK: 11
...
The issues:
- dynamic vars aren't dynamic (due to loss of metadata during processing). Workaround: alter-meta!
.
- each run
runs in a random namespace, but all tasks should run in the same namespace. Workaround: set namespace manually and use fully qualified symbols for task built-ins.
Both issues should be fixed.
Made an issue here: https://github.com/babashka/babashka/issues/865
but does the idea make sense now @ben.sless (aside from the inconvenient issues)?
So run-all
sets the environment and the rest of the tasks just work as if there is one value at a time
ok, this makes sense, it's just the implementation I was hoping to avoid Ideally, I wouldn't want to spell out the iteration and binding manually
It isn't, it just expands the more options are involved. I'm trying to figure out the correct idiom
although I'm not sure if the .edn syntax can withstand the macro syntax. you could put it in a file on your classpath instead and then require it
There are two pieces which are orthogonal - one is the ask ordering which is handled correctly by tasks, one is creating the combination of options
This is an idea, and I'm not saying it should be integrated into the task running, but what do you think about the following enhancement to the syntax:
Tasks take an additional key, :matrix
. Besides *command-line-arguments*
add another global variable, *matrix*
which starts off as an empty map
Every task which has a :matrix
will run in the context of a doseq
over the parameters where *matrix*
is bound to (assoc *matrix* task-name param)
Roughly
{server {:depends [jdk]
:matrix [:a :b :c]}
jdk {:matrix [8 11 15]
:task (setup-jdk (*matrix* jdk))}
run {:depends [server]}}
This needs more hammock time, so I would prefer if this can be done in "user" space first. I was trying this:
(def matrix [{:jdk 11 :server "foo"} {:jdk 8 :server "bar"}])
(defn var-name [k]
(symbol (str "*" (name k) "*")))
(doseq [k (keys (first matrix))]
(intern *ns* (with-meta (var-name k)
{:dynamic true})))
(defn run-matrix []
(doseq [row matrix]
(let [ks (keys row)
vars (map (fn [k]
(resolve (var-name k))) ks)
vals (vals row)]
(with-bindings* (zipmap vars vals)
(fn []
(println (map deref vars)))))))
(run-matrix)
but somehow I get: Can't dynamically bind non-dynamic var: user/*jdk*
(in normal Clojure)
user=> (intern *ns* (with-meta '*foo* {:dynamic true}))
#'user/*foo*
user=> (binding [*foo* 2])
Execution error (IllegalStateException) at user/eval233 (REPL:1).
Can't dynamically bind non-dynamic var: user/*foo*
:thinking_face:run-matrix
loses the notion of task dependency order 😞
Also, why not just use a single dynamic variable *matrix*
then work in its context?
run-matrix was just an out of context demo, not related to tasks. you can use run
within the body of run-matrix.
"Why not just" assumes that this is a trivial thing to quickly add, which in my opinion it isn't, I have to think about it more for a while and see how often this comes up
so in user space I guess you could read from the *matrix*
your task's entries and run n times
yes. My hope was for a means to make that doseq implicit and still maintain the task ordering without having to run
them "manually"
well, you only have to run
the top-level one manually, but it's the bindings you would have to manage
I may have figured out a way to hack it, just need to break free of a recursive loop
This sort-of works:
{:min-bb-version "0.4.0"
:tasks
{:init (do
(ns user
(:require
[babashka.tasks :refer :all]))
(def ^:dynamic *matrix* {})
(alter-meta! (var *matrix*) assoc :dynamic true)
(defn $ [k] (get *matrix* k))
(defn on-enter []
(prn (:name (current-task)) *matrix*))
(defn enter-the-matrix
[]
(let [{:keys [name matrix]} (current-task)]
(when matrix
(when-not ($ name)
(doseq [e matrix]
(binding [*matrix* (assoc *matrix* name e)]
(run name))))))))
:enter (do (on-enter) (enter-the-matrix))
a {:depends [b]
:matrix [+ -]}
b {:depends [c x]
:matrix [:a :b]}
x {:matrix ["foo" "bar"]}
c {:matrix [1 2 3]}}}
I need a little help on my workflow. I'm using babashka with vim-iced. I open my script and connect with :IcedInstantConnect babashka
using nrepl. In my main script I want to leverage some util fns I have in utils.clj
which looks like this:
#!/usr/bin/env bb
(ns utils)
(def bar "barrrr")
Now in my main script I require it like so:
(ns my-script
(:require [babashka.fs :as fs]
[clojure.java.shell :as shell :refer [sh]]
[clojure.string :as str]
[utils :as ut]))
So, immediately after connect, in my my-script
buffer, I use :IcedRequire
and my stdout buffer shows me:
;;
;; Iced Buffer
;;
clojure.lang.ExceptionInfo: Could not find namespace: utils.
Now, I can switch to the utils.clj
buffer and use :IcedRequire
there, switch back to the main script buffer, and then use :IcedRequire
there successfully. I'm just not sure this is the correct workflow.
Could anyone offer any pointers as to how to do the above properly? As a follow-up, how do I then deal with changes to either the main script or the utils file? Do I have to reconnect vim-iced to a new babashka buffer and repeat the dance in order for it to see my latest changes?@borkdude I have a ~/bb.edn
but it only has:
{:tasks
{foo (shell "echo foo!!!")}}
I'm on babashka v0.4.0.And that particular bb.edn
should be in the same directory as the script and utils file?
if you don't like that, you can also set the classpath manually using babashka.classpath/add-classpath
The reason I don't have a src/
or something for this is because I'm just writing a small script in my repos folder to do a bit of housekeeping. I suppose I could move it to a repos/bb_scripts/
subfolder or to a different parent altogether.
the takeaway: if you want to use require
with your own scripts, make sure the classpath is set