missionary

braai engineer 2025-02-19T09:53:30.384269Z

Would be cool if there was a Missionary version of https://docs.celeryq.dev/en/stable/getting-started/introduction.html to run tasks and handle retries, support parent/child tasks.

xificurC 2025-02-19T09:57:22.446859Z

Do you mean helper functions to work with missionary tasks in the same process or distribute tasks across several processes?

braai engineer 2025-02-19T09:59:44.873929Z

Several processes (clones of the same codebase) that talk over a distributed queue. I want to spin up multiple workers to sync stuff, send email, etc. We have a very gross internal tasks module implementation that seems fraught with deadlocks at the outset, and I want to get rid of it. To-date there isn't really anything like Celery in the Clojure space. It seems to me that Missionary would be well-suited to gracefully handling errors, retrying external APIs with backpressure 🙂

xificurC 2025-02-19T10:33:15.916059Z

If you already have a message queue set up for distributing your "very gross internal tasks", I guess you could e.g. send messages containing a task builder var's symbol + arguments to build a missionary task and run it

(defn my-task-builder [x y] (m/sp (+ x y)))
(mq/send [`my-task-builder 1 2])
(let [[f$ & args] (mq/recv), f (resolve f$)]
  (run (apply f args)))