Clojurians Log v2

Last update from my side: it's not generic enough.

(.addMethod ^clojure.lang.MultiFn handle-step :default (fn [step step-fns opts]
                                                               (let [[f _] (wire-fn step step-fns)]
                                                                 (f opts))))

Gemini fixed. I'm impressed.

I don't understand what you mean by "workflows" in this context, and I don't understand what the example is trying to convey. Could you start by defining what a "workflow" is?

Hi @weavejester, thank you for Integrant. Big fan of your work. To clarify, in the context of BigConfig, a "workflow" is a data-driven pipeline for managing infrastructure. Instead of manual orchestration—where you execute scripts in a specific order (e.g., terraform → ansible → helm)—a workflow allows you to: 1. Define a Desired State: You provide a static data structure representing what the infrastructure should look like. 2. Execute Transformations: This data is passed through a sequence of functions called steps. 3. Handle Side Effects: Much like a Clojure threading macro ->, each step in the workflow takes the state, performs a specific action (like provisioning a VPC or deploying a container), and passes the updated context to the next step. Essentially, the workflow is the engine that translates your declarative configuration into real-world resources.

Are the transformations of the data separate? i.e. do you have the data itself describing the infrastructure, and some series of transformation steps defined elsewhere that determine how to turn the data into side-effects that build the infrastructure?

Yes, that’s exactly right. There is a clean separation of concerns: 1. The Input (Pure Data): A static description of your infrastructure (the "What"). This is similar to a file or a Kubernetes manifest. 2. The Workflow (The Engine): A predefined series of transformation steps (the "How"). These steps are the logic that knows how to interpret your data and interact with external APIs to create resources. To use the Terraform analogy: the Data is your configuration code, and the Workflow is the Provider/CLI logic that determines the order of operations and handles the side effects. This separation allows you to swap out or test the transformation logic (the workflow) independently of the actual infrastructure definition (the data).

So the "steps" are part of the workflow engine. Presumably these are ordered, and you said that they each had an identifier?

This is exactly where BigConfig and Integrant diverge. While Integrant resolve a static graph of dependencies, BigConfig utilizes a `loop/recur` and qualified identifiers for its engine. The primary reasons for this are: • Error Handling & Cleanup: If a step fails, we need the flexibility to jump straight to a final 'clean-up' or 'teardown' phase rather than just halting or proceeding blindly. • Dynamic Execution: There are many use cases where the `next-step` is determined at runtime based on the output of the current step, rather than being hard-coded in a fixed order. I should also mention that you can find the detailed documentation for the ->workflow function https://bigconfig.it/api/core/#-workflow.

So the wire-fn connects the identifying keywords with their corresponding functions? Why not have that as a map or multimethod?

The wire-fn does connect keywords to their functions, but it also store the next step. I chose a function over a simple map or multimethod to allow for closures. In the case of ->workflow*, this is essential—it’s a https://github.com/amiorin/big-config/blob/7876edc8fc7f25fbef2f1d10cefc191c6672b764/src/clj/big_config/workflow.clj#L416 that uses closures to maintain state and scope under the hood while executing a workflow of workflows.

I've updated the implementation to use .addMethod programmatically. The new ->workflow-multi accepts the same arguments as before but produces a workflow where steps are resolved via multimethods with a fallback to functions.

(comment
  (debug tap-values
    (defmulti step-method (fn [step _step-fns _opts] step))

    (defmethod step-method ::start [step _ opts]
      (merge opts (ok) {step :custom-start}))

    (remove-method step-method ::start)

    (defn ->workflow-multi
      [{:keys [first-step last-step wire-fn]}]
      (fn [step-fns opts]
        (let [new-wire-fn (fn [step step-fns]
                            (let [[_ next-step] (wire-fn step step-fns)]
                              [(partial step-method step step-fns) next-step]))
              wf (->workflow {:first-step first-step
                              :last-step last-step
                              :wire-fn new-wire-fn})]
          (.addMethod ^clojure.lang.MultiFn step-method :default (fn [step step-fns opts]
                                                                   (let [[f _] (wire-fn step step-fns)]
                                                                     (f opts))))
          (wf step-fns opts))))

    (let [wf (->workflow-multi {:first-step ::start
                                :last-step ::end
                                :wire-fn (fn [step _]
                                           (case step
                                             ::start [#(merge % (ok) {step :default}) ::second]
                                             ::second [#(merge % (ok) {step :default}) ::end]
                                             ::end [identity]))})]
      (wf [] {})))
  (-> tap-values))