Looks very interesting. I know you're building your tasks on top of AWS Step Functions, but are you planning to target different platforms? I'm interested in orchestrating work in Kubernetes, the Clojure way, but haven't invested much time and effort on that yet. Very cool to see more infrastructure tooling written and designed with the Clojure mindset.

Aside from this tool, in your post you mentioned you're solely using Clojure for everything, dropping other technologies. I'm particularly interested in your replacement to Terraform. I've invested much time with that tool and I found it quite compelling. Did you write a Clojure layer on top of it? Or are you using a different approach for infrastructure management?

Great suggestion, @U02BATFPN78, about keeping the library platform-agnostic. We're currently using Step Functions and don't have plans to support other platforms at the moment. While I'm not very familiar with Kubernetes, my understanding is that Step Functions operates at a higher layer, closer to the business logic than the infrastructure. I'm curious, could you explain what you mean by orchestrating work in Kubernetes? We're only using Stepwise.Next to replace Terraform for these state machines and associated assets, such as a couple of IAM roles and an S3 bucket. We'll continue using Terraform to manage the other 90% of our assets. If you're interested in replacing Terraform with Clojure, you might want to check out https://github.com/hashicorp/terraform-cdk. They already support using Java to declare Terraform resources. Someone is likely already working on porting that to Clojure.

What is the reason to have run-here and run-on-lambda as fns rather than part of the configuration?

(sfn/run-here client
              {make-sauce               {:concurrency 5}
               put-ingredients-on-dough {:concurrency 1}
               bake                     {:concurrency 1}})


(sfn/run-on-lambda client
                   {make-dough {:timeout         30
                                :memory-size     512
                                :max-concurrency 100}})

(defn make-dough ...
)  ;; the steps

(def pizza-making-sm ...
)  ;; the wiring / state machine

(def config {make-sauce {:platform :local :concurrency 5 ...}
             bake {:platform :lambda ...}
) ;; the technical details / infra / config

(sfn/run config pizza-making-sm)

> SFN error handling configuration as metadata this is interesting 🙂

Sounds interesting!! Have you considered “data driven” approach to describe the FSMs? I mean something similar what malli and reitit does? E.g.

[:sfn/name :pauls-pizza-making-machine
 [:sfn/parallel
  [[:fn make-dough :config {,,,}]
   [:fn make-sauce :config {,,,}]]]
 [:fn put-ingredients-on-dough :config {,,,}]
 [:sfn/wait 2 :minutes]
 [:sfn/choice
  [:test-fn (comp not is-pizza-acceptable?)
   [:sfn/fail]]]]

Nothing wrong with the DSL approach though. 🙂

@U8ZE1VBSS, am I reading your intention correctly that using a configuration map with a single entry point would make a more consistent API and follow principle of configuration as data? It would also make extensions to additional platforms (as Joshua suggested above) trivial for the API, as it's just adding additional acceptable values for :platform key. I chose run-* as fns because they are first-class operations of this API that produces a critical side effect. Saying it another way, the JAR that's meant to run on AWS Lambda should only spin up the Lambda workers. The JAR that runs in a container should only run the local processes. So we'd end up needing a separate interface between the different platforms. Moreover, we found from Stepwise v1 that we often want to isolate our local workers into different containers. e.g. we currently put (run-here client urgent-workers) in one container, and (run-here client batch-workers) in another container. We do that with a -main with CLI actions that points to each (i.e. $ my-app start urgent-workers and $ my-app start batch-workers from command line). Thoughts?

@U6N4HSMFW, Stepwise v1 actually takes an EDN as a state machine definition. We deliberately moved away from that because it was hard to read the gist of a control flow apart from the configurations. Libraries like reitit are routing libraries designed to manage the intricacies at a single junction in a procedure. From readability perspective, they have benefit of not needing to deal with time and surfacing details is a benefit but for us was a cost. The "aha" moment for this new interface was realizing that Clojure already has a fantastic procedural programming operation in -> and others. Therefore, we want the new interface to work as you would expect from ->. One compromise we made involves ops like sfn/parallel and sfn/map, which are similar but not exactly the same as map and apply. We could hide them behind map and apply to make the interface more Clojure-like, but we chose not to because 1) they are not exactly the same ops, and 2) they produce a side effect. I could put more thought into making them "just work" with Clojure fns, though...

Earlier this year, I wrote an ETL workflow in clojure that uses AWS Batch with spot instances. Stepwise seems really neat. Any idea how to compare the pricing of using stepwise vs Batch+Spot instances? It seems like Stepwise is focused more on an incremental approach. Is that right? Does it also support batching and scheduling?

@U7RJTCH6J workflows (AWS Step Functions) differ from ETL (AWS Batch) by that workflow is meant to be run many times but each execution is relatively small conceptually, e.g. make pizzas for customers. Whereas ETL is meant to process a big blob of a single thing but in many pieces, e.g. make me 1 ton of dough. So the two are not mutually exclusive. You can use SFN to coordinate Batch jobs (AWS provide built-in integrations of the two) as part of a larger workflow. e.g. a SFN workflow that runs BatchJob1 -> BatchJob2 -> Clojure Lambda Fn to cleanup meta data -> trigger SageMaker training -> ...

Stepwise also supports processing large blobs of things in many pieces, correct? My understanding is that your worker functions can simply return regular maps which will get serialized to persistent storage if they are large, and can be broken up into chunks using the step functions map task type. All with a convenient clojure interface. Is that right? I am really impressed by this by the way and looking forward to the next version!

@UDF1WUJTH that's correct! we have a feature to selectively/entirely offload the payload map to S3