@U01UPFK1M29 > pmaps sprinkled here and there, but it's not very managable • Since pmap is automatically configured relative to the number of cores - try to only have one pmap at a time and have the pmap fully utilise your machine (ie. be careful not to nest pmaps inside of pmaps inside of pmaps) • The https://github.com/clj-commons/claypoole is fantastic for giving the ease pmap with a bit more control - make sure you use the lazy namespace versions. > Is there a framework or tool that allows breaking down my data processing pipeline into small tasks • I would usually recommend integrant for this. It is single threaded - but you can use futures and claypoole/upmap to run steps in parallel with each other. You can visualise the graph. ◦ At https://sevva.ai/ with process huge datasets every day with integrant. • Alternatively https://www.lambda.cd/ (open source Clojure tool) is a pretty good at this and also provides nice UIs for starting stopping jobs. > I'd like to submit tasks in code dynamically, • Check if you really need this - it might be better to declaratively define your pipeline in a config file (e.g. with integrant) rather than have it dynamically in memory. Graphs structures that are changing or modifying in memory are harder to reason about. (e.g. we don't change production code in memory normally).

My goal is for the parts of Oz that handle notebook evaluation be sufficiently abstracted for programmatic use. But it may be a while longer before that's the case.

Good luck!