Clojurians Log v2
beginners 2025-06-07
Hi, I am intrigued by what https://github.com/plumatic/plumbing?tab=readme-ov-file#graph-the-functional-swiss-army-knife has to offer. I am trying to figure out whether this is intended to be used for pure function workflows (e.g. complex data manipulation and calculation), or if it can also be useful when stitching together functions with side-effects. E.g. a web request comes in, does some data validation, pulls some additional contextual data from DB, and/or goes through a ELT pipeline. Does it make sense to use Graph for this kind of stuff?
I ended up using https://github.com/nubank/nodely. Pathom introduces too many new concepts for my liking when the thing that I want to achieve is very simple. Works like a charm
Pathom. ?
Thanks for your input. Maybe I am just put off a bit by the fact that I am repeating this pattern over and over again in the "normal" threading solution:
(defn foo
[{:keys [a b] :as ctx}]
(->> (do-something a b)
(assoc ctx :foo-result)))
(defn bar
[{:keys [b foo-result] :as ctx}]
(->> (do-another b foo-result)
(assoc ctx :bar-result)))
;; pipeline
(-> {...} ; initial ctx
foo
bar)ctx seems wrong somehow, as it kind of cheats on compositionality requirements, because any function in the pipeline can simply accept and return ctx and there is no key validation (which would not be the case with normal positional argument based functions). In your photo processing pipeline, do you simply add a bunch of key assertions to work around this? Or do you not use a big context map but instead have explicit function arguments (this makes the flow much more rigid, as data needed from a few steps back needs to be passed through all functions)?
We have a big context map, yes, and each function takes the map (and sometimes other arguments) and computes values based on what's in the map and then adds more values to the map.
I assume you've simplified the code to the point that foo and bar don't really look like that? Since in those cases, I'd do:
(let [result (do-something a b)]
(assoc ctx :foo-result result))(assoc ctx :foo-result (do-something a b))Sure, a let binding instead of the thread-last macro looks more concise here. I actually wanted to rather put emphasis on the repeating need for pulling out keys, then potentially having to assert the existence of the values of such keys, then having to assoc back to the ctx.
Plumatic's defnk/fnk actually solves this problem nicely:
(defnk foo [a b]
(do-something a b))
(defnk bar [b foo-result]
(do-another b foo-result))
(def pipeline
(graph/compile
{:foo-result foo
:bar-result bar}))
(pipeline {:a 1 :b 2})As I said yesterday "If Graph is a good fit for that part of the problem, use it." For us, our pipeline is strictly sequential. Graph will determine its own order to evaluate the functions based purely on data flow through the arguments. If every fn except the first explicitly depends on a result key from a "previous" fn, then you'll end up with the same sequential order, but if there are multiple valid orderings you might get something different. That's what I meant about our pipeline at work not being graph-like.
Got it. Thanks!
I guess the question is: are the data dependencies in your example workflow complex enough that you want/need help stitching it together?
Coming at this sort of problem fresh, I might look at the recently released core.async.flow maybe... See #core-async for some discussions of that.
My current use case is streamlining a data processing pipeline (which sometimes needs to wait for data from an API / has side-effects). It’s currently a more than twenty step long threading macro passing a state ctx object from function to function and building it up over time like that. It’s a lot of boilerplate grabbing the correct keys on function entry and adding the result to ctx, which I would like to simplify (also makes debugging a little hard). Graph’s keyword functions solve this problem at the least.
An alternative to these long hairy ctx pipelines would be a procedural „monster let“ statement, which also seems unnatural.
I furthermore looked at using the interceptor concept (e.g. https://github.com/exoscale/interceptor).
But maybe I am thinking about the problem the wrong way. How do you all handle complex pipelines, if not with the above approaches (will take a look at core.async.flow)?
If Graph is a good fit for that part of the problem, use it.
At work, we have photo processing pipelines that are a bit like this, but we just -> context through a whole bunch of named functions.
We don't have graph-like dependencies -- it's a purely sequential process, although one step runs a few things concurrently but still not graph-like.