Clojurians Log v2

OTOH, after 2 days of refactoring a small part of my handlers as a test... I'm starting to warm up to this kind of approach after just now trying to read code in other parts. The side-effect portions of the handlers are way more obvious in this way, don't have to carefully read the whole code and hope to not miss the db function calls or others...

yeah, that makes sense. for making data accessible to subsequent steps, are thinking of cases where the document/record/entity ID isn't known until the effect takes place? if so I think your approach makes sense. if it's possible to have the ID already in the pure step before the DB effect runs, you could also have that step assoc it onto a separate key on the result like my-id. E.g. in yakread I set things up so that the ctx map includes a seed value for generating random numbers, so then my pure functions can generated the numbers/values directly without needing to have an effectful step before/after do it. There are some loose ends I still need to tie up though. E.g. if I'm doing an upsert operation by querying for a document in the pure step and then generating a new ID because the document didn't exist yet, then if the document does get created by a different, concurrent request before the effectful state runs, the effectful state would have to fail instead of simply switching to the new ID generated by the concurrent request. I'm thinking the solution is to have the effect handler for XTDB have a way to go back to the previous pure state if if that happens. actually maybe I already implemented that... not sure.

are thinking of cases where the document/record/entity ID isn't known until the effect takes place?That's exactly it, inserting stuff in the DB or doing something with the results of an http call, etc.. > if it's possible to have the ID already in the pure step before the DB effect runs Hmm actually sometimes it IS possible (since I use uuid v4 which aren't native to mariadb/mysql, so I have to generate them in Clojure); other times though I don't know the ID or the result of what I'm looking for until the get effect completes (cuz I need the data from the DB before doing other things). > you could also have that step assoc it onto a separate key on the result like my-id. E.g. in yakread I set things up so that the ctx map includes I tried doing something like that and for some reason the key (assoc'd on the ctx map) wasn't making it through multiple chained effects. I'm not sure why and I should revisit because I might have missed something the first time around. EDIT: I think I did miss something the first time around, I'll review...

Nice, I've been wondering about that kind of thing and have a few aborted attempts. Not in Biff, mind you, in my own project. Instead of a "globals" map of state functions, I've experiment with using multimethods so that it would be more exensible by other components (haven't extended anything yet, just thinking ahead). This led to a small change in the orchestrator.. Something like:

(defn orchestrate [req f]
  (loop [{[state & remaining] :biff.chain/queue :as result} (f req)]
    (cond (nil? state) result

          (contains? (methods effect) state)
          (recur (effect (merge req result {:biff.chain/state state
                                            :biff.chain/queue remaining})))

          :else
          (recur (f (merge req result {:biff.chain/state state
                                       :biff.chain/queue remaining}))))))

Note: it took me a few iterations to understand that even though there are two calls to f, it's totally fine because it's being called with a different state, essentially, so it's going through different branches of the handler function. I thought at first it must be a mistake, but it wasn't.

Forgot to respond to this-- > I tried doing something like that and for some reason the key (assoc'd on the ctx map) wasn't making it through multiple chained effects. The way I've currently implemented it, things you return from one state only get propagated to the next state. If you want the return value to keep getting propagated, you need to explicitly have each state return it. That's why all the global effectful states return things like (assoc ctx :.../output (do-some-effect ...)). If you want you can have your pure states do that too. and/or modify the orchestrator function so it propagates all the returned values to the end of the chain for you.

Oh ok so in the end I was doing it the right way, whew. Thanks for confirming it!

yep no problem!

Nice, multimethods are also a good way. Note that the map-of-functions approach is also extensible since orchestrate (or https://github.com/jacobobryant/yakread/blob/355ad7c2ed2bd6f1e546a64b3b1ea4f04fd005f1/src/com/yakread/lib/pipeline.clj#L18 in its latest iteration) takes in the handlers map as an argument (it's expected to part of the ctx/request map), so you can pass in different things there.

Right, mind you my comments were from the perspective of my application and its more involved structure, so multimethods worked better --- especially when it came to frequently recompiling them to figure out how the whole thing works. my larger concern came with chaining multiple effects and keeping the data of each effect available so that the final effect (rendering a page) would have access to all this data. Or another example, one DB effect to get a record, then another to add another record refering to that one. Possibly I should combine those DB operations in a single function/effect? So far I've come up with adding an extra key (beyond the output), :biff.chain/results which is a map of each result, accreting over each effect being called. I'll use it for a little while in a section of my project and see how it goes/if it helps tackle some of the other complexity I accrued over time...