two separate supervision trees stitched together by side effect

can you post code?

I'm modeling "scraping" an api that require a lot of request some in paralles some seq wih retry ect.. so the code is quite spread. I'm trying to reproduce with a simple example but it's draining, repl can hang, need to restart often so it's just slow to get there.

I wonder if clj-kondo could help with the missionary edge case.

(defn batch-request-operations!
  "Take a list of operations, request each with pagination when required.
  tap> a report on each step when fetching paginated data
  Return a map of name->response"
  ([transaction cookie]
   (batch-request-operations! transaction cookie {}))
  ([operations cookie {:keys [connection-manager http-client step]
                       :or {step 1000}}]
   (->> (m/ap
         (let [transaction (operations->transaction (map :request (apply concat operations)))
               res (m/? (post-transaction! transaction
                                           cookie
                                           connection-manager
                                           http-client))
               d (m/?> (m/seed (map vector (apply concat operations) (:body res))))
               [op original-res] d]
           (if (more-rows? original-res)
             (->> (m/ap
                   (loop [op'  op
                          meta original-res]
                     (let [end-row (get meta "endRow")
                           start-row (get meta "startRow")
                           step                   (or step
                                                      (- end-row
                                                         start-row))
                           [start-row' end-row']  (page-range end-row step)
                           operations'            (paginate-operation op' start-row' end-row')
                           res                    (m/? (post-transaction!
                                                        (operations->transaction operations')
                                                        cookie connection-manager http-client))]
                       (fetched-rows-report res)
                       (m/amb (:body res)
                              (when (more-rows? (-> res :body first))
                                (recur operations' (-> res :body first)))))))
                  (m/eduction
                   (mapcat (fn [r] (->> r
                                        (mapcat #(get % "data"))
                                        (reduce conj [])))))
                  (m/reduce conj [])
                  m/?)
             (get original-res "data"))))
        (m/reduce conj [])
        m/?
        (map vector (map :name (apply concat operations)))
        (chunk-split (map count operations))
        (map #(into {} %))
        m/sp)))

This is stupid complex 😅

how do you know an exception is thrown ?

I was catching -> printing and re throwing. I think I refactored my out of the issue for now. Would you have an example of that ? https://clojurians.slack.com/archives/CL85MBPEF/p1662809302216159?thread_ts=1662808593.677959&cid=CL85MBPEF

why not m/observe?

Here is a continuous time (m/cp) object lifecycle - https://nextjournal.com/dustingetz/missionary-object-lifecycle - i think should work in m/ap as well

you can construct a service in the m/observe constructor and the m/observe destructor will be called per the supervision tree

Do you specifically not want to reuse the http client? Typically these are expensive to create and get reused wherever possible. Sometimes the consumer part can be extracted to a separate fn too, in which case I'd rather create the client there and send it in as an arg, which would make cleanup trivial in an m/sp block

Thanks for the response, in my "example" do-async-stuff is reusing http client for many request, I was looking for a way to do that without leaking the complexity to the consumer.

My point was the http client might get reused across several do-async-stuff calls, or a do-another-async-stuff etc, in which case its lifecycle won't be bound to your flow at all. If that's not your case then either try what Dustin suggested

(m/ap (let [http-manager (m/?> (m/observe ...))] ...))

(defn do-async-stuff [cookie]
  (let [http-manager (build-http-client ...)]
    (m/sp (try (m/reduce conj [] (m/ap ...)) (finally (.close http-manager))))))

Interesting idea to reuse the client, you could even keep it in a dependencies management tool like a db connection i guess.

The fetching I'm doing with this project is seconds long on each call, so time wise it's actually negligible. But maybe in term of CPU cycle / warming the data-center it might be helpful.

it's not just about time but also about clean code. If you find the correct lifetime of an object you'll know where to deal with it in the most efficient way, in LOC, failure handling etc

I'm probably naively reasoning that reusing the client -> more complex code -> better perf. But it lead to more complex code -> more bug.

maybe, in this case. Don't be too hard on yourself 😉