Clojurians Log v2

I get where you're coming from and maybe that really is the answer but I don't feel like it fully addresses my question. What I specifically don't understand is why the info that the entities in :xs have a property :x is relevant here and I see issues with having to add it. Let's extend my example a bit by adding an attribute :a to the entities in :xs . {::pco/output [{:xs [:x :a]}]} {::pco/input [:x] ::pco/output [:y]} {::pco/input [{:xs [:x :a :y]}] ::pco/output [:filtered-xs]} Just as before I need to add :a to the input of the :filtered-xs resolver to be able to query for [{:filtered-xs [:a]}] . Now let's say that after some time we refactor and move the addition of property :a to it's own resolver: {::pco/output [{:xs [:x]}]} {::pco/input [:x] ::pco/output [:y]} {::pco/input [:x] ::pco/output [:a]} {::pco/input [{:xs [:x :a :y]}] ::pco/output [:filtered-xs]} If we forget to remove the :a from the input of the :filtered-xs resolver we're going to needlessly resolve it every time, even if it's not in our query. This is brakes local reasoning big time imo and it's the kind of dependency tracking I would expect pathom to be able to do for me. But again, there's a lot I don't understand about how pathom works so maybe there're good reasons for having to model it this way, hence the question.

Regarding your linked thread: not sure if it's related but it was an interesting read and I learned from it, thanks for sharing.

Thinking about it a bit longer you might be right and the problem I'm seeing is completely orthogonal to what pathom does. If I were to separately define my schema's and refer to them in my inputs and outputs, the stated issue with the refactoring in my example should by easily resolved by updating the schema, which restores local reasoning. I guess I'll give this approach a try.

In a sense pathom (in query mode) is not reasoning locally about dependencies. It needs to see the whole tree of inputs and outputs to make an efficient plan. It’s not proceeding a step at a time and then seeing what is available. (In smart-map mode it is doing this because each access is like a separate query, so often things can work as smart maps but not in a larger query)

Ime it doesn’t take long before you are doing considerable higher-order resolver creation—eg generating resolvers from other code, or abstracting out bits of eql generation to ease or share input and output generation

Re your refactor of :a example: the rule of thumb I always use is: always mention in :input anything that is read by the resolver body and only those things, and always mention anything that may be included in the output

So filtered-xs should only mention :a in its input if it in fact needs to read it to filter, and that is a local decision made by looking at the resolver body

It’s the outputs where you need to make compromises. You can abstract out “base attributes of an :xs record” somewhere and refer to it in output anywhere you produce an xs record (including filtered-xs), and then don’t mention derived attrs in outputs in eg filtered-xs, even if they were already calculated

What pathom doesn’t have and can’t is “I output something that is the same shape as my input, where that shape is determined at runtime for a particular query.” I think that’s what you are trying to express with your :xs —> :filtered-xs resolver. But the plan is made before running the query, so filtered-xs really does need to say something static about its output. As a consequence you may repeat work for derived keys within an xs record

So filtered-xs should only mention :a in its input if it in fact needs to read it to filter, and that is a local decision made by looking at the resolver bodyThis is actually exactly my issue, :a is not needed for the filtering (nor :x) only :y. I believe I mentioned this in my original post but that context got lost in my attempt to distil my question. So the signature I've been striving for is {::pco/input [{:xs [:y]}] ::pco/output [:filtered-xs]} but this seems impossible for all the mentioned reasons. > What pathom doesn’t have and can’t is “I output something that is the same shape as my input, where that shape is determined at runtime for a particular query.” I think that’s what you are trying to express with your :xs —> :filtered-xs resolver. This not entirely what I'm after. For example: when I query [{:filtered-xs [:z]}] I don't want pathom to first resolve :z for all :xs and then filter them in :filtered-xs, but instead do the filtering first (resolving :y for each :xs because it's needed for the filtering) and then resolve :z for the remaining :xs. This is actually the behaviour that I see when I use the {::pco/input [{:xs [:x :y]}] ::pco/output [:filtered-xs]} > Ime it doesn’t take long before you are doing considerable higher-order resolver creation—eg generating resolvers from other code, or abstracting out bits of eql generation to ease or share input and output generation Yeah I think this is the direction I'll be exploring. I don't feel like my question is completely settled (see first comment) but I've definitely got a better understanding of what does/doesn't work. Will digest it a bit more and see what I can come up with in this space.

> So the signature I've been striving for is {::pco/input [{:xs [:y]}] ::pco/output [:filtered-xs]} but this seems impossible for all the mentioned reasons. Yeah, the chief problem is an incomplete output--pathom does not know that this map has :x :y or :a (or whatever) on it; it only knows about them from :xs so if it can resolve it, it's doing it before filtering. For your query for [{:filtered-xs [:y]}] to work, and assuming that :x is something like the "base" attribute of an xs record (e.g. maybe an id), you need at least {::pco/input [{:xs [:x :y]}] ::pco/output [{:filtered-xs [:x]}] so that pathom knows a path to anything via :x , but ideally`{pco/input [{:xs [:x :y]}] pco/output [{:filtered-xs [:x :y]}]` so that it knows :y is already available in the output.

Does pathom assume that undefined (nested) output structures are equal to the input structure? Because what you’re saying makes sense to me, but then I don’t understand why my examples work; i.e. just adding :x to the input. Edit: will try to verify this myself when I get back to my pc, on the move now

> Does pathom assume that undefined (nested) output structures are equal to the input structure? No

Then I don’t understand why my examples are working 🙂 Haha. Will experiment some more later, might be after the weekend. Appreciate you’re taking the time to have this discussion 🙏

It may be helpful to use pathom viz tool to see the query plan and step-wise evaluation

But bottom line is that everything works as intended only when inputs and outputs connect up

I some circumstances it may come to an answer anyway but by accident of data

Consider this data:

{:employees [{:id         "1"
              :first-name "John"
              :last-name  "Smith"}
             {:id         "2"
              :first-name "John"
              :last-name  "Doe"}
             {:id         "3"
              :first-name "Jane"
              :last-name  "Smith"}]}

{:input [{:employees [:last-name]}]
 :output [{:smiths [:last-name]}]}

{:employees [{:last-name "Smith"}
             {:last-name "Smith"}
             {:last-name "Doe"}]}

{:smiths [{:last-name "Smith"}
          {:last-name "Smith"}]}

[{:smiths [:first-name]}]

{:input [{:employees [:id :last-name]}]
 :output [{:smiths [:id]}]}

; result
{:smiths [{:id "1"}
          {:id "3"}]}

Back to your example: :id is :x :first-name & :last-name are :y & :z :x is used to lookup/determine :y & :z To conclude: You must provide :x , and it makes sense to do so

Here is an example that doesn’t use :x as a lookup:

(ns io.erical.sandbox.pathom.yabmas
  (:require
   [clojure.test :refer :all]
   [com.wsscode.pathom3.connect.indexes :as pci]
   [com.wsscode.pathom3.connect.operation :as pco]
   [com.wsscode.pathom3.interface.eql :as p.eql]
   [datascript.core :as d]))

(deftest example1
  (let [tx-data   [[:db/add "a" :x "ax"]
                   [:db/add "b" :x "bx"]
                   [:db/add "c" :x "cx"]]

        {:keys [db-after tempids]} (-> (d/empty-db)
                                       (d/with tx-data))

        eids      (-> tempids
                      (dissoc :db/current-tx)
                      vals)

        resolvers [
                   (pco/resolver '>xs
                     {::pco/output [{:xs [:db/id]}]}
                     (fn [{:keys [db]} _input]
                       (->>
                         db
                         (d/q '[:find [?e ...]
                                :where [?e :x]])
                         (mapv (partial hash-map :db/id))
                         (hash-map :xs))))

                   (pco/resolver 'eid>x
                     {::pco/input  [:db/id]
                      ::pco/output [:x]}
                     (fn [{:keys [db]} input]
                       (let [eid (get input :db/id)]
                         (d/pull db [:x] eid))))

                   (pco/resolver 'x>y
                     {::pco/input  [:x]
                      ::pco/output [:y]}
                     (fn [_ input]
                       (let [{:keys [x]} input]
                         {:y (str x "y")})))

                   (pco/resolver 'x>z
                     {::pco/input  [:x]
                      ::pco/output [:z]}
                     (fn [_ input]
                       (let [{:keys [x]} input]
                         {:z (str x "z")})))

                   (pco/resolver 'xs>filtered-xs
                     {::pco/input  [{:xs [:db/id :y]}]
                      ::pco/output [{:filtered-xs [:db/id]}]}
                     (fn [_ input]
                       (let [pred (comp (partial not= "cxy") :y)]
                         (->>
                           (get input :xs)
                           (filter pred)
                           (mapv #(select-keys % [:db/id]))
                           (hash-map :filtered-xs)))))]

        env       (-> {:db db-after}
                      (pci/register resolvers))]

    (is (= [1 2 3]
           eids))

    (is (= {:xs [{:db/id 3}
                 {:db/id 2}
                 {:db/id 1}]}
           (p.eql/process env '[:xs])))

    (is (= {:xs [{:x "cx"}
                 {:x "bx"}
                 {:x "ax"}]}
           (p.eql/process env [{:xs [:x]}])))

    (is (= {:xs [{:y "cxy"}
                 {:y "bxy"}
                 {:y "axy"}]}
           (p.eql/process env [{:xs [:y]}])))

    (is (= {:xs [{:z "cxz"}
                 {:z "bxz"}
                 {:z "axz"}]}
           (p.eql/process env [{:xs [:z]}])))

    (is (= {:xs [{:x "cx" :y "cxy" :z "cxz"}
                 {:x "bx" :y "bxy" :z "bxz"}
                 {:x "ax" :y "axy" :z "axz"}]}
           (p.eql/process env [{:xs [:x :y :z]}])))

    (is (= {:filtered-xs [{:db/id 2}
                          {:db/id 1}]}
           (p.eql/process env [:filtered-xs])))

    (is (= {:filtered-xs [{:x "bx"}
                          {:x "ax"}]}
           (p.eql/process env [{:filtered-xs [:x]}])))

    (is (= {:filtered-xs [{:y "bxy"}
                          {:y "axy"}]}
           (p.eql/process env [{:filtered-xs [:y]}])))

    (is (= {:filtered-xs [{:z "bxz"}
                          {:z "axz"}]}
           (p.eql/process env [{:filtered-xs [:z]}])))

    (is (= {:filtered-xs [{:x "bx" :y "bxy" :z "bxz"}
                          {:x "ax" :y "axy" :z "axz"}]}
           (p.eql/process env [{:filtered-xs [:x :y :z]}])))))

Hey @caleb.macdonaldblack, thanks for your reply! Yeah, what you say makes sense and that part actually already clicked after my discusion with favila. My confusion then shifted to why my examples worked at all, as I never defined :x in my output signature (`:id` in your example) it just happend to be there because I defined it in the input and never filtered it within the resolver: I detailed my findings https://clojurians.slack.com/archives/C87NB2CFN/p1719906481363549?thread_ts=1719395702.973629&cid=C87NB2CFN. Like favila I was under the impression that when pathom is given a query it looks at the signatures of all resolvers, makes the optimal plan and then resolves it. But that's demonstrably not the case. Do you happen to know of a good source to get a better mental model of what pathom is doing? E.g. docs I missed, blog post, talk, etc.

Isn’t the issue that the shape of filtered-xs is not described? My intuition is the resolver that takes xs and produces filtered-xs should have the same input and output so that pathom knows that filtered-xs has (or could have) :x :y and :z on it

{::pco/input [{:xs [:x :y]}] ::pco/output [{:filtered-xs [:x :y]}]}

Possibly related: https://clojurians.slack.com/archives/C87NB2CFN/p1714503643534199

> But the plan is made before running the query, so filtered-xs really does need to say something static about its output. After experimenting some more I doubt this is true for the following reasons: • If I dont define :x on the inputs {::pco/input [{:xs [:y]}] ::pco/output [:filtered-xs]}, but do return it in the actual output; I'm still able to query for it. • If I do define :x on the inputs {::pco/input [{:xs [:x :y]}] ::pco/output [:filtered-xs]} but dont return it in the actual output (filter it internally); my examples stop working. To me this means that pathom does actually a lot more then making a plan based on the defined in/out puts and then executing it. Not sure if that's positive tbh, I think I would rather have it fail; but then again I obviously don't have a good mental model of what's going on so can't really judge. It also means my initial question in this thread is sort-of answered: because I added :x to the input and didn't filter it internally it became part of the output and pathom picked it up from there for further processing. Again; I'm really a bit lost as to what pathom does exactly and don't feel like the docs help much here (or I missed the relevant parts). I think I'll try to examine some query processing with flowstorm when I find the time, but I'm gonna leave it for now. All that being said; I still think your advise is sound and I'll be looking into generating my in/outputs from schema's. Thanks again, have a nice day!