Clojurians Log v2

@holyjak hey man, just wanna say sorry for the delay on this one, I didn't forgot about it, I'm on the last weeks of renovation in my apt and that's been very madness time to me, I hope this will be over by next week and I'll have enough head space to get to it

No problem! I know you do this as a volunteer, in your free time. And I have a workaround so no big deal for me. Good luck with the renovation, I know how those are!

@wilkerlucio Not exactly sure what to look for 😅 but it seems to work - run-capturing returns lot of data and running reproduce fails with the same error as my original query. BTW I will make an issue for the problem I reported so that I have a good way to track it. Hope it is ok!

A Pathom graph can be unintentionally constructed is such a way to cause the issue you’re having. In this example, it’s intuitive to think that Pathom could return both child and age at the same time. But Pathom can’t merge nested entities, so it only goes down one path.

While this might not explain your specific issue, it does highlight how easy it can be to make a mistake without realising it. You would have to figure out what path pathom is trying to take in your query that doesnt work. You’ll have to do this with lenient mode and the pathom vis too. You can also inspect the plan and execution metadata but it’s harder to grok.

Other easy things to look for are resolvers that describe their output as just :product/production-method , without providing any nested information.

Also I’ve run into similar issues that i’ve resolved by updating Pathom to the latest version.

I see. Thank you!

Let me know how you go. I’ve run into this sort of stuff heaps

Will do!

Pathom resolver nested input specs are selections not queries. They can select among the nested keys already present in the input map, but they do not trigger a query/resolution of nested attributes that are reachable but not yet present in the input map.

One way I've tried to tackle this problem is to simplify the input spec to the outer entity only, then inside the resolver body I issue a nested query for the nested attrs:

(defresolver my-resolver
  [env input]
  {::pco/input [{:order-line/product [:product/production-method]}]}
  (let [num-id (p.eql/process-one env (get-in input [:order-line/product :product/production-method]) :production-method/num-id)]
    ...))

Hey @ben.grabow, in Pathom 3, nested queries in resolver inputs can call new resolvers if needed (just like a normal query), https://pathom3.wsscode.com/docs/resolvers#nested-inputs. Or maybe you are talking about something different, let me know

@pfeodrippe I've tried that before and it didn't work how the docs describe, but maybe I used it wrong. I'll go back and try again.

Got it, yeah, I use it a lot in may daily job and it works as expected, if you have some example that does not work, you can ping me o/

Thank you!

I'll see if I get time tonight!

FYI I got it working. It turns out that I did not read the (verbose) output from Pathom correctly. The real problem was with another input to my resolver, that lacked some dependencies. I just have to troubleshoot Pathom more to learn what to look for… . Possibly there was also some other problem I unwittingly fixed 🤷

Actually, it is more interesting than I thought. While this EQL works (notice catalogue-number is derived among others via order-line/id -> order/id -> …, and order-line/id -> order-line/product -> product/production-method -> :production-method/num-id):

[{[:order-line/id 123]
  [:order-line/catalogue-number]}]
; => {... {:order-line/catalogue-number "0002 0001"}} ; yay!

{[:order-line/id 123]
 {:order-line/label-count 950}} ; booo, no more cat. number! 😭 

@holyjak You can merge those paths into one. It’s not ideal but it should work

What do you mean? Make a dedicated single resolver that can resolve both?

Yeah

For some more insight, Pathom will treat nested data as seperate entities, even if the parent keyword is the same. It won’t merge them.

For all it knows path A returns an entity with attribute X and path B returns an entity with attribute Y. It sees them as two seperate entities. You can either get X or Y. Not both. Because no path resolves an entity with both. IMO this isn’t ideal. Maybe it’s not possible but I would love to try because it opens up so many possibilities.

I guess I start to understand it. I have 2 derived attributes (`catalogue-number` and label-count) that both depend on a third sibling, :order-line/`product` , but each one needs a different subset of product’s properties. And Pathom is not able to merge those two subqueries to request both subsets at once. Instead, it follows down one path and end up with a partial product in the environment. Then when it gets to resolving the 2nd derived attribute (cat.nr.), it thinks “nice, I already have a product, no need to resolve it” but it does not have the necessary data, and Pathom stops at that. Is that somewhat correct?

Ah yeh, you just posted that while I was typing > path A returns an entity with attribute X and path B returns an entity with attribute Y. It sees them as two seperate entities.

That’s exactly how I understand it

@wilkerlucio What Caleb and me describe just above is a limitation of Pathom that really surprised me. Could you be so kind and point me to where it is documented? And do you think there are ways to alleviate it in the future? Thank you!

@holyjak @wilkerlucio Agreed, this surprises me also.

IMO, a keyword with a nested entity should be treated like a relationship and not a value, and Pathom should auto merge (the data or the resolver) them because they’re semantically the same thing. This does get tricker however with one-many relationships. Pathom would need to track entity ids for this sort of thing to work

And I’m sure there are other nuances.

It is little more complicated in my case. The product itself has all the properties I need but in this case one of the derived attributes also needs properties of its sub-entity, but that one is only returned as an id (b/c the first derived attribute did not care about it). I guess the same problem would pop up if my product/id-resolver was smarter and only fetched the attributes that it was asked for, instead of all of them.

I’ve read that response probably over 5 times so forgive me if I’ve misunderstood. I think you’re describing that a resolver might return X & Y attributes and a dependant resolver only needs X. But later on in the query another resolver needs Y. AFAIK, Pathom3 will know about all outputs (X & Y) of a single resolver, even if it’s not immediately needed.

Ok, so the problem indeed is contained to situations where 2+ levels of subentities are involved, i.e. when we have to go over a nested reference? In my case I have order-line: • catalogue-number - needs production-method/num-id & more • label-count - needs product/labels-per-sheet & more • product ◦ product/labels-per-sheet ◦ product/production-method ▪︎ production-method/num-id and the problem is IMO with product/production-method being returned just as {production-method/id 345} , i.e. without any properties. If my catalogue-number resolver only depended on direct product attributes, it would be all fine.

So i have this and it all works. The num-id resolver being a simple alias should be pretty resilient. However if I change the product resolver output to this:

{:com.example.jakub2/product
                  [:com.example.jakub2.product/labels-per-sheet
                   :com.example.jakub2.product/production-method]}

So what is the key thing here? That the product resolver declares not just :com.example.jakub2.product/production-method as its output but elaborates it to { :com.example.jakub2.product/production-method [:com.example.jakub2.product.production-method/id]} ?

Yeah, I’m trying to find out what could be causing your issue. That would definitely be doing it if you’re not describing your nesting in your output. If that’s not your issue then I’d be curious what your input and output look like for the num-id resolver

hello folks, I'm trying to understand the issue here, from what I gather the issue seems to be around situations where the same attribute is provided with different sub-queries, this is short example I can think:

(ns com.wsscode.pathom3.demos.nested-multiple
  (:require
    [com.wsscode.pathom3.connect.built-in.resolvers :as pbir]
    [com.wsscode.pathom3.connect.indexes :as pci]
    [com.wsscode.pathom3.connect.operation :as pco]
    [com.wsscode.pathom3.interface.eql :as p.eql]))

(pco/defresolver sub-a []
  {:thing {:a 1}})

(pco/defresolver sub-b []
  {:thing {:b 1}})

(def env
  (-> {}
      (pci/register
        [sub-a sub-b])))

(comment
  ; fine
  (p.eql/process env [{:thing [:a]}])

  ; fine
  (p.eql/process env [{:thing [:b]}])

  ; not fine, because Pathom would take 2 different paths for the same attribute
  (p.eql/process env [{:thing [:a :b]}]))

is it the case?

Maybe. Not sure. @holyjak is able to exchange a single :production-method/id for his desired :production-method/num-id

I have a repor here:

(ns com.tmp
  (:require
    [clojure.test :refer :all]
    [com.wsscode.pathom3.connect.indexes :as pci]
    [com.wsscode.pathom3.connect.operation :as pco]
    [com.wsscode.pathom3.interface.eql :as p.eql]))

(pco/defresolver catalogue-number
  [_env input]
  {::pco/input  [{:order/product [
                                  :product/production-method #_ ; TODO delete this line to see the test break
                                  {:product/production-method
                                   [:production-method/num-id]}]}]
   ::pco/output [:order/catalogue-number]}
  {:order/catalogue-number
   (->> input :order/product :product/production-method :production-method/num-id
        (str "catalogue-number-"))})
(pco/defresolver num-id
  [_env input]
  {::pco/input  [:production-method/id]
   ::pco/output [:production-method/num-id]}
  {:production-method/num-id
   (->> input
        (:production-method/id)
        (str "num-id-"))})
(pco/defresolver label-count
  [_env input]
  {::pco/input  [{:order/product [:product/labels-per-sheet]}]
   ::pco/output [:order/label-count]}
  {:order/label-count
   (-> input :order/product :product/labels-per-sheet)})
(pco/defresolver product-id
  [_env input]
  {::pco/input  [:product/id]
   ::pco/output [:product/id
                 :product/labels-per-sheet
                 :product/production-method]}
  {:product/id 222
   :product/labels-per-sheet 1337
   :product/production-method {:production-method/id "production-method-1"}})
(pco/defresolver order
  [_env input]
  {::pco/input  []
   ::pco/output [:order/id {:order/product [:product/id]}]}
  {:order/id 111
   :order/product {:product/id 222}})
;; TEST IT
(-> (pci/register [order product-id label-count num-id catalogue-number])
    (p.eql/process
      {}
      [:order/id
       :order/label-count
       :order/catalogue-number
       {:order/product [:product/labels-per-sheet
                        {:product/production-method [:production-method/num-id]}]}]))

See the TODO line. With this line in, the call at the bottom succeeds. If I delete it and thus have the more elaborate input definition, it fails with > Pathom can’t find a path for the following elements in the query: [:production-method/num-id] …

(pco/defresolver product-id
  [_env input]
  {::pco/input  [:product/id]
   ::pco/output [:product/id
                 :product/labels-per-sheet
                 {:product/production-method
                  [:production-method/num-id]}]}
  {:product/id 222
   :product/labels-per-sheet 1337
   :product/production-method {:production-method/id "production-method-1"}})

That works.

Vs this which doesn’t

(pco/defresolver product-id
  [_env input]
  {::pco/input  [:product/id]
   ::pco/output [:product/id
                 :product/labels-per-sheet
                 :product/production-method]}
  {:product/id 222
   :product/labels-per-sheet 1337
   :product/production-method {:production-method/id "production-method-1"}})

My bad, I should have ☝️ in my tests, as I do have it in my prod code…

With that fixed up, the example is working now.

yes, so it must be another factor. Looking for it…

the output should always reflect the data, so product-id would be better as:

(pco/defresolver product-id
  [_env input]
  {::pco/input  [:product/id]
   ::pco/output [:product/id
                 :product/labels-per-sheet
                 {:product/production-method
                  [:production-method/id]}]}
  {:product/id 222
   :product/labels-per-sheet 1337
   :product/production-method {:production-method/id "production-method-1"}})

and it does work in with this setup

Woops sorry my example had num-id returned. Should’ve been as wikerlucio had it

something that I did in the past was take notes of attributes that show up in the index as values (something opaque, like :product/id) vs as references (when the value has something declared inside of it), same value appearing as value and as reference is an indicator of a mistake

seems time to return it, rsrs, we can make an alert when registering an attribute in a differnet format from which the value is known to be (value vs ref)

I am trying to make the test case closer to my production code but so far failing to replicate the problem

So it seems to work just fine in the test case but not in production. The tricky part now is to finding the difference between the two that matters (likely a mistake in prod config)

There could be a resolver shadowing those ones.

And the troublemaker is ….. ::pco/batch? true In the following resolver, if batch is true then I get the “insufficient input” error but if it is false then everything works out just fine:

(pco/defresolver fake-order-id
  [_env input]
  {::pco/input  [:order/id]
   ::pco/output [:order/yearly-sequence-number]
   ::pco/batch? false}
  (cond-> {:order/yearly-sequence-number 33}
          (sequential? input) vector))

Interesting. How do the attributes in that resolver relate to the ones from before?

@caleb.macdonaldblack @wilkerlucio can you spot what is wrong here? It is not exactly the same problem as I have in prod (this one complains about catalogue-number, not product-method/*) but still it fails. Notice the two lines marked with TODO - changing either as suggested makes the query to succeed (ie. either batch to false or rm label-count from the query):

(ns com.tmp
  (:require
    [clojure.test :refer :all]
    [com.wsscode.pathom3.connect.indexes :as pci]
    [com.wsscode.pathom3.connect.operation :as pco]
    [com.wsscode.pathom3.interface.eql :as p.eql]))

(pco/defresolver derived-catalogue-number
  [_env {:keys [:order/yearly-sequence-number :order-line/product]}]
  {::pco/input [:order/yearly-sequence-number
                {:order-line/product [{:product/production-method [:production-method/num-id]}]}
                #_{:order-line/product [:product/production-method]}]
   ::pco/output [:order-line/catalogue-number]}
  {:order-line/catalogue-number
   (str "catalogue-number-seq" yearly-sequence-number "-numid"
        (-> product :product/production-method :production-method/num-id))})
(pco/defresolver derived-label-count
  [_env {:order-line/keys [product package-count]}]
  {::pco/input  [{:order-line/product [:product/labels-per-package]} :order-line/package-count]
   ::pco/output [:order-line/label-count]}
  {:order-line/label-count
   (* package-count (-> product :product/labels-per-package))})
(pco/defresolver fake-order-line-id
  [_env input]
  {::pco/input  [:order-line/id]
   ::pco/output [:order-line/id :order-line/package-count {:order-line/product [:product/id]}]
   ::pco/batch? true}
  (cond-> #:order-line{:id 111
                       :package-count 10
                       :product {:product/id #uuid "cf8b66fb-b963-433f-ab37-af3134061e49"}}
          (sequential? input) vector))
(pco/defresolver fake-order-id
  [_env input]
  {::pco/input  [:order/id]
   ::pco/output [:order/yearly-sequence-number]
   ::pco/batch? true} ; TODO Either set to false to remove label-count from the query to fix
  (cond-> {:order/yearly-sequence-number 33}
          (sequential? input) vector))
(pco/defresolver fake-product-id
  [_env input]
  {::pco/input [:product/id]
   ::pco/output [:product/labels-per-package {:product/production-method [:production-method/id]}]
   ::pco/batch? true}
  (cond-> {:product/labels-per-package 8
           :product/production-method {:production-method/id #uuid "a0269c9d-63a1-4e14-afa3-08f81ab0c002"}}
          (sequential? input) vector))
(pco/defresolver fake-production-method-id
  [_env input]
  {::pco/input  [:production-method/id]
   ::pco/output [:production-method/num-id]
   ::pco/batch? true}
  (cond-> {:production-method/num-id 3}
          (sequential? input) vector))
(pco/defresolver order-line->order
  [_env input]
  {::pco/input  [:order-line/id]
   ::pco/output [:order/id]}
  {:order/id #uuid"72e65f1b-088c-4817-82c9-622aa542ed06"})

(-> (pci/register [derived-catalogue-number derived-label-count fake-order-line-id fake-order-id order-line->order
                   fake-product-id fake-production-method-id])
    (p.eql/process
      {}
      [{[:order-line/id "fake-ol1"]
        [:order-line/id
         :order-line/label-count ; TODO Either remove this or fake-order-id's batch to fix
         :order-line/catalogue-number]}]))

thanks for the report Jakub, I was able to repro and there is something strange about the batch not working there, but I need to dig deeper and understand better

in the meantime, I was playing with an idea want to check if you wanna try it. I was thinking about a way to make easier to generate repros, without having to keep doing those manual mocks

so I worked on this:

(defn capture-cache-key [env input]
  [input (pco/params env)])

(p.plugin/defplugin repro-capture
  {:com.wsscode.pathom3.connect.runner/wrap-root-run-graph!
   (fn [run-root]
     (fn [env ast-or-graph entity-tree*]
       (run-root env ast-or-graph entity-tree*)))

   :com.wsscode.pathom3.connect.runner/wrap-resolve
   (fn [resolve]
     (fn [{::keys [capture-context*] :as env} input]
       (let [op          (-> env ::pcp/node ::pco/op-name)
             capture-key (capture-cache-key env input)]
         (ctry
           (clet [response (resolve env input)]
             (swap! capture-context* assoc-in [::op-capture op capture-key] response)
             response)
           (catch Throwable e
             (swap! capture-context* assoc-in [::op-capture op capture-key]
                    {::throw {:msg (ex-message e) :data (ex-data e)}})
             (throw e))))))})

(defn mocked-op [{::keys [capture-context] :as env} input]
  (if-let [[_ response] (find (get-in capture-context [::op-capture (-> env ::pcp/node ::pco/op-name)]) (capture-cache-key env input))]
    response
    (throw (ex-info "Unexpected input to mocked call" {}))))

(defn mock-resolver [resolver]
  (-> resolver pco/operation-config
      (assoc ::pco/resolve mocked-op)))

(defn mock-mutation [mutation]
  (-> mutation pco/operation-config
      (assoc ::pco/mutate mocked-op)))

(defn extract-indexes [env]
  (-> env
      (select-keys [::pci/index-oir
                    ::pci/index-resolvers
                    ::pci/index-mutations])
      (update ::pci/index-resolvers update-vals pco/operation-config)
      (update ::pci/index-mutations update-vals pco/operation-config)))

(defn run-capturing [env entity request]
  (let [capture* (atom {})
        env'     (-> env
                     (assoc ::capture-context* capture*)
                     (p.plugin/register repro-capture))
        result   (try
                   (p.eql/process env' entity request)
                   (catch Throwable _ ::exception-result))]
    {::env     (-> (extract-indexes env')
                   (assoc ::capture-context @capture*))
     ::entity  entity
     ::request request
     ::result  result}))

(defn reproduce [{::keys [env entity request result]}]
  (let [env' (-> env
                 (coll/update-if ::pci/index-resolvers update-vals
                                 (comp pco/resolver #(assoc % ::pco/resolve mocked-op))))
        res  (p.eql/process env' entity request)]
    (if (= result res)
      res
      (throw (ex-info "Unexpected process result" {:expected result
                                                   :actual   res})))))

use case example:

;; use case

(pco/defresolver some-resolver []
  {:a 1})

(pco/defresolver b-resolver [{:keys [a]}]
  {:b (+ a 10)})

(pco/defresolver err []
  {:err (throw (ex-info "Err" {}))})

(pco/defresolver miss []
  {:miss ::pco/unknown-value})

(def env
  (-> {}
      (pci/register
        [some-resolver
         b-resolver
         err
         miss])))

(comment
  (-> (run-capturing env {:x [{:a 1} {:a 2} {:a 1} {:a 3}]} [{:x [:b]}])
      (reproduce))

  (-> (run-capturing env {} [:err])
      (reproduce))

if you can try this in your setup, I think this version is still naive, but I like to check if it works in a setup like yours