there is a middle ground solution you can use to avoid loading attributes you dont need without having to resort to dynamic resolvers, that is using the planner ::pcp/expects data, you can pull this from your resolver like: (-> env ::pcp/node ::pcp/expects) the value there will be a https://pathom3.wsscode.com/docs/shape-descriptor telling you what is required (considering the whole request) from that resolver, you can use that to "mask" your pull from datomic so, in summary • make a resolver that exposes every possible attribute of that entity • in the resolver execution, see which of these attributes are really required via ::pcp/expects • pull only these attributes please let me know if this seems to work for your case

Thank you! I will try it out and get back to you.

Having played with it, this seems like a reasonable compromise. Probably only necessary if you have entities with large number of attributes, but typically pull small subsets of those attributes.

I found an issue with using (-> env _::pcp/node ::pcp/expects)_ Here’s a minimal example to illustrate:

(defresolver example
 [env {:keys [id]}]
 {::pco/output [:a :b]}
 (-> (get {1 {:a "a" :b "b"}} id)
     (select-keys (-> env ::pcp/node ::pcp/expects keys))))

(def env-example
 (-> {:com.wsscode.pathom3.error/lenient-mode? true}
   (pci/register [example])))

=> 
{:ref {:b "b"},
:com.wsscode.pathom3.connect.runner/attribute-errors {:a #:com.wsscode.pathom3.error{:cause :com.wsscode.pathom3.error/node-errors,
                                           :node-error-details {1 #:com.wsscode.pathom3.error{:cause :com.wsscode.pathom3.error/attribute-missing}}}}}

=> {:a "a", :ref {:a "a", :b "b"}}

@U066U8JQJ

hello @U07V9T94Z, just got to read and understand, its a tricky one, and its related to caching as you notice

once a resolver is cached, the next run (using same input and params) will hit the cache, which means it wont run your content

for Pathom to know that expects across different entities is quite tricky

one way out would be to not cache the resolver as a whole, you can do it by:

(ns demos.unexpected-expects
  (:require [com.wsscode.pathom3.connect.indexes :as pci]
            [com.wsscode.pathom3.connect.operation :as pco]
            [com.wsscode.pathom3.connect.planner :as pcp]
            [com.wsscode.pathom3.interface.eql :as p.eql]))

(pco/defresolver example
  [env {:keys [id]}]
  {::pco/output [:a :b]
   ::pco/cache? false}
  (-> (get {1 {:a "a" :b "b"}} id)
      (select-keys (-> env ::pcp/node ::pcp/expects keys))))

(def env-example
  (-> {:com.wsscode.pathom3.error/lenient-mode? true}
      (pci/register [example])))

(comment
  (p.eql/process env-example
    {:id 1 :ref {:id 1}}
    [:a {:ref [:b]}]))

but of course, if the actual computation is expensive, you don't want it to run multiple times, to fix that you can use an internal caching for that, like:

(ns demos.unexpected-expects
  (:require [com.wsscode.pathom3.cache :as p.cache]
            [com.wsscode.pathom3.connect.indexes :as pci]
            [com.wsscode.pathom3.connect.operation :as pco]
            [com.wsscode.pathom3.connect.planner :as pcp]
            [com.wsscode.pathom3.interface.eql :as p.eql]))

(defn expensive-db-get [id]
  (Thread/sleep 1000)
  (get {1 {:a "a" :b "b"}} id))

(pco/defresolver example
  [env {:keys [id]}]
  {::pco/output [:a :b]
   ::pco/cache? false}
  (-> (p.cache/cached :com.wsscode.pathom3.connect.runner/resolver-cache* env [`example id] #(expensive-db-get id))
      (select-keys (-> env ::pcp/node ::pcp/expects keys))))

(def env-example
  (-> {:com.wsscode.pathom3.error/lenient-mode? true}
      (pci/register [example])))

(time
    (p.eql/process env-example
      {:id 1 :ref {:id 1}}
      [:a {:ref [:b]}]))
"Elapsed time: 1.161333 msecs"
=> {:a "a", :ref {:b "b"}}

I think that's the best option here, so you separate the cache of the expansive operation while you still allow the resolver to have multiple outputs depending on the expectation in that context

another way to "trick" pathom into running the resolver again is by changing the params, given they are also a part of the cache key:

;; you can try this with the same config you already have for the resolver:

(time
    (p.eql/process env-example
      {:id 1 :ref {:id 1}}
      ['(:a {:custom "param"}) {:ref [:b]}]))

but I think that's less effective, because if you have an expensive computation inside the resolver, it will run multiple times

please let me know how it goes 🙂

Thanks. Let me think about this and see how I want to handle this.

@U066U8JQJ I’ve had a bit of time to think about this. If I’m reading it correctly, the “internal caching” approach still fetches all the ::pco/output attributes and caches them. Then the resolver selects a subset of keys. I was trying to only fetch those attributes that are in (-> env ::pcp/node ::pcp/expects keys). Basically, I have one resolver that fetches all possible attributes from a datomic db. I was supplying (-> env ::pcp/node ::pcp/expects keys) to the datomic pull query to limit how much data I fetch from datomic on each invocation of the resolver. The ideal solution would fetch (and cache!) only those attributes that are “expected” on each invocation of the resolver. If the resolver is called again with the same input it would get the cached values for cached attributes and pull any remaining attributes from datomic. So in our example it might first pull :b from datomic and cache it, and then pull :a from datomic the second time it’s called. It seems this doesn’t fit pathom’s caching mechanism, which is by [resolver input]. And I am not sure that having a single datomic resolver is necessarily what I want. I was just exploring the options. One solution might be for me implement my own layer of caching for the resolver that is per [id attribute]. At this point I would almost be building my own EAV index for each datomic db basis as obtained with (d/db conn) (I pass this in env)! This seems a little crazy. :) I could also try to use parameters to somehow convey to the resolver which attributes are already cached and which need to be pulled from datomic, but this is a kludge. I might just go back to having to having a datomic resolver for each type of entity. But having to define entity “types” kind of goes against datomic’s open/flexible entities. For now I’ve just removed (select-keys (-> env ::pcp/node ::pcp/expects keys)) from the resolver, which ends up fetching every existing attribute for each entity.

This also got me thinking, does pathom ever purge the cache? How does it limit the size of the cache? What about when working with traditional relational DBs that have no concept of DB basis? In other words something like Postgres is place oriented, so we’re not caching values, but references. Do you just turn off the cache for those types of DBs?

hi George, thanks for the elaborated response, this makes me have some thoughts too, but lets start with the simpler question about caching purge: the resolver-cache (which is the default one that you get automatically) has a lifecycle of a single request, so for each p.eql/process the cache is initiated (its a simple atom) and discarded at the end of that execution, so it won't hold your caches for long (so it shouldn't have a long impact on memory usage), you can override that cache with some other one if you want, but I recommend keeping this one as-is

now, about the expects and internal caching, you right, if we just load everything and filter the output, we are not doing much (would be better to don't filter the expects at all, since Pathom already does it), but we do have some interesting possible strategies here, one of them, as you said, could be caching each demanded attribute as a separate thing, so we could re-use it across different demands (lets say different entities that require different things from the same resolver)

another one that I'm not if would work now (because I haven't tested, just something that came into my head now, but needs more thinking) would be to convert that resolver in a batch resolver, which will centralize all the entities in a single request, having that I think we could "merge" the need across different entities that share attributes and make a single request for those, I like to try it here, I think I can do it later today and see how it goes

Thanks, Wilker. The batch resolver idea sounds promising. If it were possible to get the whole tree from datomic in one go, that would be ideal.