hello Ben, I suggest you look at https://cljdoc.org/d/com.wsscode/pathom3/2022.10.19-alpha/api/com.wsscode.pathom3.connect.indexes#reachable-paths

the reachable-paths will take an env and some entity, and will tell you all reachable paths

this is what Pathom Viz uses under the hood to provide completions

It looks like I get the results in shape descriptor format, and I can pull the top-level attributes from the keys of this map. Is there a way to branch out to all nested attributes too? Or do I write the recursion myself?

I can't remember on the top of my head if that also does deep diving on nested, I guess no, but if that's case you can just repeat the process with the attributes at that level

Hmm, I realize now the to-many resolver I have doesn't declare the nested attributes in the output spec, so I don't think I can make the jump automatically at planning time. I'll have to run the query first then iterate over the results to query into the nested attrs. reachable-paths does a ton of the heavy lifting for me though. Thanks!

Still pretty hacky, but this "works" for simple cases:

(defn recursive-resolve
  [env entity]
  (let [entity (p.eql/process
                 env
                 entity
                 (vec (keys (pci/reachable-paths env entity))))]
    (->> (for [[k v] entity]
           [k (cond
                (instance? PersistentArrayMap v) (recursive-resolve env v)
                (vector? v) (map #(recursive-resolve env %) v)
                :else v)])
         (into {}))))

This approach does not forward parent attributes to the child query context, although that may be useful for some cases.

I suggest you replace (vec (keys ... with (pfsd/shape-descriptor->query ...

https://cljdoc.org/d/com.wsscode/pathom3/2022.10.19-alpha/api/com.wsscode.pathom3.format.shape-descriptor#shape-descriptor-%3Equery

This doesn't seem to work smoothly since my input entity has some data in it:

(pfsd/shape-descriptor->query {:my-provided-entity {:body "hello world"}
                               :derived-from-my-entity {}})
Execution error (UnsupportedOperationException) at com.wsscode.pathom3.format.shape-descriptor/shape-descriptor->query$fn (shape_descriptor.cljc:138).
nth not supported on this type: Character

maybe you removed the reachable-paths by accident? since that should return a valid shape descriptor

as: (pfsd/shape-descriptor->query (pci/reachable-paths env entity))

Eliding the real data, my situation looks something like this:

(pci/reachable-paths env {:my-provided-attr {:body "hello world"}})
=>
{:my-provided-attr {:body "hello world"}
 :derived-from-my-attr {}}

thats odd

Edited entity -> attr to be more clear

humm, or maybe something I missed, let me try it here

ah, this is what you can do: (pci/reachable-paths env (pfsd/data->shape-descriptor entity))

which is the right way to use reachable-paths, it expects a shape descriptor as input, but I only remembered it now looking it its sources, hehe:

(>defn reachable-paths
  "Discover which paths are available, given an index and a data context.

  Also includes the attributes from available-data."
  [{::keys [index-io] :as env} available-data]
  [(s/keys) ::pfsd/shape-descriptor
   => ::pfsd/shape-descriptor]

(pfsd/data->shape-descriptor {:my-provided-entity {:body "hello world"}})
=>
{:my-provided-entity {:body {}}}

Is there a way I can tell this to only parse keys that are in the index? :body is not a pathom attribute, so I think I don't want it mentioned in the query. (It seems to work fine since the parent attr is provided instead of resolved.)

its fine to mention it in the query because its already present in the data, so Pathom will just get it from the entity, no attempt look it up in the index

you could make that filtering, but its fine to keep them there

I think pfsd/data->shape-descriptor-shallow gets me what I want in this particular case but I'm not sure about downsides in the general case.

if you don't care about the nested parts, then it should be fine

The recursive query handles it 🙂

yup, makes sense

v2:

(defn recursive-resolve
  [env entity]
  (let [entity (p.eql/process
                 env
                 entity
                 (pfsd/shape-descriptor->query 
                   (pci/reachable-paths env 
                     (pfsd/data->shape-descriptor-shallow entity))))]
    (->> (for [[k v] entity]
           [k (cond
                (instance? PersistentArrayMap v) (recursive-resolve env v)
                (vector? v) (map #(recursive-resolve env %) v)
                :else v)])
         (into {}))))

another suggestion, use (coll/native-map? v) instead of (instance? PersistentArrayMap v)

Very nice!

(defn recursive-resolve
  [env entity]
  (let [entity (p.eql/process
                 env
                 entity
                 (pfsd/shape-descriptor->query
                   (pci/reachable-paths env
                     (pfsd/data->shape-descriptor-shallow entity))))]
    (->> (for [[k v] entity]
           [k (cond
                (coll/native-map? v) (recursive-resolve env v)
                (vector? v) (map #(recursive-resolve env %) v)
                :else v)])
         (into {}))))

also, since you are reading form vector, mapv instead of map, to keep it as a vector

yeah lots of rough edges here (including consuming stack!) that would need to be cleaned up before shipping to prod

one thing though, there is big drawback in mapping over the entities yourself, that is you lose the batch capatibility, since each entity gets its own request

also, if you are not using a persistent cache for the planner (you should do it in general), it means its planning again for each instance, also not very efficient

Luckily I am doing ad-hoc analytical work today so we get to be sloppy about performance 😆

the caching is easy and will help a lot, you just need to make something like this:

(def env
  (-> {:com.wsscode.pathom3.connect.planner/plan-cache* (atom {})}
      (pci/register ...)))

Planning and query resolution has been so dang fast every time I've used Pathom that it's never seemed like a priority to me. I have a little over 100 attributes and 100 resolvers in my index. At the REPL queries come back apparently instantly unless my resolvers are doing IO, and in production my typical query takes about 15 ms to plan (which is dwarfed by time spent fetching input data). One day I hope to have a graph big enough and complex enough that I will need the excellent performance optimizations you've made, but I don't need them today!

I am glad to learn about the plan-cache though, and I will reach for it when I need it!

great to hear 🙂

yeah, take your time, when you do so you will see a drop to 1ms after the first run for planning 🙂

@UANMXF34G trying to use your function above, maybe i just don’t understand how to invoke it.

@UH13Y2FSA can you show what you have tried?

Error printing return value (IllegalArgumentException) at clojure.lang.RT/seqFrom (RT.java:553). Don't know how to create ISeq from: java.lang.Double (recursive-resolve env {:some/id {:body "hi"}}) I’ve used env with p.eql/process and it works. I didn’t follow what :body is.