Fork me on GitHub
#pathom
<
2022-11-11
>
David Yang18:11:51

hey all - I have a resolver that describes a deeply nested output structure but I’m unable to join against the keys in that nested output here’s the gist: https://gist.github.com/davidyang/b492acf0024f480139eadf79bc5e214a

dehli18:11:21

Did you specify ::pc/output using nested output? Ex: {:my/output [:nested/a :nested/b]}

David Yang18:11:03

hey @U2U78HT5G - yes I specified the ::pco/output option

David Yang18:11:30

hmmm, this seems to work correctly:

David Yang18:11:33

(p.eql/process env [{[:user/id 1] [:user/id
                                     :user/full-name
                                     {'(:user/feed {:start-date 1 :end-date 2})
                                      [{:feed/day 
                                        [:feed/date
                                         {:feed/user-days 
                                          [:feed/food :user/id :user/full-name]}]}]
                                      }]}])

David Yang18:11:47

to have them at the same level of nesting - just the joining doesn’t seem to work

David Yang18:11:45

but this doesnt:

(p.eql/process env [{[:user/id 1] [:user/id
                                     :user/full-name
                                     {'(:user/feed {:start-date 1 :end-date 2})
                                      [{:feed/day 
                                        [:feed/date
                                         {:feed/user-days 
                                          [:feed/food {:user/id [:user/full-name]}]}]}]
                                      }]}])

dehli18:11:21

The problem with your second query is that you're asking for :user/full-name underneath :user/id which doesn't make sense since :user/id is a string. If instead you made the response look like:

:feed/user-days [{:feed/user {:user/id 2}
                  :feed/date ....
You could then ask for [:feed/food {:feed/user [:user/id :user/full-name]}]

dehli18:11:02

or your first query is another valid way to ask for the data

dehli18:11:39

generally it's better to keep pathom data flatter as it gives your resolvers more input data they can use

David Yang18:11:38

thanks for the help - I don’t understand what you mean when you say that :user/id is a string?

dehli18:11:05

oops, sorry in your example it was a number

dehli18:11:11

but the main point is that it's not an object

dehli18:11:30

nested queries only make sense when the response is an object

David Yang18:11:30

my mental model is that the returned attributes can be converted to their references, but it sounds like they need to return idents (the map of {:user/id 2})

David Yang18:11:03

so if I have an ident as the value of the map then I can join on that

dehli18:11:05

that is true, which is why you can ask for both :feed/date and :user/full-name

David Yang18:11:45

and when I ask for :user/full-name it seems to be smart enough to use the user/id at it’s sibling level or I guess whatever is lower in the tree

dehli18:11:12

it would be sibling, pathom won't look for attributes lower in the tree

David Yang18:11:41

so pathom doesn’t go up the tree looking for idents, it needs it at the sibling level

dehli18:11:45

The only reason you would include :feed/user {:user/id 1} is if you needed the user attributes to be in an isolated key For example, the response to the query would be:

:feed/food "food"
:feed/user {:user/id 1 
            :user/full-name "Rich"}
but if you don't need the user attributes to be in their own key, it's preferable to have all the attributes at the same level which means your response would look like:
:feed/food "food"
:user/full-name "Rich"

David Yang18:11:25

ok - so just rely on pathom’s smart resolver to look for a sibling when I join on new attributes

dehli18:11:27

correct, in the example you posted you had :user/id as a sibling to :feed/food which is why you could also for :user/full-name at the same level.

David Yang18:11:47

so I join on user/full-name at that level it’ll look for the :user/id at that level (that level only right?) and resolve it that way

David Yang18:11:51

that’s very cool

dehli18:11:40

yep! and if it already has that resolver response cached it won't need to do another lookup (say that resolver required a database lookup)

David Yang18:11:46

but that means that if I need a date parameter I need to pass it down the tree - I can’t rely on it to find the attribute from higher in tree

dehli18:11:34

Potentially, if you're able to keep your resolvers pretty flat you might not need to pass the data down. It also might make more sense as input depending on how it's being used

David Yang18:11:41

is there a performant way to break out these nested resolvers? maybe but a look-through cache in env so that when the nested resolvers are called they can get the data from the cache instead of having to fetch it again from DB? Can I separate out the resolvers from the data nesting itself without multiplying the calls out?

dehli18:11:36

There are ways to do that. One thing I've done before is have a top-level resolver that does one query to get all needed data and then hydrates pathom's resolver cache so that downstream resolver calls will already have the data accessible. Pathom has an optional parallel parser that will parallelize all the requests it can, which might be enough for what you need.

David Yang18:11:15

interesting - so if I do this big resolve it’ll normalize it for subsequent queries?

dehli19:11:25

You could (doesn't do that out of the box). I was able to get it working by creating one resolver that would interface with my db and then I just had to normalize to that db resolver

David Yang19:11:27

I guess to clarify - if I have a query that returns a deeply nested structure, but I define only the top level outputs, if I have separate resolvers that access that nested structure pathom could figure it out?

David Yang19:11:42

Or are you saying I would have that top level resolver put it’s substructure into the resolver cache somehow

dehli19:11:52

If you return a deeply nested structure, downstream resolvers would be able to access all the nested data. Every "nested level" is a new context though which means attributes above or below wouldn't be accessible (just that level itself)

dehli19:11:25

but your query would also need to closely match the response (which takes away from the value of pathom)

David Yang19:11:41

and I’d have to define the output with all the nesting correct?

David Yang19:11:24

OK - that makes sense to me - the batching is cooler than I first understood from reading, it sounds like it aggregates many places in the tree (so it can combine multiple queries for comments and then I can group those into one SQL query)

David Yang19:11:51

for some reason I thought it was just passing in a sequence of items from one level to another - not all the requests from the different parts of the EQL tree

dehli19:11:57

yeah, I'm not 100% sure tbh. i haven't used batch very much

David Yang19:11:42

OK - lots to experiment with. thanks so much for your help! very much appreciate your time/insight

dehli19:11:17

happy to help! 🙂 feel free to post again if you have any other questions!

caleb.macdonaldblack20:11:41

@wilkerlucio From my own experience and perusing the slack history, GitHub issues and documentation, Pathom3 seems to stuggle with solving cyclic problems among others like resolving back up a nested datastructure (maybe also cyclic?). I lack the experience, but could you provide any insight into this? Is there something fundamental wrong with trying to solve these cyclic problems in this paradigm? Perhaps it’s impossible? Or maybe it’s a design choice to avoid solving problems this way? For example Clojure struggles with cyclic dependency resolution and community consensus is that good design shouldn’t need cyclic dependencies. (Apologies for no references). If none of those, maybe this functionality just isn’t here yet? I’m eager to dive into these interesting problems but with my lack of experience in Pathom, I just hope I’m not wasting my time solving the unsolvable. I could put together some use-cases if anything needs clarity.

wilkerlucio19:11:56

hello @U3XCG2GBZ, I love if you can provide some use cases so I have a better understanding of the problem

wilkerlucio19:11:28

in general, Pathom has cycle detection and that will case a failure in planning, since there is no way to resolve the cycle

caleb.macdonaldblack03:11:11

@wilkerlucio Thanks for getting back to me on this one. https://github.com/wilkerlucio/pathom3/issues/168 I am running into a stack overflow with the example in that issue. If the cycle detection was working I believe it would log a warning and stop the planner from cycling here. Is there something fundamentally wrong with the approach here in the example? Perhaps the cyclic warning log is a clue of that?

wilkerlucio12:11:23

I started looking at it yesterday, I was able make your example work, but it also made a test fail, so Im trying to get all working, its in progress ;)

caleb.macdonaldblack05:11:35

Oh thanks. Would you mind pushing your WIP changes? I would like to take a look if I could.

David Yang23:11:29

Hey - I’m planning on running the results of my EQL queries through normalization - but if I use put references to :user/id in other objects then the normalizer will percieve that as a user object. Is the solution to put something like :feed/user-id in that object and then put an alias in the resolver chain?