pathom 2022-02-03 | Slack Archive

Hi, I’m wondering if this is expected behavior of priority. Here’s an example:

(-> {}
      (pci/register [@(pco/defresolver nested-single-attr [{in :input}]
                        {::pco/output [{:output [:sub-out-1]}]
                         ::pco/priority 1}
                        (prn "called single")
                        {:output {:sub-out-1 (format "(single-sub-1 %s)" in)}})
                     @(pco/defresolver nested-extra-attrs [{in :input}]
                        {::pco/output [{:output [:sub-out-1
                                                 :sub-out-2]}]}
                        (prn "called double")
                        {:output {:sub-out-1 (format "(double-sub-1 %s)" in)
                                  :sub-out-2 (format "(double-sub-2 %s)" in)}})])
      (p.eql/process [{[:input "my-input"]
                       [{:output [:sub-out-1
                                  :sub-out-2]}]}]))

returns

{[:input "my-input"]
 {:output
  {:sub-out-1 "(double-sub-1 my-input)", :sub-out-2 "(double-sub-2 my-input)"}}}

always, but when nested-single-attr has higher priority than nested-extra-attrs , it always prints “called single” and “called double”. Otherwise, it only prints “called double”. I’d hope there’d be a configuration so it only prints “called double”. But it seems like resolution happens individually for the nested attributes i.e. it selects a path for :sub-out-1 and then selects for :sub-out-2 if the resolver for :sub-out-1 didn’t already resolve :sub-out-2. I’d hope it would determine earlier that “cost” of the whole query would be more for calling nested-single-attr first because it would imply an additional call to nested-extra-attrs.

KevinK00:02:06

Maybe this is a topic for https://github.com/wilkerlucio/pathom3/discussions/57? but it might be something different entirely

wilkerlucio03:02:34

this is an interesting case, Pathom will try its best to fulfill the demand, so in this case, when single has priority, its called first, but there still the need for :sub-out-2, which will cause the other resolver to get called as well. its arguable if Pathom should optimize in this case, for example, imagine if nested-extra-attrs is some sort of cache that's a fallback for :sub-out-1, in this scenario is desirable to still call both resolvers.

wilkerlucio03:02:12

I don't really like the current priority system, as described in the discussion you pointed, it doesn't work that well

wilkerlucio03:02:42

I plan to get the default priorty to the same way pathom 2 did, based on path cost (which is dynamically computed as resolvers run)

wilkerlucio03:02:16

I hope some better ideas may come up in the future to deal with manual priority

KevinK18:02:56

Could one get a cost estimation at planning time? Like in this case, it’d be nice if the planner could see that the query requires both resolvers and sum of cost of nested-single-attr and nested-extra-attrs and compare it to the alternate of just calling nested-extra-attrs . I understand the cache argument being applicable when the outputs specified in the two resolvers are the same, and at runtime the prioritized resolver doesn’t return one of the attrs so it falls back to calling another. But in this case, it should know ahead of time that the latter resolver will need to be called regardless since the specified output of the first resolver doesn’t satisfy the query.

wilkerlucio20:02:59

it is possible to calculate the cost at planning time, one caviat to be careful is because plans in pathom 3 are expected to be cached, and since the cost is dynamic, we need to be sure that's a separate step, the simplest solution is to always do that at running time

wilkerlucio20:02:33

also, priority never modifies the plan, the plan is always the same, the priority is about which branch from an OR node (which indicates alternatives for something) should be taken first

wilkerlucio20:02:01

that said, Pathom 3 priority algorithm is plugabble, you can write a custom one and test some alternatives

imre20:02:33

Does anyone know what would be the equivalent or closest approximation of pathom2's ::p/wrap-parser in p3? Is it ::pcr/wrap-root-run-graph!?

wilkerlucio20:02:53

:com.wsscode.pathom3.interface.eql/wrap-process-ast

wilkerlucio20:02:34

I think this needs a new helper, I have made one for the viz connector, feels like this should be ported back to Pathom 3

wilkerlucio20:02:42

here is the ws connector code you can copy to do that:

wilkerlucio20:02:49

https://github.com/wilkerlucio/pathom-viz-connector/blob/master/src/com/wsscode/pathom/viz/ws_connector/pathom3.cljc#L43-L60

imre20:02:42

thanks a million!

roklenarcic22:02:16

in pathom 2 what’s teh difference between parser and async-parser in terms of resolver returns? if I use async, must i return chan? can I return chan?

souenzzo01:02:34

in pathom2, when using an async or parallel parser, your resolvers need to return: • or something that satisfies chan? https://github.com/wilkerlucio/wsscode-async/blob/master/src/com/wsscode/async/async_cljs.cljs#L7 • or something that is a map? We can confirm this information on this line of code: https://github.com/wilkerlucio/pathom/blob/main/src/com/wsscode/pathom/parser.cljc#L282

roklenarcic15:02:28

I assume that returning promesa promise won’t work though

souenzzo00:02:59

nope. promesa is only supported in pathom3 and I'm not sure how it works exactly

wilkerlucio22:02:49

in Pathom 3 everything is driven by Promesa promises, they proved to be much more efficient then core.async for the particular way Pathom processes things, except for the parallel processor which does use core.async for queuing

roklenarcic18:02:38

core async is a more general tool, channels with queues are a very general approach to async… promises are a special case. For most async flows if you don’t need queues I assume that CompletableFuture that is wrapped by promesa should be more efficient. Did you test performance @U066U8JQJ

wilkerlucio18:02:26

yes, performance using Promises is far superior than using promise-chan, I guess specially the part of the error handling/error propagation

wilkerlucio18:02:26

I tried for a long time to just use core.async for everything, but after the experience with Promesa I know see two distinct concepts, for single-hit async processes is way better to use a focused tool like Promesa (CompletableFuture), I still like to use core.async when I need an async queue, but for single hit I'll always pick Promises

imre22:02:17

Is there an equivalent to p2's ::pp/key-process-timeout in p3?

wilkerlucio13:02:23

no, did you find a situation with some sort locking going on?

imre14:02:31

Oops, sorry I forgot to respond to this. No, I was converting some test cases and it appears this key was used to set some sort of a global timeout to abort an async resolver that too long to produce a result

wilkerlucio14:02:13

the reason this was created initially was more about not letting pathom screw itself, was to prevent hard to find locksteps in the process (usually do to Pathom's itself falt)

wilkerlucio14:02:06

for Pathom 3, so far I prefer to leave this in the users hands, one way to do something similar is to write a wrap-resolve plugin, and for any async thing returned, wrap that in a timeout

wilkerlucio14:02:30

that said, parallel process still new to Pathom 3 and more info/use cases can better tell the direction to handle these kinds of problems, feedback is much appreciated

imre14:02:53

Thank you. For now this was just a quick spike but we have plans to go ahead with the conversion so I'm sure we'll have feedback sooner or later

2022-02-03

Channels