Fork me on GitHub
#pathom
<
2022-06-26
>
mbjarland17:06:49

Generic noob question about pathom. So I’m playing with mapping the star wars api with pathom and also connecting imdb data via the films represented in the swapi. My question is general, but to use the swapi as an example: the swapi has a “film” entity with a bunch of attributes, I would like to connect the swapi film entity with the imdb film entity and pull in some film information from imdb (like user rating etc) to “decorate” the swapi film entity. How would you typically model this in pathom? I’m guessing you would fully qualify everything and do swapi.film/id and imdb.film/id etc and then connect the two. This essentially makes the two film entities totally separate things, i.e. not so much decorating as just querying a different entity. I guess you would then add something like a synthetic entity swapi.film/imdb.film…but I’m not sure this feels right. So how do you typically model two different sources providing information about essentially the same thing (let’s say the 1977 star wars movie)?

nivekuil17:06:56

I would just do them as separate entities, same way https://www.wikidata.org/wiki/Q17738 does it. How you want to combine them into a "canonical" entity is up to the client really

wilkerlucio17:06:03

great question! the first step is to map them separate, like you just said, the second one would be to connect those entities, one way to do it is via ID's, would be great if the SWAPI already had that info, but since it doesn't we can add this mapping ourselves (a bit manual, but considering its less than a dozen items, works fine), with something like:

(def swapi->imdb-id-mapping
  {1 "tt0076759"
   2 "tt0080684"
   3 "tt0086190"
   ; ...
   })

(pco/defresolver swapi->imdb [{:keys [swapi.film/id]}]
  {:imdb.movie/id
   (get swapi->imdb-id-mapping id ::pco/unknown-value)})

; inverse direction

(def imdb->swapi-id-mapping
  (set/map-invert swapi->imdb-id-mapping))

(pco/defresolver imdb->swapi [{:keys [imdb.movie/id]}]
  {:swapi.film/id
   (get imdb->swapi-id-mapping id ::pco/unknown-value)})

wilkerlucio17:06:10

this will allow you to query both entities at the same level, like:

(p.eql/process env 
  ; via swapi ID
  {:swapi.film/id 1}
  [:swapi.film/title :imdb.movie/rating])

(p.eql/process env 
  ; same query, via imdb id
  {:imdb.movie/id "tt0076759"}
  [:swapi.film/title :imdb.movie/rating])

wilkerlucio17:06:45

a third step you might want to take is to abstract those specific services name away from your client, for example, if we were creating the "Star Wars Fan Site" (`swfs` for short) service, we might want to use names like :swfs.movie/title :swfs.movie/rating, so we shield our client from having to know about the different services, this is more a design decision, in some cases you might prefer using the names strait from the services, but its a consideration to make, if a layer on top will be helpful for the case, and if so, you can use an https://pathom3.wsscode.com/docs/built-in-resolvers#aliasing, like:

(pbir/alias-resolver :swapi.film/title :swfs.movie/title)
(pbir/alias-resolver :imdb.film/rating :swfs.movie/rating)

mbjarland17:06:07

Perfect. This is way cleaner and answers my question perfectly. I did not realize that adding two resolvers for the IDs would allow me to query the items on the same level directly

wilkerlucio17:06:28

no worries, this is where you get the real leverage from the attribute modelling, because it makes trivial to "merge" entities like these, combining different sources at the same level

mbjarland17:06:49

One more question…in general, would you advocate adding an id field to the information sent to the client or would you rather keep that inforamtion internal and just feed the client the data

wilkerlucio20:06:38

if its a public id, I think its fine to keep, also you need it to make the relations

roklenarcic20:06:14

Is there a way to short-circuit resolving a subtree with a plugin or something? I’ve made an attempt to that effect and it always resolves the whole tree.

wilkerlucio22:06:16

can you give an example demo? what kind of flow you are looking for?

roklenarcic09:06:37

basically let’s say you’ve got a resolver that given a ::github/repo-name will return ::github/repo-id , you can see how for some names it might not have a valid return as there is not repository of that name. Then I hafve a resolver that given ::github/repo-id can produce ::github/star-count . Now obviously I don’t want to code all my resolvers that take in repo-id to check for nil input or ::missing input values.

roklenarcic10:06:43

So far I’ve tried to code around this by introducing a filter that does: 1. if input parameter contains any ::missing values skip calling the resolver and return a response constructed from pcp/expect 2. if resolver return is ::missing return a response constructed from pcp/expect 3. the response from expect is constructed by replacing all {} in pcp/expect with ::missing This kinda works, you get the shape requested by all the properties and if somewhere in the chain we stopped gettign data, the properties are show as ::missing . The main problem there is that the expects shape and EQL in general don’t specify whether the return is a collection or a single item, the resolver can return either. So when I do something like this:

{:projects/current-user [:github/id :github/name {:github/contributors [:contributor/name]}]}
Those projects that don’t have a linked github repository will get the wrong shape:
{:projects/current-user
       [{:github/id 1
         :github/name "test 1"
         :github/contributors [{:contributor/name "guy 1"} {:contributor/name "guy 2"}]}
        ; this project has no github link so it returns ::missing value and this result
        ; is contructed from expect
        {:github/id ::missing 
         :github/name ::missing
         :github/contributors {:contributor/name ::missing}}]}

roklenarcic10:06:10

note that second contributors is not a vector

roklenarcic10:06:22

I’ve also tried using pco/final-value to short circuit processing but that seems to do nothing

wilkerlucio21:06:47

I get a bit concern about the ::missing thing, because Pathom already has semantics for it (which are: dont provide the key), adding ::missing makes you whole system in need to handle that everywhere, mixing with actual values, while by not providing a value, pathom will stop the execution (and give an error)

wilkerlucio21:06:22

and to go around items that misses things, you can either make some parts of the query optional, or use lenient mode (which will allow for problems in the middle of the query, and report errors by attribute, while keeping what was resolved)

roklenarcic11:07:14

This part doesn’t work well for me. Not providing a value is not a very good solution. You’ve mentioned two options, making part of query optional and using lenient mode. I don’t find these sufficient. 1. using pco/? pco/? doesn’t work when the attribute it wraps is not the optional attribute but it is sourced by an optional attribute. Your planner is just not sufficiently capable to make this realistic. Consider this example:

(pco/defresolver a1 [{}]
  {::repos
   [{::repo "repo1"}
    {::repo "repo2"}]})

(pco/defresolver a2 [{::keys [repo]}]
  {::pco/output [::project-id]}
  (if (= repo "repo1") {::project-id 1} {}))

(pco/defresolver a3 [{::keys [project-id]}]
  {::project-name
   (str "project" project-id)})
So some repos have a project and some don’t. I did it as you suggested, omiting attribute when it is not available. Then if I use your optional query advice:
(ask [{::repos [(pco/? ::project-id)]}])
=> #:resolvers{:repos [#:resolvers{:project-id 1} {}]}
Seems it works. But it doesn’t really, when I request a different attribute as optional the planner will request all attributes leading up to it as mandatory:
(ask [{::repos [(pco/? ::project-name)]}])
=>
#:resolvers{:repos [#:resolvers{:project-name "project1"}
                                        #:com.wsscode.pathom3.connect.runner{:attribute-errors #:resolvers{:project-name #:com.wsscode.pathom3.error{:cause :com.wsscode.pathom3.error/node-errors,
.....
:error-data {:required #:resolvers{:project-id {}},
It firmly requires ::project-id even though it is needed only to compute an optional attribute. Another thing: does it work with joins? (pco/? {::some-attr [::more-attr ::more-attr]}) I haven’t tried but I suspect it doesn’t. So this pco/? is not very useful, it only really works for leaves of the query tree, and only those that don’t have optional / potentially missing inputs. I also suspect that pco/? doesn’t play well with fulcro defsc 2. Lenient mode and ignoring the errors it creates This kinda works but it is very messy: • Suddenly my missing data errors that are expected are lumped in with other serious errors. • It generates a shitload of output in REPL and it makes results hard to read • if there’s an OR node involved I get bunch of stack traces in my output and it makes for a large output, also generating that much garbage cannot be cheap. This is especially bad with large collections. I just get pages and pages and pages of errors. I have to wonder how much does that cost to create.

👀 1
wilkerlucio00:07:02

hi @U66G3SGP5, I tried to reproduce the issue you said with 1, but in the tests I did an indirect dependency doesn't blow up when a previous part fails, based on this example:

(pco/defresolver a [{::keys []}]
  {::a ::pco/unknown-value})

(pco/defresolver b [{::keys [a]}]
  {::b (str a " x")})

(def env
  (pci/register [a b]))

(comment
  (p.eql/process env [(pco/? ::b)]))

wilkerlucio00:07:50

you can see, ::b will fail due to no value in A, but in this case, no error is thrown, but if you found a case where this isn't working as expected, please let me know with a repro

wilkerlucio00:07:28

about 2, I preferred to give more information at start, so you can see what you have available, but you can make a plugin to process the error and tune it down for something that makes more sense to your scenario

roklenarcic06:07:20

Does my example not reproduce the error? This is all in lenient mode btw.

roklenarcic10:07:53

my example doesn’t use unknown value

roklenarcic10:07:26

rather it doesn’t include the attribute as you’ve suggested

wilkerlucio10:07:27

unknown value is same as not having the key in the output

roklenarcic10:07:09

right…you’ve suggested that when I don’t have a value to omit the key in the return and use pco/? when doing a query

roklenarcic10:07:55

I gave you an example of resolvers a1 a2 a3 where this approach produces errors

roklenarcic10:07:30

You’ve said tha tyou cannot reproduce, so I am wondering if you’ve tried this example… the one you’ve listed uses a special pco/unknown value instead

roklenarcic10:07:26

I guess I have poorly communicated the issue

wilkerlucio10:07:27

let me try that

wilkerlucio10:07:16

because the premisse of optional is that its ok to fail in a dependency, that should work, so I made a simples example to try it

roklenarcic10:07:45

but your example doesn’t fail to provide a dependency, does it? It provides ::a just fine it’s just ::pco/unknown-value. Does pco/unknown-value have a special meaning?

wilkerlucio10:07:01

yeah, ::pco/unknown-value is same as not having the key in the output (different than nil)

wilkerlucio10:07:23

I can rewrite my same example as:

wilkerlucio10:07:25

(pco/defresolver a [{::keys []}]
  {::pco/output [::a]}
  {})

(pco/defresolver b [{::keys [a]}]
  {::b (str a " x")})

(def env
  (pci/register [a b]))

(comment
  (p.eql/process env [(pco/? ::b)]))

roklenarcic10:07:42

right, now try to put another resolver in between

wilkerlucio10:07:55

I just tried your demo, it doesn't trigger an error

wilkerlucio10:07:20

(do
  (pco/defresolver a1 [{}]
    {::repos
     [{::repo "repo1"}
      {::repo "repo2"}]})

  (pco/defresolver a2 [{::keys [repo]}]
    {::pco/output [::project-id]}
    (if (= repo "repo1") {::project-id 1} {}))

  (pco/defresolver a3 [{::keys [project-id]}]
    {::project-name
     (str "project" project-id)})

  (def env
    (pci/register [a1 a2 a3]))

  (p.eql/process env [{::repos [(pco/? ::project-name)]}]))
=>
{:com.wsscode.pathom3.demos.repro-opt-follow-up/repos [{:com.wsscode.pathom3.demos.repro-opt-follow-up/project-name "project1"}
                                                       {}]}

roklenarcic10:07:40

enable lenient mode

wilkerlucio11:07:42

right, that's something to check, it shouldn't, but I see it

roklenarcic11:07:08

your example with just lenient mode turned on returns error

roklenarcic11:07:09

(pco/defresolver a [{::keys []}]
  {::pco/output [::a]}
  {})

(pco/defresolver b [{::keys [a]}]
  {::b (str a " x")})

(def env
  (-> {::p.error/lenient-mode? true}
      (pci/register [a b])))

(comment
  (p.eql/process env [(pco/? ::b)]))

wilkerlucio11:07:15

I though you were comparing between using strict mode with optionals vs lenient mode

wilkerlucio11:07:51

because usually on lenient mode, optionals can kind be ignored (since you still get all the payload), but I'll look into this, this cases shouldn't add errors (if I remember correctly)

wilkerlucio11:07:16

and to illustrate what I was saying in point 2, this is how you can simplify the error, in this demo I'm just outputting the cause of it:

wilkerlucio11:07:18

(do
  (pco/defresolver a1 [{}]
    {::repos
     [{::repo "repo1"}
      {::repo "repo2"}]})

  (pco/defresolver a2 [{::keys [repo]}]
    {::pco/output [::project-id]}
    (if (= repo "repo1") {::project-id 1} {}))

  (pco/defresolver a3 [{::keys [project-id]}]
    {::project-name
     (str "project" project-id)})

  (def env
    (-> {:com.wsscode.pathom3.error/lenient-mode? true}
        (p.plugin/register
          {::p.plugin/id 'err-simplifyer
           :com.wsscode.pathom3.error/wrap-attribute-error
           (fn [attr-error]
             (fn [entity k]
               (let [err (attr-error entity k)]
                 (:com.wsscode.pathom3.error/cause err))))})
        (pci/register [a1 a2 a3])))

  (p.eql/process env [{::repos [(pco/? ::project-name)]}]))
=>
{:com.wsscode.pathom3.demos.repro-opt-follow-up/repos [{:com.wsscode.pathom3.demos.repro-opt-follow-up/project-name "project1"}
                                                       {:com.wsscode.pathom3.connect.runner/attribute-errors {:com.wsscode.pathom3.demos.repro-opt-follow-up/project-name :com.wsscode.pathom3.error/node-errors}}]}

roklenarcic11:07:53

wait, wrap-attribute-error can change how errors are returned? The docs made it seem like it was for side-effects only (like logging)

wilkerlucio11:07:43

what part of the docs? lets see if we can improve it, because this point is intended for the user to modify the error

wilkerlucio11:07:19

so I give a huge thing at first, so you can tune it down if you have too, but start from full data

wilkerlucio11:07:15

that's a different extension point, I actually don't see the :com.wsscode.pathom3.error/wrap-attribute-error there, need to add

roklenarcic11:07:26

all the error handling samples elide the param

roklenarcic11:07:34

in the outer fn

roklenarcic11:07:47

make it look like it’s just for sideeffects

wilkerlucio11:07:36

the resolver-error is mostly for side effects

wilkerlucio11:07:48

but the wrap-attribute-error was missing, I'm adding it now