IIRC, Pathom2 was about the same speed — but the majority of time was spent in elide-not-found.

Well, wow... @UGBKV7HHP opened up this issue https://github.com/fulcrologic/fulcro-rad/issues/98, where @U0CKQ19AQ suggests eliminating (or turning into no-op) remove-emissions. Let's give that a try!

Status update: created src/patch and made my own versions of pathom3.clj, and pathom-common.clj, and skipped the postwalk. I think it’s now 30% faster, as those two last peaks in flame graph are gone. Interestingly, GraalVM Java 19 makes it 20% faster than Azul Java 17! (Something about its optimizer does better on either EDN parsing or pathom3.eql transduce/reduce.) I’ll post source code next week. (My thanks to @U04V15CAJ for teaching me the trick of adding “src/paths” to :paths in deps.edn!)

And thanks to @UGBKV7HHP for the hints!

Middlewares aren’t cheap. Middlewares that transform the result will take apart a huge persistent datastructure and put it back together. That is incredibly expensive.

My main reservation with Pathom is the scenario you describe. EQL defining the data fetches as a tree is great in the abstract, but the reality is that we need to limit payload sizes, so paging is critical, then on top of that the tree transform Pathom does isn’t cheap, and worse, updating any single field in the tree is super expensive compared to updating a key in list of maps that SQL produces.

hello @U6VPZS1EK! yes, indeed there is an overhead to Pathom that might get significant depending on the use case. Pathom is more optimal when dealing with complex data source, but with limited number of entities. also there is a significant difference if your main overhead is dealing with IO or if its computational. if its very IO driven, Pathom can do a good job on paralellizing, and given in those scenarios the IO time tend to dominate the cost, it ends up being a good tradeoff

but when you have large collections that need to handle CPU intensive operations, then Pathom overhead might play a significant percentage of the total cost in the operation

pagination is a good strategy, if that's something that fits your scenario, but sometimes it really doesn't, like when you need to make some reduction over a full list, delegating the process of each item also to the pathom process will have its cost

IME in most cases we can avoid this scenario, but when we cant, there is a scape path in Pathom, that is to use ::pco/final true in the sequence meta. when you do that, Pathom will not process that value at all, and then you have to do it instead, if the process you need is simple enough in each item, that might just work fine

but I'm always looking for ways to improve performance, if/when we can spot some specific situation to optimize that can handle some specific but common scenarios, those are great candidates

Wow! This is fantastic , @wilkerlucio — I will try this out sometime next week. And I think Pathom is awesome — my posts were just documenting my discoveries, and I’m delighted I’ve learned more about how Pathom works. I’ll keep you posted! PS; I’ve done pagination, and am delighted by performance then. But I have use cases where it’s “quickly pull together a resolver / db query, and show it with minimal work” — which means not implementing paginated loads. I’m thinking of how I can make that easier with RAD. Catch y’all soon!

cool, one suggestion though, I would put the final in retval instead of the whole map, should have the same effect, to be honest I wonder how its working on the outside map, rsrs

Thanks! Time to load is in same neighborhood, maybe even faster?

(pc/defresolver all-tweets [env _]
       {; no ::pc/input
        ::pc/output [;:search-results/search-text
                     ;:search-results/matched
                     {:tweet/all-tweets
                      [:tweet/id
                       :tweet/tweet
                       :tweet/media
                       :tweet/text
                       :tweet/created-at-inst]}]}
       (let [
             ;search-text (-> env :ast :params :search/search-query)
             _           (log/warn "*** all-tweets: ")
             retval      (->> (queries/get-all-tweets)
                           vec)]
         {:tweet/all-tweets
          (with-meta
            retval
            {::p/final true})}))))