Clojurians Log v2

hello @pppaul, yes, looking at the data, its a lot of keys, and Pathom does have overhead to merge (a simple merge in pathom has a lot of checks it needs to do, like if has something nested to process, check for plugins running, etc...)

so if your real data is gonna look like that, Pathom overhead will be significant

if i turn each key into it's own resolver, will that fix things? cus otherwise i might as well build my own interface over a graph

depends on how you load/merge, if you load in smaller pieces, yes, Pathom will be able to only fetch the nescessary parts for a given process, and avoid a lof the overhead in this case

i don't need pathom to know about most of this data, maybe i could do something indirect.

actually, I just did some profiling, and I found out most of the time is being spent at the map-select-ast (which is the part of the process that cleans up keys that were not requested)

results after removing that part:

(let [pathom-lookup  (labeled-time "pathom-lookup" (run-eql ::resource))
        clojure-lookup (labeled-time "clojure-lookup" (pathom-lookup :resource/project-equity-edit-page))])
"Elapsed time: 0.466875 msecs for: pathom-lookup"
"Elapsed time: 0.00175 msecs for: clojure-lookup"

I think something might be wrong there, because in this case there is no specific request for a nested thing, it should short circuit

I'll investigate that

humm, looking at the code, I remember now, Pathom will enforce the manual re-merge of everything so its sure to apply the plugin to merge ast data for every entry

I think we can optimize that, if the user doesn't have any plugin installed to merge-ast, we can short circuit this step

just to help clear this up, without any changes to Pathom, if your query is more specific and only requests a part of the ::request data, you already get a better result:

(let [pathom-lookup  (labeled-time "pathom-lookup" (run-eql {::resource [:resource/files-manager-cashflow]})) ; !!! note the change here
        clojure-lookup (labeled-time "clojure-lookup" (pathom-lookup :resource/project-equity-edit-page))])
"Elapsed time: 13.872583 msecs for: pathom-lookup"
"Elapsed time: 0.001041 msecs for: clojure-lookup"

this is because now every other key is being ignored during the map-select process, so it only has to traverse the :resource/files-manager-cashflow part

that is what I expected. hope that you are able to implement the optimization soon. I want to have more specific queries, but I just need more time getting used to pathom, and maybe pathom isn't good for this use case. I'll attempt this implementation again soon, but right now I converted the code to just use a clojure map

hello, do you think you can make a repro so I can see in all detail what is going on? are you making any plugins?

i'm not making plugins, i have only made the most basic resolver that i could think of just to deal with some caching, and get my feet wet with pathom

i need to figure out how to write this data to a file that you can use

my data is 21 megs, is that ok?

i'm setting up a repo, i'll invite you to it (don't really want it to be public)

https://github.com/boxxxie/debugging_pathom

i made it public, since your email isn't on slack

the repo exhibits the same performance problems that i get in my app codebase

it's not a map lookup you're timing it's read-string and slurp. also try with the planner cache too

it's not that @kevin842 as i said in my app code, that slurp is replaced with code that takes 3-4 seconds to run

also, i'm using cache. i don't understand what you are suggesting

i'm very new to pathom, this is my first use of it. i'm not able to easily understand the terminology you use

i created this code via reading all the related docs for the pathom functions in the code. i'm not sure what to do as i feel like i've exhausted the docs

@kevin842 i'm not sure where a planner cache would go, but i'm only calling 1 key in my current setup, what planning is going on?

not just calling 1 key, there is 1 key total

https://pathom3.wsscode.com/docs/cache/ have you read the first section there

ok, maybe i skipped this because i thought there was nothing to plan

i'll try this

i tried with plan cache, there is no difference

(pco/defresolver ^:private resource-resolver []
  {::pco/cache-store ::my-cache*}
  {::resource (init {})})

i put some output on the init

"Elapsed time: 1029.986377 msecs for: init"
"Elapsed time: 1182.23776 msecs for: pathom-lookup"
"Elapsed time: 9.62E-4 msecs for: clojure-lookup"
"Elapsed time: 147.967815 msecs for: pathom-lookup"
"Elapsed time: 0.001102 msecs for: clojure-lookup"

init take 1 second

i don't think this is my problem

I thought init would take most of those 3 seconds, not sure where the other 2 might be coming from. If the planner isn't taking any time I guess the overhead comes from the extra steps pathom has to do, like merge the resolver output into the result accumulator and that just takes 150ms to merge a huge (20mb?) map. You can plug in a profiler like tufte to check but if that's the case I doubt there's much to optimize

the resolver cache caches a single revolver's output, not an eql query so there is going to be overhead still as pathom has to process each of them to run things like plugins

no, init takes 1 sec in the debug repo. in my real app it takes longer. the lookup takes 100ms. the 100ms is the problem I need solved. I can't use pathom if this is how long it takes to do a query that is on a resolver with no real logic happening

I'm not sure why the resolver is doing any work. derefing an atom is less than 1 ms.

the resolver isn't doing any work, the runner is. It starts with an empty map and merges resolver outputs, cached or not, into it and returns that map. That could be the cost you're seeing