Clojurians Log v2

If you’re using Pathom3, try turning that into a https://pathom3.wsscode.com/docs/resolvers/#batch-resolvers

@caleb.macdonaldblack it's a resolvers that returns a collections of users, output is {:admin/users [:user/id]} and there is no subquery needed to resolve further. this little green blocks seem to be purely pathom processing the outputs

marking the result as pco/final does remove these but ofc it's not suitable if eg I want extra attributes that are not resolved immediately by this resolver which is the case for a bunch of other queries that I have that do return a big amount of items in a seq

If it’s consuming time on simply processing the data (and not fetching from a db or something) then I’m not too sure. If you’re trying to take a large number of entities and process them by filtering or aggregating, doing that work db side will almost always be better option as apposed to using Pathom. Pathom just doesn’t have the performance for that sort of task.

However, if it’s just a data fetching problem you’re trying to speed up, your {:admin/users [:user/id]} should work just fine, and you’ll want to turn your user/id -> user/name, ... resolver into a batch resolver.

Also try out the p.a.eql/parallel? option and see if that helps.

Make sure you’re using the p.a.eql too, and not p.eql

resolvers you want running in parallel should return futures or promises or whatever

I didn’t know about ::pco/final. That’s pretty handy

hello Eric, yeah, this is an overhead that pathom adds because even if the entity has all the data, Pathom must check it (scan over results, compare the data with required query, etc...)

if you know you wont have any futher process, you can use ::pco/final to mark the data as complete, doing so will tell pathom to not try anything, this way you can avoid the scan

I was thinking the parallle processor could try to split a vector in blocks and process them in parallel, but its tricky, because I also dont want to be spawing too many threads, a solution like that involves also adding some sort of thread management/configuration

@wilkerlucio would it be possible to have something like “final” but instead of doing nothing it does the job on the first item of the sequence and extrapolates to all the others? Because when the resolvers is getting a lot of rows from a table you are 100% sure all items have the same shape of outputs?

not impossible, but its a considerable change, something maybe to test and see if there are enough gains to compensate the add complexity