Fork me on GitHub
#pathom
<
2019-09-20
>
kszabo09:09:28

any ideas to improve performance of trivial resolvers (think alias-resolver) better when using core.asyns’s threadpool for execution? My usecase is that on my entity I have a lot of entries with keys where I wrote an alias resolver for each one. Pathom is essentially doing the job of clojure.set/rename-keys at this point, but the context switches kill the performance (since there are a lot of tiny resolvers to execute)

kszabo09:09:19

I could write a single resolver which does the job with clojure.set/rename-keys but since there is no support for optional inputs (there is an issue for that) it cannot be currently expressed

wilkerlucio17:09:19

@thenonameguy one question first, why do you need so many renames? if you are loading from some source that doens't have them, is it possible to convert when you fetch it on the first time?

kszabo17:09:31

it’s part of our domain

kszabo17:09:42

we had a short uuid based name for some of our fields

kszabo17:09:48

and there is a corresponding human name

wilkerlucio17:09:19

ok, that's a valid case, just not optimized if you using a lot of them

wilkerlucio17:09:51

another thing to consider is, do you need the parallel parser? although I keep recommending it as the default, I'm starting to believe that for most people the serial parser may be a better option

wilkerlucio18:09:41

the question is, does the queries you run have parallelism opportunities? usually if its a single or a few datasources that may not be the case, and them you are just paying the higher overhead and complexity price of using the parallel parser

kszabo09:09:39

I think a better alternative would be to tag certain resolver as ‘sync’ in the resolver map. These resolvers then could be evaluated inline in the go-loop to avoid asynchronous overhead.

wilkerlucio14:09:45

yeah, I was imagining that this could be realised automatically, after the first run pathom could "notice" this is sync, and its fast, using those variables it could automatically choose to not use the thread pool

wilkerlucio14:09:46

what you think?

kszabo15:09:18

I think that works, also this improvement could be automatically applied to some resolver helpers like alias-resolver* (if you don’t consider this a breaking change)

kszabo15:09:43

an alternative would be to use the weights of resolvers and have a cutoff where resolver-weight < weight-cutoff is executed serially

kszabo15:09:33

this could be really small by default

wilkerlucio17:09:28

yeah, that's what I was thinking 🙂

wilkerlucio17:09:01

an even bolder impl could cache "common paths", kind like an internal JIT, get long processing chains and pre-comp the fns

kszabo18:09:21

specter does something like that AFAIK for it’s transforms

wilkerlucio18:09:46

yeah, I guess it does, but there you have to manually build the transformation pipeline, what I'm thinking here is pathom seeing common paths (because they will naturally tend to repeat) and cache that based on usage, makes sense?

👍 4
kszabo09:09:39

I think a better alternative would be to tag certain resolver as ‘sync’ in the resolver map. These resolvers then could be evaluated inline in the go-loop to avoid asynchronous overhead.