Fork me on GitHub
#pathom
<
2023-01-27
>
kendall.buchanan00:01:42

Second question of the day! Our index now has a couple hundred resolvers with a few hundred keys. We’re noticing that small changes to queries can have big differences on the performance of the planning step. Are there any principles to keep in mind with respect to improving planning performance?

wilkerlucio01:01:50

a big one is to use a persistent cache for the planner, its just adding an atom to the env at ::pcp/plan-cache* key so the plan cache is preserved between requests

wilkerlucio01:01:25

since the plan does care about the data values (just their shapes instead), the same query (even with different data) will reuse that cache

kendall.buchanan02:01:47

Yeah, I added that after I sent over the question and it made a radical difference.

kendall.buchanan21:01:39

Does the plan cache work independently of the values provided by the query’s ident? If, for example, a :user/username is provided, will the plan be cached for all possible values of :user/username?

wilkerlucio21:01:02

it supposed to, but I went to check after you said it here, and I see how it isn't working that way

wilkerlucio21:01:16

but I think there is a simple fix to it, can you please open an issue?

kendall.buchanan21:01:18

No problem, I’ll do that now. Can you imagine any side-effects of using a persistent cache for the planner? I can with the resolver cache, but it seems like the planner cache is more or less a free lunch.

wilkerlucio21:01:46

thanks for the issue!

wilkerlucio21:01:07

the major side effect is memory usage that will increase, but the tradeoff weights a lot more on the gains here

wilkerlucio21:01:03

in case the memory usage of the plan cache becomes an issue, then you can switch the basic atom with some more sophisticated caching implementation, an atom is just a type that supports caching, but for pathom what matters is that it implements the cache protocol: https://pathom3.wsscode.com/docs/cache#the-cache-protocol

kendall.buchanan21:01:05

Gotcha, but with this patch the space taken up in the cache will be considerably less, I imagine.

wilkerlucio21:01:20

fix almost done 🙂

kendall.buchanan21:01:25

Excellent, thank you.

wilkerlucio21:01:24

fixed on main

kendall.buchanan21:01:11

Thanks again. I think Pathom is, hands down, one of the best OSS libraries in the Clojure community, by the way.

❤️ 2
wilkerlucio21:01:30

humm, Im thinking my solution may have introduced a bug

wilkerlucio21:01:36

let me check here

wilkerlucio21:01:45

yup, bug confirmed, working on it

wilkerlucio21:01:05

I was naive here, because I just changed the cache key, but in the way I did it will always use the same ident value facepalm

wilkerlucio21:01:26

already have a test exposing it, now time for the fix

wilkerlucio22:01:10

current main is working again, but with the ident optimization removed, have to give a bit more thinking to the solution. the issue is that I need to adapt the graph back after, to re-inject the proper ident values there

wilkerlucio22:01:35

ok, it should be good now on main

kendall.buchanan16:01:06

As a recap, it’s all working as you’d like now?

kendall.buchanan16:01:15

If so, any chance you could cut a new release for Clojars?

kendall.buchanan16:01:35

(There are two patches I’m hoping to apply.)

wilkerlucio16:01:06

yes, it's all running as expected, I plan to cut a release soon, prob today, but max tomorrow

kendall.buchanan16:01:16

Excellent, thank you!

wilkerlucio14:01:55

ups, at least is a simple one, do you mind opening an issue please?

👍 2
wilkerlucio21:01:43

fixed on main

❤️ 2