Fork me on GitHub
#pathom
<
2021-07-16
>
markaddleman16:07:44

Should we expect the planner to discover cycles in the attribute graph? I accidentally this situation and ended up with a StackOverflowError. I'm not too surprised that the planner doesn't detect it but I thought I would ask.

wilkerlucio16:07:23

it could be stack overflowing more because of too deep branching

wilkerlucio16:07:39

can you make a repro?

wilkerlucio16:07:59

there is another debug method using the snapshots

wilkerlucio16:07:48

if you use pcp/compute-plan-snapshots, even if there is an exception it still captures the snapshots as it goes, so you can see the steps before the failure

👍 2
wilkerlucio03:07:10

I just remembered that I had a case like that in Pathom 2, and a simple thing that may work for your case is increasing the stack size on the JVM

wilkerlucio03:07:24

(in case its not a cycle loop of some kind)

markaddleman04:07:47

Thanks. I'll look into that. In my particular case, the resolver has an input join where the joined attributes are also output from the same resolver.

markaddleman04:07:39

I haven't tried a repro case yet but Im pretty sure that no acyclic plan can be created

wilkerlucio04:07:40

does it has nested inputs?

markaddleman13:07:33

Here's the resolver input/output declaration:

{::pco/input    [:event :portfolioKey :appKey :event/type {:aliased.event/events [:event/name-expr :event/filter-expr]}]
   ::pco/output   [:event/filter-expr]
   ::pco/priority 1}

markaddleman13:07:07

If I remove :event/filter-expr from the nested input, there is no stack overflow.

markaddleman13:07:11

I'm working on a repro case now

wilkerlucio19:07:46

thanks to your example I was abre to reproduce it here, and yes, Pathom is missing cycle detention when its about nested inputs

markaddleman21:07:24

Oh great! You just saved me an hour of creating a repro case. Thanks 😊

wilkerlucio22:07:53

please note main has a big breaking change related to the strict requests, other than that it should be good to go

wilkerlucio22:07:33

check changelogs for more details on the things at it

markaddleman22:07:31

Thanks. I have played with strict and discovered that I have some application fixes. I think I'll just turn off strict mode for now

mauricio.szabo18:07:50

Hi, @wilkerlucio! Recently, I was investigating what I thing it's a memory leak on the Chlorine project and found this situation below. When I tried to go deeper, every node that consumes more memory points to $com$wsscode$pathom3$connect$planner$compute_missing_chain_deps$$. I'm only asking if you have anything in mind, like, "ok, maybe I'll have to refactor this code in the future" so I can try to check if that's the code that's giving problems, etc... 🙂 (Just to be clear: I'm not sure that Pathom is at fault here yet, I'm just debugging 😄)

wilkerlucio00:07:11

one thing about planning, did you setup a persistent cache for planning? that can reduce a lot the cost of planning, giving queries are usually consistent (same queries, which means same plan result, even if the input/output data is different)

mauricio.szabo01:07:03

Not really. In fact, wouldn't setup a persistent cache take a bigger hit on memory?

wilkerlucio02:07:25

could be, but it would avoid the process of compute-run-graph completly (just enter, read cache, exit), I expect that to be a good tradeoff for most usages

mauricio.szabo02:07:38

A question: if I have 2 resolvers for the same data, and one sometimes fails and sometimes don't, do the persistent cache for planning that that into account when resolving things?

mauricio.szabo02:07:01

(the one that fails have higher priority)

wilkerlucio03:07:04

the plan is a variable of indexes + query + available data shape (not the data specifically, just the shape of it, for instance, if you have the data {:foo "bar" :baz {:deep "data"}} the shape of that is {:foo {} :baz {:deep {}}}, so similar data like {:foo "other thing" :baz {:deep "different here"}} will have the same shape, although the values are different)

wilkerlucio03:07:21

so, if something fails sometime,s that's always after the planning is done

wilkerlucio03:07:29

because the plan always have every possible option

wilkerlucio03:07:02

in other words: the plan is the graph before running it

wilkerlucio03:07:13

and the plan never changes after it start running

wilkerlucio03:07:06

so for the case of 2 resolvers for the same data, the plan will always include both options, connected via an OR node

nivekuil21:07:21

has anyone thought of using pathom to replace integrant? connecting integrant pieces together feels too manual

👍 6
nivekuil01:07:21

you replied too late 😞 already invested in this thing

nivekuil01:07:21

I think my thing is a little different actually since it's an integrant-like API instead of component. do you use summon in prod anywhere?

souenzzo01:07:44

To be honest, I deprecated this library. The idea is nice, but deal with "smart connect" AND "state management" at the same time is really complicated.

wilkerlucio03:07:14

@U797MAJ8M interesting way you got for the halting, one suggestion, you could put done as part of the env instead of a global variable (an atom at env), so you could control it better, and maybe add a way to avoid starting an already started system (or maybe an auto-restart in that case? by halting everything and starting over)

nivekuil04:07:27

I made a few changes actually, I think it's actually very ergonomic

(nx/bind! server [{::keys [ring-handler aleph-opts executor]}]
  {::nx/halt #(.close %)}
  (start-server ring-handler (assoc aleph-opts :executor executor)))

(nx/bind! executor []
  {::nx/halt (fn [x] (.shutdown x)
               (.awaitTermination x 15 java.util.concurrent.TimeUnit/SECONDS))}
  (utilization-executor 0.9 256 {}))

(nx/init {:app.server/ring-handler  (reitit.ring/create-default-handler)
          :app.server/aleph-opts {:port 1234}}
         [:app.server/server])
to start an aleph server + executor

nivekuil04:07:19

problem is I can only call nx/init in the same namespace as the one in which I define the resolvers (`nx/bind!`) because I don't know how to get pathom to know what start-server , inside the :resolve fn, is at resolver run time, to the macro it's just a symbol like the ones from pco/input

wilkerlucio04:07:06

what if you put start-server on the env?

nivekuil04:07:12

how do you mean? right now it comes from [aleph.http :refer [start-server]]

wilkerlucio04:07:23

like

(pco/defresolver resolver [{::keys [start-server]} _]
  (start-server))

(p.eql/process
  (-> {::start-server (fn [] "implementation")}
      (pci/register resolver))
  query)

nivekuil04:07:29

ideally the macro body feels exactly like defresolver, i.e. a closure

wilkerlucio04:07:56

are you familiar with the resolver taking 2 arguments?

nivekuil04:07:44

yeah, but the defresolver body doesn't have to be a pure function of env + params right

wilkerlucio04:07:46

mostly they should be, caching and logging are common exceptions

wilkerlucio04:07:15

to what end you are considering the pure sense of it?

nivekuil04:07:42

i.e. you can close over vars in the namespace, like from :refer

nivekuil04:07:26

you don't have to put them into env to use them inside the resolver, though maybe that is a good idea? looking through my code to see how often that even happens

wilkerlucio04:07:48

correct, and most of the time won't be in the env

wilkerlucio04:07:10

its nice to put on the env to create some indirection, in case you want to fill that up later

wilkerlucio04:07:03

most common when writing some sort of generic resolver, or to place things like database connections, this gets a bit meta in your case because you are building the config itself on top of that

wilkerlucio04:07:40

are you using the result of this setup as config for another Pathom thing, or as a stand alone system build up / shut down?

nivekuil04:07:36

I actually have no idea how pathom even manages do to this.. how does the runner know to run the :resolve fn as if it were in the same namespace as it's defined?

nivekuil04:07:49

I want to replace integrant completely, so stand alone

wilkerlucio04:07:10

the fn binding happens when we make the resolver

wilkerlucio04:07:51

looking at the resolver form:

(pco/resolver 'foo
  {::pco/output [:foo]}
  (fn [env input]
    ...))

wilkerlucio04:07:11

the resolver stores that fn defined there, so the bindings are all in place

nivekuil04:07:57

ahh, finally got it. that's a good hint, the eval needs to be done in resolver instead of defresolver

nivekuil04:07:03

basically I was calling eval at runtime instead of defresolver time.. oops

nivekuil04:07:07

going back to what you said about putting done in env, can I dynamically modify env from a resolver call in pathom3?

nivekuil04:07:20

I know there's a ::p/env in p2 but don't know what it is in p3

nivekuil04:07:55

and as far as not starting an already started system, I think resolver cache might be the solution?

wilkerlucio05:07:22

you can't modify env, but you can have mutable things on it

👍 3