Fork me on GitHub
#om
<
2016-03-07
>
jlongster05:03:08

@anmonteiro: this is how far I've gotten with the path-meta optimizations https://gist.github.com/jlongster/1c661626a0860022ef26

jlongster05:03:30

a bit messy, but it solves my use case. I'm not using union or recursive queries yet though, and those will break

jlongster05:03:48

I don't know what to do with union queries because I don't know which subquery to use while descending

jlongster05:03:53

note that this won't help much if you're still querying across a lot of data, as it will still descend into it. It allows you to "bypass" the system though by returning a lot of data for a single key (don't query into it) and optimize it yourself

akiel13:03:02

Can I use om.next code inside an om app? Can I somehow mount a om.next root inside a om controlled vdom?

danielstockton13:03:03

Don't see why not, if you're happy with them being separate apps.

akiel14:03:20

Yes I have to mount them separately in a real dom. But mounting the om.next part in a om controlled vdom is not possible?

anmonteiro14:03:31

@dnolen: the work that @jlongster is doing on path-meta will probably need to use the query-template & co. helper stuff. Would you be open to move these to a query-util namespace or something?

dnolen14:03:56

@anmonteiro: if we can avoid that it’s preferable - it’s not clear to me why path-meta needs query-template at the moment.

anmonteiro14:03:46

@dnolen: I might be misunderstanding what has to be done in path-meta

anmonteiro14:03:20

but I think we would need that stuff to choose union branches & join entries etc.

dnolen14:03:57

hrm that might be true

jlongster14:03:58

yeah I figured there might already be code for all this logic. basically path-meta needs to walk through the query so it knows when it can stop tagging data

jlongster14:03:36

I'm trying to think if there's another way we could do this, like dynamically constructing the path as props are passed down

anmonteiro14:03:30

@jlongster: I'm looking at your code and might have something for you soon

anmonteiro14:03:15

something that immediately jumps to mind is that you can't have that top-level if

anmonteiro14:03:39

or rather, you need to put x in the else branch

jlongster14:03:49

@anmonteiro: awesome, thanks! I've studied the zippers & templates before but not enough to know how to leverage them (if possible). I don't know how to move forward with union queries.

anmonteiro14:03:13

@jlongster: I'm looking into it, I'll ping you when I have something

anmonteiro14:03:41

but this would involve having the zipper / query-template stuff in a .cljc file

anmonteiro14:03:56

which I did locally

jlongster14:03:04

@anmonteiro: I see what you mean about the if, I wonder how it was working for me in my app, strange

jlongster14:03:10

also this doesn't really help for most cases, it only allows you to not query across data (just use :items instead of {:items [:x :y]} but still return a big list for the :items key, and then you can optimize that part of the app yourself. If it still descends into each item it's still a bit slow

jlongster14:03:45

so I don't know if there's another approach altogether that is faster for the general case

anmonteiro14:03:09

@jlongster: I don't think we can avoid descending into that stuff altogether with this approach

anmonteiro14:03:37

I'm thinking that we can, however, reduce by one the number of recursion steps

anmonteiro14:03:46

by looking ahead into a query and seeing that it doesn't have joins

anmonteiro14:03:08

and not recurse into that

stuartsierra14:03:23

Are there any problems with using the Om.next-style normalized application state in an Om.now application?

stuartsierra14:03:41

(Other than having to do the normalization manually.)

jlongster14:03:23

@anmonteiro: if you have a list of 1000 items and a component that renders each item, you still have to tag each item individually right?

jlongster14:03:55

I think it's the descending into each item that is slow for me, but maybe I just need to use smaller page sizes

anmonteiro14:03:21

yes you do have to tag each one in that case, but we can avoid recursing into the single properties, it's what I'm saying

anmonteiro14:03:12

e.g. in [{:item 1 :foo 2} {:item 2 :foo 3}] you need to tag the maps but not their properties

anmonteiro14:03:17

which is happening currently, I think

jlongster14:03:34

oh sure, my naive patch should have achieved that as well

jlongster14:03:11

it's a bit unfortunate imho that I can return the same data as the previous read and even with a memoized db->tree the query can be slow because of this step

jlongster14:03:20

but maybe most apps won't run into this

jlongster14:03:30

and I need to structure things a little better

jlongster15:03:50

@anmonteiro: here's why this is important to me, I don't know if I should be doing things differently: I have a big list of data, but I load it incrementally in chunks of, say, 100. Loading 100 at a time is fast enough and works great. But if I've scrolled down enough and the list had loaded 5000 items by now, if I do anything in the top-level components (`set-state!` etc) it re-reruns the query which now returns 5000 items, and path-meta is the bottleneck there

jlongster15:03:07

need to focus on work today so not doing much more, just explaining my use case

anmonteiro15:03:47

@jlongster: I also can't spend any more time on it today

anmonteiro15:03:00

I've added a comment to your gist with my working stuff

anmonteiro15:03:18

for posterity

anmonteiro15:03:30

I'm thinking that another approach could be follow the query and add meta to whatever matches it and leave everything else alone

anmonteiro15:03:04

we currently follow the data. wouldn't it be more performant to follow the query instead?

jlongster15:03:20

not sure, I'd have to see the code, I suspect if there's a path that has to descend at all it still might be a problem. but I'll think about it some more.

anmonteiro16:03:32

I was just laying my thoughts here, I think it's something worth pondering. more of a rhetoric question

jlongster16:03:09

@anmonteiro: last thought: following the query might make it easier for optimizations, like detecting if the the components that use this part of the query don't need path at all, maybe we could skip it (we don't need it if it doesn't use dynamic queries right?)

anmonteiro16:03:38

we always need the path meta

anmonteiro16:03:46

full-query will use it, for example

anmonteiro16:03:54

and that's triggered after any mutation

jlongster16:03:19

I made a test case for the situation I described before: imagine components A->B->C, A and C have queries and B doesn't, but B has local state. If B calls set-state! to rerender and C calls set-query! to change its query, it seems undefined what props B passes to C https://gist.github.com/jlongster/df77ec0d37818162557f

jlongster16:03:43

strangely, I can set the query once, and continue updating the number and C continues to render bar fine. but if I set the query again (it still stays bar, but does another set-query! call) and then increment the number C displays foo

jlongster16:03:11

I don't know if the new dynamic query work makes this better

anmonteiro16:03:49

@jlongster: haven't looked at it, but set-query! also accepts a list of reads as transact! does

anmonteiro16:03:51

would this solve the issue?

jlongster16:03:28

@anmonteiro: doesn't seem like it. C changes its own query and only it needs to rerender the new query anyway

jlongster16:03:19

the problem is that B rerenders because it changes local state, but it has no queries so it doesn't re-run any queries, but it passes props down to C. It should be passing down old props before set-query! but I'm seeing inconsistent behavior, sometimes it does and others not

kenbier17:03:12

@tony.kay im working on a similar issue using c3. I was thinking of diffing the next-props to be loaded with the current props in shouldComponentUpdate, and returning false is there is no difference, otherwise true. I then put the render logic in didUpdate. Is this what you were thinking? It seems reasonable.

tony.kay17:03:00

@kenbier: well, but then you're telling React to update the DOM as well

kenbier17:03:08

@tony.kay I see, I didn’t think of that. Why is that an issue? What does your render function look like?

tony.kay17:03:17

well, that's why I was asking the channel about it. If I don't touch shouldComponentUpdate at all and put my rendering in other lifecycle methods it works fine; however, technically React could mess with it.

tony.kay17:03:12

@kenbier: For whatever reason, my sample works when I do it that way, but it seems like doing it via componentWillReceiveProps and always returning false from shouldComponentUpdate is better.

kenbier18:03:48

I see. I didn’t touch shouldComponentUpdate and did not notice any performance issues either simple_smile. So I’ll probably leave it as well. Thanks for explaining.

jlongster18:03:46

@anmonteiro: (can't avoid thinking about this) it seems like we could tag the data path inside db->tree as it generates it, and in path-meta don't recurse into data that already has the tag

jlongster18:03:24

it's bothering me that we are the ones generating the data and should be able to do that on the fly instead of recursing the data structure afterwards

jlongster18:03:23

users also could then selectively "pre-tag" data in their parsing functions if they really wanted to micro-optimize something and path-meta wouldn't recurse into it

jlongster18:03:18

that also means that work would be memoized since I memoize db->tree

tony.kay18:03:15

@kenbier: I hadn't tried the alternative...just did, and it works fine and is almost certainly the way to do it.

tony.kay18:03:12

where render-squares is D3 code that uses the props to render/update the UI (idempotent)

jlongster18:03:58

@anmonteiro: alternatively, this actually would be a very simple solution that solves most of my concerns: expose path-meta to the user and allow them to preemptively apply it in the read function, and make path-meta not recurse if it's already been tagged. that way I could memoize this work, and my worries go away

jlongster18:03:27

because I'm fine doing it once, but when any parent component changes it re-queries the big data and re-does all that work making a simple query change slow

jlongster18:03:05

(I understand you're working, no need to response any time soon)

anmonteiro18:03:20

@jlongster: hah, I know the feeling simple_smile

anmonteiro18:03:37

there's a knob to disable path-meta in the parser

jlongster18:03:59

@anmonteiro: yeah, but like you said we actually need it, right? wouldn't things just break?

anmonteiro18:03:14

yes we need it. I meant if you want to do it in your read fns

jlongster18:03:25

I basically want to disable it on just part of the state...

anmonteiro18:03:45

also, putting it in db->tree might not be sufficient

anmonteiro18:03:53

because we also need it in non-normalized data

anmonteiro18:03:05

it's a cool problem to think about

jlongster18:03:34

db->tree returns non-normalized data

jlongster18:03:00

but yeah, in the end I basically want to disable it for part of the app state, knowing that I won't use features that will break with it

anmonteiro18:03:03

I meant people might not be using normalized data, and so they're not using db->tree at all

jlongster18:03:06

right now it's all or nothing

jlongster18:03:18

oh, yeah, well we'd still use path-meta normally then

anmonteiro18:03:30

this is something that needs to go in the parser

anmonteiro18:03:56

I'll prototype a version of path-meta that doesn't follow the data, but the query instead, as I suggested earlier

anmonteiro18:03:59

whenever I have the time

anmonteiro18:03:04

(might be only on Friday though)

jlongster18:03:15

ok, no worries, thanks!

jlongster18:03:10

I can live it for now, just trying to avoid any refactoring down the road, but I'll bet that we can just optimize this somehow later

iwankaramazow19:03:57

I was just going through the source: why does (reconciler) `(queue! [_ ks] (swap! state update-in [:queue] into ks))` trigger a rerender?

anmonteiro19:03:39

@iwankaramazow: because reconcile! is run on requestAnimationFrame

iwankaramazow19:03:38

explains a lot why my undo/redo now rerenders

matthavener19:03:21

is there a way to determine the current path into the denormalized state tree in a mutate?

matthavener19:03:00

i see private functions for path in the source, but i’m not sure if its exposed publicly anywhere?

iwankaramazow19:03:22

@matthavener: just curious, why do you need this?

matthavener19:03:54

iwankaramazow: i want to store some ui state for a component down in the tree

matthavener19:03:41

and i definitely dont want to write a mutate handler for every possible path down the tree.. its not even possible for some dynamically defined paths (like a component tied to some entity)

iwankaramazow19:03:55

so 'thinking-with-links' style queries don't suffice in your case?

matthavener19:03:49

let me read that, it seems useful simple_smile

iwankaramazow19:03:08

that's what I'm referring to

iwankaramazow19:03:24

the graph magic 🙏

seanirby20:03:36

hey i've been slowly walking through the om next todomvc tutorial to try and understand it all, and I'm not sure what the

:find
part of this query is doing. can someone explain?

seanirby20:03:19

(q [:find [(pull ?eid selector) ...]
          :in $ selector
          :where 
          [?eid :todo/created]] (db conn) [:todo/title :todo/id])

seanirby20:03:49

i know based on the :in and :where that we're querying for the entities that have :todo/created :todo/title and :todo/id attributes

anmonteiro20:03:52

@seanirby: finding all titles and ids for all todos

anmonteiro20:03:33

@seanirby: what part don't you understand

taylor.sando20:03:42

Pull will pull all attributes that has entity id ?eid, using the selector to specify the attributes

seanirby20:03:02

anmonteiro: well, why isn't it just :find ?eid

anmonteiro20:03:35

because we wanna use the query that the client has sent

anmonteiro20:03:14

we could be only asking for the title

taylor.sando20:03:59

:find ?eid
would just return a vector of ids [[id]], whereas what is is doing will return
[{:todo/title <val> :todo/id <val>}]

seanirby20:03:49

i thought the :in $ selector narrows it sufficiently?

anmonteiro20:03:08

@seanirby: that's just "declaring" variables

anmonteiro20:03:12

you need to actually use them

anmonteiro20:03:17

e.g. in pull

seanirby20:03:36

is it correct to say that the pull expression in the :find field determines the format of the output response?

taylor.sando20:03:33

It will return a map

anmonteiro20:03:14

actually the ...] says it'll return a vector

skardan21:03:19

Hi, have anybody experienced following error after upgrade to alpha31?

skardan21:03:28

Compilation of om/next defui fails with :advanced optimization

skardan21:03:46

WARNING: Use of undeclared Var foo.bar.baz/x36291 at line 38

skardan21:03:04

(there is defui on line 38)

skardan21:03:41

maybe defui now refers some react symbol with missing externs?

matthavener22:03:34

Is normalization recursive? If I had a ident to a top level unique ident that contains a vector of idents, will those idents get normalized calling tree->db?

matthavener23:03:13

basically, the contents of the map normalized don’t get normalized themselves