Did you measure where the problem starts to be? For example, high nodes should probbably return true from shouldComponentUpdate to prevent a large comparison that will mostly return true anyway. Do you know it is denormalization that is the problem? If it is, then the next question is: Is this read-only data? Does it need to be normalized?

certainly you will eventually reach a point of performance problem if you have enough nodes. Avoiding normalization for those cases is an easy fix if the nodes are not actually written to...e.g. keep a set of expanded nodes in some other place for rendering, etc.

Did you read the book sections on performance?

But at the end of the day if you want to plug in a renderer of your own, you can...and that rendering algorithm could use a custom db->tree that you write for your purposes that stops when it realizes there's no more useful work to do.

just look at keyframe or keyframe2 renderer as a base. The multi-root renderer should be considered deprecated in favor of hooks

> Did you read the book sections on performance? I have, though this time around I was looking for denormalization-related stuff. I see now what you mean about shouldComponentUpdate. > Do you know it is denormalization that is the problem? No 🙂 Denormalization is one thing that came to mind and wanted to check whether it was worth pursuing. It is not read-only data. What I'm hearing is that I should try other optimizations first, but if I decide to follow my idea the way to do it would be a custom renderer. (Probably end up making just a few modification to defaults.) Thanks, Tony!

Keep us posted 🙏

Happy New Year! I've enabled https://book.fulcrologic.com/#_render_middleware and have been doing tests. The setup is: • keyframe2 renderer • I'm using only-refresh on the toggle transaction. • I'm measuring time passed between toggle button press and a console.log call in modified component's shouldComponentUpdate, trying to isolate denormalization times. (This will include the time to run the toggle mutation itself, but that is around 10ms.) • Using a performance-dev build. • Standard transaction processing. • SCU returns true for top-most components. (Though this is not important with only-refresh.) Observations: 1. The measured interval scales with the amount of nodes relevant to the denormalization algorithm (e.g. from where in the data tree the UI starts rendering from). For 4k relevant nodes, this interval is ~90ms, and it roughly doubles from there when doubling the number of nodes. So, it seems there is room for tailoring the denormalization algorithm. 2. However, given the (correct) FDG advice that we should change CSS instead of DOM, I've realised that my original idea of modifying the algorithm would not work – it would change the DOM (potentially thousands of nodes). 3. Given that the denormalization times are constant (for fixed amount of nodes) and the reasonable expectation that the UI will always render orders of magnitude less nodes than what is in db, there might be an inflection point where my original idea makes sense. But I'm doubtful – for each data node there will be many DOM nodes, DOM is slow, etc. 4. only-refresh doesn't seem to work with synchronous transaction processing (still re-renders from root), so it ends up being slower than async. I'm assuming this is not surprising? (I have to read more on the transaction system.) So, I'm left not knowing what to do. On the one hand, it is not terribly slow. On the other – it's a constant penalty paid on every interaction. One obvious thing is to try to keep the client db light. This is a much more complicated task, balancing recursion limits (UI and load), incremental loading, the fact that trees can grow very fast, etc.

Don't know why I haven't thought of this before, but: I could get a feel for the size of the problem by measuring db->tree. So, denormalizing a tree with 4k nodes takes about 60ms (`simple-benchmark`). (Just as a reference point, pulling the same tree directly from Datascript (1.3.0) is 3 times faster.)

@U0CKQ19AQ, I've tracked down why the following happens: > only-refresh doesn't seem to work with synchronous transaction processing (still re-renders from root)... The local action works as expected, but the remote action triggers a render from root.

that option definitely depends on the optimized renderer installed

oh, you mean you install sync tx processing as the tx processor? not called a !! variant?

Yeah, that's probably a bug in passing through the txn options

On the performance: I'm very open to an optimized db->tree. I have neither had the time nor inclination (since it has always been fast enough for my uses) to do it. I don't remember if I left a hook for configuring that (I don't think I did), but we could revamp the system to let you plug in a custom version, and then it would be guaranteed not to be a breaking change. At one point in the source we had generative tests against that algorithm...possibly still there. Wilker did them. We were A-B comparing a db->tree written with pathom vs the stock one. The fact that Datascript is 3x faster is surprising to me...are you certain you are doing an equivalent test, and that you're getting the exact same output from the two?

> oh, you mean you install sync tx processing as the tx processor? not called a !! variant? Yup, that's what I mean. Everything else is equal (`transact!` with only-refresh option), I just switch out the standard tx processing with synchronous.

> The fact that Datascript is 3x faster is surprising to me...are you certain you are doing an equivalent test, and that you're getting the exact same output from the two? I double-checked and yes – same output (not counting Fulcro-specific stuff like routing data). Datascript 1.3.0 improved pull performance drastically. There's https://github.com/tonsky/datascript/releases/tag/1.3.0: "pull has much more information ahead of time, which could be leveraged for optimizations", which I wonder if could be relevant for Fulcro (pull syntax ~ EQL and all that). I imagine Fulcro's algorithm has more stuff it needs to do. > we could revamp the system to let you plug in a custom version, and then it would be guaranteed not to be a breaking change. That would be awesome! (It would also allow me to fiddle with recursion, though there's stuff I still need to figure out there.) Though, I'm pessimistic about how much could I optimize. As I've mentioned before, I've realised that making the algorithm forego joining nodes which would be invisible on screen wouldn't work – it would change the DOM too much (and that must be more expensive than needless denormalization.)

"I double-checked and yes – same output (not counting Fulcro-specific stuff like routing data)." is not making me very confident that you did an apples-to-apples comparison

e.g. you use the exact same query against the exact same graph of data? The output should be identical, otherwise you are not measuring an equivalent thing

I have thought of one thing that might be beneficial, maybe for other users of Fulcro as well. As I understand it, even when using only-refresh the whole subtree of the component being refreshed still gets denormalized (and tunnelled through), correct? Maybe there's a way to short-circuit that?

in 99% of apps it really makes no appreciable difference

I can count the number of times I've used only-refresh on, like, one finger

most applications do not have 4k nodes...that is an extremely rare case

and since it is a rare case, I have a general practice of NOT complicating the platform to make an extreme corner case better

the way React works, you simply cannot update a parent of a subtree without it, by definition, rendering the children. There is no way to have that "automatically controlled".

the optimizations around that are built-in and automatic, but the fact is YOUR code has to render the VDOM so it can diff it

shouldComponentUpdate and memoization (in hooks) are the tools we've got

It would be useful if you could publish your performance comaprison code as a stand-alone repo

(Datascript vs db->tree)

> in 99% of apps it really makes no appreciable difference ... most applications do not have 4k nodes...that is an extremely rare case I don't doubt that at all. And even with 4k nodes, it's quite fast! 🙂 > I can count the number of times I've used `only-refresh` on, like, one finger And I think I'll be using it on most user interactions 🙂 (Which is why I've considered using the ident-optimized renderer, but realise that's not it reading more closely.) > and since it is a rare case, I have a general practice of NOT complicating the platform to make an extreme corner case better I absolutely understand :) > It would be useful if you could publish your performance comaprison code as a stand-alone repo Will do.

a runnable repo would be best instead of a gist

Here it is: https://github.com/zeitstein/fulcro-datascript-denormalization-benchmark

I have to do a bunch of reading to digest you deep tree demo, but wanted to get back to this in the meantime, try to make it clear: > As I understand it, even when using `only-refresh` the whole subtree of the component being refreshed still gets denormalized (and tunnelled through), correct? Maybe there's a way to short-circuit that? So: deep recursive tree, just want to update a single node (and I mean literally only the data in its (normalized) db entity). only-refresh only updates the single node (other instances of it, as well) and the only issue remaining is that it needs to denormalize the whole subtree starting at that node. So, what I'm thinking, for my use case: take the previous props of the node, simply assoc new stuff into it (and update db correspondingly), pass that as new props to the node. No denormalization required, and React does its diffing from there. That sounds quite doable? (Maybe traced-db->tree does something similar, but I'm yet to actually read through the code.) > Here it is: https://github.com/zeitstein/fulcro-datascript-denormalization-benchmark Let me know when you've had a chance to look at this, whether I'm making a mistake somewhere. If not, I'd be interested to know whether you think this means there's room to optimize the denormalization algorithm. (Which, to be clear, I would not expect you to do. Just wondering whether you think it might be worth pursuing.)

So, you are right that Datascript manages to be about 2.4x faster on a raw pull, which is surprising to me on the surface. I assume it is because it has indexes to work with, so that some of the work is pushed off at transaction time instead of query. It may also have some smarter ways of doing loop detection.

That said, 35ms for a 4k node denormalization is very fast. Dropping it to 15ms or so is nice, but of the same basic magnitude in user space. E.g. the scalability of Datascript isn't really any better (it goes up linearly with number of nodes). So, while you can say it is 2x faster in a micro benchmark, it really doesn't matter. At some point there is a limit where there are too many nodes, and denormalization during a render (of all of them) is just too expensive.

Most UIs have 100s of nodes, not thousands.

When you get to this specific problem, your choice is clear: reduce the number of nodes that have to be denormalized in a render. That said, you do not have to be so extreme as to make that number "1". If you can get it down into the 100's, then it'll generally be fine...which is what my demo shows. The tree can have 100,000 nodes in it, and as long as you don't try to open them all, it works fine, because it only has to denormalize what is open.

Did you compose Main into root, and Main should have initial-state of at least {}

Your merge is fine (obviously)...it's the path from root that is causing you a problem. Things HAVE to compose in DATA AND QUERY to ROOT. Everything is relative to root...

Unless you use floating roots or react hooks use-component

The rendering composition, or course, will run on render and can partially succeed with nil props, thus you may see parts of your UI even thought they have no "data". This is fooling you into thinking you've composed Main in correctly...when in fact if you trace data data from root you'll find a "gap" in the linkage, because you did not provide initial state for Main, therefore it did not compose it into the data model.

I'm pretty sure Fulcro is giving you warnings about components that don't have initial state in the console

Here is the link to the code: https://github.com/sittim/fulcro-test/blob/4c633ad404e28060d0b561d8fb515d2ab6863fca/src/main/app/ui/root.cljs#L168

add :initial-state {} to Main and reload

> I'm pretty sure Fulcro is giving you warnings about components that don't have initial state in the console Absolutely right:

does that fix it?

Here is what I get in console, but still no data:

(merge/merge-component! SPA Problem {:problem/id    1
                                       :problem/title "First Problem Title"
                                       :problem/summary "First Problem Summary"}
                          :append [:component/id :main  :problems])

does NOT match your query

[:component/id :main :main/problems] is the path your code is expecting

(for the :append)

OK, that worked!

WOW, thank you so much, will have to digest it.

yeah, you have to be meticulous about your keywords 😄

and almost anything that has a static ident should have initial state

Most problems I diagnose where "it isn't rendering what I expect" boil down to: 1. Did you compose your query to root? 2. Did you compose your initial state from root down to the statically-available leaves of the UI? 3. Did you properly destructure the props you received at each node, and pass them to the children? That's literally the full checklist.

It's like the joke that all IT problems boil down the the same diagnostic: "Did you turn it off and on again?"

Yeh, Fulcro is awesome, but there is a lot to know all at once to use it effectively. Thats the price you have to pay for its power.

well, the point I'm trying to make is that rendering is very very very simple. it's those three things.

which allow the system to, on startup create a data graph, then at render pull a tree of props that match your tree of UI. It's all the same composed tree...once represented in a data graph, and once represented (in mirror form) in a code graph

that's it

app starts, make a data graph (initial state). App renders, pull the data graph, and pass it to the code, which pulls the bits apart as it walks the tree of DOM

Mutations: Refine the tree...UI code can use conditional logic to reshape/respond

Understand, actually, what I think tripped me is problem/id is just a single keyword, I was thinking that Fulcro is parsing it into 2 branches [prblem id] thats is why I have been having a hard time dealing with the data tree.

https://book.fulcrologic.com/#_ident_generation

you mean this?

using a single keyword is just a shorthand

:k -> (fn [this props] [:k (:k props)])

Yes, looking back now, my assumption was silly.

Major progress today, this is awesome, I have been at this for few days. I only get to learn it about 1 hr per day.

Have your followed https://blog.jakubholy.net/2020/troubleshooting-fulcro/ when troubleshooting your problem? How far did you get/where did you get stuck? => what can I improve?

Actually, I did not, I forgot about it.

Frontend 3.a.i would lead you to trying db->tree uchu would show you it is a data problem not UI/rendering one. Though spotting the keyword error is not trivial I guess...

Have you tried to use https://github.com/holyjak/fulcro-troubleshooting?

Yes, actually, the way the keywords work together is super easy once you get it. I was making a wrong assumption about keywords that is why I had so much trouble. This cleared it up.

I almost think that there could be a more visual way to tie it all together. I have been thinking of a way to create a diagram that would show how the keyword paths work, but have not been able to come up with something clear and easy to understand.