Fork me on GitHub
#fulcro
<
2021-12-17
>
zeitstein14:12:03

Need some help with recursive queries. I want to render the following tree:

- Folder 1
- Folder 0
- - Folder 1
But the UI only renders:
- Folder 1
- Folder 0
I'll put more details in the thread.

zeitstein14:12:30

I want to render the same data in two places. I have a main tree and a side tree. The side tree actually renders as expected, but main does not. I am using the default renderer (not ident optimized). Here are the components:

(defsc Folder [this {:keys [id text open subfolders]}]
  {:query (fn [] [:id :text :open {:subfolders '...}])
   :ident :id}
  ...)

(defsc RootFolder [this {:keys [id text subfolders]}]
  {:query [:id :text {:subfolders (comp/get-query Folder)}]
   :ident :id}
  (div
   (when subfolders
     (map ui-folder subfolders))))

(defsc Side [this {:side/keys [id list]}]
  {:query [:side/id {:side/list (comp/get-query Folder)}]
   :id :side/id}
  (div
   (when list
     (map ui-folder list))))

(defsc UI [this {main-tree :main-tree side-tree :side-tree}]
  {:query [{:main-tree (comp/get-query RootFolder)}
           {:side-tree (comp/get-query Side)}]}
  (div
   (ui-root-folder main-tree)
   (ui-side side-tree)))
The idea is that RootFolder will become a target for routing and is rendered differently from Folder. The only significant difference between main and side trees I see is that RootFolder has the same ident as Folder and uses the same :ids? Can't figure it out. Probably relevant, I get this INFO in console:
component app.ui/RootFolder's ident ([:id nil]) has a `nil` second element. This warning can be safely ignored if that is intended. Props were nil
I don't understand why this happens. Relevant bits from the client db (note differences between main-tree and side-tree):
{:main-tree [:id "main-tree-root"]
 :side-tree {:side/id   "side-tree-root"
             :side/list [[:id "01"] [:id "00"]]}
 :id        {"01"             {:id   "01"
                               :text "folder01"
                               :open true}
             "00"             {:id   "00"
                               :text "folder00"
                               :open true
                               :subfolders [[:id "01"]]}
             "main-tree-root" {:id   "main-tree-root"
                               :text "Home"
                               :subfolders [[:id "01"] [:id "00"]]}}}
Clearly, I could handle main as I do side, but I need to understand why this doesn't work before introducing complications (routing, incremental loading, etc.)

zeitstein14:12:28

I've also noticed manually setting recursion depth in Folder works. But I don't understand why – there are no infinite loops here? I understand that the denormalize algorithm detects a loop at folder01 (and warns about it).

xceno15:12:43

I've recently implemented a UI similar to yours. It's displaying a tree view made up of certain nodes and edges of my graph. I noticed that pathom doesn't like infinite recursion because you can run into cycles of your graph easily. In that case it'll return an empty result (or crash, I don't remember right now). Fulcro's denormalize algorithm handles this gracefully as you've noticed. Your [:id nil] property might come from your initial props of the root folder. Do I understand you correctly that you're basically building a master/detail layout and die Side is your sidebar whereas the RootFolder is your "master" view and will include a router?

zeitstein15:12:19

Hm, still working with a small data set, so haven't come across that about Pathom. There should be ways to circumvent Pathom there, in my case, anyway. > Your [:id nil] property might come from your initial props of the root folder. Okay, added a conditional, so that warning goes away. The main issue remains, so not connected. > Do I understand you correctly that you're basically building a master/detail layout and die `Side` is your sidebar whereas the `RootFolder` is your "master" view and will include a router? Yes, exactly. Though, I might end up wanting to have two separate trees with same behaviour side-by-side, independent routing, etc.

xceno15:12:45

I just copied your example code and look at it. One thing though: I think the open flag is ui only(?), so you should prefix it with :ui. This will also exclude it from queries to the backend. In general I'd recommend to namespace your keys, even if you probably couldn't care less right now 😉

xceno15:12:04

Oh and AFAIK you still need the lamda version for :query if you're calling functions like comp/get-query

zeitstein15:12:05

> I think the `open` flag is ui only(?) Nope 🙂 > In general I'd recommend to namespace your keys, even if you probably couldn't care less right now Ah, this is due to a limitation of Asami, so I have a different convention, which I will not bore you with. > AFAIK you still need the lamda version for `:query` if you're calling functions like `comp/get-query` Thanks! I've tried with that as well, of course. Though, I think the non-lambda version works as well. I did notice that the lambda version is required for recursive queries on routing targets.

xceno15:12:34

(defsc Folder [this {:keys [id text open subfolders]}]
  {:query         (fn [] [:id :text :open {:subfolders '...}])
   :initial-state (fn [{:keys [id text open subfolders]}]
                    {:id         id
                     :text       text
                     :open       open
                     :subfolders (or subfolders [])})
   :ident         :id}
  (dom/div (str "FOLDER: " id)))

(def ui-folder (comp/factory Folder))

(defsc RootFolder [this {:keys [id text subfolders]}]
  {:query         (fn [] [:id :text {:subfolders (comp/get-query Folder)}])
   :initial-state (fn [_] {:id         "main-tree-root"
                           :text       "Home"
                           :subfolders [(comp/get-initial-state Folder
                                          {:id   "01"
                                           :text "folder01"
                                           :open true})
                                        (comp/get-initial-state Folder
                                          {:id         "00"
                                           :text       "folder00"
                                           :open       true
                                           :subfolders [[:id "01"]]})]})
   :ident         :id}
  (dom/div
    (dom/p "Root Folder id: " id)
    (when subfolders
      (map ui-folder subfolders))))

(def ui-root-folder (comp/factory RootFolder))

(defsc Side [this {:side/keys [id list]}]
  {:query         (fn [] [:side/id {:side/list (comp/get-query Folder)}])
   :initial-state (fn [_] {:side/id   "side-tree-root"
                           :side/list [[:id "01"] [:id "00"]]})
   :ident         :side/id}
  (dom/div
    (dom/p "Side id:" id)
    (when list
      (map ui-folder list))))

(def ui-side (comp/factory Side))

(defsc ZeitsteinUI [this {:keys [main-tree side-tree]}]
  {:query         (fn [] [[:id '_]
                          {:main-tree (comp/get-query RootFolder)}
                          {:side-tree (comp/get-query Side)}])
   :route-segment ["zeitstein"]
   :ident         (fn [] [:component/id :ZeitsteinUI])
   :initial-state (fn [_] {:main-tree (comp/get-initial-state RootFolder)
                           :side-tree (comp/get-initial-state Side)})}
  (div
    (dom/h4 "UI ROOT")
    (ui-root-folder main-tree)
    (dom/h4 "UI SIDE")
    (ui-side side-tree)))
(def ui-example-ui (comp/factory ZeitsteinUI))

xceno15:12:57

I changed your queries a bit and put your example state into the initial state of the components

xceno15:12:02

This results in the following DB:

xceno15:12:46

You should be able to play around with this a bit and figure things out from there. But I try to get your subfolder to display. I see where you're stuck now

❤️ 1
Jakub Holý (HolyJak)15:12:51

BTW when you have a problem, ignore the UI and focus on the query and data first. Only when that works as expected, look at / troubleshoot rendering. Less stuff => simpler. See the https://blog.jakubholy.net/2020/troubleshooting-fulcro/

zeitstein15:12:02

Thanks for your hard work, @U012ADU90SW. But now even the side tree doesn't render as wanted 🙂 Though, at least it's consistent? 🙂 I did go through the Troubleshooting guide, @U0522TWDA 🙂 It seems to me that the query and the data work well, it's rendering I'm confused about.

xceno16:12:52

> But now even the side tree doesn't render as wanted I just noticed yeah, gonna fix that up real quick, sorry!

clojure-spin 1
Jakub Holý (HolyJak)16:12:24

So db->tree returns all the data it expect but it isn't rendered? If a component gets in its props :some-subtree and rendered (UI-tree some-subtree) then the data should show up...

xceno16:12:19

Does that look correct? I forgot to actually render the subfolders xD

zeitstein16:12:11

Nope, still have the same problem. I saw the first version of your code and you had {:subfolders 5} in Folder's query (I think. EDIT: now it appears as 5 again... Slack seems to be acting up). I had said my example worked when I set the recursion limit manually. Setting 5 to '... in your example produces the image below.

zeitstein16:12:05

Also, notice that <ul> gets rendered to the DOM but it's empty at the place one would expect FOLDER: 01 to appear – suggesting that subfolders is not nil, but doesn't get rendered?

xceno16:12:39

yeah subfolders is an empty vector in that case

👍 1
xceno16:12:20

I embedded this inside a view in my current app. let me pull it up into a separate view, so I can make sure we execute the same code

zeitstein16:12:07

You don't get the same problem with ... instead of 5?

xceno16:12:02

I do! Just made that edit. So that's the actual problem here. Because we're going in circles indefinitely: > react_devtools_backend.js:2540 WARN [com.fulcrologic.fulcro.algorithms.denormalize:124] - Loop detected in data graph at {:id "01", :text "folder01", :open true, :subfolders []} . Recursive query stopped.

xceno16:12:22

I just remember that I had this problem last week with my tree as well. Sorry that I didn't remember that straight away

❤️ 1
xceno16:12:23

now the question is why it loops on an empty vector of subfolders, if I interpret this data correctly

zeitstein16:12:00

It seems that @U0522TWDA was right – data is not denormalized as expected. I'll take a look at denormalize.cljc. I wonder if it's time to ping Tony? 🙂

xceno17:12:21

yeah just ping him 😉 It gotta has something to do with the fact, that the sidebar has it's own "table" whereas the "main-tree-root" lives in the :id table. I've written a mutation an can push the craziest recursive folder structures into the DB - They render well in the sidebar, but not in your RootFolder. I think I never came across this particular issue in my app because I always cycle between nodes and edges in my recursive query EDIT: > They render well in the sidebar Scratch that as well. At certain points with infinite recursion ('...) it'll stop as well eventually

zeitstein17:12:07

Right. Thanks immensely for your kind help, @U012ADU90SW! You really went above and beyond ❤️

xceno17:12:59

At least we tried huh! I hope we can still figure it out. Might be worth it to fork the fulcro-rad-demo and put the example in there so others can have a shot at it without reading this entire thread

❤️ 1
zeitstein17:12:15

Hey @U0CKQ19AQ, could you take a peek here? It's a long thread, but you can start with my first couple of posts. xceno has provided a https://clojurians.slack.com/archives/C68M60S4F/p1639757601160300?thread_ts=1639752543.151800&amp;cid=C68M60S4F as well (just let the recursion be unlimited in Folder). We've determined that the main tree https://clojurians.slack.com/archives/C68M60S4F/p1639758962162300?thread_ts=1639752543.151800&amp;cid=C68M60S4F (precisely folder 01 is missing from folder 00, as in the UI). The side tree does. denormalize warns:

WARN [com.fulcrologic.fulcro.algorithms.denormalize:124] - Loop detected in data graph at  {:id "01", :text "folder01", :open true} . Recursive query stopped.

zeitstein17:12:13

> Might be worth it to fork the `fulcro-rad-demo` and put the example in there so others can have a shot at it without reading this entire thread I've no experience with RAD 😕

tony.kay18:12:15

I can say the following: 1. I'm way too busy for this deep of a convo right not 😄 2. Recursive queries are very well tested 3. The book talks about using them in detail If you feel you have a bug and can reproduce it, you need to make a complete gist. NOTE: merge-component! is a great way to take raw data (a tree) and normalize it into the db just like a load would, if you need that. You can also directly use db->tree to see that the props come out correctly. That would isolate the problem to the db logic if there is one; otherwise you're doing the UI incorrectly.

zeitstein21:12:20

Thanks, Tony, completely understandable. I've created https://gist.github.com/zeitstein/21257931fea29c1d0e737705a02f1bb2*. Notes: 1. Indeed, the problem is also present in db->tree. 2. Problem only present for infinitely recursive queries. 3. There is a 'fix' in the gist, perhaps pointing to a solution. *To be clear, what I expect to see in the UI or db->tree is:

Home
- Folder 1
- Folder 0
- - Folder 1     (<- this is missing from denormalization)
Hopefully that helps.

Jakub Holý (HolyJak)08:12:49

The gist is very helpful. It would be also nice if it described what you expect vs. what you see, i.e. what you posted above + the root folder (since you also expect it in the ui).

❤️ 1
zeitstein10:12:57

Thanks for looking it over and suggestions. Edited the gist.

xceno11:12:20

Hey guys, thanks for debugging this. I thought I was just too tired last friday and missed a mistake or misunderstood the data model somehow. This behavior might also explain some stuff that happened to me months ago. But back then I thought I did something wrong, because my graph data became pretty complex, and just reworked that part of the UI.

zeitstein11:12:10

Awesome, @U0522TWDA! Thank you so much <3 > BTW have you read https://book.fulcrologic.com/#warn-denormalize-loop-detected ? No, thanks. So, that's why using a manual recursion works. Then, should I just... use a very large explicit limit? :) So, can you help me understand these denormalization and recursive queries more, since recursive queries are fundamental to my UI. Denormalization doesn't detect infinite loops, but stops when it has seen an ident before (for ...)?

zeitstein11:12:47

What happens if the :Leaf1 in your issue, Jakub, also has children? Your proposed solution would still make denormalization "detect a loop"?

Jakub Holý (HolyJak)11:12:57

You can do what I did, namely read the algorithm (which I think you already have) and play with code and debugging logs 🙂 I have never used recursive queries and have not looked at the denormalize code before today. > use a very large explicit limit Sounds as a good workaround. > Denormalization doesn't detect infinite loops, but stops when it has seen an ident before Yes. The problem is it does not distinguish between leafs (where seeing it again does not mean a problem) and non-leafs. If Leaf also has children then you are right, F. will still suspect a loop. Good catch!

zeitstein11:12:04

> You can do what I did, namely read the algorithm (which I think you already have) and play with code and debugging logs I tried doing that, but was overwhelming for a newbie 🙂 > If `Leaf` also has children then you are right, F. will still suspect a loop. Good catch! Right, so it boils down to "if this ident will result in a infinite loop, stop". Of course, easier said than done. I wouldn't mind helping with this. But I wonder whether Tony had a reason for not doing it that way in the first place?

xceno11:12:19

I mean, it complicates the logic to check for "special" infinite recursions like in your example (and thinking about it, my use case is also pretty "out there"). And when you think about, do you really want to render possibly infinite DOM Elements, without any kind of paging, lazy loading and whatnot? So maybe, specifying a sane recursion limit and lazy loading stuff at certain points is actually the right way to go

👍 2
xceno11:12:21

I don't know how big your data pool is of course, but with infinite recursive queries you're of course loading all that stuff into your UI DB, if you display it or not

zeitstein11:12:47

> "if this ident will result in a infinite loop, stop" First thought: a stopping point should be when an ident repeats in the same branch. Examples:

Root
- Folder 0
- - Folder 1
- Root (<- inf loop)

Root
- Folder 0
- - Folder 1
- - - Folder 0 (<- inf loop)

zeitstein11:12:34

> So maybe, specifying a sane recursion limit and lazy loading stuff at certain points is actually the right way to go > The big question is whether it is worth doing something about it. How common is the situation? And there is a feasible workaround: Set (a large) limit on the recursion depth. Fair points 🙂

Jakub Holý (HolyJak)11:12:19

Yes, checking for repetition in the current branch (of the depth-first query processing) would be "optimal" but I am not sure how much work it would require in practice and how important it actually is, given there is already a simple workaround by setting an explicit limit. As xceno points out, you hardly want to display infinite amount of data 🙂 I suspect Tony would be hesitant to touch a crucial algorithm that has worked just fine for years and on many project to satisfy a corner case.

👍 1
zeitstein11:12:33

Next question would be: are there performance penalties to a setting an explicit limit on recursion depth? 🙂 I mean, if the https://github.com/fulcrologic/fulcro/blob/52d8d8aea63bea26304ee65d8004a0add4273b45/src/main/com/fulcrologic/fulcro/algorithms/denormalize.cljc#L91?

xceno11:12:08

The performance degrades as your limit increases. But the real performance problems start when rendering this stuff. I'd suggest you use tufte or one of the other gazillion profiles and the react profiler and start measuring different scenarios for your use case. Also: As you increase your recursion limit you might also run into problems with pathom timeouts Side note: Keep in mind that you have roughly 12-16 ms of time "per render loop" to finish whatever you're doing and put pixels on the screen, so that your app still feels snappy. A bunch of those ms already go to stuff outside your direct control.

zeitstein12:12:09

Thanks, xceno. Though, I've probably started worrying about performance way too early 🙂 One lingering niggle I have: an explicit limit causes a loading-rendering mismatch. Consider

Root
- Folder 1
- - Folder 2
- Folder 0
- - Folder 1
- - - Folder 2  (<-- will not render)
Setting the limit to 1 will load all the data, but won't render all. (Hopefully you follow this quick example?)

zeitstein12:12:54

Also, @U0522TWDA, does the issue you've opened address the fact that when I change the root's ident, things render as they should (https://gist.github.com/zeitstein/21257931fea29c1d0e737705a02f1bb2)? I don't understand why that would be a factor here?

xceno12:12:23

> Setting the limit to `1` will load all the data, but won't render all I'm currently working on a tree view similar to yours for some graph data, and I just started to play around with lazy loading /rendering stuff X levels deep. So I don't have an answer yet for you, but I think I'll figure it out soon and share my findings with you. My initial feeling is to use dynamic queries. With those I'm able to say "Load/render X levels recursively starting at Y"

zeitstein12:12:10

Would love to collaborate on that, xceno. Let's continue that discussion in DM, though.

👍 1
Jakub Holý (HolyJak)12:12:33

> when I change the root's ident, things render as they should that is because the two components that share the folder1 child in this case are 2 different entities (`:root/id` vs. :folder/id , if you allow me to add namespace also to the latter). When doing recursive query on folders then folder1 is not repeated, it only appears inside one other folder. Its occurrence inside RootFolder does not matter as that is in a different "table".

gratitude 1