This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2021-12-17
Channels
- # adventofcode (25)
- # announcements (2)
- # babashka (16)
- # babashka-sci-dev (16)
- # beginners (213)
- # calva (15)
- # clj-kondo (126)
- # clj-on-windows (1)
- # cljdoc (5)
- # cljfx (1)
- # cljs-dev (6)
- # clojure (230)
- # clojure-europe (38)
- # clojure-nl (3)
- # clojure-uk (3)
- # conjure (10)
- # core-async (15)
- # cursive (33)
- # fulcro (58)
- # hyperfiddle (4)
- # jobs-discuss (1)
- # kaocha (5)
- # lsp (46)
- # meander (3)
- # off-topic (30)
- # polylith (10)
- # portal (9)
- # re-frame (5)
- # reitit (7)
- # releases (2)
- # ring (17)
- # sci (8)
- # shadow-cljs (6)
- # specter (1)
- # sql (1)
- # testing (9)
- # tools-deps (4)
- # vim (12)
Need some help with recursive queries. I want to render the following tree:
- Folder 1
- Folder 0
- - Folder 1
But the UI only renders:
- Folder 1
- Folder 0
I'll put more details in the thread.I want to render the same data in two places. I have a main tree and a side tree. The side tree actually renders as expected, but main does not. I am using the default renderer (not ident optimized). Here are the components:
(defsc Folder [this {:keys [id text open subfolders]}]
{:query (fn [] [:id :text :open {:subfolders '...}])
:ident :id}
...)
(defsc RootFolder [this {:keys [id text subfolders]}]
{:query [:id :text {:subfolders (comp/get-query Folder)}]
:ident :id}
(div
(when subfolders
(map ui-folder subfolders))))
(defsc Side [this {:side/keys [id list]}]
{:query [:side/id {:side/list (comp/get-query Folder)}]
:id :side/id}
(div
(when list
(map ui-folder list))))
(defsc UI [this {main-tree :main-tree side-tree :side-tree}]
{:query [{:main-tree (comp/get-query RootFolder)}
{:side-tree (comp/get-query Side)}]}
(div
(ui-root-folder main-tree)
(ui-side side-tree)))
The idea is that RootFolder
will become a target for routing and is rendered differently from Folder
. The only significant difference between main and side trees I see is that RootFolder
has the same ident as Folder
and uses the same :id
s? Can't figure it out. INFO
in console:component app.ui/RootFolder's ident ([:id nil]) has a `nil` second element. This warning can be safely ignored if that is intended. Props were nil
main-tree
and side-tree
):
{:main-tree [:id "main-tree-root"]
:side-tree {:side/id "side-tree-root"
:side/list [[:id "01"] [:id "00"]]}
:id {"01" {:id "01"
:text "folder01"
:open true}
"00" {:id "00"
:text "folder00"
:open true
:subfolders [[:id "01"]]}
"main-tree-root" {:id "main-tree-root"
:text "Home"
:subfolders [[:id "01"] [:id "00"]]}}}
Clearly, I could handle main as I do side, but I need to understand why this doesn't work before introducing complications (routing, incremental loading, etc.)I've also noticed manually setting recursion depth in Folder
works. But I don't understand why – there are no infinite loops here?
I understand that the denormalize
algorithm detects a loop at folder01 (and warns about it).
I've recently implemented a UI similar to yours. It's displaying a tree view made up of certain nodes and edges of my graph. I noticed that pathom doesn't like infinite recursion because you can run into cycles of your graph easily. In that case it'll return an empty result (or crash, I don't remember right now). Fulcro's denormalize algorithm handles this gracefully as you've noticed.
Your [:id nil]
property might come from your initial props of the root folder.
Do I understand you correctly that you're basically building a master/detail layout and die Side
is your sidebar whereas the RootFolder
is your "master" view and will include a router?
Hm, still working with a small data set, so haven't come across that about Pathom. There should be ways to circumvent Pathom there, in my case, anyway.
> Your [:id nil]
property might come from your initial props of the root folder.
Okay, added a conditional, so that warning goes away. The main issue remains, so not connected.
> Do I understand you correctly that you're basically building a master/detail layout and die `Side` is your sidebar whereas the `RootFolder` is your "master" view and will include a router?
Yes, exactly. Though, I might end up wanting to have two separate trees with same behaviour side-by-side, independent routing, etc.
I just copied your example code and look at it. One thing though:
I think the open
flag is ui only(?), so you should prefix it with :ui
. This will also exclude it from queries to the backend. In general I'd recommend to namespace your keys, even if you probably couldn't care less right now 😉
Oh and AFAIK you still need the lamda version for :query
if you're calling functions like comp/get-query
> I think the `open` flag is ui only(?) Nope 🙂 > In general I'd recommend to namespace your keys, even if you probably couldn't care less right now Ah, this is due to a limitation of Asami, so I have a different convention, which I will not bore you with. > AFAIK you still need the lamda version for `:query` if you're calling functions like `comp/get-query` Thanks! I've tried with that as well, of course. Though, I think the non-lambda version works as well. I did notice that the lambda version is required for recursive queries on routing targets.
(defsc Folder [this {:keys [id text open subfolders]}]
{:query (fn [] [:id :text :open {:subfolders '...}])
:initial-state (fn [{:keys [id text open subfolders]}]
{:id id
:text text
:open open
:subfolders (or subfolders [])})
:ident :id}
(dom/div (str "FOLDER: " id)))
(def ui-folder (comp/factory Folder))
(defsc RootFolder [this {:keys [id text subfolders]}]
{:query (fn [] [:id :text {:subfolders (comp/get-query Folder)}])
:initial-state (fn [_] {:id "main-tree-root"
:text "Home"
:subfolders [(comp/get-initial-state Folder
{:id "01"
:text "folder01"
:open true})
(comp/get-initial-state Folder
{:id "00"
:text "folder00"
:open true
:subfolders [[:id "01"]]})]})
:ident :id}
(dom/div
(dom/p "Root Folder id: " id)
(when subfolders
(map ui-folder subfolders))))
(def ui-root-folder (comp/factory RootFolder))
(defsc Side [this {:side/keys [id list]}]
{:query (fn [] [:side/id {:side/list (comp/get-query Folder)}])
:initial-state (fn [_] {:side/id "side-tree-root"
:side/list [[:id "01"] [:id "00"]]})
:ident :side/id}
(dom/div
(dom/p "Side id:" id)
(when list
(map ui-folder list))))
(def ui-side (comp/factory Side))
(defsc ZeitsteinUI [this {:keys [main-tree side-tree]}]
{:query (fn [] [[:id '_]
{:main-tree (comp/get-query RootFolder)}
{:side-tree (comp/get-query Side)}])
:route-segment ["zeitstein"]
:ident (fn [] [:component/id :ZeitsteinUI])
:initial-state (fn [_] {:main-tree (comp/get-initial-state RootFolder)
:side-tree (comp/get-initial-state Side)})}
(div
(dom/h4 "UI ROOT")
(ui-root-folder main-tree)
(dom/h4 "UI SIDE")
(ui-side side-tree)))
(def ui-example-ui (comp/factory ZeitsteinUI))
I changed your queries a bit and put your example state into the initial state of the components
You should be able to play around with this a bit and figure things out from there. But I try to get your subfolder to display. I see where you're stuck now
BTW when you have a problem, ignore the UI and focus on the query and data first. Only when that works as expected, look at / troubleshoot rendering. Less stuff => simpler. See the https://blog.jakubholy.net/2020/troubleshooting-fulcro/
Thanks for your hard work, @U012ADU90SW. But now even the side tree doesn't render as wanted 🙂 Though, at least it's consistent? 🙂 I did go through the Troubleshooting guide, @U0522TWDA 🙂 It seems to me that the query and the data work well, it's rendering I'm confused about.
> But now even the side tree doesn't render as wanted I just noticed yeah, gonna fix that up real quick, sorry!
So db->tree returns all the data it expect but it isn't rendered? If a component gets in its props :some-subtree and rendered (UI-tree some-subtree) then the data should show up...
Nope, still have the same problem. I saw the first version of your code and you had {:subfolders 5}
in Folder
's query (I think. EDIT: now it appears as 5
again... Slack seems to be acting up). I had said my example worked when I set the recursion limit manually. Setting 5
to '...
in your example produces the image below.
Also, notice that <ul>
gets rendered to the DOM but it's empty at the place one would expect FOLDER: 01
to appear – suggesting that subfolders
is not nil
, but doesn't get rendered?
I embedded this inside a view in my current app. let me pull it up into a separate view, so I can make sure we execute the same code
I do! Just made that edit. So that's the actual problem here. Because we're going in circles indefinitely: > react_devtools_backend.js:2540 WARN [com.fulcrologic.fulcro.algorithms.denormalize:124] - Loop detected in data graph at {:id "01", :text "folder01", :open true, :subfolders []} . Recursive query stopped.
I just remember that I had this problem last week with my tree as well. Sorry that I didn't remember that straight away
And here's the code that stops it: https://github.com/fulcrologic/fulcro/blob/develop/src/main/com/fulcrologic/fulcro/algorithms/denormalize.cljc#L117
now the question is why it loops on an empty vector of subfolders, if I interpret this data correctly
It seems that @U0522TWDA was right – data is not denormalized as expected. I'll take a look at denormalize.cljc
. I wonder if it's time to ping Tony? 🙂
yeah just ping him 😉
It gotta has something to do with the fact, that the sidebar has it's own "table" whereas the "main-tree-root" lives in the :id
table.
I've written a mutation an can push the craziest recursive folder structures into the DB - They render well in the sidebar, but not in your RootFolder
.
I think I never came across this particular issue in my app because I always cycle between nodes
and edges
in my recursive query
EDIT:
> They render well in the sidebar
Scratch that as well. At certain points with infinite recursion ('...) it'll stop as well eventually
Right. Thanks immensely for your kind help, @U012ADU90SW! You really went above and beyond ❤️
At least we tried huh! I hope we can still figure it out. Might be worth it to fork the fulcro-rad-demo
and put the example in there so others can have a shot at it without reading this entire thread
Hey @U0CKQ19AQ, could you take a peek here? It's a long thread, but you can start with my first couple of posts. xceno has provided a https://clojurians.slack.com/archives/C68M60S4F/p1639757601160300?thread_ts=1639752543.151800&cid=C68M60S4F as well (just let the recursion be unlimited in Folder
).
We've determined that the main tree https://clojurians.slack.com/archives/C68M60S4F/p1639758962162300?thread_ts=1639752543.151800&cid=C68M60S4F (precisely folder 01 is missing from folder 00, as in the UI). The side tree does. denormalize
warns:
WARN [com.fulcrologic.fulcro.algorithms.denormalize:124] - Loop detected in data graph at {:id "01", :text "folder01", :open true} . Recursive query stopped.
> Might be worth it to fork the `fulcro-rad-demo` and put the example in there so others can have a shot at it without reading this entire thread I've no experience with RAD 😕
I can say the following:
1. I'm way too busy for this deep of a convo right not 😄
2. Recursive queries are very well tested
3. The book talks about using them in detail
If you feel you have a bug and can reproduce it, you need to make a complete gist. NOTE: merge-component!
is a great way to take raw data (a tree) and normalize it into the db just like a load would, if you need that. You can also directly use db->tree
to see that the props come out correctly. That would isolate the problem to the db logic if there is one; otherwise you're doing the UI incorrectly.
Thanks, Tony, completely understandable. I've created https://gist.github.com/zeitstein/21257931fea29c1d0e737705a02f1bb2*. Notes:
1. Indeed, the problem is also present in db->tree
.
2. Problem only present for infinitely recursive queries.
3. There is a 'fix' in the gist, perhaps pointing to a solution.
*To be clear, what I expect to see in the UI or db->tree
is:
Home
- Folder 1
- Folder 0
- - Folder 1 (<- this is missing from denormalization)
Hopefully that helps.The gist is very helpful. It would be also nice if it described what you expect vs. what you see, i.e. what you posted above + the root folder (since you also expect it in the ui).
BTW have you read https://book.fulcrologic.com/#warn-denormalize-loop-detected ?
Hey guys, thanks for debugging this. I thought I was just too tired last friday and missed a mistake or misunderstood the data model somehow. This behavior might also explain some stuff that happened to me months ago. But back then I thought I did something wrong, because my graph data became pretty complex, and just reworked that part of the UI.
Awesome, @U0522TWDA! Thank you so much <3
> BTW have you read https://book.fulcrologic.com/#warn-denormalize-loop-detected ?
No, thanks. So, that's why using a manual recursion works. Then, should I just... use a very large explicit limit? :)
So, can you help me understand these denormalization and recursive queries more, since recursive queries are fundamental to my UI. Denormalization doesn't detect infinite loops, but stops when it has seen an ident before (for ...
)?
What happens if the :Leaf1
in your issue, Jakub, also has children? Your proposed solution would still make denormalization "detect a loop"?
You can do what I did, namely read the algorithm (which I think you already have) and play with code and debugging logs 🙂 I have never used recursive queries and have not looked at the denormalize code before today.
> use a very large explicit limit
Sounds as a good workaround.
> Denormalization doesn't detect infinite loops, but stops when it has seen an ident before
Yes. The problem is it does not distinguish between leafs (where seeing it again does not mean a problem) and non-leafs.
If Leaf
also has children then you are right, F. will still suspect a loop. Good catch!
> You can do what I did, namely read the algorithm (which I think you already have) and play with code and debugging logs I tried doing that, but was overwhelming for a newbie 🙂 > If `Leaf` also has children then you are right, F. will still suspect a loop. Good catch! Right, so it boils down to "if this ident will result in a infinite loop, stop". Of course, easier said than done. I wouldn't mind helping with this. But I wonder whether Tony had a reason for not doing it that way in the first place?
I mean, it complicates the logic to check for "special" infinite recursions like in your example (and thinking about it, my use case is also pretty "out there"). And when you think about, do you really want to render possibly infinite DOM Elements, without any kind of paging, lazy loading and whatnot? So maybe, specifying a sane recursion limit and lazy loading stuff at certain points is actually the right way to go
I don't know how big your data pool is of course, but with infinite recursive queries you're of course loading all that stuff into your UI DB, if you display it or not
> "if this ident will result in a infinite loop, stop" First thought: a stopping point should be when an ident repeats in the same branch. Examples:
Root
- Folder 0
- - Folder 1
- Root (<- inf loop)
Root
- Folder 0
- - Folder 1
- - - Folder 0 (<- inf loop)
> So maybe, specifying a sane recursion limit and lazy loading stuff at certain points is actually the right way to go > The big question is whether it is worth doing something about it. How common is the situation? And there is a feasible workaround: Set (a large) limit on the recursion depth. Fair points 🙂
Yes, checking for repetition in the current branch (of the depth-first query processing) would be "optimal" but I am not sure how much work it would require in practice and how important it actually is, given there is already a simple workaround by setting an explicit limit. As xceno points out, you hardly want to display infinite amount of data 🙂 I suspect Tony would be hesitant to touch a crucial algorithm that has worked just fine for years and on many project to satisfy a corner case.
Next question would be: are there performance penalties to a setting an explicit limit on recursion depth? 🙂 I mean, if the https://github.com/fulcrologic/fulcro/blob/52d8d8aea63bea26304ee65d8004a0add4273b45/src/main/com/fulcrologic/fulcro/algorithms/denormalize.cljc#L91?
The performance degrades as your limit increases. But the real performance problems start when rendering this stuff. I'd suggest you use tufte
or one of the other gazillion profiles and the react profiler
and start measuring different scenarios for your use case.
Also: As you increase your recursion limit you might also run into problems with pathom timeouts
Side note:
Keep in mind that you have roughly 12-16 ms of time "per render loop" to finish whatever you're doing and put pixels on the screen, so that your app still feels snappy. A bunch of those ms already go to stuff outside your direct control.
Thanks, xceno. Though, I've probably started worrying about performance way too early 🙂 One lingering niggle I have: an explicit limit causes a loading-rendering mismatch. Consider
Root
- Folder 1
- - Folder 2
- Folder 0
- - Folder 1
- - - Folder 2 (<-- will not render)
Setting the limit to 1
will load all the data, but won't render all. (Hopefully you follow this quick example?)Also, @U0522TWDA, does the issue you've opened address the fact that when I change the root's ident, things render as they should (https://gist.github.com/zeitstein/21257931fea29c1d0e737705a02f1bb2)? I don't understand why that would be a factor here?
> Setting the limit to `1` will load all the data, but won't render all I'm currently working on a tree view similar to yours for some graph data, and I just started to play around with lazy loading /rendering stuff X levels deep. So I don't have an answer yet for you, but I think I'll figure it out soon and share my findings with you. My initial feeling is to use dynamic queries. With those I'm able to say "Load/render X levels recursively starting at Y"
Would love to collaborate on that, xceno. Let's continue that discussion in DM, though.
> when I change the root's ident, things render as they should
that is because the two components that share the folder1 child in this case are 2 different entities (`:root/id` vs. :folder/id
, if you allow me to add namespace also to the latter). When doing recursive query on folders then folder1 is not repeated, it only appears inside one other folder. Its occurrence inside RootFolder does not matter as that is in a different "table".