Fork me on GitHub

I'm thinking about showing a table with potentially thousands of rows as a tree of group by aggregates. Each level would have folding, so that at first you would see totals for A, B and C. Then expand A to see A1 A2 A3 and so on. Easy enough to do so that the frontend would load all the data and then just render some of it. But how would it work to load it on demand, so that if A has {open? false} it's query will not have the children, but if it is open it will, and so on, recursively? Dynamic queries could kinda work, I guess, but sounds hairy if the open? flag could be changed from many places. Then again, I would like the grouping to be dynamic too (so the user can choose order of groups), so I don't even know how to id the "rows". I guess I'll start from loading all the data and doing the aggregation outside the app db

Jakub Holý (HolyJak)11:02:00

I cannot wrap my had about the rows vs tree. What are you loading? All the raw rows or just the A, B, C aggregates? Who does the aggregation, backend or frontend? Is the data a tree of same or different entities = components? If same, use recursive query with a limit (starting at 0, increasing as they drill in, plus perhaps params to communicate the subtree you want). If heterogenous then each level has a separate component and you can load! with :without and load-field! to exclude / load data.


So basically rows with data like manufacturer, type, item, cost (way more, but this should be enough). Instead of just showing every item, the users could see it as a tree of different types under manufacturers, or manufacturers under types (and then items in the last level). The higher levels would be true aggregates, not just groups, so it would show the whole cost of everything under the type, for example. It's easy enough to load everything to the frontend and aggregate there, but I'm wondering how to make it work on the backend instead, and components loading deeper levels when they are shown.


Hmh. Maybe a statemachine and loads are the way to go, not trying to play with queries

Jakub Holý (HolyJak)14:02:49

It might be useful to think in practical terms. What would the resolvers look like? What would the data look like in the client DB? You would need some kind of composed IDs to distinguish manufactures by type from manufactures by some other attribute. Not thinking about the UI and queries at all, just thinking about what the data tree from Pathom would look like and what would the data in the db look like. I think I can easily imagine the resolver and data tree but am unsure about how to store the different inner nodes of the different trees in the client db. Or perhaps have the raw data (in the Product "table") and single one table for Aggregations where the ID encodes the path (eg. [:aggreg/id [:manufacturer "Skoda" :type :car ...]] where an aggregation has either :sub-aggregations [agg idents] or :row-data [product idents] ?


FYI: There was a similar discussion some weeks ago which revolved around displaying multiple layers with many (possibly recursive) items, which ultimately lead to this: Maybe it helps

Jakub Holý (HolyJak)18:02:06

@U8ZQ1J1RR really interesting problem. Let me know what solution you arrive at! IMO you have multiple ways to model this in the client DB. Do you want each node of the tree normalized, i.e. have a dedicated "table" for each level of the aggregation tree? Since normalization is primarily used to make changing data simpler and reflecting the change everywhere, IMO you don't need it b/c you are not going to edit the hierarchies. If you want them normalized, you could e.g. have a single aggregation/id table, for each node's ident I would then use a vector to represent the path from the root as shown above, i.e.: [:manufacturer "Skoda" :type :car ...] If you don't care about normalization, you could have in the root a single, a denormalized property :hierarchies which holds the tree of all the possible aggregations. You would have a single, global, parametrized resolver where the param is a vector with the path of the node to load and would use a post-mutation, pre-merge or something to make sure that the data is (assoc-in client-db [:hierarchies <the path>]) . Not 100% sure it would work but I believe it would.


I was thinking that I would have at least the leaves normalized, since those would correspond to the actual rows, and having them in the db in that format would lessen the data use between the front and back when changing aggregation. Probably the aggregated levels would also be normalized, so that changing to another aggregation and coming back would not cause a reload of the data, though a different level of caching would work for that too.


I think I will go with aggregation in the browser for now. Our data amount is in thousands of rows right now, but handling millions of rows for internal analysis is waiting for me to make it work. But it doesn't need to be performant as long as it's internal and we know that we have the RAM and bandwith for it. Have to revisit it during the spring. But who knows, maybe the internal tool will show that the whole approach is untenable from UX point of view, and I can just remove the whole table…


@U012ADU90SW I'm reading that code, what is the 3 doing in

{:query [:ui/open?
           {:node/children 3}]


It's a recursion limit, so fulcro will return X levels deep of node/children


Is there a way to place the results of load-field! in a different location, other than by using the :post-mutation config option? I want to have a single parameterized resolver that provides data generically - say, ::chart-data. Since the query for the resolver always uses the same key, multiple calls to it override previous results. Using :target doesn't work because load-field! loads by ident, and custom targeting just adds entity idents at the target path instead of the result data. > When loading an entity (by ident), then this option will place additional idents at the target path(s) that point to that entity.

Jakub Holý (HolyJak)14:02:40

It would surprise me. By nature it should load the field of the current entity, into that entity. If you could have the data coming back be an entity with ident such as [:chart/id [123 :type :bar]] then you could store them separately. But you would have problem if you want allow the user to switch to :type :xy-plot ...


Do you think it's misuse of the model to "move" such results using a post-action? I suppose the easy answer here is to just use different resolvers instead of a generic one that supports calls like [(::chart-data {:grouping :weekly})].

Jakub Holý (HolyJak)15:02:43

If it works and fulfils you need it is not a misuse 😅 I am really not sure what is optimal / good enough. If you find out what works fine, let me know 🙏 🙂


why not just load it then? Why load-field?


you wanting to narrow the query...just read the source of load-field...make your own little customization for ease-of-use