I cannot wrap my had about the rows vs tree. What are you loading? All the raw rows or just the A, B, C aggregates? Who does the aggregation, backend or frontend? Is the data a tree of same or different entities = components? If same, use recursive query with a limit (starting at 0, increasing as they drill in, plus perhaps params to communicate the subtree you want). If heterogenous then each level has a separate component and you can load! with :without and load-field! to exclude / load data.

So basically rows with data like manufacturer, type, item, cost (way more, but this should be enough). Instead of just showing every item, the users could see it as a tree of different types under manufacturers, or manufacturers under types (and then items in the last level). The higher levels would be true aggregates, not just groups, so it would show the whole cost of everything under the type, for example. It's easy enough to load everything to the frontend and aggregate there, but I'm wondering how to make it work on the backend instead, and components loading deeper levels when they are shown.

Hmh. Maybe a statemachine and loads are the way to go, not trying to play with queries

It might be useful to think in practical terms. What would the resolvers look like? What would the data look like in the client DB? You would need some kind of composed IDs to distinguish manufactures by type from manufactures by some other attribute. Not thinking about the UI and queries at all, just thinking about what the data tree from Pathom would look like and what would the data in the db look like. I think I can easily imagine the resolver and data tree but am unsure about how to store the different inner nodes of the different trees in the client db. Or perhaps have the raw data (in the Product "table") and single one table for Aggregations where the ID encodes the path (eg. [:aggreg/id [:manufacturer "Skoda" :type :car ...]] where an aggregation has either :sub-aggregations [agg idents] or :row-data [product idents] ?

FYI: There was a similar discussion some weeks ago which revolved around displaying multiple layers with many (possibly recursive) items, which ultimately lead to this: https://github.com/fulcrologic/deep-tree-optimization-demo Maybe it helps

@U8ZQ1J1RR really interesting problem. Let me know what solution you arrive at! IMO you have multiple ways to model this in the client DB. Do you want each node of the tree normalized, i.e. have a dedicated "table" for each level of the aggregation tree? Since normalization is primarily used to make changing data simpler and reflecting the change everywhere, IMO you don't need it b/c you are not going to edit the hierarchies. If you want them normalized, you could e.g. have a single aggregation/id table, for each node's ident I would then use a vector to represent the path from the root as shown above, i.e.: [:manufacturer "Skoda" :type :car ...] If you don't care about normalization, you could have in the root a single, a denormalized property :hierarchies which holds the tree of all the possible aggregations. You would have a single, global, parametrized resolver where the param is a vector with the path of the node to load and would use a post-mutation, pre-merge or something to make sure that the data is (assoc-in client-db [:hierarchies <the path>]) . Not 100% sure it would work but I believe it would.

I was thinking that I would have at least the leaves normalized, since those would correspond to the actual rows, and having them in the db in that format would lessen the data use between the front and back when changing aggregation. Probably the aggregated levels would also be normalized, so that changing to another aggregation and coming back would not cause a reload of the data, though a different level of caching would work for that too.

I think I will go with aggregation in the browser for now. Our data amount is in thousands of rows right now, but handling millions of rows for internal analysis is waiting for me to make it work. But it doesn't need to be performant as long as it's internal and we know that we have the RAM and bandwith for it. Have to revisit it during the spring. But who knows, maybe the internal tool will show that the whole approach is untenable from UX point of view, and I can just remove the whole table…

@U012ADU90SW I'm reading that code, what is the 3 doing in

{:query [:ui/open?
           :node/id
           :node/content
           {:node/children 3}]

It's a recursion limit, so fulcro will return X levels deep of node/children

It would surprise me. By nature it should load the field of the current entity, into that entity. If you could have the data coming back be an entity with ident such as [:chart/id [123 :type :bar]] then you could store them separately. But you would have problem if you want allow the user to switch to :type :xy-plot ...

Do you think it's misuse of the model to "move" such results using a post-action? I suppose the easy answer here is to just use different resolvers instead of a generic one that supports calls like [(::chart-data {:grouping :weekly})].

If it works and fulfils you need it is not a misuse 😅 I am really not sure what is optimal / good enough. If you find out what works fine, let me know 🙏 🙂

why not just load it then? Why load-field?

you wanting to narrow the query...just read the source of load-field...make your own little customization for ease-of-use