Fork me on GitHub
#meander
<
2022-08-12
>
Peter Nagy07:08:46

Is there a clear pattern to transform a deeply nested part of a data structure? E.g. specter ($/transform [:x :y :zs $/ALL] inc {:foo :bar :x {:y {:zs [1 2 3]}}}) returns {:foo :bar :x {:y {:zs [2 3 4]}}}

noprompt16:08:40

No. The library wasn’t designed for those kinds of transforms.

noprompt16:08:54

Somewhat intentionally too.

Peter Nagy07:08:51

thanks. Why intentionally, is it just out of scope? Do you have something else you use for this case? I think lenses are quite useful on their own, just wanted to know if meander also covers that use case

noprompt06:08:48

Lenses can be useful but in my personal experience I find they tend to code that is difficult for people to read (myself included), and favor brevity at the expense of clarity. They provide great abstraction but not great semantics. In a language where that approach is already over represented, I have organized this experiment of Meander around other goals. tl;dr semantics are more important to me than abstraction. Another reason is that the evaluation semantics of Meander style rewriting -- and this is more pronounced on the zeta branch -- don't map to lenses neatly. Many people, including myself, have schemed ways of building an update-in-place kind of thing with Meander but nothing came out of it. That said, there's nothing stopping anyone from using Meander in conjunction with a lens library. > Do you have something else you use for this case? I try to break problems down in other ways that don't require nested transformations. If I'm using Meander, I use things like cata to do those nested transformations. If I have control over the data model, I like to keep things flat and shallow. If I don't, I transform it into clean semantic piles of data that favor simple operations. When deeply nested transformations are employed, my gut tells me I should investigate. Nested data structures are hell to work with and code often mirrors that hell (a big part of my day job right now is trying to gradually resolve a hell created by a massive nested data structure). Tying this back to lenses: lenses seem like they mitigate that hell but, I think, they just provide an alternative view of it. ;)

👍 1
noprompt06:08:12

On a more personal level, I have ADHD which means my working memory runs out pretty quickly. A lot of Clojure code and stuff like Spectre just torch my brain. Though it wasn't a goal of mine to address that problem with Meander, I think unintentionally I intentionally built it to help with that issue by orienting around externalizing information.

Peter Nagy07:08:45

thanks for the detailed response. What I value about lenses is that they compose well. And while I try to make data shallow/normalized, there's code out there that isn't. I was looking into meander for finding all actions in a https://lucywang000.github.io/clj-statecharts/docs/get-started/ . As you can see it is deeply nested and rebuilding the whole data structure by hand is just tedious, something the machine should do for me. Another nice thing about specter is that it goes out of its way to keep everything the same, so a vector stays a vector, a set a set etc. I understand the distinction between lenses and meander better now, which will help me use meander in a better way. Thank you! Also, looking forward to zeta!

❤️ 1
timothypratley06:08:43

Just chiming in with my opinion... I think that Meander is better than Specter for deeply nested data. If we look at the example described, a simple solution would be:

(m/rewrite {:foo :bar :x {:y {:zs [1 2 3]}}}
           {& ?m :x {& ?m2 :y {& ?m3 :zs [!x ...]}}}
           {& ?m :x {& ?m2 :y {& ?m3 :zs [(m/app inc !x) ...]}}})
=> {:foo :bar, :x {:y {:zs [2 3 4]}}} ^^ Easy and very clear what is happening. Specter is more compact, but I'd much rather see the shape of my data, thanks. In terms of clarity, the difference is huge. I like patterns. And patterns are just fine with deeply nested data. Moving on from the contrived example... I've written at least two large project that are entirely based on Meander for data transformation of deeply nested data structures and it worked great: 1. Kalai (transpiler Clojure to Rust and other languages) basically taking an AST (the mother of all nested data) and rewritting it into many other forms. https://github.com/echeran/kalai/blob/main/src/kalai/pass/kalai/b_kalai_constructs.clj Take a look at this, its really easy to read, clear what tranforms are taking place, and recursive. 2. HappyGAPI https://github.com/timothypratley/happygapi/blob/main/dev/happy/beaver.clj Here again I'm dealing with a pretty scary deeply nested data set (The schemas for Google's APIs) and Meander is able to express them in an extremely clear and recursive way. All that to say IMO regardless of the design goals, Meander is the best solution to complex data transformations including deeply nested structured data. I'd add that a) I do think it's best to avoid complex data by simplifying it into a triple store, b) Complex data does exist though, and not everything goes into a triple store, and even if it does you still need to do transformations at the edges, so it's kinda more a question of how much mapping you have to do. c) When it comes to mapping, I think the closest thing to a competitor to Meander is Instaparse, but they occupy quite different spaces in that Instaparse concerns itself with taking text->AST and Meander concerns itself with taking data->data. Coming back to the main question of "what's the pattern for deeply nested data structures?" My answer would be: 1. If it's an explicit path, then match the explicit path you expect to find, and collect everything else with & 2. If it's a nested recursive scenario then just use your favorite recursion tool (I like m/app, maybe you prefer m/cata, or ~. 3. There's nothing wrong with the old (update-in megamap [:k :k2 :k3 :k4] meander-transform) either Hope that helps