Fork me on GitHub
#meander
<
2020-05-11
>
jeroenvandijk11:05:39

I’m playing with Meander. It looks like a useful tool to transform arbitrary data. So now I’m trying to transform this JSON data with a deeply nested structure. There are many optional keys in the map. I wonder what’s the best way to describe this The following transforms the data recursively, but I don’t how to solve the optional keys. With & I can catch the rest, but I probably need to use this to capture optional keys

(require '[meander.strategy.epsilon :as m*])

(def rewrite-child
  (m*/rewrite 
    {"title".      !node-title
     "create-time" !create-time
     "children" !node-children

     & !others 
     }
    {:create/email !create-email
     :create/time !create-time
     :node/title !node-title
     :node/children !node-children
     
     & 
     })

(def rewrite-children
      (m*/bottom-up
       (m*/attempt rewrite-child)))    
Ideally I would like to have the following transformation:
{"create-time" 1582030610649,
 "title" "",
 "children" [{"create-time" 1582030610649,
              "string" ""}]}
=> 
{:create/time 1582030610649,
 :node/title "",
 :node/children [{:create/time 1582030610649,
                  :node/string ""}]}
Now it becomes
{:create/time 1582030610649,
 :node/title "",
 :node/children [{:create/time 1582030610649,
                  :node/title nil
                  :node/childeren nil}]} 
I wonder if I can get rid of the nil’s somehow. There are also more complex cases where I basically need wildcards for keys. Any suggestion how I should approach this? Is there a particular syntax for maps? Thank you :)

Jimmy Miller17:05:30

I'll have to do some thinking about how I'd do that. Are there certain keys that are optional and you don't want them if they aren't there? Or all the keys?

noprompt18:05:21

Yah more detail would be helpful. I can also jump on later today and lend a hand. 👍

jeroenvandijk20:05:16

Thanks! I’m trying to import a json file. The data structure in the file has an a recursive structure of variable depth. The first level has slightly different keys than the child levels. And there are some special childs with others fields. I guess I could write multiple rewrite rules. I’m wondering what the best way would be to compose that. I’ll add a snippet of the import file to give you an idea

jeroenvandijk20:05:14

This is the file. Eventually I’m trying to put it in a Datomic database by flattening the structure, but my first concern is mapping the right fields

jeroenvandijk20:05:53

So if I could combine at least 3 rewrite rules with attempt and bottom-up it could work I guess. Maybe attempt is composable? I’ll give it a try

jeroenvandijk21:05:31

I’m try rewrite with multiple rules now, one for each map type. But it seems the first one is always chosen. I had the understanding that ? would require the match to be non-nil. I’m probably missing something. Will try more

Jimmy Miller21:05:11

(m/some ?x) is what you want to use for requiring something to be non-nil. Will take a look at this later tonight. Sorry can't get to it earlier.

jeroenvandijk21:05:42

np, thank you!

jeroenvandijk21:05:03

Ok maybe multiple rewrite rules did work

jeroenvandijk21:05:58

Now I’m wondering if I can do transformation on values. E.g. I would like to change timestamps to an instant. I could do a postwalk over the data, but maybe this is something Meander can do as well, saving another data iteration

Jimmy Miller04:05:19

You can use (m/app my-fn ?x) to apply functions to your data.

👌 4
Jimmy Miller04:05:29

Glad that multiple rewrite rule worked for you. We are thinking more about if keys should match even if they don’t exist in the map in the next version of meander. It is a gotcha people often run into.

Jimmy Miller04:05:54

I think multiple matches is probably the best answer for what you are looking to do.

jeroenvandijk07:05:08

Thank you. If all goes well, I'll post a link to the result later :)