Fork me on GitHub
#meander
<
2020-03-23
>
grounded_sage16:03:28

How would you tackle this transform

(m/rewrite  {:Data
               {:FirstName "Bob"
                :CommunicationChannels [{:ChannelType {:Name "EMail"}
                                         :Content ""}
                                        {:ChannelType {:Name "Phone"}
                                         :Content "5555-0000"}
                                        {:ChannelType {:Name "EMail"}
                                         :Content ""}]}}
              {:Data
               {:FirstName ?first-name
                :CommunicationChannels [{:ChannelType {:Name !comm-channel-type}
                                         :Content !comm-channel-content} ...]}}
              {:first_name ?first-name
               :CommunicationChannels [{!comm-channel-type !comm-channel-content} ...]})
Such that the result is.
{:first_name "Bob"
 :email ""
 :email_2 ""
 :phone "5555-0000"
 :phone_2 nil}

jimmy16:03:32

Is it just up to two emails and phones or n emails and phones?

grounded_sage17:03:43

That is the annoying part. I am shaping it to a flat structure which goes into SQL and CSV. Perhaps a solution which has up to 2 and one that can have variable would be good.

grounded_sage17:03:52

This is what exists in the db actually.

phone_number,
   phone_number_2,
   mobile_number,
   mobile_number_2,
   email,
   email_2,

grounded_sage17:03:14

At present I’m not 100% sure where placement of nil values will occur. Process is. Folder full of JSON files -> Meander to get values of interest -> Turn into single or multiple CSV’s from that JSON -> Generate db tables from CSVs -> Query them with SQl and perform secondary transform with Meander if necessary.

jimmy17:03:55

That makes it a lot easier. I won't be able to write up a solution right now. There will be a tiny complication with map ordering. But other than that I think it shouldn't be two bad. My recommendation for a first stab at it is to use or to capture the different types in different memory variables.

grounded_sage17:03:16

So first Meander step is flattening JSON files and splitting up the data so it fits with the rest of our CSV etl workflow.

grounded_sage17:03:31

I have a lot of other very nested JSON files so I am hoping when I see a solution it enlightens me as to how I can do the rest.

jimmy19:03:12

@grounded_sage

(m/rewrite {:Data
            {:FirstName "Bob"
             :CommunicationChannels [{:ChannelType {:Name "EMail"}
                                      :Content ""}
                                     {:ChannelType {:Name "Phone"}
                                      :Content "5555-0000"}
                                     {:ChannelType {:Name "EMail"}
                                      :Content ""}]}}
  {:Data
   {:FirstName ?first-name
    :CommunicationChannels [(m/or
                             {:ChannelType {:Name "EMail"}
                              :Content !emails}
                             {:ChannelType {:Name "Phone"}
                              :Content !phones}) ...]}}
  {& [[:first_name ?first-name]
      [:email !emails]
      [:email_2 !emails]
      [:phone !phones]
      [:phone_2 !phones]]})

grounded_sage08:03:21

Is there a reason for the vector pairs here?

grounded_sage08:03:49

As I just tried

{& {:first_name ?first-name, 
                 :email !emails, 
                 :email_2 !emails, 
                 :phone !phones, 
                 :phone_2 !phones}}
and it appears to give the same result.

jimmy13:03:00

Yes because as your map grows it will be parsed out of order and the memory variables will be in the wrong place. I think it is 8 elements when that happens?

👍 4
niclasnilsson20:03:32

Hi! A little bit of an odd example (it’s been minimized from a much larger example). Isn’t m/app called from the action part of a match?

(defn ->h [v]
  (case v
    1 {:info "some info"}
    2 {:extra "something else"}))

(defn foo [data]
  (m/match
    data

    {:a ?a
     :b ?b}

    {:foo
     (m/app str ?a
       merge
       {}
       (m/app ->h ?a)
       (m/app ->h ?b))}))

(foo {:a 1 :b 2})

;; expected
;; {:foo {:info "some info" :extra "something else"}}

niclasnilsson20:03:18

The use case is that I have a value in the pattern part (like ?a and ?b) that I need to merge (after some transformation) in the action.

niclasnilsson20:03:37

Never mind, solved it with m/rewrite!

niclasnilsson21:03:09

And then realized that it could be used with m/match using just merge, mot m/app.

jimmy21:03:35

Yeah for match the right hand side is just normal clojure code. With rewrite it is a substitution.

noprompt22:03:42

Just wanted to share something fun from progress being made on the zeta branch showing a properly greedy * and the concept of the fold (which is still a work in progress).

(solve '[1 8 2 -5 3 4]
       (mz/with [%min (mz/fold *min 0 clojure.core/min)
                 %max (mz/fold *max 0 clojure.core/max)]
         [(mz/* (mz/and %min %max)) . !zs ...]))
;; =>
({#meander.runtime.zeta/fold-variable [*min ,,,] -5,
  #meander.runtime.zeta/fold-variable [*max ,,,] 8,
  #meander.runtime.zeta/memory-variable !zs []})

noprompt22:03:42

*min is -5, *max is 8, and !zs is empty because the greedy star ate its lunch. 🙂