Fork me on GitHub
#specter
<
2018-03-16
>
Marcus Pemer12:03:40

Hi - my first question to this channel, hope you all are doing well. I am processing some CSV input - fairly large file with four types of records in them. All records are grouped with a header line and they are separated with blank (empty) lines. I have come up with the following attempt to use Specter for the transformation:

(let [data (->>
            ;; my data comes in as text from a csv file
            (csv/parse-csv (slurp "some-file-name.csv")
            ;; group by number of elements and remove group of empty rows
            (dissoc (group-by count csv) 1)
            ;; remove trailing empty cells within records
            (setval [MAP-VALS ALL LAST empty?] NONE)
            ;; prepare fields that will become map keys
            (transform [MAP-VALS FIRST ALL] keify)
            ;; turn the groups into maps
            (transform [MAP-VALS] sc/mappify)
            ;; lastly, read our data fields
            (transform [MAP-VALS ALL ALL] read-field)
            )]
  (def a-records (get data 6))
  (def b-records (get data 7))
  (def c-records (get data 11))
  (def d-records (get data 15)))
While this works for my immediate needs, I find myself navigating to MAP-VALS over and over again, thinking this can probably be done in a more elegant way. Is there a way in Specter to group the four transformations (setval + 3 transforms) into a sub-select of some sort? Should I even care about this type of optimization?

drowsy12:03:58

@marcus165 There is multi-transform with multi-path to group setval and transform together.

(multi-transform [MAP-VALS (multi-path [ALL LAST empty? (terminal-val NONE)]
                                       [FIRST ALL (terminal keify)]
                                       [(terminal sc/mappify)]
                                       [ALL ALL (terminal read-file)])])

Marcus Pemer12:03:32

awesome - thank you @drowsy!