Fork me on GitHub
#specter
<
2021-04-05
>
h0bbit05:04:15

Hello all! I’m new to specter and excited to learn how to use it properly. My question is: Is it possible to select the subset of data you want and then to transform it such that the output is just the transformed subset? And if so, how? To clarify, my understanding is as follows: transform is useful for transformations in deeply nested data, and will return the full data-structure back to me. select is useful for arbitrary data selection in a given structure and will only return the selected data. What I want is to select and also transform. We have deeply nested YAML files containing dependency data for our microservices. Each service defines what component it depends on, and exactly which tables/topics etc in that component it reads/writes to. I’m parsing this data to extract information from it and build dependency graphs. The parser in standard Clojure looks like this:

(->> yaml-data
       (mapcat :systems)
       (mapcat :containers)
       (map (juxt :name :dependencies))
       (mapcat (fn [[c ds]]
                 (map (fn [d] {:container c
                              :dependency (:name d)
                              :transports (:transport d)
                              :entities (:data_entity d '())})
                      ds))))
I rewrote this with specter like this:
(->> yaml-data
       (s/select [s/ALL :systems s/ALL :containers s/ALL (s/submap [:name :dependencies])])
       (s/transform [s/ALL (s/collect-one :name) :dependencies s/ALL]
                    (fn [c d]
                      {:container c
                       :dependency (:name d)
                       :transports (:transport d)
                       :entities (:data_entity d '())}))
       (s/select [s/ALL :dependencies s/ALL]))
The problem is that the specter version is more than 2X slower as compared to the previous version. I suspect it’s because the transform step has to re-stitch everything when I don’t need the stitching. What mistake am I making / How can I rewrite the pipeline?