Fork me on GitHub
#specter
<
2017-06-15
>
timgilbert21:06:29

Hey, I've got a newbie question. I have a big arbitrarily-nested ball of data (vectors, maps, etc). I want to traverse it looking for maps that pass a certain predicate, and then assoc a new key into those maps.

timgilbert21:06:53

Here's what my current implementation looks like:

(defn decorate
  "walk/postwalk version of above that adds get urls in-place"
  [data]
  (walk/postwalk
   (fn [item]
     (if-not (right-thing? item)
       item
       (my-transform item)))
   data))

timgilbert21:06:23

Can someone advise me on what the equivalent specter calls would be?

nathanmarz21:06:18

@timgilbert it would look something like (setval [(walker right-thing?) :my-new-key] my-new-val data)

nathanmarz21:06:27

or to match your code exactly (transform (walker right-thing?) my-transform data)

nathanmarz21:06:51

depending on what my-transform is you may be able to express it within the path

timgilbert21:06:30

It's basically just (assoc my-map :my/key (other-fn my-map))

nathanmarz21:06:00

you can do (transform [(walker right-thing?) (collect-one (view other-fn)) :my/key] (fn [m _] m) data)

timgilbert21:06:37

Cool. Thanks for the help, I'll play around with it some. I'm benchmarking a few different approaches

nathanmarz21:06:24

the walker in latest specter release is a lot faster than clojure.walk and has same functionality

nathanmarz21:06:40

i benchmarked 70% improvement

timgilbert21:06:16

Cool. I'll let you know how it turns out

timgilbert22:06:08

FWIW, I'm seeing about the same speeds on my test data set, which has a lot of stuff I don't care (about 750K of EDN) about and 0-90 target maps

nathanmarz22:06:37

care to share your benchmark code?

timgilbert22:06:36

It would take me some time to clean it up and put it into a gist, but I'll try to do it tomorrow afternoon sometime

timgilbert22:06:17

But it's basically running criterium against a bunch of random data from an EDN file plus a bunch of generated data from specs

timgilbert22:06:57

The context is I need to go through a potentially large API response and decorate some data with derived values, and I'm trying to see how much of a perf hit I'm setting myself up for

nathanmarz22:06:33

i'd be curious to see benchmark of (transform (walker right-thing?) my-transform data) approach as well (without using collect-one)

timgilbert22:06:21

Looks like about the same. In my benchmarks I've got 0.2927 for the collect-one version, 0.2994 for the transform one and 0.2962 for the postwalk version

timgilbert22:06:09

(I'm not going absolutely crazy with the methodology here, I should note, just looking for back-of-the-envelope stuff)

nathanmarz22:06:01

if your target maps are never within another map's keys, you can gain big performance improvement by using a custom recursive path

timgilbert22:06:32

Sadly, they are deeply and somewhat arbitrarily nested

timgilbert22:06:47

(Which is something I'm looking to fix but that's another story)

nathanmarz22:06:48

(def optimized-walker
  (recursive-path [afn] p
    (cond-path (pred afn) STAY
               map? [MAP-VALS p]
               coll? [ALL p]
               )))

nathanmarz22:06:04

the data has maps within map keys?

nathanmarz22:06:23

e.g. {[{:a 1}] 2}

timgilbert22:06:43

Yeah, so it could be like {:company "foo" :people [{:person "bob" :avatar {:type :thumbnail :image-path "/dir/file.png"}}]}

timgilbert22:06:55

I'm looking for :image-path

timgilbert22:06:44

But it could also be {:product "foo" :images [{:type :product-img :image-path "foo-bar"}]}

nathanmarz22:06:52

yea, use optimized-walker

timgilbert22:06:55

...and those things can all nest in various ways

nathanmarz22:06:19

avoiding traversing over key/value pairs and keys should be very significant

timgilbert22:06:54

Cool. Thanks again for the help. I'll play around with it more tomorrow and let you know if I kick up anything interesting