Fork me on GitHub
#specter
<
2016-06-23
>
aengelberg00:06:41

@nathanmarz just saw your post about multi-transform, looks interesting. Have you considered adding a terminal-val which sets the value instead of transforms it (like setval is to transform)?

nathanmarz00:06:44

@aengelberg: good idea, i'll add that to the branch

aengelberg01:06:05

just realized that setval does not work with collect... is this a bug?

boot.user=> (setval [:a (collect STAY)] 2 {:a 1})

clojure.lang.ArityException: Wrong number of args (2) passed to: impl/compiled-setval*/fn--2959

aengelberg01:06:36

I was about to say, why not switch constantly to (fn [_] val) like setval uses, for efficiency...

nathanmarz01:06:43

ah, just looked at the implementation of constantly

nathanmarz01:06:56

but yea that's a bug

nathanmarz01:06:19

though it's a weird bug to hit

aengelberg01:06:31

user> (let [f (constantly 1)] (dotimes [i 5] (time (dotimes [i 10000000] (f 2)))))
"Elapsed time: 178.060125 msecs"
"Elapsed time: 164.164117 msecs"
"Elapsed time: 170.549443 msecs"
"Elapsed time: 165.947481 msecs"
"Elapsed time: 163.606619 msecs"
nil
user> (let [f (fn [_] 1)] (dotimes [i 5] (time (dotimes [i 10000000] (f 2)))))
"Elapsed time: 55.36876 msecs"
"Elapsed time: 47.140944 msecs"
"Elapsed time: 46.974314 msecs"
"Elapsed time: 46.51298 msecs"
"Elapsed time: 44.286403 msecs"
nil
user> (let [f (fn [_ & _] 1)] (dotimes [i 5] (time (dotimes [i 10000000] (f 2)))))
"Elapsed time: 86.562498 msecs"
"Elapsed time: 89.103274 msecs"
"Elapsed time: 86.074677 msecs"
"Elapsed time: 87.669437 msecs"
"Elapsed time: 89.690679 msecs"
nil
(edited with more accurate benchmark)

aengelberg01:06:58

There's a spectrum

nathanmarz01:06:57

hand-rolled a fast-constantly and also fixed that bug

luxbock08:06:49

are (transform [<selector> (view f)] identity coll) and (transform <selector> f coll) interchangeable? is the latter going to be faster?

luxbock08:06:28

using specter feels a bit like concatenative programming to me

luxbock08:06:45

I wonder if there are some ideas there that could be applicable

luxbock08:06:18

you could think of VAL as an analog of dup and the discussed OMIT as being drop

luxbock08:06:34

concatenative languages can be extremely expressive and allow for easy meta-programming

luxbock09:06:11

I played around with combining this idea with destructuring: https://gist.github.com/luxbock/f93529b82792ef35db12fcf5e5a78fdb

nathanmarz13:06:35

@luxbock: they are not interchangeable as f in the latter form will receive collected values while f in the former will not

nathanmarz13:06:40

the latter will also be faster

nathanmarz14:06:31

@luxbock: interesting ideas, but can't the starting example be written as this?

(transform
  [MAP-VALS
   (view frequencies)
   (collect-one (subselect MAP-VALS) (view #(apply max %)))
   MAP-VALS]
  (fn [mx e] (/ e mx))
  {:a [1 1 3 4] :b [1 1 2 2 2 3]})

luxbock14:06:48

yes you are correct, my specter-fu is still far from perfect đŸ™‚

nathanmarz14:06:01

you'll get there đŸ™‚

nathanmarz14:06:12

just realized it can be made even more concise:

(transform
  [MAP-VALS
   (view frequencies)
   (subselect MAP-VALS)
   (collect-one (view #(apply max %)))
   ALL]
  (fn [mx e] (/ e mx))
  {:a [1 1 3 4] :b [1 1 2 2 2 3]})

luxbock15:06:11

if I wanted to do the same thing, but wanted to inc all the numbers before using frequencies and the rest of the selectors, can I somehow fit that into the selector?

luxbock15:06:43

I have this type of situation where I had a nested transform like in my example, and I'm now trying to re-write it using your approach

luxbock15:06:21

but instead of numbers in a vector, I have a collection of maps where I need to fetch a nested value and then round those up before calling frequencies

luxbock15:06:56

so I need to start with MAP-VALS, ALL, (view inc), but then I'd need to go back up a level in the selection

nathanmarz15:06:30

@luxbock: is transformed what you're looking for?

(transform
  [MAP-VALS
   (transformed ALL inc)
   (view frequencies)
   (subselect MAP-VALS)
   (collect-one (view #(apply max %)))
   ALL]
  (fn [mx e] (/ e mx))
  {:a [1 1 3 4] :b [1 1 2 2 2 3]})

luxbock15:06:12

@nathanmarz: yeah it works for my toy example, thanks

luxbock15:06:27

my actual use case is still a bit more complicated so I need to figure out a few more things

luxbock15:06:48

does view only take one argument? it seems to accept many but I'm not sure if they are doing anything

nathanmarz15:06:09

it only takes one argument

nathanmarz15:06:18

that it doesn't error with zero or more than one is an implementation artifact

luxbock15:06:43

what do you think if multiple arguments would use an implicit comp?

nathanmarz15:06:02

i think it's better to be explicit about that

nathanmarz15:06:33

comp and view are completely orthogonal to each other

luxbock15:06:32

(view f) (view g) (view h) is the same as (view (comp f g h)) right? just making sure I understand how it works

nathanmarz15:06:10

i think it would be (comp h g f)

luxbock16:06:39

are there any noticeable anti-patterns in here?

Chris O’Donnell16:06:41

@luxbock: you could use (.intValue (double 12.543)) instead of the string business if that's clojure

nathanmarz16:06:10

@luxbock: you should make the transform function argument for transformed static – otherwise it can't inline cache

nathanmarz16:06:27

factor it into a defn

luxbock16:06:59

@codonnell: yeah good point

nathanmarz16:06:20

that's the only case in specter where anonymous functions to a navigator won't factor automatically

luxbock16:06:43

@nathanmarz: ah, why is that?

luxbock16:06:08

that's a good thing to know

nathanmarz16:06:42

it would be possible to re-engineer transformed to do so, but that will be a lot of work

luxbock16:06:39

is it generally true that I want to do as much of the work in the selector as possible?

luxbock16:06:57

rather than nesting in another call to transform

nathanmarz16:06:57

I find that's generally more elegant

luxbock16:06:22

is there an easier way to do this: (transform [(collect-one :a :b :c) (view (comp :d :b :a))] + {:a {:b {:c 1 :d 2}}}) ?

nathanmarz17:06:20

(reduce + (select [:a :b (multi-path :c :d)] {:a {:b {:c 1 :d 2}}})

nathanmarz17:06:46

or in 0.12.0 (reduce + (traverse [:a :b (multi-path :c :d)] {:a {:b {:c 1 :d 2}}})

nathanmarz17:06:51

don't know if that's easier

Chris O’Donnell17:06:59

I think it's clearer

Chris O’Donnell17:06:19

comp feels unnatural for accessing nested maps, because the keys have to be backwards

luxbock17:06:35

yeah I like the traverse one

luxbock17:06:10

one problem that I run into with these large selectors is that they break the cider debugger

luxbock17:06:21

"Method code too large!"

luxbock17:06:09

running macroexpand-all on the transform I can see why

nathanmarz17:06:13

that's the inline caching implementation you're seeing

nathanmarz17:06:30

odd that cider would have a limit like that

luxbock18:06:43

I benchmarked my specter implementation of a function vs a transducer / vanillla core version of the same function

luxbock18:06:01

results are quite close speed wise but in this case I think I prefer the core version slightly for I think it's easier to go and modify to change its behavior

luxbock18:06:32

for example I actually prefer to have the (filter filter-fn) happen between the two map-steps, and I can't think off the top of my head how I'd need to modify the selector to accomplish this

nathanmarz18:06:23

you probably want a new pathed navigator

nathanmarz18:06:40

then it would look like:

(defn round-frequencies
  [iso-map find-size round-fn filter-fn]
  (transform [MAP-VALS
              (selectview [ALL (view find-size) (view round-fn) (pred filter-fn)])
              (view frequencies)
              (transformed
                [(subselect MAP-VALS)
                 (collect-one (view #(apply + %)))
                 ALL]
                revdiv)]
    #(into (sorted-map)
       (keep (fn [[k v]]
               (when (> v 1/100)
                 [k (* 100 v)])))
       %)
    iso-map))

luxbock18:06:33

thanks, yeah I hadn't seen selectview before

nathanmarz18:06:38

it doesn't exist

nathanmarz18:06:44

but it would be easy to make

luxbock18:06:52

my third version which uses (transform MAP-VALS ...) instead of (into {} ...) is the fastest of them all

luxbock18:06:07

also I realized I was doing some uneccessary extra work in the vanilla core version

nathanmarz18:06:31

If #117 was implemented then it could be simplified further

luxbock18:06:15

yeah I think that'd be a really useful feature

Chris O’Donnell18:06:57

@luxbock: are you using comp so your maps and filters are composing as transducers rather than generating intermediate sequences?

Chris O’Donnell18:06:56

have you checked how much of a performance impact that has?

luxbock18:06:06

no, but I imagine it's a fair amount, as the bulk of the work is done there

luxbock18:06:23

about to head to bed but I'll run that out of curiosity before I log off

Chris O’Donnell18:06:26

I'd love to hear the results; I'm curious

nathanmarz20:06:21

@codonnell: specter has a benchmark for that

nathanmarz20:06:25

Benchmark: even :a values from sequence of maps (500000 iterations)

Avg(ms)		vs best		Code
93.478 		 1.00 		 (into [] xf data)
113.85 		 1.22 		 (select [ALL :a even?] data)
156.08 		 1.67 		 (into [] (comp (map :a) (filter even?)) data)
253.74 		 2.71 		 (->> data (mapv :a) (filter even?) doall)

nathanmarz20:06:20

note that comp with more than two arguments will slow things down a lot

nathanmarz20:06:24

Benchmark: even :a values from sequence of maps (500000 iterations)

Avg(ms)		vs best		Code
155.98 		 1.00 		 (into [] (comp (map :a) (filter even?)) data)
164.82 		 1.06 		 (into [] xf data)
203.00 		 1.30 		 (select [ALL :a even? even? even?] data)
331.59 		 2.13 		 (into [] (comp (map :a) (filter even?) (filter even?) (filter even?)) data)
525.34 		 3.37 		 (->> data (mapv :a) (filter even?) (filter even?) (filter even?) doall)

nathanmarz20:06:44

here xf is (comp (map :a) (filter even?) (filter even?) (filter even?))

Chris O’Donnell20:06:27

wow, that's a huge slowdown

nathanmarz20:06:11

it slows down because it does a reduce after more than two args

nathanmarz20:06:35

if the implementation of comp was unrolled up to 20 arguments it would work much better

Chris O’Donnell20:06:12

I wonder why it isn't unrolled like that; other functions in core certainly are.

Chris O’Donnell20:06:49

Perhaps they've assumed that people aren't going to be calling comp over and over like in the benchmark.

nathanmarz20:06:28

well it seems like with transducers it would be a common pattern to do it inline like that

Chris O’Donnell20:06:42

that's true, though transducers weren't in existence when that patch was submitted

aengelberg21:06:32

@nathanmarz It seems like the multi-transform functionality of doing a variety of operations in a single transform is still "doable" with transform, if you collect values along the way and use that to dispatch on some operation in the transform fn.

aengelberg21:06:42

it's probably not as performant as multi-transform but might be fun to add to your benchmark.

nathanmarz22:06:23

@aengelberg: true but it's a bit convoluted and definitely would be less performant

nathanmarz22:06:56

the cases I consider for the benchmarks are the fastest impls, the most concise impls, and the most idiomatic impls