Fork me on GitHub
#clojure-uk
<
2019-10-01
>
thomas07:10:27

I am very pleased, and proud, to announce that I'll be joining http://Avisi.nl as a Clojure developer on Nov 1st.

sheepy 16
partywombat 20
bananadance 20
mario-star 16
otfrom07:10:42

@thomas w00t! congrats!

dharrigan07:10:24

Ooh! Well done Thomas!

dharrigan07:10:35

It's good to get paid for what you love 😉

3Jane07:10:53

Congrats!

dharrigan08:10:19

tee hee, I once again remind myself that map is lazy 🙂

dharrigan08:10:24

but on the repl it's eager 🙂

Ben Hammond08:10:04

> Works in my REPL > Your server must be wonky

dharrigan08:10:04

so WIMR instead of WFM?

👍 4
otfrom09:10:00

well, not eager so much as evaluated for the printing

dharrigan10:10:13

Yup, that's what I mean, the act of printing it realises the data 🙂

otfrom10:10:10

I'm trying to sum up a vector of vector of maps

otfrom10:10:01

the bottom level vector of maps is an ordered list that I want to keep ordered (they are counts per calendar year)

otfrom10:10:48

the top level vector of vectors is made up of each simulation run of a model, so I want to be able to compute some things for each run (median, mean, interquartile ranges etc)

otfrom10:10:55

I've got some dummy code that is doing the summing

otfrom10:10:09

(def foo-runs [[{:a 11 :b 11} ;; run 1
                  {:a 12 :b 12}
                  {:a 13 :b 13}]
                 [{:a 21 :b 21} ;; run 2
                  {:a 22 :b 22}
                  {:a 23 :b 23}]])

otfrom10:10:25

(defn merge-with-+
  [x1 x2] (merge-with + x1 x2))

(defn sum-reducef
  ([] [])
  ([x] x)
  ([acc x]
   (if (empty? acc)
     x
     (into [] (map merge-with-+ acc x)))))

otfrom10:10:38

(transduce
   (map identity)
   sum-reducef
   foo-runs)

otfrom10:10:46

and this produces what I expect.

otfrom10:10:39

what bugs me is my (if (empty? acc) in the sum-reducef. I feel like I'm doing something wrong here but I'm not sure of the "right" way to produce the initial value

otfrom10:10:01

so, my code is producing the right results, but I feel I'm going about it the wrong way. Any thoughts?

otfrom10:10:31

the results should look like this:

(def expected [{:a 32 :b 32}
                 {:a 34 :b 34}
                 {:a 36 :b 36}])

otfrom11:10:38

from reading around I'd need to do that empty check in my reducing step function with either transduce or clojure.core.reducers/fold

folcon11:10:16

Morn’ and congrats @thomas =)…

maleghast11:10:29

Morning All! Congrats @thomas

thomas11:10:09

thank you all...

otfrom12:10:01

breakthrough with simplifying some complicated code at exactly 1337 today. 😄

danielneal12:10:19

hacker confirmed

otfrom13:10:37

I'd still like to figure out that empty? bit tho. It feels strangely wrong to me

danielneal14:10:08

maybe….

(def foo-runs [[{:a 11 :stuck_out_tongue: 11} ;; run 1
                {:a 12 :stuck_out_tongue: 12}
                {:a 13 :stuck_out_tongue: 13}]
               [{:a 21 :stuck_out_tongue: 21} ;; run 2
                {:a 22 :stuck_out_tongue: 22}
                {:a 23 :stuck_out_tongue: 23}]
               [{:a 21 :stuck_out_tongue: 1} ;; run 3
                {:a 22 :stuck_out_tongue: 9}
                {:a 23 :stuck_out_tongue: 23}]])

(defn merge-with-+
  [ms]
  (apply merge-with + ms))

(apply sequence
  (map (fn [& colls]
         (merge-with-+ colls)))
  ... other transducers
  foo-runs)

danielneal14:10:07

aw man slack’s replaced all my b keys with stuck out tongues

Ben Hammond14:10:47

(apply map merge-with-+ foo-runs) also works ..?

4
Ben Hammond14:10:09

but you are trying to work with ragged data?

Ben Hammond14:10:06

so I know two ways to tackle ragged data; (concat (repeat {}) onto your inner vectors, and then take-while inputs are not all empty maps

Ben Hammond14:10:01

I vaguely remember once writing a transducer for ragged data, just looking to see if I can dig it out

Ben Hammond14:10:53

oh I had something like

(defn slice-ragged-data
  "like (apply mapv vector data) but works with ragged data;
    exhausted colls are overlooked"
  [colls]
  (if (seq colls)
    (cons (vec (keep first colls))
          (lazy-seq (slice-ragged-data (keep next colls))))
    '()))

Ben Hammond14:10:00

so, for example

(mapv (partial apply merge-with +) (slice-ragged-data foo-runs))
=> [{:a 32, :b 32} {:a 34, :b 34} {:a 36, :b 36}]

Ben Hammond14:10:02

(mapv (partial apply merge-with +) 
  (slice-ragged-data 
    [[{:a 11, :b 11} {:a 12, :b 12} {:a 13, :b 13} {:a 2 :b 2}]
     [{:a 21, :b 21} {:a 22, :b 22} {:a 23, :b 23}]]))
=> [{:a 32, :b 32} {:a 34, :b 34} {:a 36, :b 36} {:a 2, :b 2}]
etc

Ben Hammond14:10:01

the reducible approach would be

(defn create-raggedly-slices
  [colls]
  (let [inext (fn [^Iterator it] (when (.hasNext it)
                                   (.next it)))]
    (reify IReduceInit
      (reduce [this f start]
        (let [iters (map #(.iterator ^Iterable %) colls)]
          (loop [acc start]
            (if (reduced? acc)
              (unreduced acc)
              (let [r (mapv inext iters)]
                (if (every? nil? r)
                  acc
                  (recur (f acc r)))))))))))
and then you can say
(into []
  (map #(apply (partial merge-with +) %))
  (create-raggedly-slices
    [[{:a 11, :b 11} {:a 12, :b 12} {:a 13, :b 13} {:a 2 :b 2}]
     [{:a 21, :b 21} {:a 22, :b 22} {:a 23, :b 23}]]))
=> [{:a 32, :b 32} {:a 34, :b 34} {:a 36, :b 36} {:a 2, :b 2}]

Ben Hammond14:10:03

if that's any good

Ben Hammond14:10:31

I guess it depends on how big your data is

Ben Hammond14:10:53

what part nil plays in the data; obviously that part can be substituted

otfrom15:10:20

ragged data isn't the problem, but there will be about 1000 runs which is why I was going on a reduce approach rather than map or apply map

Ben Hammond15:10:22

you can write

(reduce
  (partial map (partial merge-with +))
  [{} {} {} {}]
  [[{:a 11, :b 11} {:a 12, :b 12} {:a 13, :b 13} {:a 2 :b 2}]
   [{:a 21, :b 21} {:a 22, :b 22} {:a 23, :b 23}]])
=> ({:a 32, :b 32} {:a 34, :b 34} {:a 36, :b 36})

Ben Hammond15:10:02

but map requires exactly rectangular data

Ben Hammond15:10:35

and will shrink to the lowest width

otfrom15:10:31

each run is the same shape tho

Ben Hammond15:10:02

> I'd still like to figure out that empty? bit tho. It feels strangely wrong to me I think that is ragged data

dominicm15:10:08

My data is wheaty.

otfrom15:10:50

ah, I think we had slightly different definitions of ragged here, but I see what you mean now

Ben Hammond15:10:13

oh what is your definition?

otfrom15:10:40

that the maps might have had different keys in them at the lowest level

otfrom15:10:20

b/c the maps are consistent, my problem was with the initial value (the bit created in the ([] ,,,) arity of the reducing step function

otfrom15:10:33

and that raggedness is indeed what is causing me the issue

otfrom15:10:43

and the nils in the map work with with merge-with +

otfrom15:10:56

so I just need to see how my real target function will work there 🙂

otfrom15:10:39

this does what I want

(defn sum-reducef
    ([] (repeat {}))
    ([x] x) 
    ([acc x]
     (into [] (map (partial merge-with +) acc x)))) 

  (def foo-runs [[{:a 11 :b 11} ;; run 1
                  {:a 12 :b 12}
                  {:a 13 :b 13}]
                 [{:a 21 :b 21} ;; run 2
                  {:a 22 :b 22}
                  {:a 23 :b 23}]])

  (def expected [{:a 32 :b 32}
                 {:a 34 :b 34}
                 {:a 36 :b 36}])

  (transduce
   (map identity)
   sum-reducef
   foo-runs)

4
Ben Hammond16:10:58

(transduce
   (map identity)
does make me wonder about life-choices though

Ben Hammond16:10:11

why not just reduce then?

Ben Hammond16:10:46

or is that waiting for the next bit

otfrom16:10:20

it is waiting for the next bit

danielneal16:10:15

I figured that there would be things that would go in there

danielneal16:10:24

yeah, I was just wondering about the mechanics

danielneal16:10:58

I’ve always been curious to write a transducing context

danielneal16:10:12

that was a stupid idea anyway

Ben Hammond16:10:40

curiousity is a good thing

Ben Hammond16:10:44

and > stupid idea is very much a hindsight judgement

Ben Hammond16:10:01

you only get to find that out if you do it

otfrom16:10:20

it is waiting for the next bit