This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2019-10-01
Channels
- # announcements (4)
- # aws (1)
- # beginners (60)
- # calva (10)
- # cider (21)
- # clj-kondo (38)
- # cljdoc (1)
- # clojure (59)
- # clojure-austin (1)
- # clojure-conj (1)
- # clojure-europe (19)
- # clojure-italy (9)
- # clojure-nl (29)
- # clojure-spec (6)
- # clojure-uk (85)
- # clojurescript (54)
- # community-development (11)
- # core-async (2)
- # cursive (21)
- # data-science (4)
- # datomic (39)
- # events (2)
- # fulcro (48)
- # funcool (1)
- # graalvm (5)
- # jackdaw (9)
- # kaocha (17)
- # luminus (2)
- # off-topic (10)
- # parinfer (22)
- # quil (1)
- # re-frame (4)
- # reagent (29)
- # shadow-cljs (7)
- # sql (9)
- # tools-deps (31)
- # yada (1)
måning
Mornin'
I am very pleased, and proud, to announce that I'll be joining http://Avisi.nl as a Clojure developer on Nov 1st.
congratulations @thomas!
> Works in my REPL > Your server must be wonky
the bottom level vector of maps is an ordered list that I want to keep ordered (they are counts per calendar year)
the top level vector of vectors is made up of each simulation run of a model, so I want to be able to compute some things for each run (median, mean, interquartile ranges etc)
(def foo-runs [[{:a 11 :b 11} ;; run 1
{:a 12 :b 12}
{:a 13 :b 13}]
[{:a 21 :b 21} ;; run 2
{:a 22 :b 22}
{:a 23 :b 23}]])
(defn merge-with-+
[x1 x2] (merge-with + x1 x2))
(defn sum-reducef
([] [])
([x] x)
([acc x]
(if (empty? acc)
x
(into [] (map merge-with-+ acc x)))))
what bugs me is my (if (empty? acc)
in the sum-reducef. I feel like I'm doing something wrong here but I'm not sure of the "right" way to produce the initial value
so, my code is producing the right results, but I feel I'm going about it the wrong way. Any thoughts?
the results should look like this:
(def expected [{:a 32 :b 32}
{:a 34 :b 34}
{:a 36 :b 36}])
from reading around I'd need to do that empty check in my reducing step function with either transduce
or clojure.core.reducers/fold
hacker confirmed
maybe….
(def foo-runs [[{:a 11 :stuck_out_tongue: 11} ;; run 1
{:a 12 :stuck_out_tongue: 12}
{:a 13 :stuck_out_tongue: 13}]
[{:a 21 :stuck_out_tongue: 21} ;; run 2
{:a 22 :stuck_out_tongue: 22}
{:a 23 :stuck_out_tongue: 23}]
[{:a 21 :stuck_out_tongue: 1} ;; run 3
{:a 22 :stuck_out_tongue: 9}
{:a 23 :stuck_out_tongue: 23}]])
(defn merge-with-+
[ms]
(apply merge-with + ms))
(apply sequence
(map (fn [& colls]
(merge-with-+ colls)))
... other transducers
foo-runs)
relying on this trick: https://www.reddit.com/r/Clojure/comments/beogpw/something_i_learned_recently_about_transducers/
aw man slack’s replaced all my b keys with stuck out tongues
but you are trying to work with ragged data?
so I know two ways to tackle ragged data; (concat (repeat {})
onto your inner vectors, and then take-while
inputs are not all empty maps
I vaguely remember once writing a transducer for ragged data, just looking to see if I can dig it out
oh I had something like
(defn slice-ragged-data
"like (apply mapv vector data) but works with ragged data;
exhausted colls are overlooked"
[colls]
(if (seq colls)
(cons (vec (keep first colls))
(lazy-seq (slice-ragged-data (keep next colls))))
'()))
so, for example
(mapv (partial apply merge-with +) (slice-ragged-data foo-runs))
=> [{:a 32, :b 32} {:a 34, :b 34} {:a 36, :b 36}]
(mapv (partial apply merge-with +)
(slice-ragged-data
[[{:a 11, :b 11} {:a 12, :b 12} {:a 13, :b 13} {:a 2 :b 2}]
[{:a 21, :b 21} {:a 22, :b 22} {:a 23, :b 23}]]))
=> [{:a 32, :b 32} {:a 34, :b 34} {:a 36, :b 36} {:a 2, :b 2}]
etcthe reducible approach would be
(defn create-raggedly-slices
[colls]
(let [inext (fn [^Iterator it] (when (.hasNext it)
(.next it)))]
(reify IReduceInit
(reduce [this f start]
(let [iters (map #(.iterator ^Iterable %) colls)]
(loop [acc start]
(if (reduced? acc)
(unreduced acc)
(let [r (mapv inext iters)]
(if (every? nil? r)
acc
(recur (f acc r)))))))))))
and then you can say
(into []
(map #(apply (partial merge-with +) %))
(create-raggedly-slices
[[{:a 11, :b 11} {:a 12, :b 12} {:a 13, :b 13} {:a 2 :b 2}]
[{:a 21, :b 21} {:a 22, :b 22} {:a 23, :b 23}]]))
=> [{:a 32, :b 32} {:a 34, :b 34} {:a 36, :b 36} {:a 2, :b 2}]
if that's any good
I guess it depends on how big your data is
what part nil
plays in the data; obviously that part can be substituted
ragged data isn't the problem, but there will be about 1000 runs which is why I was going on a reduce approach rather than map or apply map
you can write
(reduce
(partial map (partial merge-with +))
[{} {} {} {}]
[[{:a 11, :b 11} {:a 12, :b 12} {:a 13, :b 13} {:a 2 :b 2}]
[{:a 21, :b 21} {:a 22, :b 22} {:a 23, :b 23}]])
=> ({:a 32, :b 32} {:a 34, :b 34} {:a 36, :b 36})
but map
requires exactly rectangular data
and will shrink to the lowest width
> I'd still like to figure out that empty? bit tho. It feels strangely wrong to me I think that is ragged data
https://www.transdatasolutions.com/what-is-ragged-data/ This is a legit term, never heard of it before
ah, I think we had slightly different definitions of ragged here, but I see what you mean now
oh what is your definition?
b/c the maps are consistent, my problem was with the initial value (the bit created in the ([] ,,,)
arity of the reducing step function
this does what I want
(defn sum-reducef
([] (repeat {}))
([x] x)
([acc x]
(into [] (map (partial merge-with +) acc x))))
(def foo-runs [[{:a 11 :b 11} ;; run 1
{:a 12 :b 12}
{:a 13 :b 13}]
[{:a 21 :b 21} ;; run 2
{:a 22 :b 22}
{:a 23 :b 23}]])
(def expected [{:a 32 :b 32}
{:a 34 :b 34}
{:a 36 :b 36}])
(transduce
(map identity)
sum-reducef
foo-runs)
(transduce
(map identity)
does make me wonder about life-choices thoughwhy not just reduce
then?
or is that waiting for the next bit
I figured that there would be things that would go in there
um, aren't those more conveniently accessed using https://clojure.github.io/clojure/clojure.core-api.html#clojure.core/sequence
or eduction
yeah, I was just wondering about the mechanics
I’ve always been curious to write a transducing context
that was a stupid idea anyway
curiousity is a good thing
and > stupid idea is very much a hindsight judgement
you only get to find that out if you do it
😄 true
it is waiting for the next bit