Fork me on GitHub
#data-science
<
2022-06-19
>
chrisn16:06:23

@surf-kid24 - I created a https://gist.github.com/cnuernber/6924075f89f686a9bf38f02569de6046. The idea is to create a new column that is index-1 or (max 0 (dec idx)). Then filter whenever the existing price is higher than the previous price. I hope this makes it clear how to do it in dataset/datatype land. These exact operations will work fine on a TC dataset.

👍 3
genmeblog19:06:08

Another way is to make pairs as groups and process these groups (sort, select lower).

(def ds (dataset {:a (clojure.core/shuffle (range 15))
                :b (repeatedly #(clojure.core/rand-nth ["a" "b" "c"]))
                :id (range)}))

ds
;; => _unnamed [15 3]:
;;    | :a | :b | :id |
;;    |---:|----|----:|
;;    | 13 |  a |   0 |
;;    |  5 |  b |   1 |
;;    |  2 |  a |   2 |
;;    | 10 |  a |   3 |
;;    | 14 |  b |   4 |
;;    |  0 |  a |   5 |
;;    | 12 |  b |   6 |
;;    |  1 |  b |   7 |
;;    |  9 |  a |   8 |
;;    |  6 |  b |   9 |
;;    |  7 |  a |  10 |
;;    |  3 |  b |  11 |
;;    |  4 |  b |  12 |
;;    | 11 |  a |  13 |
;;    |  8 |  a |  14 |

(-> ds
    (group-by (zipmap (range) (partition 2 1 (range (row-count ds))))) ;; make pairs
    (order-by :a) ;; order each pair
    (first) ;; select first record from each pair
    (ungroup) ;; combine back groups
    (order-by :id)) ;; sort by
;; => _unnamed [14 3]:
;;    | :a | :b | :id |
;;    |---:|----|----:|
;;    |  5 |  b |   1 |
;;    |  2 |  a |   2 |
;;    |  2 |  a |   2 |
;;    | 10 |  a |   3 |
;;    |  0 |  a |   5 |
;;    |  0 |  a |   5 |
;;    |  1 |  b |   7 |
;;    |  1 |  b |   7 |
;;    |  6 |  b |   9 |
;;    |  6 |  b |   9 |
;;    |  3 |  b |  11 |
;;    |  3 |  b |  11 |
;;    |  4 |  b |  12 |
;;    |  8 |  a |  14 |

👍 1
genmeblog19:06:14

group-by here creates groups from row indexes.

chrisn17:06:29

I was reminded this morning here was already a datatype function - https://cnuernber.github.io/dtype-next/tech.v3.datatype.functional.html#var-shift - that did the operation requested.

👍 1
SK22:06:39

Thanks to both of you! I'll look into these in detail.