This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2017-03-30
Channels
- # bangalore-clj (1)
- # beginners (23)
- # boot (1)
- # cider (23)
- # cljsjs (2)
- # cljsrn (4)
- # clojure (251)
- # clojure-dusseldorf (2)
- # clojure-gamedev (23)
- # clojure-italy (5)
- # clojure-russia (53)
- # clojure-sanfrancisco (5)
- # clojure-spec (7)
- # clojure-uk (66)
- # clojurescript (169)
- # community-development (21)
- # core-async (10)
- # cursive (15)
- # data-science (1)
- # datomic (7)
- # docker (1)
- # emacs (24)
- # events (1)
- # funcool (6)
- # hoplon (24)
- # liberator (1)
- # luminus (6)
- # lumo (62)
- # mount (7)
- # off-topic (1)
- # om (12)
- # om-next (5)
- # onyx (14)
- # overtone (2)
- # pedestal (58)
- # powderkeg (27)
- # protorepl (1)
- # re-frame (1)
- # ring-swagger (16)
- # rum (51)
- # spacemacs (25)
- # uncomplicate (7)
- # unrepl (22)
- # untangled (7)
- # yada (109)
I just returned from the Spark codebase and I’m more hopeful than before about Encoders
ExpressionEncoder is the type to use https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/ExpressionEncoder.scala#L219
But now we need to provide two arguments `serializer: Seq[Expression], deserializer: Expression,`
Expressions will be built using https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala which provides StaticInvoke
, ùInvoke`, and NewInstance
.
I think we can achieve an API like (df src spec & xforms-then-options)
very close to what we have for rdd
on todo list for rdd I guess: https://github.com/HCADatalab/powderkeg/blob/master/src/main/clojure/powderkeg/core.clj#L662
I do remember. Transducers assume linear traversal. So you have to solve transducers+fold first.
Rich's strange loop talk on transducers mentioned parallellism on one slide, wonder where it is at now :)