Fork me on GitHub
#flambo
<
2019-05-04
>
takis_17:05:48

hello,i am reading a spark book,and i am trying it with Clojure,everything is simple so far,i also saw the sparkling/flambo library,the question i have is this i can wrap select of spark to make the use more simple like this (defn select [dataset & col-names] (.select ^Dataset dataset (into-array (map (fn [c] (if (string? c) (functions/col c) c)) col-names)))) but when i use it,catalyst knows what my select function is? or i loss the optimizations? in general i was wondering if using spark from Clojure is simple or problems will occur ,but so far it seems simple to use spark from Clojure

takis_18:05:53

i found that the plans are exactly the same,so spark i guess has the way to find what my function will do to the dataframe thanks