Clojurians Log v2

@jiyouyou125 has joined the channel

@ccann: how are you creating the schema?

seems like the dataframe api is way better for this on both ends

like

way better

(:import [org.apache.spark.sql.types DataTypes StructField StructType])

(defn my-schema
  []
  (let [coll [(StructField. "id" DataTypes/FloatType true (empty))
              (StructField. "field_a" DataTypes/FloatType true (empty))
              (StructField. "field_b" DataTypes/StringType true (empty))]
              fields (into-array StructField coll)]
    (StructType. fields)))

^ e.g. @jrotenberg

cool

i think i figured out a (really hacky) way to create it dynamically

using the the json reading stuff

not pretty

but then

ah nice

working with spark dataframes from clojure has been a nightmare, for what it’s worth 🙂

I’m being dramatic, but it’s not very pleasant

yeah

basically all of our code that gets touched by anyone else is in scala right now

so there are other nightmares to be had

a nightmare for every season

but i just inherited a codebase from a guy who is bailing to do something more interesting

lucky bastard

@ccann: nightmare because it requires a ton of java interop?