Fork me on GitHub
#powderkeg
<
2017-05-09
>
viesti03:05:07

@kenny could you post your project.clj, looks like kryo version issue

kenny03:05:16

@viesti Here are my deps:

[org.clojure/clojure "1.8.0"]
[hcadatalab/powderkeg "0.5.1"]
[org.apache.spark/spark-core_2.11 "2.1.1"]
[org.apache.spark/spark-streaming_2.11 "2.1.1"]

kenny03:05:02

I have also tried with these deps:

[org.clojure/clojure "1.8.0"]
[hcadatalab/powderkeg "0.5.1"]
[org.apache.spark/spark-core_2.11 "2.1.0"]
[org.apache.spark/spark-streaming_2.11 "2.1.0"]
and running the master and slave with 2.1.0 as well.

viesti05:05:17

hmm, we have a problem with version of kryo used

viesti05:05:47

@kenny as a workaround, could you add [com.esotericsoftware/kryo-shaded "4.0.0"] explicitly

kenny06:05:48

@viesti That did it! I tried updating kyro myself before but I must've used the wrong artifact or version.

viesti09:05:06

so I remember that powderkeg/spark 1.5 works with kryo 3.0.3

viesti09:05:21

powderkeg/spark 2.x works with kryo 4.0.0

viesti09:05:52

either we ship two jars (powderkeg-kryo3/powderkeg-kryo4) or instruct to add direct dependency

cgrand11:05:03

shipping both makes building a classpath management harder no?

viesti11:05:48

and one still has to add dependency with different name actually 🙂