Fork me on GitHub
#flambo
<
2016-09-01
>
turbopape10:09:02

Hi guys, I have a problem

turbopape10:09:23

When using mllib, I run into incmpatibilities with kryo serialization

turbopape10:09:43

Seems it’s a known bug, and I need to be going with spark 2.X

turbopape10:09:11

problem is, When I reaquire spark 2.X, it tells me that it can’t find logger class...

turbopape10:09:49

here is the relevant trace:

turbopape10:09:51

com.esotericsoftware.kryo.KryoException: java.lang.IllegalArgumentException: Can not set final scala.collection.mutable.ListBuffer field org.apache.spark.mllib.fpm.FPTree$Summary.nodes to scala.collection.mutable.ArrayBuffer Serialization trace: nodes (org.apache.spark.mllib.fpm.FPTree$Summary) org$apache$spark$mllib$fpm$FPTree$$summaries (org.apache.spark.mllib.fpm.FPTree) at com.esotericsoftware.kryo.serializers.FieldSerializer$Obje

turbopape10:09:02

seems it does not like fptree

turbopape10:09:45

seems to have been better if I register the missing classes at kryo’s level :

turbopape10:09:03

… (conf/set "spark.kryo.classesToRegister" "scala.collection.mutable.ArrayBuffer,scala.collection.mutable.ListBuffer")

turbopape10:09:39

But still waiting for it to finish ...

turbopape11:09:40

This definitely worked !