scala migration backward compatibility issue

88 Views Asked by At

I am facing serialisation issue when trying to read the binary object file which is stored using scala 2.11.11 with scala 2.12.17. We're migrating our MLpipeline from Spark 2.4(scala 2.11.11) &to Spark 3.3.0(scala 2.12.17) we were not able to read the existing objects with spark 3. This is also because scala won’t support BC with major release upgrade. Seems this is not a Spark issue, As I was able to load the PipelineModel. The issue that I am facing now is we're storing a case class as binary object which is having Map datatype in scala 2.11.11 and trying to read that object file back with scala 2.12.17. While doing that I am getting the below exception,

java.io.InvalidClassException: scala.collection.immutable.Map$Map4; local class incompatible: stream classdesc serialVersionUID = -7746713615561179240, local class serialVersionUID = -7992135791595275193

Sample case class(just adding one column here): case class LearningModelOutput( transformerState: Map[String, Any])

The object that we're storing is like

_1 =transformerState
_2= {Map$Map4}
    0 = {Tuple2@8285} (targetColumn,)
    1 = {Tuple2@8286} (consideredColumns, [Ljava.lang.String;@35b49c11)
    2 = {Tuple2@8287}(schema,StructType(StructField(col1,IntegerType,true),StructField(col2,LongType,true))
    3 = {Tuple2@8288} (transformerStages,Map())

When trying to read this objectfile using spark.sparkcontext.objectfileMap[String, Any] getting the above mentioned serialization error.

0

There are 0 best solutions below