I am facing serialisation issue when trying to read the binary object file which is stored using scala 2.11.11 with scala 2.12.17. We're migrating our MLpipeline from Spark 2.4(scala 2.11.11) &to Spark 3.3.0(scala 2.12.17) we were not able to read the existing objects with spark 3. This is also because scala won’t support BC with major release upgrade. Seems this is not a Spark issue, As I was able to load the PipelineModel. The issue that I am facing now is we're storing a case class as binary object which is having Map datatype in scala 2.11.11 and trying to read that object file back with scala 2.12.17. While doing that I am getting the below exception,
java.io.InvalidClassException: scala.collection.immutable.Map$Map4; local class incompatible: stream classdesc serialVersionUID = -7746713615561179240, local class serialVersionUID = -7992135791595275193
Sample case class(just adding one column here): case class LearningModelOutput( transformerState: Map[String, Any])
The object that we're storing is like
_1 =transformerState
_2= {Map$Map4}
0 = {Tuple2@8285} (targetColumn,)
1 = {Tuple2@8286} (consideredColumns, [Ljava.lang.String;@35b49c11)
2 = {Tuple2@8287}(schema,StructType(StructField(col1,IntegerType,true),StructField(col2,LongType,true))
3 = {Tuple2@8288} (transformerStages,Map())
When trying to read this objectfile using spark.sparkcontext.objectfileMap[String, Any] getting the above mentioned serialization error.