I have a Spark job which reads data from MongoDB, everything is fine when I connect to MongoDB 3.x, But when I upgrade to MongoDB 4.x, runtime error happens.
now my spark & casbah version is as follows(all are of latest version):
- spark-core_2.12:3.0.1
- spark-sql_2.12:3.0.1
- mongo-spark-connector_2.12:3.0.0
- casbah_2.12:3.2.1
but I got errors when running the spark job connecting to MongoDB 4.0.5:
Exception in thread "main" java.lang.NoClassDefFoundError: com/mongodb/MongoOptions
at com.mongodb.casbah.MongoOptions$.<init>(MongoOptions.scala:40)
at com.mongodb.casbah.MongoOptions$.<clinit>(MongoOptions.scala)
at com.mongodb.casbah.BaseImports.$init$(Implicits.scala:154)
at com.mongodb.casbah.Imports$.<init>(Implicits.scala:141)
at com.mongodb.casbah.Imports$.<clinit>(Implicits.scala)
at com.mongodb.casbah.MongoClient.apply(MongoClient.scala:218)
Seems casbah needs MongoOptions, but this class has been removed from MongoDB 4.x. I noticed the casbah version has not upgraded since Dec 06, 2016, it stays in 3.2.1 for 3 years.
Does anyone has any suggestions?