Apache Spark Mongo-Hadoop Connector class not found

Question

So im trying to run this example https://github.com/plaa/mongo-spark/blob/master/src/main/scala/ScalaWordCount.scala

But i keep getting this error

 Exception in thread "main" java.lang.NoClassDefFoundError: com/mongodb/hadoop/MongoInputFormat
at ScalaWordCount$.main(ScalaWordCount.scala:27)

Im not sure why its having a hard time finding the class. I built the project with maven and it seems to be building fine.

      /usr/local/spark/bin/spark-submit \
--class ScalaWordCount \
--master local target/scalawordcount-0.0.1-SNAPSHOT.jar  \
    --jars /home/daniel/.m2/repository/org/mongodb/mongo-java-driver/2.12.3 \/mongo-java-driver-2.12.3.jar, \
    /home/daniel/mongo-hadoop/core/build/libs/mongo-hadoop-core-1.3.3-SNAPSHOT.jar

This is the command i am using to run it. Im working within my home directory. Thanks in advance.

I used this tutorial https://github.com/crcsmnky/mongodb-spark-demo to set up mongodb with apache spark

ps ive read a few things online about there being a bug in the class path that will be fixed in a newer realese...

Answer 1

I just add the jars path to spark-env.sh of SPARK_CLASSPATH. I know it's not a good solution, but it works.