I have written a following Spark program in Scala. I exported it as JAR file. When I run it in Cloudera Spark cluster using spark-submit command, it is giving me Class Not found error.
import org.apache.spark.SparkConf
import org.apache.spark.sql.SaveMode
import org.apache.spark.sql.SparkSession
object AirflowFileReader extends App {
val sparkConf = new SparkConf()
sparkConf.set("spark.app.name" , "load and filter a csv file")
sparkConf.set("spark.master" , "local[2]")
val spark = SparkSession.builder()
.config(sparkConf)
.getOrCreate()
val ordersDf = spark.read
.format("csv")
.option("header", true)
.option("inferSchema", true)
.option("path", "airflow_input_nw/orders.csv")
.load();
val ordersDf1=spark.sql("select * from orders where order_status='CLOSED'")
ordersDf1.write.format("csv").mode(SaveMode.Overwrite).option("path","airflow_output").save()
}
Command to execute in Cloudera Quickstart terminal -
spark-submit --class AirflowFileReader /home/cloudera/sparkpractice.jar
Error -
[root@quickstart cloudera]# spark-submit --class AirflowFileReader
/home/cloudera/sparkpractice.jar Exception in thread "main"
java.lang.NoClassDefFoundError: org/apache/spark/sql/SparkSession at
java.lang.Class.getDeclaredMethods0(Native Method) at
java.lang.Class.privateGetDeclaredMethods(Class.java:2570) at
java.lang.Class.getMethod0(Class.java:2813) at
java.lang.Class.getMethod(Class.java:1663) at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:715)
at
org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused
by: java.lang.ClassNotFoundException:
org.apache.spark.sql.SparkSession at
java.net.URLClassLoader$1.run(URLClassLoader.java:366) at
java.net.URLClassLoader$1.run(URLClassLoader.java:355) at
java.security.AccessController.doPrivileged(Native Method) at
java.net.URLClassLoader.findClass(URLClassLoader.java:354) at
java.lang.ClassLoader.loadClass(ClassLoader.java:425) at
java.lang.ClassLoader.loadClass(ClassLoader.java:358) ... 9 more