SAP HANA Vora shell can't find org.apache.spark.launcher.Main

Question

I'm currently doing my first steps with SAP HANA Vora on Cloudera Express 5.5.0.

The Vora server is up and running and I would now like to use the Vora spark shell but this is what I get:

sh start-spark-shell.sh 
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/launcher/Main
Caused by: java.lang.ClassNotFoundException: org.apache.spark.launcher.Main
    at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
Could not find the main class: org.apache.spark.launcher.Main.  Program will exit.

This is how my environment looks like:

export LD_LIBRARY_PATH=/opt/cloudera/parcels/CDH/lib/hadoop/lib/native
export JAVA_HOME=/usr/java/default
export HADOOP_PARCEL_PATH=/opt/cloudera/parcels/CDH
export HADOOP_CONF_DIR=/etc/hadoop/conf
export SPARK_HOME=/usr/lib/spark
export SPARK_CONF_DIR=$SPARK_HOME/conf
export PATH=$PATH:$SPARK_HOME/bin

SPARK_DIST_CLASSPATH=$SPARK_HOME/lib/spark-assembly.jar
SPARK_DIST_CLASSPATH=$SPARK_DIST_CLASSPATH:$HADOOP_PARCEL_PATH/lib/hadoop/lib/*
SPARK_DIST_CLASSPATH=$SPARK_DIST_CLASSPATH:$HADOOP_PARCEL_PATH/lib/hadoop/*
SPARK_DIST_CLASSPATH=$SPARK_DIST_CLASSPATH:$HADOOP_PARCEL_PATH/lib/hadoop-hdfs/lib/*
SPARK_DIST_CLASSPATH=$SPARK_DIST_CLASSPATH:$HADOOP_PARCEL_PATH/lib/hadoop-hdfs/*
SPARK_DIST_CLASSPATH=$SPARK_DIST_CLASSPATH:$HADOOP_PARCEL_PATH/lib/hadoop-mapreduce/lib/*
SPARK_DIST_CLASSPATH=$SPARK_DIST_CLASSPATH:$HADOOP_PARCEL_PATH/lib/hadoop-mapreduce/*
SPARK_DIST_CLASSPATH=$SPARK_DIST_CLASSPATH:$HADOOP_PARCEL_PATH/lib/hadoop-yarn/lib/*
SPARK_DIST_CLASSPATH=$SPARK_DIST_CLASSPATH:$HADOOP_PARCEL_PATH/lib/hadoop-yarn/*
SPARK_DIST_CLASSPATH=$SPARK_DIST_CLASSPATH:$HADOOP_PARCEL_PATH/lib/hive/lib/*
SPARK_DIST_CLASSPATH=$SPARK_DIST_CLASSPATH:$HADOOP_PARCEL_PATH/lib/flume-ng/lib/*
SPARK_DIST_CLASSPATH=$SPARK_DIST_CLASSPATH:$HADOOP_PARCEL_PATH/lib/parquet/lib/*
SPARK_DIST_CLASSPATH=$SPARK_DIST_CLASSPATH:$HADOOP_PARCEL_PATH/lib/avro/lib/*
export SPARK_DIST_CLASSPATH

Answer 1

Solved it.

Just needed to upgrade Java from JDK6 to JDK7. Make sure you have the following environment variables set (check the values with your installation):

export LD_LIBRARY_PATH=/opt/cloudera/parcels/CDH/lib/hadoop/lib/native
export JAVA_HOME=/usr/java/default
export HADOOP_PARCEL_PATH=/opt/cloudera/parcels/CDH
export HADOOP_CONF_DIR=/etc/hadoop/conf
export SPARK_HOME=/usr/lib/spark
export SPARK_CONF_DIR=$SPARK_HOME/conf
export PATH=$PATH:$SPARK_HOME/bin

Answer 2

Thanks for answer https://stackoverflow.com/users/1867854/michael-kunzmann. I found JDK7 already installed under /usr/java/jdk1.7.0_67-cloudera directory. I think it's a step in the Cloudera Manager installation. Minimum supported version is 1.7.0_55 for CDH 5.3, 5.5, 5.6 at time of writing (e.g. see https://www.cloudera.com/documentation/enterprise/5-6-x/topics/cdh_ig_req_supported_versions.html#concept_pdd_kzf_vp)

I was trying with the stanard (non-Vora) spark-shell and got the same issue on CDH. So JDK7 is required for the standard spark-shell too. The Vora spark-shell script..

$VORA_SPARK_HOME/bin/start-spark-shell.sh

...just adds the Vora datasources jar as an additional library.

FYI here's an example for standard spark-shell on Cloudera CDH..

~> cd /usr/java
/usr/java> ls -l
total 8
lrwxrwxrwx 1 lroot root   16 Dec 17  2015 default -> /usr/java/latest
drwxr-xr-x 9 lroot root 4096 Dec 17  2015 jdk1.6.0_31
drwxr-xr-x 8 lroot root 4096 Dec 17  2015 jdk1.7.0_67-cloudera
lrwxrwxrwx 1 lroot root   21 Dec 17  2015 latest -> /usr/java/jdk1.6.0_31
/usr/java> export JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera
/usr/java> export SPARK_HOME=/usr/lib/spark
/usr/java> cd $SPARK_HOME
/usr/lib/spark> ./bin/spark-shell

FYI I also have Vora on HortonWorks. Java 7 was already on the PATH via /usr/bin/java sybolic link and this just worked..

source /etc/vora/vora-env.sh
$VORA_SPARK_HOME/bin/start-spark-shell.sh