I'm working in finding the cause for java.lang.NoClassDefFoundError: org/mortbay/util/ajax/JSON$Convertor, when running dse spark job from azure instance to read data from azure blob storage.
Dse version: 6.8/ Spark version: 2.4.0.29/ java : 1.8/ hadoop: 2.10.2(native)
I have tried many options like giving --jars jetty-util-6.1.26.jar, --packages org.mortbay.jetty:jetty-util:6.1.25 and also build a fat jar using mvn-assembly-plugin. But none of them worked when submitting the spark job into node using the spark-submit command.
Then When I gave --driver-class-path jetty-util-6.1.26.jar it worked!. My question is --jars also placing jars to both driver and executors, if --driver-class-path is working then --jars should work, what I'm missing here. Someone please explain. I have a 3 node dse cluster and I submit my java spark job in 3rd node directly with master as local.
working cmd: dse spark-submit --class com.bbl.app.App --driver-class-path jetty-util-6.1.26.jar azure-bbl-1.0-SNAPSHOT-jar-with-dependencies.jar
IGNORE: I have checked the source code of hadoop-azure-pom.xml they changed from mortbay to eclipse in later versions. Also the version 2.10.2 have the mortbay-jetty-util under hadoop-comman which is in compile scope, and in later version 3.3.1 ecplise-jetty-util-ajax is used. However the one inside hadoop-common should work(I used mvn dependency:tree) but spark job couldn't find it.