Running same JAR on different machines using spark-submit crashes on one machine

Question

I am trying to run a JAR file using spark-submit on:

1. My local machine (Mac OS)
2. Another local machine (Windows)
3. IBM Cloud Analytics Engine Console

PS: All of them run the same Spark and Scala versions (Spark - 2.3.2 and Scala - 2.11.12)

This is my build.sbt:

name := "CMDW-Security"

version := "19.11.25"

scalaVersion := "2.11.12"

libraryDependencies ++= Seq(
    "org.scalaj" %% "scalaj-http" % "2.4.2",
    "org.scalatest" %% "scalatest" % "3.0.5" % "test",
    "com.typesafe.play" %% "play-json" % "2.7.3",
    "org.passay" % "passay" % "1.5.0"
)

The command to run the jar file I gave everywhere is the same:

spark-submit --master local[*] --deploy-mode client --class com.ibm.cmdwldap.executable.Test <jar-name>.jar

While it ran succesfully on both local machines, it gave an error trying to run on the Analytics Engine spark, with this crash log:

Exception in thread "main" java.lang.NoSuchMethodError: play.api.libs.json.JsLookup$.apply$extension1(Lplay/api/libs/json/JsLookupResult;Ljava/lang/String;)Lplay/api/libs/json/JsValue;
    at com.ibm.cmdwldap.restapi.Authorization.getAuthToken(Authorization.scala:41)
    at com.ibm.cmdwldap.executable.Test$.main(Test.scala:18)
    at com.ibm.cmdwldap.executable.Test.main(Test.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:904)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:198)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:228)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

I'm not sure as to what is causing this error on Analytics Engine, given that the same JAR file ran on the local machines using spark-submit.

I also don't need to specify the library JARs as all of them have got compiled in the executable JAR when I ran sbt assembly command.

EDIT 1: Also need to mention that, when I run with

java -jar <name of jar>

the program executes successfully on all 3 machines, but when executing with spark-submit, it fails on the Analytics Engine. I am looking for scheduling with Oozie that's why it is important for me to get spark-submit working for my job on Analytics Engine.

Answer 1

For the job to run on local machine, you might need to give master Local[*] in spark submit, but when you run on other machine, this command can be replaced with master yarn i guess. This worked for me.