I am running a spark standalone cluster in which I have 3 nodes. One is master and other two are workers. I copied the job in all three nodes at location /home/bizruntime/***.jar in all three and when I submit the job using spark-submit in master the job is running.
But my cluster is on remote location and I can not access master node terminal every time. So I think of using Spark Launcher to trigger the job form my laptop which is a Windows machine. But when I run my spark launcher program in Eclipse in my laptop then it starts a driver which I can see in Spark UI but then it gives error. When I check the error in log it because of not finding jar in nodes.
The path which I am giving in appResource() function Spark launcher is /home/bizruntime/***.jar this. But they are searching the jar in $Eclipse_Worksapce/home/bizruntime/***.jar
My question is how to give absolute path in it without $Eclipse_Worksapce.
package com.biz.Remote_Start;
import java.io.IOException;
import org.apache.spark.launcher.SparkAppHandle;
import org.apache.spark.launcher.SparkLauncher;
/**
* Hello world!
*
*/
public class App {
public static void main(String[] args) {
try {
Process handle = new SparkLauncher()
.setAppResource("home/bizruntime/Read_from_ADL-0.0.1-SNAPSHOT.jar")
.setMainClass("com.biz.Read_from_ADL.App")
.setMaster("spark://192.168.1.154:7077")
.setDeployMode("cluster")
.launch();
handle.waitFor();
} catch (Exception e) {
System.out.println(e.getMessage());
}
}
}
Incase jar file exists in local file system, please specify the the location like file:///home/bizruntime/Read_from_ADL-0.0.1-SNAPSHOT.jar