Is there any way to submit spark job using API

Question

I am able to submit spark job on linux server using console. But is there any API or some framework that can enable to submit spark job in linux server?

Answer 1

You can use port 7077 to submit spark jobs in you spark cluster instead of using spark-submit.

val spark = SparkSession
.builder()
.master(spark://master-machine:7077)

Answer 2

you can look into Livy server. It is in GA mode in Hortonworks and Cloudera distros of Apache Hadoop. We have had good success with it. its documentation is good enough to get started with. Spark jobs start instantaneously when submitted via Livy since it has multiple SparkContexts running inside it.