Spark Submit Default Command line options

309 Views Asked by At

How can we change the parameters in Spark Submit Default Command line options in Qubole.

Though there is a option to override the values if needed under "Spark Submit Command Line Options" but this option is not available in Spark "Command Line".

1

There are 1 best solutions below

0
On

In case of Spark Command Line, you can pass the configuration in the code itself, as below:

/usr/lib/spark/bin/spark-submit --class org.apache.spark.examples.SparkPi -- 
master yarn-client /usr/lib/spark/spark-examples*.jar --conf 
spark.executor.instances=2 --conf spark.logConf=true

Reference Link: https://docs.qubole.com/en/latest/user-guide/analyze/compose-spark.html#compose-a-spark-application-using-the-command-line