When using spark-submit
to submit a Spark app to Yarn, I can pass java options to the driver via the --driver-java-options
, for example:
spark-submit --driver-java-options "-Dlog4j.configuration=file:///conf/log4j.properties" ...
How do I achieve the same when submitting via SparkLauncher
? In particular, is there a way to achieve it with Spark 1.4?
Not familiar with
SparkLauncher
but from looking at the code it appears you can pass configuration withsetConf()
. In this if you add the propertySparkLauncher.DRIVER_EXTRA_JAVA_OPTIONS
this should have the same effect.For example