Spark 2.1 with cluster mode ignores extraJavaOptions

Question

We are running a HortonWorks Spark Distribution 2.1. We noticed that when we run Spark with yarn-cluster as master, whatever we pass as executors.extraJavaOptions is completely ignored, and the job maintains the default values in the executors (as seen through the Spark UI).

I was wondering whether anyone has noticed anything similar and has some wisdom to share. I would be glad to provide any more details needed.

Update: Here is the command executed, along with the options, as requested:

spark-submit --name streaming-driver --num-executors 40 
    --conf spark.metrics.conf=streaming.metrics.properties 
    --files kafka_client_jaas.conf#kafka_client_jaas.conf,
            appconf.conf#appconf.conf,
            streaming.metrics.properties#streaming.metrics.properties,
            kafka.conf#kafka.conf,log4j.properties#log4j.properties 
    --conf "spark.executor.extraJavaOptions=
               -Dcom.sun.management.jmxremote 
               -Dcom.sun.management.jmxremote.port=0 
               -Dcom.sun.management.jmxremote.rmi.port=0 
               -Dcom.sun.management.jmxremote.authenticate=false 
               -Dcom.sun.management.jmxremote.ssl=false 
               -Dcom.sun.management.jmxremote.local.only=false 
               -Djava.security.auth.login.config=./kafka_client_jaas.conf 
               -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode" 
    --conf spark.metrics.conf=streaming.metrics.properties 
    --conf spark.hadoop.yarn.timeline-service.enabled=false 
    --driver-java-options=" 
           -Dhdp.version=2.5.3.0-37 -Dconfig.file=appconf.conf 
           -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode 
           -Dlog4j.configuration=log4j.properties 
           -Djava.security.auth.login.config=./kafka_client_jaas.conf 
           -Dcom.sun.management.jmxremote 
           -Dcom.sun.management.jmxremote.port=0 
           -Dcom.sun.management.jmxremote.rmi.port=0 
           -Dcom.sun.management.jmxremote.authenticate=false 
           -Dcom.sun.management.jmxremote.ssl=false 
           -Dcom.sun.management.jmxremote.local.only=false" 
    --master yarn-cluster 
    --driver-memory "6g" 
    --properties-file spark.conf --class < class> application.jar

Note this is slightly modified because of some other security parameters like keytabs etc. No newline exists as well, put here only so that it is more readable.

Answer 1

Looks like you are wrapping spark.executor.extraJavaOptions proery also inside double inverted quotes. I think only the value should be inside quotes some thing like this:

 --conf spark.executor.extraJavaOptions="
                   -Dcom.sun.management.jmxremote 
                   -Dcom.sun.management.jmxremote.port=0 
                   -Dcom.sun.management.jmxremote.rmi.port=0 
                   -Dcom.sun.management.jmxremote.authenticate=false 
                   -Dcom.sun.management.jmxremote.ssl=false 
                   -Dcom.sun.management.jmxremote.local.only=false 
                   -Djava.security.auth.login.config=./kafka_client_jaas.conf 
                   -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode"