Can I change SparkContext.appName on the fly?

Question

I know I can use SparkConf.set('spark.app.name',...) to set appName before creating the SparkContext.

However, I want to change the name of the application as it progresses, i.e., after SparkContext has been created.

Alas, setting sc.appName does not change how the job is shown by yarn application -list.

Is there a way?

Answer 1

This is not possible: any update to the sparkConf, including spark.app.name, is only taken into account before the instance of SparkConf is used to instanciate a SparkContext:

Note that once a SparkConf object is passed to Spark, it is cloned and can no longer be modified by the user. Spark does not support modifying the configuration at runtime.

https://spark.apache.org/docs/1.3.1/api/scala/index.html#org.apache.spark.SparkConf