Configuring Spark logging with log4j.xml

Question

I guesss its a silly question, but I couldnt find answer anywhere.

Can I configure logging in spark using log4j.xml?

In spark documentation its mentioned you can configure logging with log4j.properties, I wish to use log4j.xml for more advance log4j capabilities such as async appender. my job will run in cluster mode over yarn (CDH) scheduled with oozie. Im aware ill need in any solution to use --files

Answer 1

You can set the spark.executor.extraJavaOptions to add -Dlog4j.configuration=log4j.xml and include your log4j.xml file on the classpath of the workers (either bundling in the application jar or adding to the files included with --files).