How to override spark conf in OLP

Question

How can I use the following flags to run a spark job in OLP?

--conf spark.driver.userClassPathFirst=true
--conf spark.executor.userClassPathFirst=true

Answer 1

OLP currently do not support setting custom Spark properties. If you are running into dependency conflicts (often what triggers this question), you will need to find another way to resolve them such as shading (when building the deployed fat jar).

Answer 2

Using a shaded jar did help, and yes it was a dependency issue, since I was using a higher version of the commons lib.

I had to put my commons lib dependency before any other conflicting dependency would be included, incase anyone else faces this issue, they could try that to resolve conflicts.