We are using CDP private cloud 7.1.7 and have configured Hue connector for pyspark using livy. By default I can see the driver launches with 1GB memory and I need to increase this as some of the code we are running is failing due to OOM errors for driver container. I tried setting the default spark driver memory to a higher value under spark config which works fine with spark-submit or directly using livy api. But When I launch a livy session via Hue its still launching with 1gb.
Any idea how I can apply this via Hue?
Thanks, Sajin
I tried setting the default spark driver memory to a higher value under spark config which works fine with spark-submit or directly using livy api. But When I launch a pyspark session via Hue its still launching with 1gb.