How to change the timeout value when running commands on QDS

Question

I've a spark-submit command that calls my python script. The code runs more than 36 hours, however because of the QDS timeout limit of 36 hours my command gets killed after 36 hours.

Can someone help me how to change this parameter value to set to 50 hours? This is how I'm calling my script in QDS

/usr/lib/spark/bin/spark-submit s3:///abc.py

Answer 1

We cannot configure the time hour limit to more than 36 hour limit. But we can remove this limit for spark commands.In order to run the Spark application from Analyze/Notebooks, you need to do the following before cluster start:

Edit Cluster configuration and update following configuration in Hadoop Configuration Over-rides

yarn.resourcemanager.app.timeout.minutes=-1

Edit Cluster configuration and update following configuration in Spark Configuration Over-rides

spark.qubole.idle.timeout=-1 

Please let me know if this helps. Also, if you are not running a streaming application, and the data being processed/accessed by your spark app is not humongous, then you may also want to reduce runtime of your app through some performance tuning as well( thereby potentially can reduce the runtime of your app less than 36 hrs) which would not require removing this 36 hour limit in that case.