I am new to EMR-serverless. I was able to successfully run a Pyspark job.
But when I tried running the spark-scala jar and submitted the job-run, the executors are unable to be launched. Below is the error I am encountering :
Executor removed. Reason : Unable to create executor due to : Unable to register with external shuffle server.
This error definitely is related to the Spark dynamic allocation where external shuffle service is required to be turned on. But I am unable to get the fix for this solution.
Would be great if someone could help me in the same. Thanks.