I am running an EMR cluster with 25 nodes and a job is running on for continuously 15-20 minutes. I realized that it would take another 30-40 minutes so I added 10 more task nodes in the cluster. But I didn't see any improvement.
Would adding these extra task nodes increase the concurrent mappers execution in current job or the extra nodes will be used for only the jobs initiated after resizing?
It depends on how the MapReduce job is structured. If there was tasks pending then the additional nodes will be used. If the job determined its number of tasks based on original size of the cluster then additional nodes will not make a difference until new job submission.