YARN Reserved Memory Issue on Dataproc

Question

When using FIFO scheduler with YARN(FIFO is default right?), I found out YARN reserve some memory/CPU to run the application. Our application doesn't need to reserve any of these, since we want fixed number of cores to do the tasks depending on user's account. This reserved memory makes our calculation inaccurate, so I am wondering if there is any way to solve this. If removing this is not possible, we are trying to scale the cluster(we are using dataproc on GCP), but without graceful decommission, scaling down the cluster is shutting down the job.

1. Is there any way to get rid of reserved memory?

2. If not, is there any way to implement graceful decommission to yarn 2.8.1? I found out cases with 3.0.0 alpha(GCP only has beta version), but couldn't find any working instruction for 2.8.1.'

Thanks in advance!

Answer 1

Regarding 2, Dataproc supports YARN graceful decommissioning because Dataproc 1.2 uses Hadoop 2.8.

Answer 2

Reserved memory is a feature of YARN which allows applications to reserve memory for future use.

Say there are 2 nodes in YARN, each has 4GB memory. App1 is running with 1 container of 1GB, so there are 7 GB left. Then app2 comes and requests 4 * 2GB containers. In this case, app2 might get 3 * 2GB containers allocated, and 1GB reserved. The 1GB is reserved by app2 to prevent it from being used by other apps, so as soon as app1 releases its 1GB, app2 can get 2GB to start a new container.

But in practice, large reserved memory is sometimes an indication of non-optimal container size. Say in the cluster, each node manager is configured with 30GB memory, and each container size is 16GB, so it is not sufficient to pack 2 containers on one node, about half (14GB) of the memory is wasted, which might show up as reserved memory. In this case, you might want to slightly reduce the container size or increase the node manager size, to reduce the waste.