Disk pressure on nodes

Question

I have been working with GKE a lot for a while, I created the standard cluster and deployed my image and some dependencies as well. However, it was running fine for sometime and all of a sudden started throwing disk pressure error.

Any thoughts on this? Additionally, I saw image pull back off error.

Answer 1

First check in the logging if you can see the error message (combined from similar events): Failed to garbage collect the required amount of images. Attempted to free 9041880678 bytes, but only found 0 bytes eligible to free. if so, then this message comes from here,indicating that the Kubelet failed to remove the unused images. This could happen if any one of the following conditions is met:

The image must not currently be in use by an existing container running on the node

The image must not have been recently pulled (to avoid thrashing), etc.

Only images that are "eligible" to be removed are actually garbage collected and end up being deleted.

As mentioned in this document, the Node condition 'DiskPressure' implies that available disk space and inodes on either the node's root filesystem or image filesystem have met an eviction threshold.

You may follow the suggestion mentioned here.

Answer 2

As per this document says that the node condition “DiskPressure” signifies that space of the disk on the nodes (on either the node's root filesystem or image filesystem) has satisfied an eviction threshold.

To fix the issue, you may provision new nodepool that has the same settings (labels and taints if any) and machine type but with a higher disk capacity (optionally higher performance e.g. SSD).

You may also check this article for additional information on possible solution in your case.