Error 403: QUOTA_EXCEEDED - Quota 'NETWORK_ENDPOINT_GROUPS' exceeded. Limit: 100.0 in region europe-west6

Question

I have a GKE environment with a GKE Ingress in europe-west6. It was always working until a couple of days ago when the Ingress started to fail with this error:

Failed to sync NEG "k8s1-484a9780-default-prova-80-0425e497" (will retry): googleapi: Error 403: QUOTA_EXCEEDED - Quota 'NETWORK_ENDPOINT_GROUPS' exceeded. Limit: 100.0 in region europe-west6.

I am pretty confused because the error says that I exceeded the limit of 100 NEGs, but there is only one NEG in the whole project. I tried to load the same environment in us-central1, and it is working as expected.

Are there problems with NEGs in europe-west6?

Thanks a lot Massimo

Answer 1

I found out the problem. It seems that when you create a GKE Ingress via Kubernetes, it creates a NEG for the Ingres. But, when you delete the Ingress, it does not remove the NEG. So I actually collected 100 NEGs! I think this is a very buggy behavior. When I shut my cluster down, I expect all the resources related to such a cluster to be removed. GCP cannot activate resources behind the curtains and leave them there forever! Al least GCP should send an alert to make the user aware that removing the Ingress will not remove the related NEG.

Thanks Massimo

Answer 2

Any time you receive an error like this: QUOTA_EXCEEDED - Quota 'NETWORK_ENDPOINT_GROUPS' exceeded. Limit: 100.0 in region europe-west6, you can check wether or not this is the case by looking up the quotas in your Cloud Console.

As for the Network Endpoint Groups not being deleted when you remove the Ingress - this is working as intended. To avoid leaking NEG, take a look at this document:

With standalone NEGs, you are responsible for managing the lifecycles of NEGs and the resources that make up the load balancer. You could leak NEGs in these ways:

When a GKE service is deleted, the associated NEG will not be garbage collected if the NEG is still referenced by a backend service. Dereference the NEG from the backend service to allow NEG deletion. When a cluster is deleted, standalone NEGs are not deleted.

If you would like this behavior to change, you can suggest it via Feature Request at the Public Issue Tracker. However, there is no guarantee that this will be introduced.