In Private GKE Cluster achieve dedicated public IP as source IP for each pod for outgoing traffic

Question

Requirement : With private GKE ( version : 1.21.11-gke.1100 ), each pod is required to have a dedicated public IP as source IP when reaching to internet. It is not required for ingress but only for egress.

Solution tried : Cloud NAT. Works partially. Meaning, suppose we have 10 pods and each of them is made to run on a distinct node. Cloud NAT does not assign an unique IP to each pod even when the Minimum ports per VM instance is set to the maximum possible value of 57344.

Experiment Done: 10 NAT gateway IPs are assigned to the NAT Gateway. 8 pods are created, each running on a dedicated node. Cloud NAT assigned only 3 Cloud NAT IPs instead of 8 even though there aee 10 IPs available.

Cloud NAT is configured as below :

Configuration	Setting
Manual NAT IP address assignment	true
Dynamic port allocation	disabled
Minimum ports per VM instance	57344. This decides how many VMs can be assigned to the same Cloud NAT IP.
Endpoint-Independent Mapping	disabled

Instead of converting to a Public GKE cluster, is there an easier way of achieving this goal?

Has anyone ever done such a setup which is proved to work?

Answer 1

You can create the NAT gateway instance and forward the traffic from there.

Here terraform script to create : https://github.com/GoogleCloudPlatform/terraform-google-nat-gateway/tree/master/examples

https://medium.com/google-cloud/using-cloud-nat-with-gke-cluster-c82364546d9e

If you are looking to use cloud NAT with route you can checkout this : https://github.com/GoogleCloudPlatform/gke-private-cluster-demo/blob/master/README.md#private-clusters

TF code for NAT : https://github.com/GoogleCloudPlatform/gke-private-cluster-demo/blob/master/terraform/network.tf#L84

Demo architecture : https://github.com/GoogleCloudPlatform/gke-private-cluster-demo/blob/master/README.md#demo-architecture

Answer 2

That's expected behavior because that's what NAT does. Network Address Translation will always hide the Private IP Address of whatever is behind it (In this case a Pod or Node IP) And will forward traffic to the Internet using the Public NAT IP. Return traffic goes back to the Public NAT IP which knows to where Pod route the traffic back.

In other terms you have no ways using Managed Cloud NAT to ensure each pod in your cluster will get a Unique Public IP on Egress.

The only i can see to solve this is to:

• Create a Public GKE cluster with 10 nodes (following your example) and using taints, tolerations and node selector run each pod on a dedicated node, this way when the pod Egress to the internet, it will use the Node Public IP.
• Create a Multi-NIC GCE instance, deploy some proxy on it (HA proxy for example) and configure it to somewhere route Egress traffic using one of the Interfaces for each of the pods behind (Note that a multi-Nic node can only have 8 Interfaces).