Nodeport Load Balancing Issue despite setting externalTrafficPolicy: Local

Question

I'm currently facing an issue with NodePort and its behavior when externalTrafficPolicy is set to Local. My understanding is that with externalTrafficPolicy: Local, the service should only route traffic to local pods on the node where the service is running, essentially bypassing any external load balancing.

However, in my setup, it seems that NodePort is still performing some form of automatic load balancing across multiple pods by sending the same request to 2 pods as opposed to just one, which is causing severe issues for our application, even though I have explicitly set externalTrafficPolicy: Local.

Here are the relevant details of my setup:

Kubernetes version: v1.26
No. of Pods at the backend : 3

Service definition:

apiVersion: v1
kind: Service
metadata:
  name: my-service
spec:
  selector:
    app: my-app
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080
  type: NodePort
  externalTrafficPolicy: Local
  internalTrafficPolicy: Local

I have also tried using a ClusterIP by setting internalTrafficPolicy: Local, and it behaves as expected by routing traffic to only one pod out of the 3 and does not send any requests to other 2 pods at all, and this is again causing design issues for our application and hence cannot be used.

For design purposes, I can't use LoadBalancer or Ingress, as this routing is intended to be handled by the application running on a new pod in the same namespace.

Is there something I am missing in the configuration, or is this an expected behavior? How can I ensure that the externalTrafficPolicy: Local setting enforces traffic routing only to local pods on the node?

Any insights or suggestions on resolving this issue would be greatly appreciated.

Answer 1

Review worker node iptables rules with iptables-save, and follow the KUBE-SERVICES chain. View packet and byte counts in the verbose output with 'iptables -t nat -L -n -v'. Follow the packet path with the help of the byte counts to see what's happening. Make sure that the IP address of the external test client doesn't overlap with pod subnet (often 192.168.0.0/16) or the iptables rules may follow an unexpected path and do load balancing instead of local processing in the service pod as you would expect with externalTrafficPolicy: Local.