Kubernetes: Load Balancer vs Readiness health check

Question

I'm running a WebService backend application in Kubernetes (GKE). It is used only by our frontend Web app. Typically there are sequences of tens of requests coming from the same user (ClientIP). My app is set up to run at least 2 instances ("minReplicas: 2").

The problem: From logs I can see situations when one pod is overloaded (receiving many requests) while the other is idle. Both pods being in Ready state.

My attempt to fix it: I tried to add a custom Readiness health check that returns "Unhealthy" status when there is too many open connections. But even after the health check returned "Unhealthy", load balancer sends further requests to the same pod while the second (healthy) pod is idle.

Here is an excerpt from service.yaml:

kind: Service
metadata:
  annotations:
    networking.gke.io/load-balancer-type: "Internal"
spec:
  type: LoadBalancer
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8080

sessionAffinity is not specified so I expect it is "None"

My questions: What am I doing wrong? Has the Readiness health check any effect on load balancer? How can I control requests distribution?

Additional information:

Cluster creation:

gcloud container --project %PROJECT% clusters create %CLUSTER% 
  --zone "us-east1-b" --release-channel "stable" --machine-type "n1-standard-2" 
  --disk-type "pd-ssd" --disk-size "20" --metadata disable-legacy-endpoints=true 
  --scopes "storage-rw" --num-nodes "1" --enable-stackdriver-kubernetes 
  --enable-ip-alias --network "xxx" --subnetwork "xxx" 
  --cluster-secondary-range-name "xxx" --services-secondary-range-name "xxx" 
  --no-enable-master-authorized-networks 

Node Pool:

gcloud container node-pools create XXX --project %PROJECT% --zone="us-east1-b" 
  --cluster=%CLUSTER% --machine-type=c2-standard-4 --max-pods-per-node=16 
  --num-nodes=1 --disk-type="pd-ssd" --disk-size="10" --scopes="storage-full" 
  --enable-autoscaling --min-nodes=1 --max-nodes=30

Service:

apiVersion: v1
kind: Service
metadata:
  name: XXX
  annotations:
    networking.gke.io/load-balancer-type: "Internal"
  labels:
    app: XXX
    version: v0.1
spec:
  selector:
    app: XXX
    version: v0.1
  type: LoadBalancer
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8080

HPA:

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: XXX
spec:
  scaleTargetRef:
    apiVersion: "apps/v1"
    kind:       Deployment
    name:       XXX
  minReplicas: 2
  maxReplicas: 30
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 40
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 70

Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: XXX
  labels:
    app: XXX
    version: v0.1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: XXX
      version: v0.1
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0

  template:
    metadata:
      labels:
        app: XXX
        version: v0.1
    spec:
      containers:
      - image: XXX
        name: XXX
        imagePullPolicy: Always        
        resources:
          requests:
            memory: "10Gi"
            cpu: "3200m"
          limits:
            memory: "10Gi"
            cpu: "3600m"
        readinessProbe:
          httpGet:
            path: /health/ready
            port: 8080
          initialDelaySeconds: 3
          periodSeconds: 8
          failureThreshold: 3                        
        livenessProbe:
          httpGet:
            path: /health/live
            port: 8080
          initialDelaySeconds: 120
          periodSeconds: 30   
      nodeSelector:
        cloud.google.com/gke-nodepool: XXX

Answer 1

Posting this community wiki answer to extend on the comment that I made on the reproduction steps.

I've reproduced your setup and I couldn't replicate the issue you're having. The request were divided evenly. As for the image I used plain nginx and all of the testing showed the usage/balancing at ~50% (logs from containers, their cpu usage). Could you please check if the same situation happens with nginx image on your setup?

---

The reproduction steps I've followed:

• Run below script that will create a network, subnetworks, cluster and will add a node pool:

project_id="INSERT_PROJECT_ID_HERE"
zone="us-east1-b"
region="us-east1"

gcloud compute networks create vpc-network --project=$project_id --subnet-mode=auto --mtu=1460 --bgp-routing-mode=regional
gcloud compute firewall-rules create vpc-network-allow-icmp --project=$project_id --network=projects/$project_id/global/networks/vpc-network --description=Allows\ ICMP\ connections\ from\ any\ source\ to\ any\ instance\ on\ the\ network. --direction=INGRESS --priority=65534 --source-ranges=0.0.0.0/0 --action=ALLOW --rules=icmp
gcloud compute firewall-rules create vpc-network-allow-internal --project=$project_id --network=projects/$project_id/global/networks/vpc-network --description=Allows\ connections\ from\ any\ source\ in\ the\ network\ IP\ range\ to\ any\ instance\ on\ the\ network\ using\ all\ protocols. --direction=INGRESS --priority=65534 --source-ranges=10.128.0.0/9 --action=ALLOW --rules=all
gcloud compute firewall-rules create vpc-network-allow-rdp --project=$project_id --network=projects/$project_id/global/networks/vpc-network --description=Allows\ RDP\ connections\ from\ any\ source\ to\ any\ instance\ on\ the\ network\ using\ port\ 3389. --direction=INGRESS --priority=65534 --source-ranges=0.0.0.0/0 --action=ALLOW --rules=tcp:3389
gcloud compute firewall-rules create vpc-network-allow-ssh --project=$project_id --network=projects/$project_id/global/networks/vpc-network --description=Allows\ TCP\ connections\ from\ any\ source\ to\ any\ instance\ on\ the\ network\ using\ port\ 22. --direction=INGRESS --priority=65534 --source-ranges=0.0.0.0/0 --action=ALLOW --rules=tcp:22
gcloud compute networks subnets update vpc-network --region=$region --add-secondary-ranges=service-range=10.1.0.0/16,pods-range=10.2.0.0/16
gcloud container --project $project_id clusters create cluster --zone $zone --release-channel "stable" --machine-type "n1-standard-2" --disk-type "pd-ssd" --disk-size "20" --metadata disable-legacy-endpoints=true --scopes "storage-rw" --num-nodes "1" --enable-stackdriver-kubernetes --enable-ip-alias --network "vpc-network" --subnetwork "vpc-network" --cluster-secondary-range-name "pods-range" --services-secondary-range-name "service-range" --no-enable-master-authorized-networks 
gcloud container node-pools create second-pool --project $project_id --zone=$zone --cluster=cluster --machine-type=n1-standard-4 --max-pods-per-node=16 --num-nodes=1 --disk-type="pd-ssd" --disk-size="10" --scopes="storage-full" --enable-autoscaling --min-nodes=1 --max-nodes=5
gcloud container clusters get-credentials cluster --zone=$zone --project=$project_id
# n1-standard-4 used rather than c2-standard-4

• Use following manifest to schedule a workload on the cluster:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - image: nginx
        name: nginx
        imagePullPolicy: Always        
        resources:
          requests:
            memory: "10Gi"
            cpu: "3200m"
          limits:
            memory: "10Gi"
            cpu: "3200m"
      nodeSelector:
        cloud.google.com/gke-nodepool: second-pool
---
apiVersion: v1
kind: Service
metadata:
  name: nginx
  annotations:
    networking.gke.io/load-balancer-type: "Internal"
  labels:
    app: nginx
spec:
  selector:
    app: nginx
  type: LoadBalancer
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80

• $ kubectl get nodes

NAME                                     STATUS   ROLES    AGE     VERSION
gke-cluster-default-pool-XYZ             Ready    <none>   3h25m   v1.18.17-gke.1901
gke-cluster-second-pool-one              Ready    <none>   83m     v1.18.17-gke.1901
gke-cluster-second-pool-two              Ready    <none>   83m     v1.18.17-gke.1901
gke-cluster-second-pool-three            Ready    <none>   167m    v1.18.17-gke.1901

• $ kubectl get pods -o wide

NAME                     READY   STATUS    RESTARTS   AGE   IP          NODE                                    NOMINATED NODE   READINESS GATES
nginx-7db7cf7c77-4ttqb   1/1     Running   0          85m   10.2.1.6    gke-cluster-second-pool-three          <none>           <none>
nginx-7db7cf7c77-dtwc8   1/1     Running   0          85m   10.2.1.34   gke-cluster-second-pool-two            <none>           <none>
nginx-7db7cf7c77-r6wv2   1/1     Running   0          85m   10.2.1.66   gke-cluster-second-pool-one            <none>           <none>

---

The testing was done with a VM in the same zone that has access to the Internal Load Balancer.

The tool/command used:

• $ ab -n 100000 http://INTERNAL_LB_IP_ADDRESS/

The logs showed the requests per pod accordingly:

NAME	Number of requests
nginx-7db7cf7c77-4ttqb	~33454
nginx-7db7cf7c77-dtwc8	~33208
nginx-7db7cf7c77-r6wv2	~33338

---

With the internal load balancer, the traffic should be split evenly between the backends (by default it uses the CONNECTION balancing mode).

There could be many possible reasons on why the traffic is not evenly distributed.

• The replica of an app is not in Ready state.
• The Node is in unhealthy state.
• The application is persisting the connection.

It could be useful to check if the same situation happens in different scenarios (different cluster, different image, etc.).

It could also be a good idea to check the details about the Service and the Pods in the Cloud Console:

• Cloud Console (Web UI) -> Kubernetes Engine -> Services & Ingress -> SERVICE_NAME -> Serving pods

---

Additional resources:

• Cloud.google.com: Kubernetes Engine: Docs: How to: Internal load balancing
• Peterfreitag.com: Using Apache Bench for Simple Load Testing