Targets not getting registered in AWS NLB target groups when creating a load balancer service in EKS

Question

I'm creating a LoadBalancer service in EKS cluster using terraform. The service is getting created as well as the NLB is created too but the targets in the target groups are empty expect one target group. I have total 6 instances in the cluster.

I'm using the below code to create the Load Balancer service from terraform

resource "kubernetes_service" "ml" {
  count = (var.enabled_environments[var.namespace] == true && var.namespace != "prod" && var.namespace != "demo" ? 1 : 0)
  metadata {
    namespace = var.namespace
    name      = "${var.namespace}-xyz-ml-service"
    labels = {
      "app.kubernetes.io/component" = "${var.namespace}-xyz-ml"
    }
    annotations = {
      "service.beta.kubernetes.io/aws-load-balancer-type"                              = "nlb"
      "service.beta.kubernetes.io/aws-load-balancer-nlb-target-type"                   = "instance"
      "service.beta.kubernetes.io/aws-load-balancer-internal"                          = "true"
    }
  }
  spec {
    type = "LoadBalancer"
    port {
      name        = "abc-0"
      port        = 8110
      target_port = 8110
    }
    port {
      name        = "abc-1"
      port        = 8111
      target_port = 8111
    }
    port {
      name        = "abc-2"
      port        = 8112
      target_port = 8112
    }
    port {
      name        = "abc-3"
      port        = 8113
      target_port = 8113
    }
    selector = {
      app = "xyz-ml"
    }
  }
}

Can you let me know what am I missing here?

I tried following these steps https://docs.aws.amazon.com/eks/latest/userguide/network-load-balancing.html

Answer 1

The issue was because of limit for security group rules. So, that is why it was not registering targets. After increasing the security group rules limit it worked fine.

Answer 2

Here are a few things to check:

1. Check your pod's selector. Based on TF code, your service will forward requests to pods with the label "app"="xyz-ml". Double-check to ensure that your pods have this label.

2. Are your pods running? To be targeted by a service, pods must be running and healthy. Check this using the kubectl get pods -n <namespace>

3. Ensure that your pods are listening on these ports: 8110, 8111, 8112, and 8113.

4. If your nodes are not in the correct subnet for the load balancer, they will not be registered as targets.

5. Ensure your pods and services are in the same namespace.

6. Verify that your Network ACLs and Security Groups are not blocking in/out traffic.