Kops nginx-Ingress controller fails to create AWS Network Load Balancer due to permission issue

Question

I created my Kubernetes cluster using kops on aws. The cluster has been successfully created.

When I try to deploy the nginx-ingress-controller with Network Load Balancer from AWS, it shows a not authorized error. I am stuck and not sure of what this error indicates.

$ kubectl -n nginx-ingress get service 
NAME                                    TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
lb-ingress-nginx-controller             LoadBalancer   100.65.99.173   <pending>     80:30319/TCP,443:31790/TCP   25m
lb-ingress-nginx-controller-admission   ClusterIP      100.65.34.134   <none>        443/TCP                      25m

$ kubectl -n nginx-ingress get service lb-ingress-nginx-controller -o yaml
apiVersion: v1
kind: Service
metadata:
  annotations:
    meta.helm.sh/release-name: lb
    meta.helm.sh/release-namespace: nginx-ingress
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
    service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
    service.beta.kubernetes.io/aws-load-balancer-type: nlb
  creationTimestamp: "2022-04-07T16:56:28Z"
  finalizers:
  - service.kubernetes.io/load-balancer-cleanup
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: lb
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.1.3
    helm.sh/chart: ingress-nginx-4.0.19
  name: lb-ingress-nginx-controller
  namespace: nginx-ingress
  resourceVersion: "5087"
  uid: bf1a7ae0-6ab4-4164-b739-8d0966ea47d6
spec:
  allocateLoadBalancerNodePorts: true
  clusterIP: 100.65.99.173
  clusterIPs:
  - 100.65.99.173
  externalTrafficPolicy: Cluster
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - appProtocol: http
    name: http
    nodePort: 30319
    port: 80
    protocol: TCP
    targetPort: http
  - appProtocol: https
    name: https
    nodePort: 31790
    port: 443
    protocol: TCP
    targetPort: https
  selector:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: lb
    app.kubernetes.io/name: ingress-nginx
  sessionAffinity: None
  type: LoadBalancer
status:
  loadBalancer: {}

In the events, I can see:

$ kubectl get events -n nginx-ingress 
26m         Warning   SyncLoadBalancerFailed   service/lb-ingress-nginx-controller                 Error syncing load balancer: failed to ensure load balancer: error creating load balancer: "AccessDenied: User: arn:aws:sts::944675846918:assumed-role/masters.kops.example.com/i-01bf8acf72b2ed01d is not authorized to perform: ec2:DescribeInternetGateways\n\tstatus code: 403, request id: 91efdfe4-5c0d-48c5-b38d-5d4c11042c43"
26m         Warning   SyncLoadBalancerFailed   service/lb-ingress-nginx-controller                 Error syncing load balancer: failed to ensure load balancer: error creating load balancer: "AccessDenied: User: arn:aws:sts::944675846918:assumed-role/masters.kops.example.com/i-01bf8acf72b2ed01d is not authorized to perform: ec2:DescribeInternetGateways\n\tstatus code: 403, request id: 30d00f40-1f23-47ef-bda5-8ec255df40fa"
26m         Normal    CREATE                   configmap/lb-ingress-nginx-controller               ConfigMap nginx-ingress/lb-ingress-nginx-controller
26m         Warning   SyncLoadBalancerFailed   service/lb-ingress-nginx-controller                 Error syncing load balancer: failed to ensure load balancer: error creating load balancer: "AccessDenied: User: arn:aws:sts::944675846918:assumed-role/masters.kops.example.com/i-01bf8acf72b2ed01d is not authorized to perform: ec2:DescribeInternetGateways\n\tstatus code: 403, request id: d346cebd-2a17-4682-a425-969d86380159"
25m         Warning   SyncLoadBalancerFailed   service/lb-ingress-nginx-controller                 Error syncing load balancer: failed to ensure load balancer: error creating load balancer: "AccessDenied: User: arn:aws:sts::944675846918:assumed-role/masters.kops.example.com/i-01bf8acf72b2ed01d is not authorized to perform: ec2:DescribeInternetGateways\n\tstatus code: 403, request id: 8942201d-a51d-4464-acc1-edc2db92e455"
25m         Warning   SyncLoadBalancerFailed   service/lb-ingress-nginx-controller                 Error syncing load balancer: failed to ensure load balancer: error creating load balancer: "AccessDenied: User: arn:aws:sts::944675846918:assumed-role/masters.kops.example.com/i-01bf8acf72b2ed01d is not authorized to perform: ec2:DescribeInternetGateways\n\tstatus code: 403, request id: c6992eff-8bcb-4613-b0de-4f51d1642fe8"
23m         Warning   SyncLoadBalancerFailed   service/lb-ingress-nginx-controller                 Error syncing load balancer: failed to ensure load balancer: error creating load balancer: "AccessDenied: User: arn:aws:sts::944675846918:assumed-role/masters.kops.example.com/i-01bf8acf72b2ed01d is not authorized to perform: ec2:DescribeInternetGateways\n\tstatus code: 403, request id: c089e12d-0e81-4c70-ba67-129f9235b0f4"
21m         Warning   SyncLoadBalancerFailed   service/lb-ingress-nginx-controller                 Error syncing load balancer: failed to ensure load balancer: error creating load balancer: "AccessDenied: User: arn:aws:sts::944675846918:assumed-role/masters.kops.example.com/i-01bf8acf72b2ed01d is not authorized to perform: ec2:DescribeInternetGateways\n\tstatus code: 403, request id: ae9c57d6-1d4c-4ec8-b5c1-e47adf681bc5"
16m         Warning   SyncLoadBalancerFailed   service/lb-ingress-nginx-controller                 Error syncing load balancer: failed to ensure load balancer: error creating load balancer: "AccessDenied: User: arn:aws:sts::944675846918:assumed-role/masters.kops.example.com/i-01bf8acf72b2ed01d is not authorized to perform: ec2:DescribeInternetGateways\n\tstatus code: 403, request id: cb3bcc2c-8ff9-4daa-99d9-6c9f1846e9b9"
11m         Warning   SyncLoadBalancerFailed   service/lb-ingress-nginx-controller                 Error syncing load balancer: failed to ensure load balancer: error creating load balancer: "AccessDenied: User: arn:aws:sts::944675846918:assumed-role/masters.kops.example.com/i-01bf8acf72b2ed01d is not authorized to perform: ec2:DescribeInternetGateways\n\tstatus code: 403, request id: 9bb449ee-b245-47c0-bc9b-20694d33ccf4"
69s         Warning   SyncLoadBalancerFailed   service/lb-ingress-nginx-controller                 (combined from similar events): Error syncing load balancer: failed to ensure load balancer: error creating load balancer: "AccessDenied: User: arn:aws:sts::944675846918:assumed-role/masters.kops.example.com/i-01bf8acf72b2ed01d is not authorized to perform: ec2:DescribeInternetGateways\n\tstatus code: 403, request id: 9f771f53-786b-4af2-b4e7-37e289084b3d"

Answer 1

Do you recognize the IAM role masters.kops.example.com? You have a component that's assuming an IAM role name masters.kops.example.com and that role does not have sufficient permissions, specifically for ec2:DescribeInternetGateways.

The kops guide for AWS states that:

the kops user will require the following IAM permissions to function properly:
AmazonEC2FullAccess
AmazonRoute53FullAccess
AmazonS3FullAccess
IAMFullAccess
AmazonVPCFullAccess
AmazonSQSFullAccess
AmazonEventBridgeFullAccess

The ec2:DescribeInternetGateways permission is a subset of the permissions in the AmazonEC2FullAccess managed role.

Did you create the correct IAM role for the kops user?

Answer 2

When creating my nginx-ingress controller on my cluster that was provisioned using kops I ran into the same issue. Two things have to be in place here:

1. The aws profile used by kops to provision the cluster will need the following policy attached to its Permission policies (IAM -> Users -> UserName -> tab Permissions -> Permission policies -> Add permissions button on the right) since the user will be provisioning an Network Load Balancer for the Ingress-Nginx Controller to use:

   ElasticLoadBalancingFullAccess

2. There is a Role in your case called masters.kops.example.com (IAM -> Roles -> Role name: masters.kops.example.com -> tab Permissions -> Permission policies -> click on Policy name masters.kops.example.com in your case) that will need to have its Permissions adjusted to include these permissions in certain part of it:

   "ec2:DescribeVpcs", "ec2:DescribeSecurityGroups", "ec2:DescribeInstances", "elasticloadbalancing:DescribeTargetGroups", "elasticloadbalancing:DescribeTargetHealth", "elasticloadbalancing:ModifyTargetGroup", "elasticloadbalancing:ModifyTargetGroupAttributes", "elasticloadbalancing:RegisterTargets", "elasticloadbalancing:DeregisterTargets"

   So this part will end up looking like this:

   ...
   {
       "Action": [
           "autoscaling:DescribeAutoScalingGroups",
           "autoscaling:DescribeAutoScalingInstances",
           "autoscaling:DescribeLaunchConfigurations",
           "autoscaling:DescribeScalingActivities",
           "autoscaling:DescribeTags",
           "ec2:DescribeAccountAttributes",
           "ec2:DescribeAvailabilityZones",
           "ec2:DescribeInstanceTypes",
           "ec2:DescribeInstances",
           "ec2:DescribeLaunchTemplateVersions",
           "ec2:DescribeRegions",
           "ec2:DescribeRouteTables",
           "ec2:DescribeSecurityGroups",
           "ec2:DescribeSubnets",
           "ec2:DescribeTags",
           "ec2:DescribeVolumes",
           "ec2:DescribeVolumesModifications",
           "ec2:DescribeVpcs",
           "ecr:BatchCheckLayerAvailability",
           "ecr:BatchGetImage",
           "ecr:DescribeRepositories",
           "ecr:GetAuthorizationToken",
           "ecr:GetDownloadUrlForLayer",
           "ecr:GetRepositoryPolicy",
           "ecr:ListImages",
           "elasticloadbalancing:DescribeListeners",
           "elasticloadbalancing:DescribeLoadBalancerAttributes",
           "elasticloadbalancing:DescribeLoadBalancerPolicies",
           "elasticloadbalancing:DescribeLoadBalancers",
           "elasticloadbalancing:DescribeTargetGroups",
           "elasticloadbalancing:DescribeTargetHealth",
           "iam:GetServerCertificate",
           "iam:ListServerCertificates",
           "kms:DescribeKey",
           "kms:GenerateRandom",
           "sqs:DeleteMessage",
           "sqs:ReceiveMessage",
           "ec2:DescribeInternetGateways",
           "iam:CreateServiceLinkedRole",
           "ec2:DescribeAddresses"
       ],
       "Effect": "Allow",
       "Resource": "*"
   },
   ...
   

   Find this part of the policy and edit it like it shows above and your nginx ingress controller will no longer be in Pending state.