Adding EKS managed windows node group failed. How to debug?

Question

At AWS Console,

1. I created an AWS EKS Node IAM role with following IAM policies:

AmazonEKSWorkerNodePolicy AmazonEKS_CNI_Policy AmazonEC2ContainerRegistryReadOnly

2. I created launch template with the AMI, ami-0e6430de0e2d50a33 (Windows_Server-English-Full-EKS-Optimized-1.16-2020.09.09)

I have an existing eks cluster created by terraform (0.11.13). It has one eks node group. I would like to add a new windows eks node group manually. At AWS console, I went to my eks cluster, clicked on "Add Node Group", use the template above, and clicked on the "Create button". But, I got "Create failed". I have no clue cause of the failure. Where can I find the logs at AWS console?

Answer 1

Not sure where to find those type of logs.

However, here is an AWS CloudFormation template we use to create a self-managed Windows Server 2019 node group that joins the given cluster. Note that it uses spot instances and the worker nodes also join an existing AD.

You will need to either export your EKS cluster name from another CF template or hard-code the value in the UserData property (or pass in your EKS cluster name).

Remove the line 'New-SSMAssociation' line if not joining the AD.

AWSTemplateFormatVersion: 2010-09-09
Description: Creates EC2 instances to support the EKS cluster worker nodes.
Metadata:
  AWS::CloudFormation::Interface:
    ParameterGroups:
    -
        Label:
          default: "EKS Worker Nodes Configuration"
        Parameters:
          - Environment
          - NodeImageIdSSMParam
          - SpotPrice
          - Subnets
          - ActiveDirectoryIdentifier
          - ActiveDirectoryName
          - DesiredCapacity
          - MaxCapacity
          - MinCapacity
Parameters:
  Environment:
    Type: String
    Description: The associated environment of the EKS cluster.
    AllowedValues:
      - preprod
      - prod
  BootstrapArguments:
    Type: String
    Default: ""
    Description: Arguments to pass to the bootstrap script.
  NodeImageIdSSMParam:
    Type: "AWS::SSM::Parameter::Value<AWS::EC2::Image::Id>"
    Default: /aws/service/ami-windows-latest/Windows_Server-2019-English-Core-EKS_Optimized-1.17/image_id
    Description: AWS Systems Manager Parameter Store parameter of the AMI ID for the worker node instances.
  SpotPrice:
    Type: String
    Description: The spot price to bid for the EKS Optimized instances.
    Default: 0.4000
  Subnets:
    Description: Select the PRIVATE subnets where workers can be created.
    Type: List<AWS::EC2::Subnet::Id>
  ActiveDirectoryIdentifier:
    Type: String
    Description: The identifier of the shared Microsoft Managed AD
  ActiveDirectoryName:
    Type: String
    Description: The name of the shared Microsoft Managed AD
  DesiredCapacity:
    Type: Number
    Description: The desired number of EC2 instances for the Autoscaling group.
    Default: 6
  MaxCapacity:
    Type: Number
    Description: The maximum number of EC2 instances for the Autoscaling group.
    Default: 6
  MinCapacity:
    Type: Number
    Description: The minimum number of EC2 instances for the Autoscaling group.
    Default: 6
Resources:
  LaunchConfiguration:
    Type: AWS::AutoScaling::LaunchConfiguration
    Properties:
      BlockDeviceMappings:
        - DeviceName: /dev/sda1
          Ebs:
            DeleteOnTermination: true
            VolumeSize: 50
            VolumeType: gp2
      LaunchConfigurationName: !Sub eks-worker-nodes-windows-${Environment}-launch-config 
      SpotPrice: !Ref SpotPrice
      AssociatePublicIpAddress: false
      ImageId: !Ref NodeImageIdSSMParam
      InstanceType: t3.large
      IamInstanceProfile: !ImportValue eks-worker-instance-profile-arn
      InstanceMonitoring: true
      KeyName: samtec-ec2-key
      SecurityGroups:
        - Fn::ImportValue: !Sub eks-${Environment}-sg   
      UserData:
        Fn::Base64: !Sub
          - |
            <powershell>
            Set-DefaultAWSRegion -Region ${AWS::Region}
            Set-Variable -name instance_id -value (Invoke-Restmethod -uri http://169.254.169.254/latest/meta-data/instance-id)
            New-SSMAssociation -InstanceId $instance_id -Name "awsconfig_Domain_${ActiveDirectoryIdentifier}_${ActiveDirectoryName}"
            [string]$EKSBinDir = "$env:ProgramFiles\Amazon\EKS"
            [string]$EKSBootstrapScriptName = 'Start-EKSBootstrap.ps1'
            [string]$EKSBootstrapScriptFile = "$EKSBinDir\$EKSBootstrapScriptName"
            [string]$cfn_signal = "$env:ProgramFiles\Amazon\cfn-bootstrap\cfn-signal.exe"
            & $EKSBootstrapScriptFile -EKSClusterName ${ClusterName} ${BootstrapArguments} 3>&1 4>&1 5>&1 6>&1
            $LastError = if ($?) { 0 } else { $Error[0].Exception.HResult }
            & $cfn_signal --exit-code=$LastError `
              --stack="${AWS::StackName}" `
              --resource="NodeGroup" `
              --region=${AWS::Region}
            </powershell>
          - ClusterName: 
              'Fn::ImportValue': !Sub 'eks-${Environment}-name'
  AutoScalingGroup:
    Type: AWS::AutoScaling::AutoScalingGroup
    Properties:
      AutoScalingGroupName: !Sub eks-worker-nodes-windows-${Environment}-autoscaler
      Cooldown: 30
      DesiredCapacity: !Ref DesiredCapacity
      HealthCheckGracePeriod: 300
      HealthCheckType: EC2
      LaunchConfigurationName: !Ref LaunchConfiguration
      MaxSize: !Ref MaxCapacity
      MinSize: !Ref MinCapacity
      MetricsCollection:
        - Granularity: 1Minute
      Tags:
        - Key: Name
          Value: !Sub eks-windows-${Environment}-worker
          PropagateAtLaunch: true
        - Key: operating-system
          Value: windows
          PropagateAtLaunch: true          
        - Key: !Sub
          - |
            kubernetes.io/cluster/${ClusterName}
          - ClusterName: 
              'Fn::ImportValue': !Sub 'eks-${Environment}-name' 
          Value: owned
          PropagateAtLaunch: true
        - Key: !Sub
          - |
            k8s.io/cluster-autoscaler/${ClusterName}
          - ClusterName: 
              'Fn::ImportValue': !Sub 'eks-${Environment}-name' 
          Value: owned
          PropagateAtLaunch: true
        - Key: k8s.io/cluster-autoscaler/enabled
          Value: true
          PropagateAtLaunch: true
        - Key: eks:cluster-name
          Value: 
            'Fn::ImportValue': !Sub 'eks-${Environment}-name'
          PropagateAtLaunch: true     
        - Key: eks:nodegroup-name
          Value: 
            'Fn::ImportValue': !Sub 'eks-${Environment}-name'
          PropagateAtLaunch: true           
      VPCZoneIdentifier: !Ref Subnets