EC2 - Auto Scaling Group - Termination Reason: Client.InvalidKMSKey.InvalidState: The KMS key provided is in an incorrect state

Question

I was trying to setup an ASG in my current AWS architecture, I did configure all the steps correctly, but the ASG does not scale as expected. I was seeing new instances in terminated state in my EC2 console. When I look at the activity logs, I found this message:

Launching a new EC2 instance: i-xxxxxxxxxxxx. Status Reason: Instance became unhealthy while waiting for instance to be in InService state. Termination Reason: Client.InvalidKMSKey.InvalidState: The KMS key provided is in an incorrect state.

Tried looking up what it means but I was not successful. GPT says that it relates to a permissions error on the key that I am using. I created a customer managed key to encrypt my instance's EBS volumes. Here is its policy:

{
"Id": "key-consolepolicy-3",
"Version": "2012-10-17",
"Statement": [
    {
        "Sid": "Enable IAM User Permissions",
        "Effect": "Allow",
        "Principal": {
            "AWS": "arn:aws:iam::303567773654:root"
        },
        "Action": "kms:*",
        "Resource": "*"
    },
    {
        "Sid": "Allow access for Key Administrators",
        "Effect": "Allow",
        "Principal": {
            "AWS": [
                "arn:aws:iam::303567773654:role/aws-service-role/elasticache.amazonaws.com/AWSServiceRoleForElastiCache",
                "arn:aws:iam::303567773654:role/aws-service-role/events.amazonaws.com/AWSServiceRoleForCloudWatchEvents",
                "arn:aws:iam::303567773654:role/c95174a2135306l4753725t1w30356777365-LambdaSLRRole-1P6X3H0TXRS9T",
                "arn:aws:iam::303567773654:role/aws-service-role/organizations.amazonaws.com/AWSServiceRoleForOrganizations",
                "arn:aws:iam::303567773654:role/aws-service-role/trustedadvisor.amazonaws.com/AWSServiceRoleForTrustedAdvisor",
                "arn:aws:iam::303567773654:role/aws-service-role/support.amazonaws.com/AWSServiceRoleForSupport",
                "arn:aws:iam::303567773654:role/EMR_EC2_DefaultRole",
                "arn:aws:iam::303567773654:role/aws-service-role/cloud9.amazonaws.com/AWSServiceRoleForAWSCloud9",
                "arn:aws:iam::303567773654:role/EMR_AutoScaling_DefaultRole",
                "arn:aws:iam::303567773654:role/EMR_DefaultRole"
            ]
        },
        "Action": [
            "kms:Create*",
            "kms:Describe*",
            "kms:Enable*",
            "kms:List*",
            "kms:Put*",
            "kms:Update*",
            "kms:Revoke*",
            "kms:Disable*",
            "kms:Get*",
            "kms:Delete*",
            "kms:TagResource",
            "kms:UntagResource",
            "kms:ScheduleKeyDeletion",
            "kms:CancelKeyDeletion"
        ],
        "Resource": "*"
    }
]

}

Seems like the policy is configured properly. Am I missing something here?

Answer 1

We ran into the same problem. Is the KMS key shared in multiple accounts? Did you try to disable KMS when you created the AMI? Make you you're using the same key when you create the AMI.

Answer 2

Somehow I had this error even when my Launch template did not specify any encryption. :( That said I did need to enable encryption, which necessitated this fix anyway.

As @jordanm said above, the AWSServiceRoleForAutoScaling role needs to have access to the keys, especially customer managed KMS keys.

https://docs.aws.amazon.com/autoscaling/ec2/userguide/key-policy-requirements-EBS-encryption.html

That link provided the following example to be placed in the KMS key policy section (not in IAM)

   "Sid": "Allow service-linked role use of the customer managed key",
   "Effect": "Allow",
   "Principal": {
       "AWS": [
           "arn:aws:iam::account-id:role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling"
       ]
   },
   "Action": [
       "kms:Encrypt",
       "kms:Decrypt",
       "kms:ReEncrypt*",
       "kms:GenerateDataKey*",
       "kms:DescribeKey"
   ],
   "Resource": "*"
}

{
   "Sid": "Allow attachment of persistent resources",
   "Effect": "Allow",
   "Principal": {
       "AWS": [
           "arn:aws:iam::account-id:role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling"
       ]
   },
   "Action": [
       "kms:CreateGrant"
   ],
   "Resource": "*",
   "Condition": {
       "Bool": {
           "kms:GrantIsForAWSResource": true
       }
    }
}```