Cloud Custodian Policy action not working - ec2 start

Question

Below is my custodian policy

policies:
- actions:
  - start
  filters:
  - key: tag:applicationid
    type: value
    value: '7461'
  - key: tag:environment
    type: value
    value: nonprod
  name: ec2-instance
  resource: ec2

Logs after executing this policy is as follows enter image description here

But the ec2 instance didnt start meaning the action didnt work. Can anyone help.

I was trying to run cloud custodian policy which didn't give me any errors but the action of starting ec2 instances didnt happen.

Answer 1

To see more verbose logging you could run the custodian binary with increased logging:

custodian run -s outputs/ custodian.yml --verbose

Happy to follow-up on the output you get to figure out the issue in the comments.

ref: command-line arguments (source): https://github.com/cloud-custodian/cloud-custodian/blob/main/c7n/cli.py#L67

Answer 2

I ran into this issue when the EC2 instance type was of the newer Graviton instances.

Older versions of CloudCustodian prior to the 0.9.21.0 release did not support the Graviton (ARM) starting.

It would say it started but the instance would still be down.

After upgrading to the latest (0.9.29) (as of this post) stopping/starting of the Graviton EC2 instance began to work again (no change in configuration was required).

Edit: After investigating further I discovered that the real reason for the instance not starting was due to the encrypted volume and the Cloud Custodian role not having the permissions to use the KMS key.

Look in the EC2 details for that instance for "State transition reason" the instance was in "Server.Internalerror"

(picture taken from the answer linked below)

Which lead to this answer.

If however, your "State transition reason" is different read about the different states in AWS EC2 Docs