Invalid Terraform AWS provider credentials when passing AWS system manager parameter store variables

Question

Background:

I'm using an AWS CodeBuild buildspec.yml to iterate through directories from a GitHub repo to apply IaC using Terraform. To access the credentials needed for the Terraform AWS provider, I used AWS system manager parameter store to retrieve the access and secret key within the buildspec.yml.

Problem:

The system manager parameter store masks the access and secret key env value so when they are inherited by the Terraform AWS provider, the provider outputs that the credentials are invalid:

Error: error configuring Terraform AWS Provider: error validating provider credentials: error calling sts:GetCallerIdentity: InvalidClientTokenId: The security token included in the request is invalid.
    status code: 403, request id: xxxx

To reproduce the problem:

• Create system manager parameter store variables (TF_VAR_AWS_ACCESS_KEY_ID=access, TF_AWS_SECRET_ACCESS_KEY=secret)

• Create AWS CodeBuild project with:

 "source": {
                "type": "NO_SOURCE",
}
"environment": {
                "type": "LINUX_CONTAINER",
                "image": "aws/codebuild/standard:4.0",
                "computeType": "BUILD_GENERAL1_SMALL"
}

buildspec.yml with the following: (modified to create .tf files instead of sourcing from github)

version: 0.2
env:
  shell: bash
  parameter-store:
    TF_VAR_AWS_ACCESS_KEY_ID: TF_AWS_ACCESS_KEY_ID
    TF_VAR_AWS_SECRET_ACCESS_KEY: TF_AWS_SECRET_ACCESS_KEY
phases:
  install:
    commands:
      - wget https://releases.hashicorp.com/terraform/0.12.28/terraform_0.12.28_linux_amd64.zip -q
      - unzip terraform_0.12.28_linux_amd64.zip && mv terraform /usr/local/bin/
      - printf "provider "aws" {\n\taccess_key = var.AWS_ACCESS_KEY_ID\n\tsecret_key = var.AWS_SECRET_ACCESS_KEY\n\tversion    = \"~> 3.2.0\"\n}" >> provider.tf
      - printf "variable "AWS_ACCESS_KEY_ID" {}\nvariable "AWS_SECRET_ACCESS_KEY" {}" > vars.tf
      - printf "resource \"aws_s3_bucket\" \"test\" {\n\tbucket = \"test\"\n\tacl = \"private\"\n}" >> s3.tf
      - terraform init
      - terraform plan

Attempts:

1. Passing creds through terraform -vars option:

terraform plan -var="AWS_ACCESS_KEY_ID=$TF_VAR_AWS_ACCESS_KEY_ID" -var="AWS_ACCESS_KEY_ID=$TF_VAR_AWS_SECRET_ACCESS_KEY"

but I get the same invalid credentials error

2. Export system manager parameter store credentials within buildspec.yml:

commands:
  - export AWS_ACCESS_KEY_ID=$TF_VAR_AWS_ACCESS_KEY_ID
  - export AWS_SECRET_ACCESS_KEY=$TF_VAR_AWS_SECRET_ACCESS_KEY

which results in duplicate masked variables and the same error above. printenv output within buildspec.yml:

AWS_ACCESS_KEY_ID=***
TF_VAR_AWS_ACCESS_KEY_ID=***
AWS_SECRET_ACCESS_KEY=***
TF_VAR_AWS_SECRET_ACCESS_KEY=***

Possible solution routes:

• Somehow pass the MASKED parameter store credential values into Terraform successfully (preferred)
• Pass sensitive credentials into the Terraform AWS provider using a different method e.g. AWS secret manager, IAM role, etc.
• Unmask the parameter store variables to pass into the aws provider (probably defeats the purpose of using aws system manager in the first place)

Answer 1

Pass sensitive credentials into the Terraform AWS provider using a different method e.g. AWS secret manager, IAM role, etc.

Generally you wouldn't need to hard-code AWS credentials for terraform to work. Instead CodeBuild IAM role should be enough for terraform, as explain in terraform docs.

Having this in mind, I verified that the following works and creates the bucket requested using terraform from CodeBuild project. The default CB role was modified with S3 permissions to allow creation of the bucket.

version: 0.2
phases:
  install:
    commands:
      - wget https://releases.hashicorp.com/terraform/0.12.28/terraform_0.12.28_linux_amd64.zip -q
      - unzip terraform_0.12.28_linux_amd64.zip && mv terraform /usr/local/bin/
      - printf "resource \"aws_s3_bucket\" \"test\" {\n\tbucket = \"test-43242-efdfdfd-4444334\"\n\tacl = \"private\"\n}" >> s3.tf
      - terraform init
      - terraform plan
      - terraform apply -auto-approve

Answer 2

Well my case was quite foolish but it might help:

So after downloading the .csv file I copy paste the keys with aws configure.

In the middle of the secret key there was a "+". In the editor I use the double click to copy, however will stop when meeting a non alphanumeric character, meaning that only the first part of the secret access key was copied.

Make sure that you had dutifully copied the full secret key.

Answer 3

I had a 403 error. Issue is - you should remove {} from example code.

provider "aws" { 
   access_key = "{YOUR ACCESS KEY}" 
   secret_key = "{YOUR SECRET KEY}"
   region = "eu-west-1"
}

it should look like,

provider "aws" { 
   access_key = "YOUR ACCESS KEY" 
   secret_key = "YOUR SECRET KEY"
   region = "eu-west-1"
}

Answer 4

I experienced this same issue when working with Terraform on Windows.

I had configured the AWS CLI using the aws configure command with an IAM credential for the terraform user I created on AWS.

However, when I run the command: terraform apply

I get the error:

Error: configuring Terraform AWS Provider: validating provider credentials: retrieving caller identity from STS: operation error STS: GetCallerIdentity, https response error StatusCode: 403, RequestID: some_id_xxxx, api error InvalidClientTokenId: The security token included in the request is invalid. 
with provider["registry.terraform.io/hashicorp/aws"], 
on main.tf line 12, in provider "aws": 
12: provider "aws" {

Here's how I fixed it:

The issue was caused as a result of the misconfiguration of my AWS CLI using the aws configure command. I had inputted the AWS Access Key ID and AWS Secret Access Key but there was no option to provide AWS Session Token. But AWS expects the AWS Session Token value to work.

To fix this you can manually add a profile to your AWS credentials file (Short-term credentials). Paste the following text in your AWS credentials file (typically located in ~/.aws/credentials). In windows, under C:\Users\your_user\.aws\credentials

You can find these values under your IAM user

aws_access_key_id=your_acces_key
aws_secret_access_key=your_secret_access_key
aws_session_token=your_aws_session_token

Good luck!

Answer 5

I was also facing this issue, and was unable to authenticate any CLI or application commands to AWS. In order to resolve it, I re-created my IAM user and generated new Access Keys.

Placing these in the /.aws/credentials file then did the trick.

Answer 6

i have faced this issue multiple times the solution is to create user in AWS from IAM Management console and the error will be fixed

Answer 7

I experienced this same issue when working with Terraform on Ubuntu 20.04.

I had configured the AWS CLI using the aws configure command with an IAM credential for the terraform user I created on AWS.

However, when I run the command:

terraform plan

I get the error:

Error: error configuring Terraform AWS Provider: error validating provider credentials: error calling sts:GetCallerIdentity: InvalidClientTokenId: The security token included in the request is invalid. status code: 403, request id: 17268b96-6451-4527-8b17-0312f49eec51

Here's how I fixed it:

The issue was caused as a result of the misconfiguration of my AWS CLI using the aws configure command. I had inputted the AWS Access Key ID where I was to input AWS Secret Access Key and also inputted AWS Secret Access Key where I was to input AWS Access Key ID:

I had to run the command below to re-configure the AWS CLI correctly with an IAM credential for the terraform user I created on AWS:

aws configure

You can confirm that it is fine by running a simple was cli command:

aws s3 ls

If you get an error like the one below, then you know you're still not setup correctly yet:

An error occurred (InvalidAccessKeyId) when calling the ListBuckets operation: The AWS Access Key Id you provided does not exist in our records.

That's all.

I hope this helps