Deploying resource to multiple regions w/ TF 0.12/13

Question

We have a rather complex environment where we have lots of AWS accounts, in multiple regions and these are all connected to a transit network via VPN tunnels.

At the moment we deploy Customer Gateways via a "VPC" module for each VPC in a region but the problem that we get is that deploying the first VPC is fine but subsequent VPC deploys cause issues with the fact that the CGW is already there and so we have to import it before we can continue which isn't an ideal place to be in and also I think there's a risk that if we tear down a VPC it might try to kill the CGW that is being used by other VPN's.

What I'm wanting to do is deploy the CGW's separately from the VPC and then the VPC does a data lookup for the CGW.

I've been thinking that perhaps we can use our "base" job to provision the CGW's that are defined in the variables file but nothing I've tried has worked so far.

The variable definition would be:

variable "region_data" {
  type = list(object({
    region = string
    deploy_cgw = bool
    gateways = any
  }))

  default = [
    {
      region = "eu-west-1"
      deploy_cgw = true
      gateways = [
        {
          name = "gateway1"
          ip   = "1.2.3.4"
        },
        {
          name = "gateway2"
          ip   = "2.3.4.5"
        }
      ]
    },
    {
      region = "us-east-1"
      deploy_cgw = true
      gateways = [
        {
          name = "gateway1"
          ip   = "2.3.4.5"
        },
        {
          name = "gateway2"
          ip   = "3.4.5.6"
        }
      ]
    }
  ]
}

I've tried a few things, like:

locals {
  regions = [for region in var.region_data : region if region.deploy_cgw]
  cgws    = flatten([
    for region in local.regions : [
      for gateway in region.gateways : {
        region = region.region
        name = gateway.name
        ip = gateway.ip
      }
    ]
  ])
}

provider "aws" {
  region = "eu-west-1"
  alias = "eu-west-1"
}

provider "aws" {
  region = "us-east-1"
  alias = "us-east-1"
}

module "cgw" {
  source = "../../../modules/customer-gateway"

  for_each = { for cgw in local.cgws: "${cgw.region}.${cgw.name}" => cgw }
  name_tag = each.value.name
  ip_address = each.value.ip

  providers = {
    aws = "aws.${each.value.region}"
  }
}

But with this I get:

Error: Invalid provider configuration reference

 on main.tf line 439, in module "cgw":
 439:     aws = "aws.${each.value.region}"

A provider configuration reference must not be given in quotes.

If I move the AWS provider into the module and pass the region as a parameter, I get the following:

Error: Module does not support for_each

  on main.tf line 423, in module "cgw":
 423:   for_each   = { for cgw in local.testing : "${cgw.region}.${cgw.name}" => cgw }

Module "cgw" cannot be used with for_each because it contains a nested
provider configuration for "aws", at

I've done quite a bit of research and the last one I understand is something that Terraform take a tough stance on.

Is what I'm asking possible?

Answer 1

for_each can't be used on modules that have providers defined within them. I was disappointed to find this out too. They do this because having nested providers does cause nightmares if that provider goes away, then you have orphaned resources in the state that you can't manage and your plans will fail. It is, however, entirely possible in https://www.pulumi.com/. I'm sick of the limitations in terraform and will be moving to pulumi. But that's not what you asked so I'll move on.

Definitely don't keep importing it. You'll end up with multiple parts of your terraform managing the same resource.

Just create the cgw once per region. Then pass the id into your vpc module. You can't iterate over providers, so have one module per provider. In other words, for each over all vpcs in the same account and same region per module call.

resource "aws_customer_gateway" "east" {
  bgp_asn    = 65000
  ip_address = "172.83.124.10"
  type       = "ipsec.1"
}
resource "aws_customer_gateway" "west" {
  bgp_asn    = 65000
  ip_address = "172.83.128.10"
  type       = "ipsec.1"
}


module "east" {
  source = "../../../modules/customer-gateway"

  for_each = map(
    {
      name = "east1"
      ip = "1.2.3.4"
    },
    {
      name = "east2"
      ip = "1.2.3.5"
    },
  )
  name_tag = each.value.name
  ip_address = each.value.ip
  cgw_id = aws_customer_gateway.east.id

  providers = {
    aws = "aws.east"
  }
}

module "west" {
  source = "../../../modules/customer-gateway"

  for_each = map(
    {
      name = "west1"
      ip = "1.2.3.4"
    },
    {
      name = "west2"
      ip = "1.2.3.5"
    },
  )
  name_tag = each.value.name
  ip_address = each.value.ip
  cgw_id = aws_customer_gateway.west.id

  providers = {
    aws = "aws.west"
  }
}