terraform for_each implementation with values from .tfvars

Question

I have a common.tfvars file with definition of a variables as:

bqtable_date_partition = [
  { dataset = "d1", table_name = "d1-t1", part_col = "partition_date",
  part_type = "DAY", schema_file = "data_tables/d1-t1.json" },

  { dataset = "d1", table_name = "d1-t2", part_col = "tran_dt",
  part_type = "DAY", schema_file = "data_tables/d1-t2.json" },

  { dataset = "d2", table_name = "d2-t1", part_col = "tran_dt",
  part_type = "DAY", schema_file = "data_tables/d2-t1.json" },
]

and I am referencing this var in main.tf file with following resource defintion:

resource "google_bigquery_table" "bq_tables_dt_pt" {
  count      = length(var.bqtable_date_partition)
  project    = var.project_id
  dataset_id = "${var.bqtable_date_partition[count.index].dataset}_${var.env}"
  table_id   = var.bqtable_date_partition[count.index].table_name
  time_partitioning {
    type  = var.bqtable_date_partition[count.index].part_type
    field = var.bqtable_date_partition[count.index].part_col
  }
  schema     = file("${path.module}/tables/${var.bqtable_date_partition[count.index].schema_file}")
  depends_on = [google_bigquery_dataset.crte_bq_dataset]
  labels = {
    env        = var.env
    ind        = "corp"
  }
}

I want to change the resource definition to use "for_each" instead of "count" to loop through the list:

My motive to change from count to for_each is to eliminate the dependency on the order in which I have written the elements of the variable "bqtable_date_partition "

I did this:

resource "google_bigquery_table" "bq_tables_dt_pt" {
  for_each   = var.bqtable_date_partition
  project    = var.project_id
  dataset_id = "${each.value.dataset}_${var.env}"
  table_id   = each.value.table_name
  time_partitioning {
    type  = each.value.part_type
    field = each.value.part_col
  }
  schema     = file("${path.module}/tables/${each.value.schema_file}")
  depends_on = [google_bigquery_dataset.crte_bq_dataset]
  labels = {
    env        = var.env
    ind        = "corp"
  }
}

I got the following error as expected:

The given "for_each" argument value is unsuitable: the "for_each" argument must be a map or set of strings, and you have provided a value of type list of map of string.

Can anyone help me with what changes I need do to make in the resource definition to use "for_each"?

Terraform version - 0.14.x

Answer 1

Error says it only accepts the map or set of strings. So we have to convert our input variable to either map or set of strings.

https://www.terraform.io/docs/language/expressions/for.html

resource "google_bigquery_table" "bq_tables_dt_pt" {
  for_each   = { for index, data_partition in var.bqtable_date_partition : index => data_partition }
  project    = var.project_id
  dataset_id = "${each.value.dataset}_${var.env}"
  table_id   = each.value.table_name
  time_partitioning {
    type  = each.value.part_type
    field = each.value.part_col
  }
  schema     = file("${path.module}/tables/${each.value.schema_file}")
  depends_on = [google_bigquery_dataset.crte_bq_dataset]
  labels = {
    env = var.env
    ind = "corp"
  }
}

So basically, here we are converting for_each input into the following format. and only referencing value in from newly created map.

{
  "0" = {
    "dataset" = "d1"
    "part_col" = "partition_date"
    "part_type" = "DAY"
    "schema_file" = "data_tables/d1-t1.json"
    "table_name" = "d1-t1"
  }
  "1" = {
    "dataset" = "d1"
    "part_col" = "tran_dt"
    "part_type" = "DAY"
    "schema_file" = "data_tables/d1-t2.json"
    "table_name" = "d1-t2"
  }
  "2" = {
    "dataset" = "d2"
    "part_col" = "tran_dt"
    "part_type" = "DAY"
    "schema_file" = "data_tables/d2-t1.json"
    "table_name" = "d2-t1"
  }
}

Answer 2

There are two main requirements for using for_each:

• You must have a collection that has one element for each resource instance you want to declare.
• There must be some way to derive a unique identifier from each element of that collection which Terraform will then use as the unique instance key.

It seems like your collection meets both of these criteria, assuming that table_name is a unique string across all of those values, and so all that remains is to project the collection into a map so that Terraform can see from the keys that you intend to use the table_name for the unique tracking keys:

resource "google_bigquery_table" "bq_tables_dt_pt" {
  for_each = {
    for o in var.bqtable_date_partition : o.table_name => o
  }

  # ...
}

Here I've used a for expression to project from a sequence to a mapping, where each element is identified by the value in its table_name attribute.

---

If you are in a situation where you're able to change the interface to this module then you could simplify things by changing the variable's declaration to expect a map instead of a list, which would then avoid the need for the projection and make it explicit to the module caller that the table IDs must be unique:

variable "bqtable_date_partition" {
  type = map(object({
    dataset     = string
    part_col    = string
    part_type   = string
    schema_file = string
  }))
}

Then you could just assign var.bqtable_date_partition directly to for_each as you tried before, because it'll already be of a suitable type. But would also require changing your calling module to pass a map value instead of a list value, and so this might not be practical if your module has many callers that would all need to be updated to remain compatible.