Terraform Module Import - Google BigQuery

223 Views Asked by At

I'm working on bringing BigQuery datasets and tables into Terraform using the "terraform import module" This is how the code looks like:

main.tf:

module "bigquery" { 
  source          = "./modules/bigquery"
  for_each        = var.dataset_config
  region          = var.region
  project_id      = var.PROJECT_ID
  bq_dataset      = each.key
  bq_tables_views = each.value.table_and_views
  bq_tables       = flatten([for k, v in each.value.table_and_views : v.table])
  bigquery_labels = each.value.labels
  dataset_config  = var.dataset_config
  env             = var.ENV
  view_path       = "./environments/environment" 
}

variables.tfvars:

dataset_config = {
"raw" = {
    "table_and_views" = [],
    "labels" = {}
    },
  "dataset1" = {
    "table_and_views" = [
       { "table" = "t_tabel1", "table_schema" = "tables.table1.json", "partition_type" = "DAY", "partition_field": "ingestion_date",  "views" = [], "view_inputs" = {} }
         ],
    "labels" = {}
  },
  "my_dataset" = {
    "table_and_views" = [
      { "table" = "t_tabel1", "table_schema" = "tables.table1.json", "partition_type" = "DAY", "partition_field": "ingestion_date",  "views" = [], "view_inputs" = {} }
    ],
    "labels" = {}
    }
}

modules/bigquery/bigquery.tf

resource "google_bigquery_dataset" "dataset" {
  dataset_id                 = var.bq_dataset
  location                   = var.region
  project                    = var.project_id
  delete_contents_on_destroy=true
  labels = merge(
    var.bigquery_labels,
    {
      "env" = "dev"
    }
  ) 
} 

I am using the following command to import the module:

terraform import -var-file="./environments/environment/variables.tfvars" google_bigquery_dataset.dataset  prj-dev-2342/my_dataset 

I encounter an issue where the logs display the message "ReferenceTransformer: reference not found: 'each.value'," and consequently, the import operation does not import any resources. It appears that the variables have not been properly initialized. However, when I execute Terraform normally, it successfully creates all the required resources. The problem only arises when attempting to perform an import operation, where the actual values of these variables cannot be retrieved.

I have a lot of tables and datasets so I am wondering if there any other approach to import the module with respect to my current terraform structure?

Thank you in advance

0

There are 0 best solutions below