Is there a way to define multiple source_file for Terraform archive provider?

Question

I am using the Terraform archive_file provider to package multiple files into a zip file. It works fine when I define the archive like this:

data "archive_file" "archive" {
  type        = "zip"
  output_path = "./${var.name}.zip"
  source_dir  = "${var.source_dir}"
}

However I don't want the archive to contain all of the files in var.source_dir, I only want a subset of them. I notice the archive_file provider has a source_file attribute so I was hoping I could supply a list of those files and package them into the archive like so:

locals {
  source_files = ["${var.source_dir}/foo.txt", "${var.source_dir}/bar.txt"]
}

data "archive_file" "archive" {
  type        = "zip"
  output_path = "./${var.name}.zip"
  count       = "2"
  source_file = "${local.source_files[count.index]}"
}

but that doesn't work, the archive gets built for each file defined in local.source-files hence I have a "last one wins" scenario where the archive file that gets built only contains bar.txt.

I tried this:

locals {
  source_files = ["${var.source_dir}/main.py", "${var.source_dir}/requirements.txt"]
}

data "archive_file" "archive" {
  type        = "zip"
  output_path = "./${var.name}.zip"
  source_file = "${local.source_files}"
}

but unsurprisingly that failed with:

data.archive_file.archive: source_file must be a single value, not a list

Is there a way to achieve what I'm after here i.e pass a list of files to the archive_file provider and have it package all of them into the archive file?

Answer 1

---- Thanks jamiet, I modified as your comment ----

1. copy files to temp dir and archive them

locals {
  source_files = ["${var.source_dir}/main.py", "${var.source_dir}/requirements.txt"]
}

data "template_file" "t_file" {
  count = "${length(local.source_files)}"

  template = "${file(element(local.source_files, count.index))}"
}

resource "local_file" "to_temp_dir" {
  count    = "${length(local.source_files)}"
  filename = "${path.module}/temp/${basename(element(local.source_files, count.index))}"
  content  = "${element(data.template_file.t_file.*.rendered, count.index)}"
}

data "archive_file" "archive" {
  type        = "zip"
  output_path = "${path.module}/${var.name}.zip"
  source_dir  = "${path.module}/temp"

  depends_on = [
    "local_file.to_temp_dir",
  ]
}

2. use source of archive_file

locals {
  source_files = ["${var.source_dir}/main.py", "${var.source_dir}/requirements.txt"]
}

data "template_file" "t_file" {
  count = "${length(local.source_files)}"

  template = "${file(element(local.source_files, count.index))}"
}


data "archive_file" "archive" {
  type        = "zip"
  output_path = "./${var.name}.zip"

  source {
    filename = "${basename(local.source_files[0])}"
    content  = "${data.template_file.t_file.0.rendered}"
  }

  source {
    filename = "${basename(local.source_files[1])}"
    content  = "${data.template_file.t_file.1.rendered}"
  }
}

3. create shell script and call it using external data resource.

locals {
  source_files = ["${var.source_dir}/main.py", "${var.source_dir}/requirements.txt"]
}

data "template_file" "zip_sh" {
  template = <<EOF
#!/bin/bash
zip $* %1>/dev/null %2>/dev/null
echo '{"result":"success"}'
EOF
}

resource "local_file" "zip_sh" {
  filename = "${path.module}/zip.sh"
  content  = "${data.template_file.zip_sh.rendered}"
}

data "external" "zip_sh" {
  program = ["${local_file.zip_sh.filename}", "${var.name}", "${join(" ", local.source_files)}"]

  depends_on = [
    "data.template_file.zip_sh",
  ]
}

Answer 2

I have a little modified version of Ripe in use:

I am using fileset to loop over a list of patterns to find excluded files i.e. *.pyc.

data "archive_file" "archive" {
  output_path = "${path.module}/${var.name}.zip"
  source_dir  = "${var.source_dir}"
  excludes    = toset(flatten([
    for pattern in var.excluded_files : [
      fileset("${local.base_path}", pattern)
    ]
  ]))
}

variable "excluded_files" {
  description = "exclude this files from the zip"
  type        = list(string)
  default = [
    "**/*.pyc",
  ]
}

Answer 3

Well theres another compact way to do is using excludes parameter

data "archive_file" "archive" {
  type        = "zip"
  output_path = "${path.module}/${var.name}.zip"
  source_dir  = "${path.module}/${var.source_dir}/"
  excludes    = setsubtract(fileset("${var.source_dir}/","*"), ["requirements.txt", "main.py"])
}

This will help you to include selective files/folders/sub-folders but your state file might look a little ugly.