How to get Terraform providers on GitLab CI runners in an air-gapped environment

Question

I am running a Gitlab CI runner on an Azure Red Hat Linux 7.9 VM which is air-gapped and cannot communicate with the outside world due to our network restrictions. In the .gitlab-ci.yml file of my GitLab pipeline, I run some Terraform commands including Terraform init which subsequently attempts to pull down the Terraform provider plugins from the default Hashicorp location.

Obviously, with the prevailing network restrictions, the aforementioned Terraform command fails and subsequently the entire pipeline job fails. From what I've now established and bearing in mind our particular network setup and circumstances, I probably have two options:

A) Configure my GitLab pipeline with a Proxy to allow access to sites like registry.terraform.io. I have been provided a proxy address by the way, but I'm not entirely sure how to get this configured in GitLab. Will therefore appreciate some assistance with this.

B) It also appears I will be able to force my Terraform configuration to reference the required provider plugins locally or on an internal share. Again, any advice on how to set this up would be greatly appreciated, having spent hours poring over endless online material on both options without any success.

[[ Case Update after recommendations by @sytech ]]

I discarded the proxy option entirely, due mainly to some resource challenges within my organisation and not at all to do with the proposed solution itself. I therefore went with the option to set up a provider mirror residing on my file system and this is how I went about it.....but still with no success.

My Implementation Summary

1. I uploaded the two required Terraform providers into our Artifactory repo, with each provider content displayed in the image below.

2. In my .gitlab-ci.yml file, I create the following directories:

*/root/.terraform.d/plugins/registry.terraform.io/hashicorp/azuread 
/root/.terraform.d/plugins/registry.terraform.io/hashicorp/azurerm*

I then updated my terraform providers.tf configuration file by amending both provider source locations accordingly (see below screenshot).

3. Next, I download each provider (excluding the *.json files) from the Artifactory repo to the GitLab runner local file system and unzip them to their respective directories, created in Step #2 above. Finally, I perform a clean-up by removing the respective provider zip files that were downloaded from Artifactory.

4. I then switch to the /root/.terraform.d/plugins/registry.terraform.io/hashicorp directory and run a Linux recursive command to verify both providers are present in the expected directories.

5. Finally, I switch to the defined terraform root directory, run the ls -a command to verify that my terraform configuration files are all present. I then run the terraform init and terraform validate commands which result in the subsequent error displayed in the last of the images displayed below.

[[ Required Providers Block in providers.tf file ]]

terraform {
  required_providers {
    azurerm = {
      source = "registry.terraform.io/hashicorp/azurerm/terraform-provider-azurerm_v3.19.1_x5"
      version = "3.19.1" 
    }

    azuread = {
      source = "registry.terraform.io/hashicorp/azuread/terraform-provider-azuread_v2.27.0_x5"
      version = "2.27.0"
    }
  }
}

[[ Snippet from my .gitlab-ci.yml file ]]

default:
  image: 
    name: xxx-local.artifactory.ourdomain:x.x.x
    entrypoint: [""]

variables:
  TF_ROOT: ${CI_PROJECT_DIR}
  
stages:
  - validate
  - plan
  - destroy

download-plugins:
  stage: validate 
  script:
    - git config --global url."https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.ourdomain".insteadOf https://gitlab.ourdomain

    # Download and copy azuread provider to the gitlab runner filesystem from our Artifactory store
    - mkdir -p /root/.terraform.d/plugins/registry.terraform.io/hashicorp/azuread      
    - curl -o /root/.terraform.d/plugins/registry.terraform.io/hashicorp/azuread/azuread.zip https://artifactory.ourdomain/artifactory/local/terraform/providers/registry.terraform.io/hashicorp/azuread/terraform-provider-azuread_2.27.0_linux_amd64.zip
    - unzip /root/.terraform.d/plugins/registry.terraform.io/hashicorp/azuread/azuread.zip -d /root/.terraform.d/plugins/registry.terraform.io/hashicorp/azuread
    - rm /root/.terraform.d/plugins/registry.terraform.io/hashicorp/azuread/azuread.zip

    # Download and copy azurerm provider to the gitlab runner filesystem from our Artifactory store
    - mkdir -p /root/.terraform.d/plugins/registry.terraform.io/hashicorp/azurerm      
    - curl -o /root/.terraform.d/plugins/registry.terraform.io/hashicorp/azurerm/azurerm.zip https://artifactory.ourdomain/artifactory/local/terraform/providers/registry.terraform.io/hashicorp/azurerm/terraform-provider-azurerm_3.19.1_linux_amd64.zip
    - unzip /root/.terraform.d/plugins/registry.terraform.io/hashicorp/azurerm/azurerm.zip -d /root/.terraform.d/plugins/registry.terraform.io/hashicorp/azurerm
    - rm /root/.terraform.d/plugins/registry.terraform.io/hashicorp/azurerm/azurerm.zip

    - cd /root/.terraform.d/plugins/registry.terraform.io/hashicorp
    - echo "Display contents of the present working directory:" `pwd`
    - ls -a -R

    - echo "Switch to the $TF_ROOT directory to verify file content and then run terraform commands"
    - cd $TF_ROOT
    - ls -a
    - terraform init
    - terraform validate

  tags:
    - "AzureVMRunner"
   

Answer 1

Configure my GitLab pipeline with a Proxy

If you have a forward proxy available in your environment and want to have your CI jobs use that proxy, it's usually simply a matter of setting your HTTP_PROXY and HTTPS_PROXY variables. You can do this in your runner environment configuration or CI configuration. For example in a .gitlab-ci.yml you could add:

variables:
  HTTP_PROXY: "http://proxy.mydomain.corp"
  HTTPS_PROXY: "http://proxy.mydomain.corp"

Refer to the official docs Running GitLab Runner Behind a Proxy for more information.

---

It also appears I will be able to force my Terraform configuration to reference the required provider plugins locally or on an internal share

You can use the filesystem mirror option to configure a terraform providers mirror that lives on your filesystem (requires terraform v0.13+).

By default, the directory .terraform.d/plugins can be used without any additional configuration (see implied mirrors). When you have the provider files in your filesystem mirror, you won't need to download them from the internet.

Let's say for example you're using the graylog provider like this:

provider "graylog" {
  version          = "1.0.4"
  web_endpoint_uri = "https://graylog.example.com/api"
  api_version      = "v3"
  auth_name        = "admin"
  auth_password    = "password"
}

All you need are the requisite files -- namely, the zip files of the providers you need (e.g., terraform-provider-graylog_1.0.4_linux_amd64.zip), then in your job copy them into your .terraform.d plugins directory (e.g., .terraform.d/plugins/...) in an expected layout (packed or unpacked).

You can make these files available by placing them on your runner and (for docker runners) mounting the files into the job using the volumes configuration.

For example, say you have the providers on your runner host, you can mount them into your jobs with this configuration in the config.toml file (only needed for docker-based runners!)

[runners.docker]
  # ...
  volumes = ["/path/to/plugins/from/host:/plugins/path/in/container:rw"]

Then in your jobs, you can copy the provider files into the .terraform directory.

my_job:
  image: 
    name: hashicorp/terraform
    entrypoint: [""]
  before_script:
    - cp /plugins/path/in/container/ .terraform.d/plugins/ 
    - terraform init  # will get plugins from your local mirror