Terraform: How to obtain VPCE service name when it was dynamically created

Question

I am trying to obtain (via terraform) the dns name of a dynamically created VPCE endpoint using a data resource but the problem I am facing is the service name is not known until resources have been created. See notes below.

Is there any way of retrieving this information as a hard-coded service name just doesn’t work for automation?

e.g. this will not work as the service_name is dynamic

    resource "aws_transfer_server" "sftp_lambda" {

      count                  = local.vpc_lambda_enabled
      domain                 = "S3"
      identity_provider_type = "AWS_LAMBDA"
      endpoint_type          = "VPC"
      protocols              = ["SFTP"]
      logging_role           = var.loggingrole
      function               = var.lambda_idp_arn[count.index]

      endpoint_details = {
         security_group_ids = var.securitygroupids
         subnet_ids         = var.subnet_ids
         vpc_id             = var.vpc_id
      }
    
      tags = {
        NAME = "tf-test-transfer-server"
        ENV  = "test"
      } 
    }

    data "aws_vpc_endpoint" "vpce" {
      count        = local.vpc_lambda_enabled
      vpc_id       = var.vpc_id
      service_name = "com.amazonaws.transfer.server.c-001"
      depends_on   = [aws_transfer_server.sftp_lambda]
    }

    output "transfer_server_dnsentry" {
        value = data.aws_vpc_endpoint.vpce.0.dns_entry[0].dns_name
    }

Note: The VPCE was created automatically from an AWS SFTP transfer server resource that was configured with endpoint type of VPC (not VPC_ENDPOINT which is now deprecated). I had no control over the naming of the endpoint service name. It was all created in the background.

Minimum AWS provider version: 3.69.0 required.

Here is an example cloudformation script to setup an SFTP transfer server using Lambda as the IDP. This will create the VPCE automatically. So my aim here is to output the DNS name from the auto-created VPC endpoint using terraform if at all possible.

example setup in cloudFormation

data source: aws_vpc_endpoint resource: aws_transfer_server

Answer 1

I've finally found the solution:

data "aws_vpc_endpoint" "transfer_server_vpce" {
  count  = local.is_enabled
  vpc_id = var.vpc_id

  filter {
    name   = "vpc-endpoint-id"
    values = ["${aws_transfer_server.transfer_server[0].endpoint_details[0].vpc_endpoint_id}"]
  }
}

Answer 2

I had a response from Hashicorp Terraform Support on this and this is what they suggested:

you can get the service SFTP-Server-created-VPC-Endpoint by calling the following exported attribute of the vpc_endpoint_service resource [a].

NOTE: There are certain setups that causes AWS to create additional resources outside of what you configured. The AWS SFTP transfer service is one of them. This behavior is outside Terraform's control and more due to how AWS designed the service.

You can bring that VPC Endpoint back under Terraform's control however, by importing the VPC endpoint it creates on your behalf AFTER the transfer service has been created - via the VPCe ID [b]. If you want more ideas of pulling the service name from your current AWS setup, feel free to check out this example [c].

Hope that helps! Thank you.

[a] https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/vpc_endpoint_service#service_name [b] https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/vpc_endpoint#import [c] https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/vpc_endpoint#gateway-load-balancer-endpoint-type

There is a way forward like I shared earlier with the imports but it not going to be fully automated unfortunately.

Optionally, you can use a provisioner [1] and the aws ec2 describe-vpc-endpoint-services --service-names command [2] to get the service names you need.

I'm afraid that's the last workaround I can provide, as explained in our doc here [3] - which will explain how - as much as we'd like to, Terraform isn't able to solve all use-cases.

[1] https://www.terraform.io/language/resources/provisioners/remote-exec [2] https://awscli.amazonaws.com/v2/documentation/api/latest/reference/ec2/describe-vpc-endpoint-services.html [3] https://www.terraform.io/language/resources/provisioners/syntax