AWS lambda connect to internal ELB with private Hostzone DNS

Question

I want to set up an EC2 instance running on a private VPC. It can connect to the Internet from the private VPC but can not access from outside. And there is a lambda function to trigger the EC2 to initiate some interactions with external resources (S3, Dynamo, Internet).

I have set up a VPC as following:

1. An EC2 instance running docker in a private VPC subnet
2. An ALB(application load-balancer) configured as internal and in private subnets (same as the EC2 subnet)
3. A NAT Gateway which is working
4. A lambda function which will do HTTPs GET and POST to the Internet and ALB
5. Route53 private Hostzone has a record set that route "abcd.internal/api" to the ALB.

Here is the problem. The lambda function can connect to the Internet with HTTPs, but when it fails to HTTPs GET to the ALB with the private Hostzone record("abcd.internal").

My understanding is my ALB, EC2, lambda, NAT Gateway and Route53 are configured in the same VPC, they should be able to talk to each other with the private DNS name. I don't know why it fails.

Note: Before setting up a internal ALB, I did try setting up a internet-facing ALB in a public subnet, then configure a public Hostzone record set "abcd.public" to this ALB. It can talk to the EC2 instance and the EC2 instance can interact with the Internet through the NAT Gateway. So the "EC2 to Internet" part is working.

Update: I finally dig some error messages in lambda log as follows:

Error: Hostname/IP doesn't match certificate's altnames: "Host: abcd.internal. is not in the cert's altnames: DNS:.public"] reason: 'Host: abcd.internal. is not in the cert\'s altnames: DNS:.public', host: 'abcd.internal.',

That is interesting. I do have a public hostzone co-exist with the private hostzone, but the public hostzone is for other purpose. I dont know why the lambda function use the public DNS rather than the private DNS since it was configured inside a private subnet.

Answer 1

Thanks for everyone who post comments and gave suggestions.

To solve this problem, I have almost found every possible solutions online. I put everything at the right position. Lambda function, ELB and EC2 are in the same VPC private subnet. Route53, NAT and IGW are properly set up. I did try playing with the DHCP options set, didn't work. Maybe I don't fully understand this DHCP and I can't find an example.

It turns out the HTTPS protocol is not working. Before I move to private VPC, I have the same thing setup in a public VPC and resources are using HTTPS to communicate. For example, the lambda function will GET/POST to the EC2 instance or ELB. After I move stuffs into a private VPC, HTTPS commands can not use the internal DNS names.

However, if I use HTTP protocol, resources finally can find each other by internal DNS names.

I still dont know why HTTPS can't be used in the private VPC, but I can live with this solution.

Answer 2

I had the same problem.

The ALB was not added as a trigger for the Lambda which was causing a similar certificate issue for me. The security group was configured wrongly in my case. I noticed that the role that I assigned to Lambda should include a policy with create/delete ENI permissions

Sometimes the ALB updates were not quick. so I recreated with the same settings, it started to work.

Answer 3

Did you make sure to check if the IAM role attached to your Lambda has access to ec2 Network related actions? Here's an example IAM policy:

{
"Version": "2012-10-17",
"Statement": [
    {
        "Action": [
            "ec2:CreateNetworkInterface",
            "ec2:DescribeNetworkInterfaces",
            "ec2:DeleteNetworkInterface",
            "ec2:DescribeSecurityGroups",
            "ec2:DescribeSubnets"
        ],
        "Resource": [
            "*"
        ],
        "Effect": "Allow"
    }
]
}