Context:
I have a simple setup with a private VPC endpoint that points to an NLB that distributes plain TCP (not HTTP) traffic to multiple EC2 instances in an autoscaling group.
We are hitting the port limit (~65k) for open TCP connections to those EC2 instances because when used as service endpoint the NLB changes the source IP to its own IP (instead of when used without service endpoint, where the original IP is kept). Workaround here is to listen on multiple ports on the target EC2 instances (e.g. 8080-8090), but we still need to have only one port open to outside traffic on the NLB (e.g. 8080).
You can add a target group to the autoscaling group config and it gets updated when the autoscaling group changes, but that only seems to work with one single port that is configured in the target group and does not work for multiple ports. You can add the same instance to a target group twice via the console with a different port, but that does not help me much as infrastructure is managed via terraform.
Question:
Does anybody know a solution where targets in an autoscaling group get updated automatically for multiple ports on the same target that are exposed on 1 port on the NLB to the outside ?
Update: as far as I could figure out this can be done if you create a lambda that calls the AWS API and adds the same instance to one target group multiple times with different ports. The lambda is triggered by an SNS event that is triggered by the scaling events of the autoscaling group.