EBS volume provisioning isn't working when my kubernetes cluster runs in a private subnet (no internet access)

Question

I have a Microk8s cluster on EC2 instances. EBS volume provisioning (both static and dynamic) is ONLY working when my cluster has internet access. Meaning, EBS provisioning works when my cluster is in a public subnet (that uses internet gateway) as well as in a private subnet (that uses NAT gateway).

Since, my Microk8s cluster is in a private subnet without any internet access, I configured PrivateLink interface endpoints for EC2. I was expecting EBS provisioning works correctly. But it is giving me following error:

I0405 11:30:26.394521       1 controller.go:104] "CreateVolume: called" args={Name:pvc-fff50e05-f28f-4d0a-9b9e-6e98a30aee67 CapacityRange:required_bytes:4294967296  VolumeCapabilities:[mount:<fs_type:"ext4" > access_mode:<mode:SINGLE_NODE_WRITER > ] Parameters:map[csi.storage.k8s.io/pv/name:pvc-fff50e05-f28f-4d0a-9b9e-6e98a30aee67 csi.storage.k8s.io/pvc/name:ebs-claim csi.storage.k8s.io/pvc/namespace:default] Secrets:map[] VolumeContentSource:<nil> AccessibilityRequirements:requisite:<segments:<key:"topology.ebs.csi.aws.com/zone" value:"eu-west-2a" > > preferred:<segments:<key:"topology.ebs.csi.aws.com/zone" value:"eu-west-2a" > >  XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
2023/04/05 11:30:26 DEBUG: Request ec2/CreateVolume Details:
---[ REQUEST POST-SIGN ]-----------------------------
POST / HTTP/1.1
Host: ec2.eu-west-2.amazonaws.com
User-Agent: aws-sdk-go/1.44.218 (go1.20.2; linux; amd64) exec-env/aws-ebs-csi-driver-v1.17.0
Content-Length: 792
Authorization: AWS4-HMAC-SHA256 Credential=AKIAR74FCIA4TYUN7QXX/20230405/eu-west-2/ec2/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-date, Signature=2286702fd8cfbefe16346b7015bbefac9dc4e476e6a662357d9a4cf3d8fefa62
Content-Type: application/x-www-form-urlencoded; charset=utf-8
X-Amz-Date: 20230405T113026Z
Accept-Encoding: gzip


-----------------------------------------------------
2023/04/05 11:30:34 DEBUG: Send Request ec2/CreateVolume failed, attempt 0/8, error RequestError: send request failed
caused by: Post "https://ec2.eu-west-2.amazonaws.com/": dial tcp: lookup ec2.eu-west-2.amazonaws.com on 10.152.183.10:53: server misbehaving
2023/04/05 11:30:34 DEBUG: Request ec2/CreateVolume Details:
---[ REQUEST POST-SIGN ]-----------------------------
POST / HTTP/1.1
Host: ec2.eu-west-2.amazonaws.com
User-Agent: aws-sdk-go/1.44.218 (go1.20.2; linux; amd64) exec-env/aws-ebs-csi-driver-v1.17.0
Content-Length: 792
Authorization: AWS4-HMAC-SHA256 Credential=AKIAR74FCIA4TYUN7QXX/20230405/eu-west-2/ec2/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-date, Signature=22b30cb07b29e41c4ccf361a1dbfa20b625ec1c957764acd25ab3e1f2782fe42
Content-Type: application/x-www-form-urlencoded; charset=utf-8
X-Amz-Date: 20230405T113034Z
Accept-Encoding: gzip


-----------------------------------------------------
2023/04/05 11:30:36 DEBUG: Send Request ec2/CreateVolume failed, attempt 1/8, error RequestCanceled: request context canceled
caused by: context deadline exceeded
I0405 11:30:36.394429       1 inflight.go:74] "Node Service: volume operation finished" key="pvc-fff50e05-f28f-4d0a-9b9e-6e98a30aee67"
E0405 11:30:36.394453       1 driver.go:120] "GRPC error" err=<
    rpc error: code = Internal desc = Could not create volume "pvc-fff50e05-f28f-4d0a-9b9e-6e98a30aee67": could not create volume in EC2: RequestCanceled: request context canceled
    caused by: context deadline exceeded
 >

Answer 1

Finally got it working. By default Microk8s dns addon forwards DNS requests to 8.8.8.8,8.8.4.4 if it cannot resolve requests locally. Since my AWS EC2 instances were in a private subnet (no NAT gateway or internet access) the requests were failing. So the solution was:

1. Disable dns addon: sudo microk8s disable dns.
2. Enable dns addon: sudo microk8s enable dns:/etc/resolve.conf