tcp 10.10.2.2:443: i/o timeout Cloud Build accessing private GKE

Question

I'm trying to get Cloud Build to access my private GKE by following the instructions described at https://cloud.google.com/architecture/accessing-private-gke-clusters-with-cloud-build-private-pools

From what I can see I've configured things as described, but when I try to deploy anything or run the test described in the link above, I get the following error:

Unable to connect to the server: dial tcp 10.10.2.2:443: i/o timeout
ERROR
ERROR: build step 0 "gcr.io/cloud-builders/kubectl" failed: step exited with non-zero status: 1

Does anyone have any ideas how I can troubleshoot this? My VPN tunnels are all established and my GKE cluster is allowing 192.168.0.0/20 in its authorized networks.

Here are some additional details on my setup for reference:

# gcloud compute vpn-tunnels list
NAME            REGION    GATEWAY   PEER_ADDRESS
ha-vpn-tunnel1  us-east1  ha-vpn-1  35.242.12.184
ha-vpn-tunnel2  us-east1  ha-vpn-1  35.220.15.95
ha-vpn-tunnel3  us-east1  ha-vpn-2  35.242.6.151
ha-vpn-tunnel4  us-east1  ha-vpn-2  35.220.14.211

# gcloud compute vpn-gateways list
NAME      INTERFACE0     INTERFACE1     NETWORK                   REGION
ha-vpn-1  35.242.6.151   35.220.14.211  demo-vpc                  us-east1
ha-vpn-2  35.242.12.184  35.220.15.95   private-pool-peering-vpc  us-east1

# gcloud compute routers list
NAME            REGION    NETWORK
ha-vpn-router1  us-east1  demo-vpc
ha-vpn-router2  us-east1  private-pool-peering-vpc
my-router       us-east1  demo-vpc

# gcloud compute routers describe ha-vpn-router1 --region us-east1
bgp:
  advertiseMode: DEFAULT
  asn: 64515
  keepaliveInterval: 20
bgpPeers:
- advertiseMode: CUSTOM
  advertisedIpRanges:
  - description: ''
    range: 192.168.0.0/20
  advertisedRoutePriority: 100
  bfd:
    minReceiveInterval: 1000
    minTransmitInterval: 1000
    multiplier: 5
    sessionInitializationMode: DISABLED
  enable: 'TRUE'
  enableIpv6: false
  interfaceName: router1-interface2
  ipAddress: 169.254.1.2
  name: router1-peer2
  peerAsn: 64516
  peerIpAddress: 169.254.1.1
- advertiseMode: CUSTOM
  advertisedIpRanges:
  - description: ''
    range: 192.168.0.0/20
  advertisedRoutePriority: 100
  bfd:
    minReceiveInterval: 1000
    minTransmitInterval: 1000
    multiplier: 5
    sessionInitializationMode: DISABLED
  enable: 'TRUE'
  enableIpv6: false
  interfaceName: router1-interface1
  ipAddress: 169.254.0.1
  name: router1-peer1
  peerAsn: 64516
  peerIpAddress: 169.254.0.2
creationTimestamp: '2023-01-03T16:55:55.711-08:00'
id: '2186172510416498804'
interfaces:
- ipRange: 169.254.1.2/30
  linkedVpnTunnel: https://www.googleapis.com/compute/v1/projects/fakeproject/regions/us-east1/vpnTunnels/ha-vpn-tunnel2
  name: router1-interface2
- ipRange: 169.254.0.1/30
  linkedVpnTunnel: https://www.googleapis.com/compute/v1/projects/fakeproject/regions/us-east1/vpnTunnels/ha-vpn-tunnel1
  name: router1-interface1
kind: compute#router
name: ha-vpn-router1
network: https://www.googleapis.com/compute/v1/projects/fakeproject/global/networks/demo-vpc
region: https://www.googleapis.com/compute/v1/projects/fakeproject/regions/us-east1
selfLink: https://www.googleapis.com/compute/v1/projects/fakeproject/regions/us-east1/routers/ha-vpn-router1

# gcloud compute routers describe ha-vpn-router2 --region us-east1
bgp:
  advertiseMode: DEFAULT
  asn: 64516
  keepaliveInterval: 20
bgpPeers:
- advertiseMode: CUSTOM
  advertisedIpRanges:
  - description: ''
    range: 10.10.2.0/28
  advertisedRoutePriority: 100
  bfd:
    minReceiveInterval: 1000
    minTransmitInterval: 1000
    multiplier: 5
    sessionInitializationMode: DISABLED
  enable: 'TRUE'
  enableIpv6: false
  interfaceName: router2-interface1
  ipAddress: 169.254.0.2
  name: router2-peer1
  peerAsn: 64515
  peerIpAddress: 169.254.0.1
- advertiseMode: CUSTOM
  advertisedIpRanges:
  - description: ''
    range: 10.10.2.0/28
  advertisedRoutePriority: 100
  bfd:
    minReceiveInterval: 1000
    minTransmitInterval: 1000
    multiplier: 5
    sessionInitializationMode: DISABLED
  enable: 'TRUE'
  enableIpv6: false
  interfaceName: router2-interface2
  ipAddress: 169.254.1.1
  name: router2-peer2
  peerAsn: 64515
  peerIpAddress: 169.254.1.2
creationTimestamp: '2023-01-03T16:56:06.404-08:00'
id: '6144519957886103625'
interfaces:
- ipRange: 169.254.0.2/30
  linkedVpnTunnel: https://www.googleapis.com/compute/v1/projects/fakeproject/regions/us-east1/vpnTunnels/ha-vpn-tunnel3
  name: router2-interface1
- ipRange: 169.254.1.1/30
  linkedVpnTunnel: https://www.googleapis.com/compute/v1/projects/fakeproject/regions/us-east1/vpnTunnels/ha-vpn-tunnel4
  name: router2-interface2
kind: compute#router
name: ha-vpn-router2
network: https://www.googleapis.com/compute/v1/projects/fakeproject/global/networks/private-pool-peering-vpc
region: https://www.googleapis.com/compute/v1/projects/fakeproject/regions/us-east1
selfLink: https://www.googleapis.com/compute/v1/projects/fakeproject/regions/us-east1/routers/ha-vpn-router2

Thanks!

Update 2022-01-04:

I have a cloudbuild.yaml file matching the example from the link above which I've just updated to run a kubectl get config instead of kubectl get nodes. Here is the result of that command:

Running: kubectl config view
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: DATA+OMITTED
    server: https://10.10.2.2
  name: gke_fake-project_us-east1_demo-gke
contexts:
- context:
    cluster: gke_fake-project_us-east1_demo-gke
    user: gke_fake-project_us-east1_demo-gke
  name: gke_fake-project_us-east1_demo-gke
current-context: gke_fake-project_us-east1_demo-gke
kind: Config
preferences: {}
users:
- name: gke_fake-project_us-east1_demo-gke
  user:
    auth-provider:
      config:
        access-token: xxx
        cmd-args: config config-helper --format=json
        cmd-path: /builder/google-cloud-sdk/bin/gcloud
        expiry: "2023-01-05T03:17:33Z"
        expiry-key: '{.credential.token_expiry}'
        token-key: '{.credential.access_token}'
      name: gcp
PUSH
DONE

Here's the output showing my authorized networks as well:

gcloud container clusters describe demo-gke \
>       --format="value(masterAuthorizedNetworksConfig)"\
>       --region=us-east1
cidrBlocks=[{'cidrBlock': '10.0.0.0/8', 'displayName': 'internal_10'}, {'cidrBlock': '192.168.0.0/16', 'displayName': 'internal_192'}, {'cidrBlock': '172.16.0.0/12', 'displayName': 'internal_172'}];enabled=True

Answer 1

Unable to connect to the server: dial tcp [IP_ADDRESS]: connect: connection timed out.

This can occur when kubectl is unable to talk to the cluster control plane.

To resolve this issue, verify the context where the cluster is set:

1. Go to $HOME/.kube/config or run the command kubectl config view to verify the config file contains the cluster context and the external IP address of the control plane.
2. Set the cluster credentials:

 gcloud container clusters get-credentials CLUSTER_NAME \
        --region=COMPUTE_REGION

Replace the following:

CLUSTER_NAME: the name of your cluster.

COMPUTE_REGION: the Compute Engine region for your cluster. For zonal clusters, use --zone=COMPUTE_ZONE.

Follow this doc which has muliple methods for troubleshooting that can't reach the control plane of a private cluster, try them and let me know if this resolves your issue.

Answer 2

Turns out the problem was I had a typo in one of my BGP advertised routes. Once I fixed that up everything works as expected.