Unable to connect to Google GKE Private cluster through bastion/cloud-shell/proxy

Question

I've setup the GKE private cluster, and trying to follow the Google's documentation and each of the documentation ends up in blockers which we need to figure out ourselves.

I have a GKE private cluster to which I figured out how to add public IP in authorized list so that I can add my home network to it to connect kubectl. But I want to deploy the applications through CICD from github actions/gitlab.

The problem I'm facing is that I have setup each of following proxies/bastions mentioned in below two documentation, but each of them ends up in a dead end.

I deployed the proxy pod and opened up with service: https://cloud.google.com/kubernetes-engine/docs/archive/creating-kubernetes-engine-private-clusters-with-net-proxies

It gives me the output from CONTROLLER_IP:

curl -k -x $LB_IP:8118 https://$CONTROLLER_IP/version
{
  "major": "1",
  "minor": "27",
  "gitVersion": "v1.27.7-gke.1121000",
  "gitCommit": "4daab1fd78c0b9aba478a19b363ab4a25bdadd79",
  "gitTreeState": "clean",
  "buildDate": "2023-11-06T09:24:38Z",
  "goVersion": "go1.20.10 X:boringcrypto",
  "compiler": "gc",
  "platform": "linux/amd64"
}

But any further kubectl command errors out in:

Unable to connect to the server: Request blocked by Privoxy

I deployed a VM according to below documentation: https://cloud.google.com/kubernetes-engine/docs/tutorials/private-cluster-bastion

Then it gives me:

ERROR: gcloud crashed (ProxyError): HTTPSConnectionPool(host='container.googleapis.com', port=443): Max retries exceeded with url: /v1/projects/<my-project-id>/locations/us-central1/clusters/autopilot-cluster-1?alt=json (Caused by ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 403 Access denied')))

Is there a way to make either of above work? All the pointless verbiage in Google documentation is taking me nowhere.

Answer 1

I suggest not using the first doc (proxy pod) approach as it is archived. The second bastion host approach should work. However, I couldn't reproduce the error you posted when trying the same approach. I assume that the error message is from the workstation/vm where the gcloud command runs, as showing the following demonstration:

client machine (forward with IAP) -> Bastion VM -> GKE control plane

Your issue is likely related to the IAP (Identity-Aware Proxy) configuration. Specifically, I would suggest checking:

• VPC firewall rule to allow IAP traffic (35.235.240.0/20) coming into your VPC on port 22 and 3389.

• IAM permission is not granted properly for the user/service account that runs the client application (gcloud i.e). In this case, it is likely that the connection between the workstation VM and bastion VM failed due to the IAM permission.

Sorry to put another document, but please follow the using IAP for TCP forwarding.