Cannot connect to GCP Compute Engine Instance - Error while connecting [4003: 'failed to connect to backend']. (Failed to connect to port 22)

Question

I'm facing a new issue where I'm unable to connect to my GCP Compute Engine instance for the past 2 days. I'd set up the VM, and the firewall rules including enabling IAP for TCP Tunneling about a week ago, and was able to SSH to my machine using this command fine up until 2 days ago.

gcloud compute ssh <vm-name> --zone=us-west1-b --quiet --tunnel-through-iap --project=<project-name>

Running the above command in my terminal issues this error:

ERROR: (gcloud.compute.start-iap-tunnel) Error while connecting [4003: 'failed to connect to backend']. (Failed to connect to port 22)

kex_exchange_identification: Connection closed by remote host
Connection closed by UNKNOWN port 65535

Here is the firewall rule I set up to enable the IAP TCP tunneling access

I can see that my VM instance is included in the "Applicable to instances" list.

I was also unable to connect via SSH to the instance using the in-built SSH tool. Here's a screenshot of the error:

According to this error, it seems that the access to VM is blocked by a firewall and I've made sure to set up the firewall correctly.

I've tried solutions outlined in these posts:

• Post 1
• This answer in the Google Cloud Community.

But I'm still unable to connect to my VM. I've also deleted and re-created the firewall rules a couple of times now to no avail.

What might be causing this? I'm new to GCP and networking concepts so if there's something I'm missing, please point me to the resources.

Appreciate your help, thanks a lot!

Answer 1

After multiple days (attempts) trying to figure out the cause of this issue, I looked through the logs of my VM, and found the error mentioned in this link

I decided to tear down the Compute Engine instance and the firewall rules I'd set, and created a new Compute Engine instance and re-created the firewall rules.

After doing this, I'm able to connect to the new VM without trouble.

Answer 2

According to the error message and configuration, it is not clear on the root cause. There are couple of things that you could do to troubleshoot the issue:

Update the glcoud SDK/CLI to latest version if possible.

1. Run gcloud troubleshoot command. It will kick off a connection test under the cover.

gcloud compute ssh <vm-name> --zone=us-west1-b --quiet --troubleshoot --tunnel-through-iap --project=<project-name>

2. Temporarily modify the allow-ingress-from-iap firewall rule to allow traffic from 0.0.0.0/0 on port 22. See whether that allows the ssh to your vm.

3. Add a public IP to the VM and try to ssh to the public IP. If it works, it is likely an issue with IAP.

4. Restart the VM or spun up a new VM in the same region same network, retry the ssh with IAP.

5. Lastly, check whether you have moved the VM among different VPCs or subnets recently that may cause the connection issue.