GRPC "i/o timeout" errors while invoking GRPC method between GCP Could Run services

Question

we have several GRPC services implemented in Go deployed as Cloud Run services. There is also an ESPv2 Envoy service proxy Cloud Run service which fronts the GRPC services.

Our mobile app doesn't directly connect to the GRPC services but instead is able to successfully connect to any of the GRPC service methods via the ESPv2 service proxy. So the App -> ESPv2 -> GRPC Services path is working just great.

Recently, we have had the need to invoke a GRPC method on Service A from Service B.

Each time the Go code in Service B attempts to invoke the GRPC method of Service A, it encounters the following error.

transport: Error while dialing: dial tcp [XXXX:XXXX:XXXX:38::35]:50051: i/o timeout

Service B uses the same service account as the ESPv2 proxy (and the ESPv2 proxy does successfully connect to all of the GRPC services including Service A).

Both Service A and Service B have the following network configuration in Cloud Run.

Service A is configured to "Allow direct access to your service from the internet" and allows unauthenticated connections.

The address that is used in the grpc.Dial function is the URL adverstised by Cloud Run for Service A (without the https:// of course). We have tried using the directly IP address with no avail.

I would like to reach out to anybody who has experienced this issue and has been able to resolve it.

Thanks

Answer 1

So it turns out that the port I was using was wrong. The containerPort defined in the Cloud Run yaml config is apparently not the port at which the requests are serviced. Once I changed the port from 50051 to 443, the gRPC client was able to connect successfully. Thanks to the comment from John Hanley which put me on the right path.