we just migrated from public cluster to private on GKE, we are facing a weird issue where a pod is unable to call an external API having https on port 9090. It becomes unresponsive and after 60 second throw Connection reset by peer.
We are using cloud NAT for enabling internet access for pods. We have tested its not a port blocking issue by temporary adding egress and ingress rules in firewall to allow all traffic on every port.
We tested the ssl handshake using curl from inside pod by running following command:
curl -k -vvv --location 'https://103.xx.xx.xx:9090/v1/users/login' POST \
--header 'Content-Type: application/json' \
--header 'Authorization: Basic YWRtaW46QnJhbmR2ZXJzZUAxMjM=' \
--data-raw ''
It gets stuck on this response:
* Trying 103.xx.xx.xx:9090...
* TCP_NODELAY set
* Connected to 103.xx.xx.xx (103.xx.xx.xx) port 9090 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
While on any other service its working fine.
Would really appreciate any help or pointer in this regard. Thanks
Update:
I have narrowed it down to TLS version, so the service I am trying to call does not support TLS 1.3. Is there anyway I can configure my VPC/cluster to enable TLS 1.2 support?
Is it possible you have an other service in the VPC using the IP address of the API ? Maybe using something like privately used public IPs for GKE ?
One other thing you can check is run a debug pod like this one which contains a bunch of debugging tools like traceroute and try to traceroute from the pod to the IP of your API