Error from server: error dialing backend: dial tcp <datanode-ip>: i/o timeout

Question

I have a problem with kubernetes local cluster recently.When I was using the command kubectl exec -it curl-- bash to run some commands on the pod called 'curl',I got some errors: error info

And here are the nodes' info: nodes info

The pod 'curl' is working nicely on the datanode-2 and kubelet is listening on the port 10250,but I don't know why I got the error info,Here is the `kubectl describe po curl': curl pod describe

And here are the pods in the namespace kube-system,the CNI is flannel: enter image description here

It's same to run kubectl exec on others pod(same on datanode-1),how to solve this?

Answer 1

This error might be related with communication of the kube-apiserver.service (on the control nodes) with the kubelet.service (port 10250 by default)

To Troubleshoot , you might want to ssh into the control node and

telnet hostname(workernode) 10250
telnet privateip(workernode) 1025

If both telnet tests failed it might be related with your firewall on the worker nodes . So you should open the port 10250 in the worker nodes . To check if the kubelet is running on this port

lsof -i :10250

If the telnet test fails with the hostname or public ip, but works with the private ip . You should add to the unit service file of the kube-apiserver (located at /etc/systemd/system/kube-apiserver.service) the flag

--kubelet-preferred-address-types InternalIP

Save it , and then just

systemctl daemon-reload
systemctl restart kube-apiserver 

Answer 2

Encountered the same error in my kubeadm cluster.

kubectl exec -it nginx -- env

Error from server: error dialing backend: dial tcp 10.0.0.3:10250: i/o timeout

I was able to resolve the issue by creating Firewall rules for port 10250.

1. Go to the Firewall page.

2. Enter a Name for the firewall rule.

3. Specify the Network in which you want to implement the firewall rule.

4. Specify the Priority of the rule.

5. Choose Ingress for Direction of traffic.

6. Choose Allow for Action on match.

7. For Targets, choose All instances in the network.

8. For Protocols and ports, choose Specified protocols and enter the port 10250.

9. Click Create

10. Run the kubectl exec command again.

kubectl exec -it nginx -- env | grep var1

var1=value1

Hope this information works for you.