NodePort conflicts wiht kube-apiserver's hostPort

Question

I have a healthy k8s cluster. The kube-apiserver is using port 8443. Then I do something like below. I create/edit a Sevice and assign a fixed nodePort 8443, too.

apiVersion: v1
kind: Service
metadata:
  name: my-service
spec:
  selector:
    app: MyApp
  ports:
    - protocol: TCP
      port: 80
      targetPort: 9376
      nodePort: 8443

The cluster becomes unreachable. If I run command kubectl get pods. It returns Unable to connect to the server: EOF

Cause I need run kubectl command to change the nodePort back. But I cannot use kubectl. It becomes a dead loop. I have 2 questions.

1. What can I do to get my k8s cluster back?
2. Why kube-apiserver or other components doesn't prevent me to change the nodePort to 8443? Indicate the port 8443 is already in use.

Answer 1

kube-apiserver pod is a static pod brought up by kubelet using /etc/kubernetes/manifests/kube-apiserver.yaml file on master nodes

so you can edit /etc/kubernetes/manifests/kube-apiserver.yaml , where 8443 port is referenced and change it to a unused port. so that kube-apiserver will be available on that port. ideally kubelet tries to restart a static pod when its manifest file is changed .

you need to edit ~/.kube/config , where 'server' address needs to be changed to new port.

delete the problematic nodeport service .

revert your changes in /etc/kubernetes/manifests/kube-apiserver.yaml & .kube/config file to go back to original port (8443)

Answer 2

It should not be possible when port range is set with default service-node-port-range api-server flag.

--service-node-port-range portRange Default: 30000-32767
A port range to reserve for services with NodePort visibility. Example: '30000-32767'. Inclusive at both ends of the range.

---

What can I do to get my k8s cluster back?

You can change /etc/kubernetes/manifests/kube-apiserver.yaml file and reaplace all occurrences of 8443 to e.g. 6443.

Then connect to apiserver with:

$ kubectl --server https://<ip>:6443 ...

Delete the my-service service and restore kube-apiserver.yaml file to its previous form.

---

Why kube-apiserver or other components doesn't prevent me to change the nodePort to 8443? Indicate the port 8443 is already in use.

I don't think many people encounter this problem because people usually use api-server with default values of --service-node-port-range flag which is: 30000-32767. In such case you would see this error:

The Service "my-service" is invalid: spec.ports[0].nodePort: Invalid value: 8443: provided port is not in the valid range. The range of valid ports is 30000-32767

Probably the best thing you can do with it is to ask k8s developers on k8s github repo. Just open an issue and as this question there.