We have set up a bare metal non-cloud kubernetes cluster with an nginx-ingress controller bound to port 443 via the hostNetwork: true directive.
After a reboot we are hitting a strange "No route to host" error:
$ curl http://127.0.0.1:443
curl: (7) Failed connect to 127.0.0.1:443; No route to host
$ curl http://127.0.0.1:444
curl: (7) Failed connect to 127.0.0.1:444; Connection refused
I would have never guessed that such an error could be happening on port-level, or even on 127.0.0.1.
Interestingly, curl -g http://[::1]:443 does work ;-)
With lsof I do see that nginx is listening to port 443 in many processes. I didn't know that was possible. Apparently Linux has this mechanism SO_REUSEPORT noways for performant network traffic towards child worker processes. Refer to https://lwn.net/Articles/542629/
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.0", GitCommit:"e19964183377d0ec2052d1f1fa930c4d7575bd50", GitTreeState:"clean", BuildDate:"2020-08-26T14:30:33Z", GoVersion:"go1.15", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.6", GitCommit:"dff82dc0de47299ab66c83c626e08b245ab19037", GitTreeState:"clean", BuildDate:"2020-07-15T16:51:04Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
We are using flannel as the CNI network overlay, image:
quay.io/coreos/flannel:v0.12.0-amd64
Nginx-ingress image:
k8s.gcr.io/ingress-nginx/controller:v0.35.0@sha256:fc4979d8b8443a831c9789b5155cded454cb7de737a8b727bc2ba0106d2eae8b
What can I do troubleshoot this further?
I have removed the whole ingress-nginx namespace, and still this behavior occurs.
lsof does not show any listening of 443.
OS: RedHat Enterprise Linux 7.8
Firewalld is switched off.