Back-off restarting failed container kubernetes

Question

After remove Kubernetes and re-install it on both master and node, I can't no longer install NGINX Ingress Controller to work correctly.

First, To remove Kubernetes I have done:

# On Master
k delete namespace,service,job,ingress,serviceaccounts,pods,deployment,services --all
k delete node k8s-node-0
sudo kubeadm reset 
sudo systemctl stop kubelet
sudo apt-get purge kubeadm kubectl kubelet kubernetes-cni kube* -y
sudo apt-get autoremove -y
sudo rm -rf ~/.kube /etc/cni


# On Node
sudo kubeadm reset 
sudo systemctl stop kubelet
sudo apt-get purge kubeadm kubectl kubelet kubernetes-cni kube* -y
sudo apt-get autoremove -y
sudo rm -rf ~/.kube 

Then, to re-install everything back, I have done:

# On Master
sudo apt install -y kubelet kubeadm kubectl
sudo kubeadm init
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
kubectl apply -f https://docs.projectcalico.org/v3.14/manifests/calico.yaml



# On Node
sudo apt install -y kubelet kubeadm kubectl
sudo kubeadm join 10.0.8.135:6443 --token 31xags.h9mr5dz6ncn632uv --discovery-token-ca-cert-hash sha256:c6b479e2130799a4e4d41c4a02dab54eedc431806171b92f4bbc1978d84bd91d

Then to install NGINX Ingress Controller:

git clone https://github.com/nginxinc/kubernetes-ingress.git
cd kubernetes-ingress/deployments 
k apply -f common/ns-and-sa.yaml 
k apply -f rbac/rbac.yaml
k apply -f common/default-server-secret.yaml
k apply -f common/nginx-config.yaml
k apply -f deployment/nginx-ingress.yaml
k apply -f daemon-set/nginx-ingress.yaml

Then when I execute k get all -n nginx-ingress i got:

NAME                      READY   STATUS             RESTARTS   AGE
pod/nginx-ingress-2thp4   0/1     CrashLoopBackOff   7          14m

NAME                           DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/nginx-ingress   1         1         0       1            0           <none>          14m

and the detail k describe pods nginx-ingress-2thp4 -n nginx-ingress:

Name:         nginx-ingress-2thp4
Namespace:    nginx-ingress
Priority:     0
Node:         k8s-node-0/10.0.8.66
Start Time:   Mon, 28 Sep 2020 15:22:01 +0700
Labels:       app=nginx-ingress
              controller-revision-hash=646bf8d696
              pod-template-generation=1
Annotations:  cni.projectcalico.org/podIP: 192.168.11.198/32
              cni.projectcalico.org/podIPs: 192.168.11.198/32
Status:       Running
IP:           192.168.11.198
IPs:
  IP:           192.168.11.198
Controlled By:  DaemonSet/nginx-ingress
Containers:
  nginx-ingress:
    Container ID:  docker://175d13f95564d98c06af5514b0519a035e5ee95872bb428fa94c9c2bfc6776a5
    Image:         nginx/nginx-ingress:edge
    Image ID:      docker-pullable://nginx/nginx-ingress@sha256:fdb07d0a639d0f2c761b4c5a93f6d5063b972b8ae33252bb7755bb5fb6da4fda
    Ports:         80/TCP, 443/TCP, 8081/TCP
    Host Ports:    80/TCP, 443/TCP, 0/TCP
    Args:
      -nginx-configmaps=$(POD_NAMESPACE)/nginx-config
      -default-server-tls-secret=$(POD_NAMESPACE)/default-server-secret
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    255
      Started:      Mon, 28 Sep 2020 15:33:09 +0700
      Finished:     Mon, 28 Sep 2020 15:33:09 +0700
    Ready:          False
    Restart Count:  7
    Readiness:      http-get http://:readiness-port/nginx-ready delay=0s timeout=1s period=1s #success=1 #failure=3
    Environment:
      POD_NAMESPACE:  nginx-ingress (v1:metadata.namespace)
      POD_NAME:       nginx-ingress-2thp4 (v1:metadata.name)
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from nginx-ingress-token-j9hjm (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  nginx-ingress-token-j9hjm:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  nginx-ingress-token-j9hjm
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/disk-pressure:NoSchedule op=Exists
                 node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                 node.kubernetes.io/not-ready:NoExecute op=Exists
                 node.kubernetes.io/pid-pressure:NoSchedule op=Exists
                 node.kubernetes.io/unreachable:NoExecute op=Exists
                 node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
  Type     Reason     Age                 From               Message
  ----     ------     ----                ----               -------
  Normal   Scheduled  15m                 default-scheduler  Successfully assigned nginx-ingress/nginx-ingress-2thp4 to k8s-node-0
  Normal   Pulled     15m                 kubelet            Successfully pulled image "nginx/nginx-ingress:edge" in 9.106863831s
  Normal   Pulled     15m                 kubelet            Successfully pulled image "nginx/nginx-ingress:edge" in 3.626387366s
  Normal   Pulled     14m                 kubelet            Successfully pulled image "nginx/nginx-ingress:edge" in 3.839665529s
  Normal   Created    14m (x4 over 15m)   kubelet            Created container nginx-ingress
  Normal   Started    14m (x4 over 15m)   kubelet            Started container nginx-ingress
  Normal   Pulled     14m                 kubelet            Successfully pulled image "nginx/nginx-ingress:edge" in 3.846965585s
  Normal   Pulling    13m (x5 over 15m)   kubelet            Pulling image "nginx/nginx-ingress:edge"
  Warning  BackOff    14s (x70 over 15m)  kubelet            Back-off restarting failed container

And logs k logs nginx-ingress -n nginx-ingress:

I0928 08:38:16.776841       1 main.go:245] Starting NGINX Ingress controller Version= GitCommit=
W0928 08:38:16.797787       1 main.go:284] The '-use-ingress-class-only' flag will be deprecated and has no effect on versions of kubernetes >= 1.18.0. Processing ONLY resources that have the 'ingressClassName' field in Ingress equal to the class.
F0928 08:38:16.802335       1 main.go:288] Error when getting IngressClass nginx: ingressclasses.networking.k8s.io "nginx" not found

Here is the kubectl version:

Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.2", GitCommit:"f5743093fd1c663cb0cbc89748f730662345d44d", GitTreeState:"clean", BuildDate:"2020-09-16T13:41:02Z", GoVersion:"go1.15", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.2", GitCommit:"f5743093fd1c663cb0cbc89748f730662345d44d", GitTreeState:"clean", BuildDate:"2020-09-16T13:32:58Z", GoVersion:"go1.15", Compiler:"gc", Platform:"linux/amd64"}

I have done a lot of searching but still couldn't find a way to fix this problem. Also have tried to re-install many time already but it never works.

Answer 1

I think this may happen because you want to use the nginx ingress controller in limited namespaces. Please try this patch applied to your ClusterRole definition nginx-ingress-clusterrole:

@@ -157,7 +160,7 @@ rules:
       - list
       - watch
   - apiGroups:
-      - "extensions"
+      - "networking.k8s.io"
     resources:
       - ingresses
     verbs:

Take a look: cluster-role-nginx-controller.

Answer 2

You also need to add

kubectl apply -f common/ingress-class.yaml

More info about IngressClass and changes done in 1.18+ can be found at https://kubernetes.io/docs/concepts/services-networking/ingress/

Answer 3

For my setup, the IngressClass wasn't created, as Zzorica answered. Specifically, this config needs to be applied.

Also, don't forget to add

spec:
  ingressClassName: nginx

to your Ingress object as well.