Kubernetes - Calico CrashLoopBack on Containers

Question

I have just started experimenting with K8S a few days back, try to learn K8S with specific emphasis on networking, service mesh etc.

I am running 2 worker nodes and 1 master on VMs with Centos 7 and K8S, installed with kubeadm.

Default CNI of Flannel. Install was OK and everything except the networking was working. I could deploy containers etc, so a lot of control plane was working.

However, networking not working correctly, even container to container in same worker node. I checked all the usual suspects, the veths, IPs, MACs, briges on a single worker and everything seemed to check out... e.g. MACs where on the correct bridges i.e. cni0, IP address assignments etc. Even when pinging from busybox to busybox, I would see the ARP caches being populated but pings not working still.... disabled all FWs, IP forwarding enabled etc. Not an expert of IPtables but looked OK..... also when logged into the worker node shell I could ping the busybox containers, but they could not ping each other....

One question I have at this point, why is the docker0 bridge still present even when flannel is installed can I delete it or are there some dependencies associated with it ? I did not notice the veths for the containers were showing connected to docker0 bridge but docker bride0 was down... however I followed this website and it show a different way of validating and show veths connected to cni0, which is very confusing and frustrating.....

I gave up Flannel as I was just using Flannel to experiment and decided to try out Calico....

I followed install procedures from Calico site... was not entirely clear on the tidy up procedures for Flannel, not sure where this is documented?... this is where it went from bad to worse...

I started getting crash loops on calico containers and coredns stuck creating, reporting liveliness issues on calico ....and this is where I am stuck......... and would like some help.......

I am have read and tried many things on web and may have fixed some issues as there may be many in play, but would really appreciate any help....

=== install info and some output...

enter image description here

enter image description here

enter image description here

enter image description here

enter image description here

Some questions...

1. The Container creating for the coredns..... is this dependent on successful install of Calico... are the issues related.... or should coredns install work independent of the CNI install ?

Answer 1

The Container creating for the coredns..... is this dependent on successful install of Calico... are the issues related.... or should coredns install work independent of the CNI install ?

Yes, it is. You need to install a CNI to have coredns working.

When you setup your cluster with kubeadm there's is a flag called --pod-network-cidr, depending on which CNI you intend to use, you need to specify the range with this flag.

For example, by default, Calico use the range 192.168.0.0/16 and Flannel use the range 10.244.0.0/16.

I have a guide how to setup a cluster using kubeadm, maybe it help you.

Please note, if you want to replace the CNI without delete the entire cluster, extras steps need to be taken in order to "cleanup" the firewall rules from the older CNI.

See here how to replace flannel to calico, for example.

And here how to migrate from flannel to calico.