I am attempting to set up a single node Kubernetes cluster for testing on an Ubuntu Server 18.04.2 VM running on my Windows Hyper-V. I install Docker with the standard apt install docker.io command, and I initialize Kubernetes the cluster with the command kubeadm init --pod-network-cidr=10.244.0.0/16
and then add the Flannel CNI with kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml and then install rabbitmq-ha using its "stable" Helm chart.
If the VM is using the Hyper-V’s “Default Switch”, then everything works as expected. I also have an “Extra Switch” configured as an “External network” with “Allow management operating system to share this network adapter” checked, pointing to my wireless adapter. When I am in the office, with the Ubuntu VM set to use this “Extra Switch”, the IP address of my laptop and the Ubuntu VM (eth0) is set to 10.X.X.X, and everything works as expected.
When I am at home, with the Ubuntu VM set to use this same “Extra Switch”, the IP address of my laptop and the Ubuntu VM (eth0) is set to 192.168.X.X, and rabbitmq pod throws errors and continuously restarts, with the logged error message:
Failed to get nodes from k8s - {failed_connect,[{to_address,{"kubernetes.default.svc.cluster.local",443}}, {inet,[inet],nxdomain}]}
Doing some google research tells me that the rabbitmq pod cannot resolve the name "kubernetes.default.svc.cluster.local".
Adding the “apiserver-advertise-address” to kubeadm inti, set to my laptop’s 192.168.X.X IP address, does not make a difference.
I also tried using Weave Works instead of Flannel, and omitting the CIDR in the ‘kubeadm init’ command, but I get the same rabbitmq error at home on "Extra Switch", while working on the home Default Switch and working on the office network.
So what am I missing? Is there a Ubuntu network setting I need when running under Hyper-V and an external switch? Or is there a Kubernetes or Docker configuration setting that I am missing? Thanks!
As @Steve solved problem, solution is:
Firstly the newer version of the RabbitMQ Helm chart does not have this problem. But if you want to find workaround the "bad" name resolution, you can add a DNS rewrite rule to the CoreDNS config map, see configmap-coredns.
Take a look at coredns-issue for a full description of the problem and how to narrowed it down to the particular DNS name in the chain that needed the "rewrite" rule.