kubectl exec error dialing backend: x509: certificate signed by unknown authority

Question

After a long struggle I just created my cluster, deployed a sample container busybox now i am trying to run the command exec and i get the following error:

error dialing backend: x509: certificate signed by unknown authority

How do i solve this one: here is the command output with v=9 log level. kubectl exec -v=9 -ti busybox -- nslookup kubernetes I also noticed in the logs that this curl command that failed is actually the second command the first GET command passed and it return results without any issues.. ( GET https://myloadbalancer.local:6443/api/v1/namespaces/default/pods/busybox 200 OK)

curl -k -v -XPOST  -H "X-Stream-Protocol-Version: v4.channel.k8s.io" -H "X-Stream-Protocol-Version: v3.channel.k8s.io" -H "X-Stream-Protocol-Version: v2.channel.k8s.io" -H "X-Stream-Protocol-Version: channel.k8s.io" -H "User-Agent: kubectl/v1.19.0 (linux/amd64) kubernetes/e199641" 'https://myloadbalancer.local:6443/api/v1/namespaces/default/pods/busybox/exec?command=nslookup&command=kubernetes&container=busybox&stdin=true&stdout=true&tty=true'
I1018 02:19:40.776134  129813 round_trippers.go:443] POST https://myloadbalancer.local:6443/api/v1/namespaces/default/pods/busybox/exec?command=nslookup&command=kubernetes&container=busybox&stdin=true&stdout=true&tty=true 500 Internal Server Error in 43 milliseconds
I1018 02:19:40.776189  129813 round_trippers.go:449] Response Headers:
I1018 02:19:40.776206  129813 round_trippers.go:452]     Content-Type: application/json
I1018 02:19:40.776234  129813 round_trippers.go:452]     Date: Sun, 18 Oct 2020 02:19:40 GMT
I1018 02:19:40.776264  129813 round_trippers.go:452]     Content-Length: 161
I1018 02:19:40.776277  129813 round_trippers.go:452]     Cache-Control: no-cache, private
I1018 02:19:40.777904  129813 helpers.go:216] server response object: [{
  "metadata": {},
  "status": "Failure",
  "message": "error dialing backend: x509: certificate signed by unknown authority",
  "code": 500
}]
F1018 02:19:40.778081  129813 helpers.go:115] Error from server: error dialing backend: x509: certificate signed by unknown authority
goroutine 1 [running]:

Adding more information: This is on UBUNTU 20.04. I went through step by step creating my cluster manually as a beginner I need that experience instead of spinning up with tools like kubeadm or minikube

xxxx@master01:~$ kubectl exec -ti busybox -- nslookup kubernetes
Error from server: error dialing backend: x509: certificate signed by unknown authority
xxxx@master01:~$ kubectl get pods --all-namespaces
NAMESPACE              NAME                                         READY   STATUS    RESTARTS   AGE
default                busybox                                      1/1     Running   52         2d5h
kube-system            coredns-78cb77577b-lbp87                     1/1     Running   0          2d5h
kube-system            coredns-78cb77577b-n7rvg                     1/1     Running   0          2d5h
kube-system            weave-net-d9jb6                              2/2     Running   7          2d5h
kube-system            weave-net-nsqss                              2/2     Running   0          2d14h
kube-system            weave-net-wnbq7                              2/2     Running   7          2d5h
kube-system            weave-net-zfsmn                              2/2     Running   0          2d14h
kubernetes-dashboard   dashboard-metrics-scraper-7b59f7d4df-dhcpn   1/1     Running   0          2d3h
kubernetes-dashboard   kubernetes-dashboard-665f4c5ff-6qnzp         1/1     Running   7          2d3h
tinashe@master01:~$ kubectl logs busybox
Error from server: Get "https://worker01:10250/containerLogs/default/busybox/busybox": x509: certificate signed by unknown authority
xxxx@master01:~$

xxxx@master01:~$ kubectl version
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.3", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-10-14T12:50:19Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.3", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-10-14T12:41:49Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"linux/amd64"}

Answer 1

**Edited for simplicity:

my cluster operator kube-apiserver was degraded, causing my certificate failures. Resolving that degradation was necessary to resolve the overarching problem, resulting in x509 errors. Validate that all masters are in READY, pods in your apiserver projects are also scheduled and ready. See below KCS for more information:

https://access.redhat.com/solutions/4849711

**removed below outdated/incorrect information about local cert pull/export.

Answer 2

I've faced a similar issue when the server's ip changed (restored a snapshot on a new vm). After lots of struggles I was able to fix that regenerating the server certificates.

In case you're running standard Kubernetes:

kubeadm certs renew all 

In my case using Microk8s:

sudo microk8s refresh-certs --cert ca.crt
sudo microk8s refresh-certs --cert front-proxy-client.crt
sudo microk8s refresh-certs --cert server.crt