"error retrieving resource lock i/o timeout" kubebuilder k8s controller deployment

Question

I'm trying to deploy a custom controller built using kubebuilder to my local k8s cluster. I have installed this k8s cluster from Docker Desktop settings.

Now, everything is working as expected when I ran it using "make run" on my machine. As it is working fine, I've ran the command "make docker-build custom-controller:0.1.0" and then ran the "make deploy custom-controller:0.1.0"

This worked fine, it has created a seperate namespace and this controller is running as a pod in that namespace and when I see the logs, I'm getting the following errors

2024-03-26T13:49:16Z    INFO    setup   starting manager
2024-03-26T13:49:16Z    INFO    starting server {"kind": "health probe", "addr": "[::]:8081"}
2024-03-26T13:49:16Z    INFO    controller-runtime.metrics  Starting metrics server
2024-03-26T13:49:16Z    INFO    controller-runtime.metrics  Serving metrics server  {"bindAddress": "127.0.0.1:8080", "secure": false}
I0326 13:49:16.092772       1 leaderelection.go:250] attempting to acquire leader lease vault-system/9737ddd2.ivzdso...
E0326 13:49:46.093895       1 leaderelection.go:332] error retrieving resource lock vault-system/9737ddd2.ivzdso: Get "https://10.96.0.1:443/apis/coordination.k8s.io/v1/namespaces/vault-system/leases/9737ddd2.ivzdso": dial tcp 10.96.0.1:443: i/o timeout
E0326 13:50:18.973156       1 leaderelection.go:332] error retrieving resource lock vault-system/9737ddd2.ivzdso: Get "https://10.96.0.1:443/apis/coordination.k8s.io/v1/namespaces/vault-system/leases/9737ddd2.ivzdso": dial tcp 10.96.0.1:443: i/o timeout

The ip 10.96.0.1 is the cluster-ip

kubernetes ClusterIP 10.96.0.1 443/TCP 4d21h

What is the reason behind this? Is it some issue with the k8s controller or the cluster itself. And is there any way to fix this?

Answer 1

There is an article by Dipesh Shah with the same error encountered. As per the article it might be a permission issue with the service account for ingress controller while acquiring leases.

They extracted the clusterrole yaml with

kubectl get clusterrole -n ingress-nginx

kubectl get clusterrole nginx-ingress-ingress-nginx -n ingress-nginx -o yaml

Once extracted, the yaml clusterrole lists down all the rules associated with different apiGroups, however you should keep a close eye with "coordination.k8s.io"

They updated their clusterrole yaml with the command and added the rule for "coordination.k8s.io" apiGroup

kubectl edit clusterrole nginx-ingress-ingress-nginx -n ingress-nginx -o yaml

Code snippet added:

-apiGroups:

— coordination.k8s.io

resources:

— leases

verbs:

— create

— get

— list

— update

You may consider checking the API server address, typically kubebuilder controllers use in-cluster configuration to discover the API server address automatically. Verify if there's any explicit configuration overriding the default behavior.