We have 4 nodes EKS cluster. We have some pods (part of Daemonset) in pending status as the Node is full and there is no capacity in the node to run the pod. The question is do we need to manually reshuffle the workloads to make Daemonset pods running in this situation or is there any configuration to overcome this issue in an automated fashion?
Note: we have also installed Cluster Autoscaler which works perfectly for deployments.
Thank you in advance.
Kubernetes has pod priorities and preemption for this specific purpose.
If EKS does not have priority classes pre-configured, you can create one yourself. For example, the one from the docs which is a preempting one:
Then you use that class on your daemon set
Note this is just a small example copied from linked docs, you should read the docs carefully and perhaps also review how this would interact with pod disruption budgets.
Also note, that this may cause disruption to other deployments, depending on various factors such as the Update Strategy. So, be careful.