Namespace PodNodeSelector no effect on existing running pods/statefulsets

Question

I have kubernetes cluster on bare-metal, and I want to have logical separation of nodes where DEV environment will be on less power machines while the production environment will be on most powerful machines. I implemented PodNodeSelector of Admission controller. It works well on newly created namespace but on existing namespace, edited and added annotation to it all existing pods on Statefulset are not going to the specified powerful machines, just keep running. How can I apply the new annotation and have effect on Statefulset without recreating or deleting the pods/Statefulsets, cant afford downtime on Elassandra

Answer 1

You cannot. This is how it works at the moment. Once a pod is scheduled on a node it stays on a node.

Same applies to affinity/antiaffinity.

But in k8s docs you can read:

In the future we plan to offer requiredDuringSchedulingRequiredDuringExecution which will be just like requiredDuringSchedulingIgnoredDuringExecution except that it will evict pods from nodes that cease to satisfy the pods' node affinity requirements.

And this is what you are looking for, but it is not implemented yet. It looks like deleting pods to force then to reschedule is the only option available at the moment.