My kubernetes cluster has a pod deployment definition to have 10 pod copies. I ran into an issue where a pod instance's container will go into an unrecoverable state where the only mitigation I found is to restart the pod deployment through the following command:
kubectl rollout restart deployment my-pod
I would like to a way to auto execute a pod deployment restart through liveness probes or even pod condition status.
Are there any current out of the box solutions for this?
Kubernetes's pod object provides you with a health check function. If the pod is unhealthy, the pod controller will automatically restart your pod.
Please check the official documentation here, https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/.
It provides all the implementations you will need in order to make the health check probe work.