As we have setup Prometheus and Grafana using helm charts and we have configured email alerts for:
- If a pod is created send me email alert
- If a pod if in pending, error, unknown state send me email notification
sum by (namespace, pod) (
kube_pod_created{job="kubernetes-service-endpoints", namespace=~".*" } > (time() - 600)
)
sum by (namespace, pod, node, instance, job, service) (
max by (namespace, pod, node, instance, job, service) (
kube_pod_status_phase{job="kubernetes-service-endpoints",namespace=~".*",phase=~"Pending|Unknown"}
)
* on (namespace, pod) group_left (node)
max by (namespace, pod, node, owner_kind, deployment, container) (kube_pod_owner{owner_kind!="Job"})
)
>
0
We are trying to get email notification for deletion of a pod at present.
Can you suggest anything?
I tried some queries to get information about the pods which are deleted:
sum by (namespace, pod) (
kube_pod_container_status_terminated{job="kubernetes-service-endpoints", namespace=~".*" } > (time() - 600)
)
but I am not able to get the proper information.