Can we get information of pod deleted In Kubernetes Environment using Prometheus

89 Views Asked by At

As we have setup Prometheus and Grafana using helm charts and we have configured email alerts for:

  1. If a pod is created send me email alert
  2. If a pod if in pending, error, unknown state send me email notification
sum by (namespace, pod) (
  kube_pod_created{job="kubernetes-service-endpoints", namespace=~".*" } > (time() - 600)
)
sum by (namespace, pod, node, instance, job, service) (
                    max by (namespace, pod, node, instance, job, service) (
                      kube_pod_status_phase{job="kubernetes-service-endpoints",namespace=~".*",phase=~"Pending|Unknown"}
                    )
                  * on (namespace, pod) group_left (node)
                    max by (namespace, pod, node, owner_kind, deployment, container) (kube_pod_owner{owner_kind!="Job"})
                )
              >
                0

We are trying to get email notification for deletion of a pod at present.

Can you suggest anything?

I tried some queries to get information about the pods which are deleted:

sum by (namespace, pod) (
  kube_pod_container_status_terminated{job="kubernetes-service-endpoints", namespace=~".*" } > (time() - 600)
)

but I am not able to get the proper information.

0

There are 0 best solutions below