Is there a way to get actual Pod hostname in Prometheus

Question

kube_pod_labels does populates pod_label_node_name and pod_label_host_ip for the given pod however the values don't correspond to the actual node name and ip of the pod. Instead it represents the node name and ip for kube-state-metrics/node-exporter node.

As a result these values are to be same for all the pods.

What is the right way to get the actual node name and ip of the Pod in Prometheus?

Answer 1

It looks like you have improper configuration for scraping kube-state-metrics target. This target must be scraped spearately from other pods, and it shouldn't have relabeling rules, which set node, pod, container and namespace labels, since these labels are already set to proper values by kube-state-metrics itself - see these docs. kube-state-metrics also sets labels based on pod annotations and pod labels, so these labels also shoudn't be set by relabeling config for kube-state-metrics target.

When kube-state-metrics is scraped correctly, then the hostname of every pod can be obtained via node and host_ip label at kube_pod_info metric - see these docs.