How can we include custom labels/annotations of K8s objects in Prometheus metrics?

Question

For example I have StatefulSet with custom labels/annotations applied on it.

Everyone mentions these two metrics should be used, because kube-state-metrics should generate labels/annotations as well.

kube_statefulset_annotations
kube_statefulset_labels

The thing is, I can see only default ones (job,instance, namespace,...) but not additionally added labels/annotations.

Example of the manifest I am testing with:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  annotations:
    label_network: "111"
  labels:
    app: testing-labels
    label_STATEFULSET_LABEL: "111"
    label_network: "111"
  name: testing-labels
  namespace: sre-test
spec:
  selector:
    matchLabels:
      app: testing-labels
  serviceName: testing-labels-headless
  template:
    metadata:
      labels:
        app: testing-labels
        label_network: "111"

I've added so many different labels/annotations but the kube_statefulset_labels{statefulset='testing-labels'} returns:

kube_statefulset_labels{container="kube-rbac-proxy-main", instance="10.2.23.229:8443", job="kube-state-metrics", namespace="sre-test", prometheus="aws-monitoring/k8s", prometheus_replica="prometheus-k8s-1", statefulset="testing-labels"}

...which certainly doesn't contain any extra label. Any idea would be helpful?

Versions:

• kube-state-metrics:v2.5.0
• kube-rbac-proxy:v0.12.0

Answer 1

Guess you are referring to these metrics. https://github.com/kubernetes/kube-state-metrics/blob/master/docs/statefulset-metrics.md

According to description I would expect a list with all labels in following format.

statefulset=<statefulset-name>
namespace=<statefulset-namespace>
label_STATEFULSET_LABEL=<STATEFULSET_LABEL>

Guess it could help when you share some more details.

Maybe the manifest of you statefullset and what you get when calling the metric.

Answer 2

In order to get custom metrics within kube_statefulset_labels we need to add

--metric-labels-allowlist as a flag in kube-state-metrics. (docs)

In this particular example it would be:

--metric-labels-allowlist=statefulsets=[label_network]