Monitor only one namespace pods - Prometheus & Kubernetes & cadvisor

Question

I have deployed istio on kubernetes, and I installed prometheus from istio addons. My goal is to only monitor some pods of one application(such as all pods of bookinfo application). The job definition for monitoring pods is as below:

    - bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      job_name: kubernetes-nodes-cadvisor
      kubernetes_sd_configs:
      - role: node
      relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)
      - replacement: kubernetes.default.svc:443
        target_label: __address__
      - regex: (.+)
        replacement: /api/v1/nodes/$1/proxy/metrics/cadvisor
        source_labels:
        - __meta_kubernetes_node_name
        target_label: __metrics_path__
      scheme: https
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecure_skip_verify: true

My problem is that I don't know how to monitor only one namespace's pods. For example, I deploy the bookinfo application in a namespace named Book. I only want the metrics of pods from namespace Book. However, prometheus will collect all pods metrics of the nodes. Instead of changing annotations of the application like Monitor only one namespace metrics - Prometheus with Kubernetes, I want know if there is a method to select only one namespace by changing the job definition above. Or is there some way to choose the monitor pods by it's labels?

Answer 1

The following will match all target pods with label: some_label with any value.

relabel_configs:
  - action: keep
    source_labels: [__meta_kubernetes_pod_label_some_label]
    regex: (.*)

If you want to keep targets with label: monitor and value: true you would do:

relabel_configs:
  - action: keep
    source_labels: [__meta_kubernetes_pod_label_monitor]
    regex: true

All pods that don't match it will be dropped from scraping.

The same you should be able to do for namespaces:

relabel_configs:
  - action: keep
    source_labels: [__meta_kubernetes_namespace]
    regex: Book

---

EDIT >

is there a way to change the [container_label_io_kubernetes_container_name] labels into "container_name"?

Try this:

relabel_configs:
  - action: replace
    source_labels: [container_label_io_kubernetes_container_name]
    target_label: container_name

It's all explained in prometheus docs about configuration

Answer 2

this worked for me.

  - job_name: "kubernetes-cadvisor"
    scheme: https
    metrics_path: /metrics/cadvisor
    tls_config:
      ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      # disable certificate verification by uncommenting the line below.
      # insecure_skip_verify: true
    authorization:
      credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    kubernetes_sd_configs:
      - role: node
    relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)
    metric_relabel_configs:
      - action: keep
        source_labels: [namespace]
        regex: tsb. #namespace name you want

Answer 3

I found a method in cadvisor's document. https://github.com/google/cadvisor/blob/master/docs/runtime_options.md It says that we can change the parameter '--docker_only' and '--raw_cgroup_prefix_whitelist' to choose the container to be monitored.