Using the kube-prometheus-stack helm chart, how can I see only the pods I want in a particular namespace as targets in Promethes?

Question

Using the kube-prometheus-stack helm chart, I want to scrape the metrics of only 3 pods out of 6 pods in the app namespace in K8S with Prometheus.

In this context, I added the additionalScrapeConfigs code blog to the values.yaml file and added the following change.

additionalScrapeConfigs:
  - job_name: new-app-api-metrics
    scrape_interval: 15s
    metrics_path: /app-api-metrics
    honor_labels: true
    kubernetes_sd_configs:
    - role: endpoints
      namespaces:
        names:
          - app
    relabel_configs:
    - source_labels: [__meta_kubernetes_endpoint_port_name]
      separator: ;
      regex: http
      replacement: $1
      action: keep
    - source_labels: [__meta_kubernetes_namespace]
      separator: ;
      regex: (.*)
      target_label: namespace
      replacement: $1
      action: replace
    - source_labels: [__meta_kubernetes_pod_name]
      separator: ;
      regex: new-app-api-.*
      target_label: pod
      replacement: $1
      action: replace
    - source_labels: [__meta_kubernetes_service_name]
      separator: ;
      regex: new-app-api-.*
      target_label: service
      replacement: $1
      action: replace
    - source_labels: [__meta_kubernetes_service_name]
      separator: ;
      regex: new-app-api-.*
      target_label: job
      replacement: ${1}
      action: replace
    - separator: ;
      regex: new-app-api-.*
      target_label: endpoint
      replacement: http
      action: replace

Then I added the additionalServiceMonitors code blog to the values.yaml file and added the following change.

additionalServiceMonitors:
- name: new-app-api-metrics
  additionalLabels:
    release: new-service-test-kube-prometheus-stack
    app.kubernetes.io/name: new-app-api
    app.kubernetes.io/instance: new-app-api
  jobLabel: new-app-api-metrics
  namespaceSelector: 
    matchNames: 
      - new-app-api
  selector:
    matchLabels:
      app.kubernetes.io/instance: new-app-api
      app.kubernetes.io/name: new-app-api
      release: new-service-test-kube-prometheus-stack
  endpoints: 
    - port: "3000"
      interval: 15s
      path: /app-api-metrics
      scheme: http

But when I look at the Prometheus UI, the new-config-app-api pods in the app namespace are also seen as targets, besides the one I want to pull their metrics from.

However, I just want to scrape the metrics of 3 new-app-api pods from 6 pods in the app namespace.

What other changes do I need to make?

Answer 1

Finally, I was able to filter the pods I wanted by adding the following source_labels to the relabel_configs section in job_name.

     - source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_name,       __meta_kubernetes_service_labelpresent_app_kubernetes_io_name]
      separator: ;
      regex: (new-app-api);true
      replacement: $1
      action: keep