2 different prometheus server to scrapping different data but shows combined in their /targets

Question

I have 2 different prometheus server (with its set of alertmanager, pushgateway, nodeExporter and kube-state-metrics)

When I port forward either of these 2 servers, in their respective /targets - I find combined details for the below job_name(s) :

1. kubernetes-nodes (20/20 up)
2. kubernetes-nodes-cadvisor (20/20 up)
3. kubernetes-pods (90/90 up)
4. kubernetes-service-endpoints (78/78 up)
5. prometheus-pushgateway (2/2 up)
6. prometheus (1/1 up)
7. kubernetes-apiservers (1/1 up)

My values.yaml file (different for both servers) contains the following part -

prometheus:
  serverFiles:
    prometheus.yml:
      rule_files:
        - /etc/config/recording_rules.yml
        - /etc/hcp/config/alerting_rules.yml
      scrape_configs:
        - job_name: prometheus
          static_configs:
            - targets:
                - localhost:9090 #exposing metrics

I want to ensure that for prometheus-server-1 should only scrape details related to it and same goes with prometheus-server-2, & have only their details in their /targets despite having the same default ports used by prometheus.

Answer 1

While going through all the prometheus docs and experimenting different ways, finally am able to get through to have prometheus-server scraping data only related to it, hence answering my own question.

This can be achieved by adding relabel_configs condition in the respective jobs under values.yaml file shared above.

Example for job_name: prometheus-pushgateway:

prometheus:
  serverFiles:
    prometheus.yml:
      rule_files:
        - /etc/config/recording_rules.yml
        - /etc/hcp/config/alerting_rules.yml
      scrape_configs:
        - job_name: prometheus
          static_configs:
            - targets:
                - localhost:9090 #exposing metrics
      - job_name: prometheus-pushgateway
        kubernetes_sd_configs:
        - role: service
        relabel_configs:
        - action: keep
          regex: pushgateway
          source_labels:
          - __meta_kubernetes_service_annotation_prometheus_io_probe
        # Added below condition to ensure it keeps only specific pushgateway
        - action: keep
          regex: first-prometheus-pushgateway.namespace.svc:9091
          source_labels:
          - __address__

**NOTE: scrape_configs overwrites all the default scrape_configs, so kindly include all default statements while adding your specific conditions to it.

For additional info, below links can be helpful:

• https://github.com/prometheus-community/helm-charts/blob/prometheus-13.8.0/charts/prometheus/values.yaml#L1296
• https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config