Thanos Querier Deduplication Issue with Prometheus

Question

We are new to Prometheus Monitoring and are facing an issue. On our Kubernetes cluster, to monitor our MongoDB application, we are using the following setup :

1. MongoDB exporter (installed using prometheus-community prometheus-mongodb-exporter Helm Chart), to get metrics from MongoDB application, convert them to prometheus-readable format and expose them at /metrics endpoint. Below values.yaml file was used to override the default chart values. This file is passed as an argument to the helm install -f command.

values.yaml:

mongodb:
    uri: "mongodb://<Username>:<Password>@pod0.service.monitoring.svc.cluster.local:27017"
serviceMonitor:
     additionalLabels:
        app.kubernetes.io/instance: prometheus-operator

2. Prometheus with thanos as sidecar (installed using bitnami's prometheus-operator Helm chart). We specified 3 replicas for Prometheus to provision scalability for the deployed Prometheus. Thanos as sidecar is being used to provide HA for the deployed prometheuses. External labels are provided so that Thanos Querier (mentioned in Point# 3 will use it to deduplicate the data). We are using the following values.yaml file to override the default prometheus-operator values. This file is passed as an argument to the helm install -f command.

values.yaml:

prometheus:
   thanos:
     create: true
     service:
      type: ClusterIP
     externalLabels:
          cluster: prometheus-ha
          replica: test
      replicaExternalLabelName: "replica"
      prometheusExternalLabelName: "replica"
  storageSpec:
    volumeClaimTemplate:
      spec:
        # Name of the PV you created beforehand
        # volumeName: prometheus-hostpath-pv
        accessModes: ["ReadWriteOnce"]
        # StorageClass should match your existing PV's storage class
        storageClassName: manual
        resources:
          requests:
            # Size below should match your existing PV's size
            storage: 3Gi
  replicaCount: 3

3. Thanos Querier (installed using bitnami's thanos Helm Chart) to query metrics from all the thanos sidecars and deduplicate the data received from all the sidecars using the replicaLabel parameter. This replicaLabel parameter should match the externalLabels "replica" label name. Following values.yaml was used during installation using helm ls command.

values.yaml:

query:
        stores:
                - prometheus-thanos-sidecar-custom-headless-service-for-pod-0.monitoring.svc.cluster.local:10901
                - prometheus-thanos-sidecar-custom-service-for-pod-1.monitoring.svc.cluster.local:10901
                - prometheus-thanos-sidecar-custom-service-for-pod-2.monitoring.svc.cluster.local:10901    
        replicaLabel:
                - replica

But the deduplication of the data is not happening, since the same data/metric from all the replicas is shown in the Thanos UI (Screenshot attached), even with deduplication enabled. Also, in Grafana for a few metrics, we get the error "Only queries that return single series/table is supported" which I guess is because of deduplication not happening.

Kindly need help us in fixing the deduplication issue.

Thanos Deduplication issue in Thanos UI

Answer 1

Deduplication in Thanos Query is based on replicaLabel config, which you have it wrong, it should be like this based on your screenshot:

        replicaLabel:
                - prometheus_replica