accessing data from aws s3 in thanos, prometheus and minio stack

22 Views Asked by At

i have 3 nodes, 2 of them in the same lan and the 3rd are remote(different country). on the 2 nodes: the 1st node runs a docker compose with some services that exports metrics, and another compose with prometheus and thanos sidecar which gather the metrics and export them to minio. on the 2nd node there is a minio instance (runs as a container) and a python script that backup the data to aws s3 bucket when there is internet connection. in the 3rd nodes i have a docker compose with thanos-query and thanos-store which configure to receive the data from aws s3 bucket. i can see the data in aws in the same format as in the minio bucket. it seems that the store is not receiving the data correctly or at all. i can see the store in thanos-query but no data. this is my prometheus-compose.yaml

services:
  prometheus:
    image: prom/prometheus:v2.51.0
    container_name: prometheus
    restart: always
    user: root
    ports:
      - "9090:9090"
    volumes:
      - ./data/prometheus:/data
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - ./alerts:/etc/prometheus/rules/
    command:
      - --config.file=/etc/prometheus/prometheus.yml
      - --web.enable-admin-api
      - --storage.tsdb.path=/data
      - '--web.console.libraries=/etc/prometheus/console_libraries'
      - '--web.console.templates=/etc/prometheus/consoles'
      - '--storage.tsdb.retention.time=2h'
      - '--web.enable-lifecycle'
      - '--web.listen-address=:9090'
      - '--storage.tsdb.min-block-duration=5m'
      - '--storage.tsdb.max-block-duration=5m'    
      
  thanos-sidecar:
    image: quay.io/thanos/thanos:v0.34.1
    container_name: thanos-sidecar
    restart: always
    user: 0:1001
    environment:
      - MINIO_BUCKET_NAME=${MINIO_BUCKET_NAME}
      - MINIO_ACCESS_KEY=${MINIO_ACCESS_KEY}
      - MINIO_SECRET_KEY=${MINIO_SECRET_KEY}
      - MINIO_ENDPOINT=${MINIO_ENDPOINT}
    command:
      - "sidecar"
      - "--log.level=debug"
      - "--tsdb.path=/data"
      - "--prometheus.url=http://prometheus:9090"
      - "--reloader.config-file=/etc/config/prometheus.yml"
      - |
          --objstore.config=type: S3
          config: 
            bucket: ${MINIO_BUCKET_NAME}
            access_key: ${MINIO_ACCESS_KEY}
            secret_key: ${MINIO_SECRET_KEY}
            endpoint: ${MINIO_ENDPOINT}
            insecure: true
    volumes:
      - ./data/prometheus:/data
      - ./prometheus.yml:/etc/config/prometheus.yml:ro
      - ./thanos/sidecar:/etc/thanos
    expose:
      - 10902
      - 10901

thanos-query.yaml:

services:
  thanos-store:
    image: quay.io/thanos/thanos:v0.34.1
    container_name: thanos-store
    restart: always
    user: 0:1001
    volumes:
      - ./thanos/store:/etc/thanos
      - ./bucket.yml:/etc/prometheus/bucket.yml
    command:
      - "store"
      - "--log.level=debug"
      - "--data-dir=/data"
      - "--log.format=logfmt"
      - "--index-cache-size=250MB"
      - "--chunk-pool-size=1GB"
      - "--store.grpc.series-max-concurrency=20"
      - "--sync-block-duration=3m"
      - "--block-sync-concurrency=20"
      - "--objstore.config-file=/etc/prometheus/bucket.yml"
    expose:
      - 10902
      - 10901
    ports:
      - '10912:10902'

  thanos-query:
    image: quay.io/thanos/thanos:v0.34.1
    container_name: thanos-query
    restart: always
    command:
      - "query"
      - "--log.level=debug"
      - "--log.format=logfmt"
      - "--store=thanos-store:10901"
      - "--store.sd-interval=5m"
      - "--query.replica-label=monitor"
    expose:
      - 10902
      - 10901
    ports:
      - "10902:10902"

my bucket.yaml:

type: S3
config:
  bucket: "<my-bucket>"
  endpoint: "s3.<my-region>.amazonaws.com"
  region: "<my-region>"
  access_key: "<my_key>"
  secret_key: "<my_secret>"

tested and double checked that the permission to the bucket is not the problem. i suspect the problem might be that the python script is not copying the data in the same manner as thanos-sidecar, though the format looks the same. when i configured thanos-query on the same host as prometheus i get the data from minio just fine (2h retention time so i know that im getting from minio and from thanos-sidecar). end to end everything looks configured ok exept that i cant read metrics from aws s3. there are no error in logs of thanos-store and query and i can see the store in thanos-query(in the 3rd node) is this setup even possible? (the use of standalone thanos-query with thanos-store just for read) thank you

0

There are 0 best solutions below