Thanos S3 Config

Question

I am struggling to get my head around of using S3 bucket for thanos.

It appears from the config that we have to provide access and secret keys for the S3 bucket, but they are being written in plain sight and as the code is stored in git so anyone can access the bucket.

Is there anyway of providing them as kubernetes secrets? i am trying to run prometheus-operator on EKS

Thanks,

Answer 1

Assuming you want to pass bucket configuration with auth data explicitely then you can use two command line options for thanos to achieve that:

1. --objstore.config-file=FILEPATH
2. --objstore.config=CONFIG_CONTENTS

With 1. you can just mount the secret and point thanos to that location With 2 you can use env variable and load secret into that variable, for example:

      - args:
        - sidecar
(...)
        - --objstore.config=$(OBJSTORE_CONFIG)                                                                                                                                                                                                                                   
        env:
        - name: OBJSTORE_CONFIG
          valueFrom:
            secretKeyRef:
              key: thanos-bucket.yaml
              name: thanos-service-account

There are some other ways to use identity information, which are cloud-specific. I'm not familiar with AWS offerring, so I'll refer you to thanos docs

Answer 2

For EKS >= 1.13, you can use IAM Role for Service Account. The gist of it is first to create an IAM role with access to the S3 bucket, and update the trust policy to trust the EKS cluster's OIDC identity provider sts:AssumeRoleWithWebIdentity. You then create and annotate a Service Account in EKS with the ARN of the IAM role (eks.amazonaws.com/role-arn: <IAM_ROLE_ARN>), and assign this Service Account to your pod. Henceforth, API calls to AWS from SDK within the pod (the SDK must support sts:AssumeRoleWithWebIdentity in the credential chain) will be done with the IAM role assumed.

For Prometheus and Thanos, you will need to annotate the Service Accounts used by Prometheus replica pods (for Thanos sidecar container to use the IAM role) and Thanos store gateway pods.

More details can be found in AWS official docs.