Testing thanos object storage upload without waiting 2 hours

Question

I was working on a docker-compose file using Prometheus to remote write data to a Thanos receiver. This data would then be queried by Thanos Querier as well as uploaded to minIO. When configuring the minIO portion, I ran into an issue where I could not tell whether or not I had correctly configured the minIO and Thanos receive. Is it possible to increase the rate at which Thanos uploads data to make it easier to test if this step in the process is working?

Answer 1

Thankfully after a few hours of plugging away at this problem and researching it, I found a solution and wanted to share it so hopefully, this could be a little easier for the next person to find.

This also works for any other object storage system compatible with Thanos receive

The solution that ended up working for me was to set the tsdb.min-block-duration=15s and tsdb.max-block-duration=15s. They have to be the same in order to not get any compaction in the receiver.

This makes it so Thanos uploads a block every 15 seconds so you can get immediate feedback if the project is working as expected. Obviously, this is not beneficial for production use but it will (hopefully) make your development process a little faster. You could also use this in a docker run command if you wish.

Here is what my Thanos receive service looks like

receive:
    user: "root"
    container_name: receive
    volumes:
      - './receive-data:/receive/data'
      - './bucket.yml:/bucket.yml'
    image: thanosio/thanos:v0.31.0
    command:
    - "receive"
    - '--tsdb.path=/receive/data'
    - "--tsdb.min-block-duration=2h"
    - "--tsdb.max-block-duration=2h"
    - "--grpc-address=0.0.0.0:10907"
    - "--grpc-grace-period=1s"
    - "--http-address=0.0.0.0:10909"
    - "--http-grace-period=1s"
    - "--label=receive=\"true\""
    - "--remote-write.address=0.0.0.0:10908"
    - "--objstore.config-file=/bucket.yml"

    expose:
      - 10907
      - 10908
      - 10909
    networks:
    - tunnel
    - cloud