Is there a way to set up alert with Prometheus storage retention size

Question

Is there a way to set a prometheus alert on storage.tsdb.retention.size, lets say if the retention size has been hit, i want an alert sent out.

Answer 1

• As per the prometheus operational flags --storage.tsdb.retention.size flag is set to the maximum number of bytes that can be stored for blocks and cannot set alerts for this . The supported units to set retention size are : B, KB, MB, GB, TB, PB, EB. Ex: "512MB". Based on powers-of-2, 1KB is 1024B. Use this with server mode only.

• As the alert configuration is unable to be set, you can use this flag --storage.tsdb.retention.time: to remove old data. By this make sure storage is free. This retention time defaults to 15d. Units Supported: y, w, d, h, m, s, ms. Use this with server mode only.

  Example : --storage.tsdb.retention.time=2h - erases old information after 2 hours. This is the lowest supported retention time.

• As this is a valid request you can take help/support from this prometheus official site or raise a git issue under the prometheus git community.

• If you need to configure the prometheus alert for disk usage then use the node_filesystem_avail_bytes flag. Refer to this How to configure alerts in Prometheus for diskspace by Abraham Dhanyaraj Arumbaka.

Answer 2

I'm not sure about storage.tsdb.retention.size, but another way to accomplish this is to create an alert rule for prometheus persistent volume being used as storage.

The example below would be triggered if disk usage > 90%:

      - alert: HighDiskUsageOnPVC
        expr: 100 * sum(kubelet_volume_stats_used_bytes) by (persistentvolumeclaim)/sum(kubelet_volume_stats_capacity_bytes) by (persistentvolumeclaim) > 90
        for: 1m
        labels:
          severity: warning
        annotations:
          summary: High disk usage on Persistent Volume {{ $labels.persistentvolumeclaim }}
          description: "More than 90% of the disk was used."