Defaulting a prometheus counter metric to 0

Question

i'm facing a nightmarishly hard problem to solve for something that seems so trivial. Ultimately, i'm trying to detect a single increment of a counter metric in PromQL when there was no data previously in order to trigger an alert.

I've hit a dead-end in PromQL and the only other solution seems to be to initialise a counter to 0 using Benthos (i.e serve the counter as a value as 0, even though it's not been incremented in the processor pipeline). Is this possible? if it is, it would allow me to detect a change from 0 -> 1 instead of [no-data] -> 1.

Any guidance would be much appreciated

Answer 1

One way around it is to use a mix of sequence and generate inputs:

input:
  sequence:
    inputs:
      - generate:
          count: 1
          mapping: root = ""
        processors:
          - metric:
             type: counter
             name: test
             labels:
               foo: bar
          - mapping: root = deleted()
      - your_actual_input:
          # ...

The message created by the generate input is used to set that test metric and then it's discarded.

Answer 2

For anyone interested, here is the solution in promql to detect a single instance of the metric:sum(max_over_time(metric{status="success", env="$env"}[5m]) or vector(0)) - sum(max_over_time(metric{status="success", env="$env"}[5m] offset 5m) or vector(0))