Gauge values in Chronosphere Pushgateway only last for a couple of minutes

Question

I have a hourly batch jobs in which I defined a gauge metrics where I'd set_to_current_time at certain time in the program.

However when I see it in chronosphere, it will show with gaps in the graph instead of a continuous curve.

The intent is to generate alerts once the gauge value become too old. In order to make this alert, I intend to use (time() - series) / 3600 > 1.5 as a criteria. However this won't work if series are bunch of empty values.

So the question is, is there a way to interpolate the curve to use the last non-empty value? Or is there a different way of doing it?

Previously, we were using open source prometheus pushgateway and it works correctly.

Answer 1

You can use last_over_time for your values. For example to fill the gaps in graph from screenshot you could use

last_over_time(series[3h])

It'll fill the gaps with latest value of series over last three hours.

Regarding your alert rule: its expr: would look something like

(time() - last_over_time(series[3h])) / 3600 > 1.5