wrong timestamp in promql

Question

In promql I perform the following query: kafka_consumergroup_group_lag_seconds{topic=~"TOPIC_PLACEHOLDER"}[1y:10s].

Then I wrap it in the max(max_over_time(...)) function to retrieve the max value. The value I retrieved is fine, but the timestamp of that value is different between the two queries. How can I retrieve the right value and timestamp from the first metric?

I've tried converting the first metric into a vector type to perform topk(1, sort_desc(FIRST_METRIC)), but it didn't work.

Answer 1

Depending on what exact goal you pursue you might want to use one of the following queries.

• To get data points that are equal to the fixed maximum over last hour respective of the moment of execution.

  metric == on() max(max_over_time(metric[1h]@end()))
  

  If time range for the query will be one hour, this will result in the following: all caught values will be the same, at least one value will be returned.

• To get data points that are equal to a sliding maximum over hour relative to the step of evaluation.

  metric == on() max(max_over_time(metric[1h]))
  

  Will produce no result for decreasing metric, but for increasing metric will return all data points. Depending on range of the query and range selector used inside of max_over_time might be effective for finding local maximums.

To get a timestamp of the moment when value was maximum wrap query with timestamp( ... )

---

Results of these queries will be most of the time scarce. I assume that you intend to use them directly in the API and that this is OK for your use case.
But if you intend to use those in some graphing solution, you'll need to make them more continuous.

For example, to show last timestamp, when some metric achieved it's maximum within last hour, you'd want to use query like

last_over_time(timestamp(
  metric == on() max(max_over_time(metric[1h]@end()))
)[1h:])

Or for this to be more Grafana way - replace both occurrences of 1h with $__range.

---

What is happening in those queries

• max_over_time(metric[1h]@end()) - calculates maximum of the metric in the one hour window ending in the time specified for evaluation.
• == on() used to compare metric with maximum, because max on the right hand side strips result of all the labels, and == by default compares only time series with matching label sets.
• [1h:] - subquery syntax. Since selector before it is not a simple vector selector, but full fledged query, we cannot use simple range selector. Subquery syntax allows for a similar (to range selector) functionality with added requirement for resolution (part after colon). If resolution is omitted, evaluation_interval is used by default.

---

Demo of mentioned queries in use can be found here. Hopefully this might additionally clarify some bits.