How to fire alerts on the basis of historical data in prometheus using alert manager?

56 Views Asked by At

I have a small use case where we have alerts firing daily on the basis of highload, cpu, memory etc etc. I would want a regression (Dynamic alerting) alerting system in prometheus for below scenario:

  1. Day1 alert1 notifies as critical with threshold for e.g. 90% cpu utilisation.
  2. Day2 same alert alert1 gets fired again for same data point.

In above case is there any mechanism where i can mute the day2 alert if threshold is below for last x days data? as in if for last x days threshold did not cross 90% then mute these alerts else if it is above > 90 and fire it and so on.

Thanks in advance.

0

There are 0 best solutions below