I have a small use case where we have alerts firing daily on the basis of highload, cpu, memory etc etc. I would want a regression (Dynamic alerting) alerting system in prometheus for below scenario:
- Day1 alert1 notifies as critical with threshold for e.g. 90% cpu utilisation.
- Day2 same alert alert1 gets fired again for same data point.
In above case is there any mechanism where i can mute the day2 alert if threshold is below for last x days data? as in if for last x days threshold did not cross 90% then mute these alerts else if it is above > 90 and fire it and so on.
Thanks in advance.