fix: suppress notifications on alert de-escalation (e.g. CRITICAL→WARNING)

Only notify on worsening transitions (OK→WARNING, OK→CRITICAL,
WARNING→CRITICAL) and recovery (any→OK). De-escalation within alert
states no longer sends a duplicate notification since the metric never
recovered.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Andreas Wrede
2026-05-02 14:27:18 -04:00
parent ce0590f015
commit 28e2180f7b
+12 -2
View File
@@ -1114,7 +1114,9 @@ class ThresholdChecker:
) -> None: ) -> None:
"""Handle a state-change transition with grace-period logic. """Handle a state-change transition with grace-period logic.
Transitioning INTO alert: defers the notification for grace_seconds. Transitioning INTO alert (worsening): defers the notification for grace_seconds.
De-escalation within alert states (e.g. CRITICAL→WARNING): no new notification;
the metric is still alerting so no RECOVER was sent.
Transitioning TO OK: Transitioning TO OK:
- Still in grace window (pending_since set): suppresses both the alert - Still in grace window (pending_since set): suppresses both the alert
and the recovery — the spike never warranted a page. and the recovery — the spike never warranted a page.
@@ -1134,12 +1136,20 @@ class ThresholdChecker:
alert_state.pending_since = None alert_state.pending_since = None
else: else:
self._send_notification(host_name, lvl, message, metric_path, old_level, new_level, value) self._send_notification(host_name, lvl, message, metric_path, old_level, new_level, value)
else: elif new_level.value > old_level.value:
# Worsening (OK→WARNING, OK→CRITICAL, WARNING→CRITICAL): schedule notification.
alert_state.pending_since = time.time() alert_state.pending_since = time.time()
logger.debug( logger.debug(
"Alert deferred (%.0fs grace): %s on %s = %s", "Alert deferred (%.0fs grace): %s on %s = %s",
self.grace_seconds, metric_path, host_name, value, self.grace_seconds, metric_path, host_name, value,
) )
else:
# De-escalation within alert states (e.g. CRITICAL→WARNING): metric is still
# alerting but did not recover, so no new notification.
logger.debug(
"De-escalation %s%s for %s on %s, no notification",
old_level.name, new_level.name, metric_path, host_name,
)
def _check_pending_or_renotify( def _check_pending_or_renotify(
self, self,