feat: flapping detection suppresses notification storms

A (host, service) pair is flapping once it exceeds flap_count warning or
critical notifications within flap_interval minutes. The notification that
trips the state carries "Now flapping!! No more messages!" and every later
one is dropped, including RECOVER. The state ends silently flap_interval
minutes after a RECOVER, provided no further alert arrived meanwhile.

Hooked into notify.send_notification, the single choke point for channel
delivery, so only outbound notifications are suppressed — eventlog keeps
recording, leaving the journal and /log with the full history of the flap.

Threshold alerts key on their metric path, so a flapping disk check cannot
silence a CPU alert; connectivity, boot and shutdown events key on the host
itself. State lives at module level in flap.py and is never pickled.

Flapping pairs surface in Host.stateinfo() and render as an amber badge on
the live dashboard. Config: flap_count (5), flap_interval (10 minutes),
0 in either disables; both editable on the settings page.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Andreas Wrede
2026-07-23 09:53:46 -07:00
co-authored by Claude Opus 4.8
parent e3b0e5041f
commit 4414967bdc
11 changed files with 402 additions and 1 deletions
+6
View File
@@ -386,6 +386,12 @@ def get_settings_sections(config: dict, threshold_checker=None, user=None) -> li
"Extra seconds to wait after a missed heartbeat before sending notifications.", editable=True),
field("threshold_renotify_interval", "Re-notify interval", "duration",
"How often to re-send notifications for ongoing threshold alerts.", editable=True),
field("flap_count", "Flap count", "number",
"Warning/critical notifications within the flap interval that mark a "
"service as flapping and silence it. 0 disables flap detection.", editable=True),
field("flap_interval", "Flap interval", "number",
"Minutes: the window flap count is measured over, and how long a service "
"must stay OK before flapping ends.", editable=True),
field("autosave_interval", "Autosave interval", "duration",
"How often the server saves its state to disk."),
field("base_url", "Base URL", "text",