feat: flapping detection suppresses notification storms
A (host, service) pair is flapping once it exceeds flap_count warning or critical notifications within flap_interval minutes. The notification that trips the state carries "Now flapping!! No more messages!" and every later one is dropped, including RECOVER. The state ends silently flap_interval minutes after a RECOVER, provided no further alert arrived meanwhile. Hooked into notify.send_notification, the single choke point for channel delivery, so only outbound notifications are suppressed — eventlog keeps recording, leaving the journal and /log with the full history of the flap. Threshold alerts key on their metric path, so a flapping disk check cannot silence a CPU alert; connectivity, boot and shutdown events key on the host itself. State lives at module level in flap.py and is never pickled. Flapping pairs surface in Host.stateinfo() and render as an amber badge on the live dashboard. Config: flap_count (5), flap_interval (10 minutes), 0 in either disables; both editable on the settings page. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
e3b0e5041f
commit
4414967bdc
@@ -386,6 +386,12 @@ def get_settings_sections(config: dict, threshold_checker=None, user=None) -> li
|
||||
"Extra seconds to wait after a missed heartbeat before sending notifications.", editable=True),
|
||||
field("threshold_renotify_interval", "Re-notify interval", "duration",
|
||||
"How often to re-send notifications for ongoing threshold alerts.", editable=True),
|
||||
field("flap_count", "Flap count", "number",
|
||||
"Warning/critical notifications within the flap interval that mark a "
|
||||
"service as flapping and silence it. 0 disables flap detection.", editable=True),
|
||||
field("flap_interval", "Flap interval", "number",
|
||||
"Minutes: the window flap count is measured over, and how long a service "
|
||||
"must stay OK before flapping ends.", editable=True),
|
||||
field("autosave_interval", "Autosave interval", "duration",
|
||||
"How often the server saves its state to disk."),
|
||||
field("base_url", "Base URL", "text",
|
||||
|
||||
Reference in New Issue
Block a user