feat: flapping detection suppresses notification storms

A (host, service) pair is flapping once it exceeds flap_count warning or
critical notifications within flap_interval minutes. The notification that
trips the state carries "Now flapping!! No more messages!" and every later
one is dropped, including RECOVER. The state ends silently flap_interval
minutes after a RECOVER, provided no further alert arrived meanwhile.

Hooked into notify.send_notification, the single choke point for channel
delivery, so only outbound notifications are suppressed — eventlog keeps
recording, leaving the journal and /log with the full history of the flap.

Threshold alerts key on their metric path, so a flapping disk check cannot
silence a CPU alert; connectivity, boot and shutdown events key on the host
itself. State lives at module level in flap.py and is never pickled.

Flapping pairs surface in Host.stateinfo() and render as an amber badge on
the live dashboard. Config: flap_count (5), flap_interval (10 minutes),
0 in either disables; both editable on the settings page.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Andreas Wrede
2026-07-23 09:53:46 -07:00
co-authored by Claude Opus 4.8
parent e3b0e5041f
commit 4414967bdc
11 changed files with 402 additions and 1 deletions
+4
View File
@@ -30,6 +30,10 @@ SERVER_DEFAULTS = {
"grace": 2, # Grace period (extra seconds before notifying after a missed heartbeat)
"threshold_renotify_interval": 3600, # Seconds between threshold re-notifications
# Flap detection (0 in either key disables it)
"flap_count": 5, # Warning/critical notifications within flap_interval that trip flapping
"flap_interval": 10, # Minutes: the counting window, and the quiet window after an OK
# User management
"users": {}, # username -> {full_name, avatar, password, admin, notification_channels}
"default_owner": None, # Username that owns hosts with no explicit owner