andreas/heartbeat
Public Access
1
0
Fork 1
Code Issues Pull Requests Actions Packages Projects Releases 50 Wiki Activity

feat: flapping detection suppresses notification storms

Browse Source 
A (host, service) pair is flapping once it exceeds flap_count warning or
critical notifications within flap_interval minutes. The notification that
trips the state carries "Now flapping!! No more messages!" and every later
one is dropped, including RECOVER. The state ends silently flap_interval
minutes after a RECOVER, provided no further alert arrived meanwhile.

Hooked into notify.send_notification, the single choke point for channel
delivery, so only outbound notifications are suppressed — eventlog keeps
recording, leaving the journal and /log with the full history of the flap.

Threshold alerts key on their metric path, so a flapping disk check cannot
silence a CPU alert; connectivity, boot and shutdown events key on the host
itself. State lives at module level in flap.py and is never pickled.

Flapping pairs surface in Host.stateinfo() and render as an amber badge on
the live dashboard. Config: flap_count (5), flap_interval (10 minutes),
0 in either disables; both editable on the settings page.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Andreas Wrede
2026-07-23 09:53:46 -07:00
co-authored by Claude Opus 4.8
parent e3b0e5041f
commit 4414967bdc
11 changed files with 402 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+7
View File
@@ -116,6 +116,11 @@ dyndomains:
# Threshold alert re-notification interval (seconds)
threshold_renotify_interval: 3600
# Flap detection — silence a service/host after flap_count warning or critical
# notifications within flap_interval minutes (flap_count: 0 disables)
flap_count: 5
flap_interval: 10
# Notification channels
notification_channels:
pushover_ops:
@@ -400,6 +405,8 @@ hosts:
Notifications are sent on state transitions (OK → WARNING, WARNING → CRITICAL, CRITICAL → OK). De-escalations (CRITICAL → WARNING) do not trigger a notification. Ongoing alerts generate a re-notification every `threshold_renotify_interval` seconds (default: 3600). Alerts can be acknowledged via the web UI or API to suppress re-notifications.
A service or host that exceeds `flap_count` warning/critical notifications within `flap_interval` minutes is marked **flapping**: the tripping notification carries `Now flapping!! No more messages!` and further notifications are suppressed until it stays OK for `flap_interval` minutes. Flapping hosts are badged on the live dashboard; the event log keeps recording throughout. See [docs/NOTIFICATIONS.md](docs/NOTIFICATIONS.md#flap-detection).
### RTT thresholds
The server measures heartbeat round-trip time and supports RTT thresholds using the same format: