feat: flapping detection suppresses notification storms

A (host, service) pair is flapping once it exceeds flap_count warning or
critical notifications within flap_interval minutes. The notification that
trips the state carries "Now flapping!! No more messages!" and every later
one is dropped, including RECOVER. The state ends silently flap_interval
minutes after a RECOVER, provided no further alert arrived meanwhile.

Hooked into notify.send_notification, the single choke point for channel
delivery, so only outbound notifications are suppressed — eventlog keeps
recording, leaving the journal and /log with the full history of the flap.

Threshold alerts key on their metric path, so a flapping disk check cannot
silence a CPU alert; connectivity, boot and shutdown events key on the host
itself. State lives at module level in flap.py and is never pickled.

Flapping pairs surface in Host.stateinfo() and render as an amber badge on
the live dashboard. Config: flap_count (5), flap_interval (10 minutes),
0 in either disables; both editable on the settings page.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Andreas Wrede
2026-07-23 09:53:46 -07:00
co-authored by Claude Opus 4.8
parent e3b0e5041f
commit 4414967bdc
11 changed files with 402 additions and 1 deletions
+39
View File
@@ -9,6 +9,7 @@ Notifications are dispatched to the **owner and managers** of a host, each via t
```
Alert event (udp.py / threshold.py)
└─ notify.send_notification(host_name, Notification)
├─ flap.observe(host, service, level) → pass | trip | suppress
├─ look up host.owner + host.managers
├─ for each user → user.notification_channels
└─ for each channel → _dispatch_to_channel (filtered by min_level)
@@ -19,6 +20,7 @@ Every notification carries:
- **body** — detail message (metric value, threshold, duration)
- **url** — link to the plugin metrics page (`{base_url}/plugins#{hostname}`)
- **level** — `RECOVER | WARNING | CRITICAL | INFO`
- **service** — flap-detection key within the host (empty = the host itself)
## Configuration
@@ -268,6 +270,43 @@ min_level: WARNING
Reminder notifications (re-notify) are sent only for CRITICAL level alerts.
## Flap detection
A check that toggles between OK and alerting produces a notification per swing. Flap
detection silences it after the first few.
A **`(host, service)`** pair is flapping once it exceeds `flap_count` WARNING/CRITICAL
notifications within `flap_interval` minutes. Threshold alerts key on their metric path,
so a flapping disk check does not silence an unrelated CPU alert; connectivity, boot and
shutdown events key on the host itself.
```yaml
flap_count: 5 # notifications within the window that trip flapping (0 disables)
flap_interval: 10 # minutes — both the counting window and the quiet window
```
Lifecycle:
| Event | Effect |
|---|---|
| Alerts 1..`flap_count` within the window | Delivered normally |
| Alert `flap_count + 1` | Delivered with ` Now flapping!! No more messages!` appended to the body |
| Every notification after that | Dropped — including RECOVER and INFO |
| RECOVER while flapping | Dropped, and starts the `flap_interval` quiet window |
| WARNING/CRITICAL during the quiet window | Restarts the quiet window; still flapping |
| Quiet window elapses | Flapping ends **silently** — no notification |
Only outbound notifications are suppressed. `notify.eventlog` keeps recording every event,
so the journal and the `/log` page retain the full history of the flap.
Flapping pairs appear in each host's `stateinfo()` under `flapping` (a list of service
keys; `""` means the host itself) and render as an amber **flapping** badge next to the
host name on the live dashboard, with the affected services in its tooltip.
State lives at module level in `hbd/server/flap.py` and is never pickled — a server restart
starts every check with a clean slate. Hosts with `watch: false` never reach the
notification path, so they never flap.
## API reference
### `send_notification(host_name, notif) -> dict`