andreas/heartbeat
Public Access
1
0
Fork 1
Code Issues Pull Requests Actions Packages Projects Releases 50 Wiki Activity

feat: flapping detection suppresses notification storms

Browse Source 
A (host, service) pair is flapping once it exceeds flap_count warning or
critical notifications within flap_interval minutes. The notification that
trips the state carries "Now flapping!! No more messages!" and every later
one is dropped, including RECOVER. The state ends silently flap_interval
minutes after a RECOVER, provided no further alert arrived meanwhile.

Hooked into notify.send_notification, the single choke point for channel
delivery, so only outbound notifications are suppressed — eventlog keeps
recording, leaving the journal and /log with the full history of the flap.

Threshold alerts key on their metric path, so a flapping disk check cannot
silence a CPU alert; connectivity, boot and shutdown events key on the host
itself. State lives at module level in flap.py and is never pickled.

Flapping pairs surface in Host.stateinfo() and render as an amber badge on
the live dashboard. Config: flap_count (5), flap_interval (10 minutes),
0 in either disables; both editable on the settings page.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Andreas Wrede
2026-07-23 09:53:46 -07:00
co-authored by Claude Opus 4.8
parent e3b0e5041f
commit 4414967bdc
11 changed files with 402 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
hbd/server/templates/live.html
+8 -1
View File
@@ -45,6 +45,7 @@
.dot.down { background: var(--st-crit); box-shadow: 0 0 0 3px var(--st-crit-soft); }
.dot.overdue { background: transparent; border: 2px solid var(--st-warn); }
.stale { color: var(--st-faint); font-size: 10px; border: 1px solid var(--st-line); border-radius: 4px; padding: 1px 4px; font-family: var(--st-mono); white-space: nowrap; }
.flap { color: var(--st-warn); font-size: 10px; border: 1px solid var(--st-warn); border-radius: 4px; padding: 1px 4px; font-family: var(--st-mono); white-space: nowrap; }
.chip.outline { background: transparent; border: 1px solid var(--st-line); }
.lat { font-family: var(--st-mono); font-size: 12px; color: var(--st-muted); font-variant-numeric: tabular-nums; }
@@ -247,11 +248,17 @@
staleTag = '<span class="stale" title="agent v' + escHtml(String(data.hbc_version)) + ' ≠ server v' + HBD_VERSION + '">v' + escHtml(String(data.hbc_version)) + '</span>';
}
var dotCls = w ? (w.state === 'up' ? sev : (w.state === 'down' ? 'down' : 'overdue')) : 'off';
var flapTag = '';
var flapping = data.flapping || [];
if (flapping.length) {
var svcs = flapping.map(function (s) { return s || 'host'; }).join(', ');
flapTag = '<span class="flap" title="notifications suppressed: ' + escHtml(svcs) + '">flapping</span>';
}
row.innerHTML =
'<span class="host' + (data.watched ? ' watched' : '') + '">'
+ '<span class="dot ' + dotCls + '"></span>'
+ '<a href="/plugins#' + encodeURIComponent(name) + '">' + escHtml(name) + '</a>' + staleTag + '</span>'
+ '<a href="/plugins#' + encodeURIComponent(name) + '">' + escHtml(name) + '</a>' + staleTag + flapTag + '</span>'
+ '<span class="tip" tabindex="0">' + statusChip(data) + tipCard(data) + '</span>'
+ '<span class="lat num">' + lat + '</span>'
+ '<span>' + alertChips(data) + '</span>'