- nagios_runner: remove overall_status/overall_status_code/plugin_count fields;
each command still reports its own <name>_status and <name>_status_code
- threshold: expose {output} and {status} aliases in display templates for
nagios_runner generic matches (mapped from <check_name>_output/status)
- alerts.html: fix scrolling by overriding html,body height/overflow (style.css
sets both); make hostname a link to /plugins/<hostname>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
_find_threshold() now returns the stripped prefix ("check_name") alongside
the ThresholdConfig, enabling a single generic entry (e.g. nagios_runner.status_code)
to cover all per-command metrics (check_disk_root_status_code, check_load_status_code,
…). The prefix is threaded through to _format_display() as {check_name}, with
{metric_name} also available in display templates. purge_stale_alerts() updated
to use generic matching so it does not incorrectly drop alerts on generic-matched
metrics. README updated with Display Format Templates and Generic Threshold
Matching sections.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The 10% default hysteresis created an unreasonably wide recovery band:
a 95% threshold would only clear once the value dropped below 85.5%,
causing alerts to linger long after the metric was well below the
trigger level.
Change default hysteresis to 2% across all threshold parsers (plugin
metrics, partitions, RTT). For a 95% threshold, recovery is now at
93.1% instead of 85.5%.
Add AlertState.hysteresis field (set on every check, cleared on OK) and
expose recovery_threshold in to_dict() so the Alerts dashboard can
display "recovers < 93.1" alongside the trigger threshold, making the
hysteresis band visible to the user. Pickle backward-compatible via
__setstate__.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
memory_monitor / hbc_mini: ZFS ARC is reclaimable but not reflected in
MemAvailable by the Linux kernel (not in SReclaimable). Read ARC size
from /proc/spl/kstat/zfs/arcstats and add it to available memory before
computing memory_percent and memory_used. No-op on systems without ZFS.
cpu_monitor: report uptime_seconds via psutil.boot_time() (full client)
and /proc/uptime (hbc_mini).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace break-after-first-iteration with next(c for c in connections if
c.transport) so the message goes to the first connection that actually
has an open transport. Falls back to connections[0] if none are open
yet (sendto will attempt reopen), avoiding silent message loss when the
leading connection is still connecting.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Settings page: pass threshold_checker to http.start so the Threshold
Configurations section has data. Use threshold_checker's already-parsed
ThresholdConfig objects instead of re-parsing the raw nested YAML.
Named (non-default) configs now display only their explicit overrides
via threshold_raw_configs, not the full merged set with defaults.
hbc/hbc_mini: send boot and shutdown messages on first connection only
to avoid duplicate packets when multiple servers are configured.
Replace print("Daemonizing...") with logging.info so output goes to
syslog in daemon mode.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace href navigation with fetch() so the server response is captured
and displayed in a slide-up toast at the bottom of the page. Delete also
removes the host card from the DOM on success without a page reload.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Host Overview (plugins.html): show Update and Delete buttons in the
host-right zone when the logged-in user is the host owner (or admin /
unauthenticated mode). Buttons link to /u?h=<host> and /d?h=<host>
with stopPropagation so they don't toggle the accordion; Delete prompts
for confirmation first.
ThresholdChecker.purge_stale_alerts(): removes alert states whose
metric_path has no matching threshold in the current config. Called
after startup pickle restore and after every SIGHUP config reload so
alerts orphaned by upgrades or config changes do not persist
indefinitely.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Override the global style.css body height/overflow that locks all pages
to the viewport height (a remnant of the old drawer-menu layout).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- ZFS monitor plugin (zfs_monitor) added to plugin list and features
- nagios_runner: async execution, stderr capture, signal handling, path validation
- Threshold alerting: de-escalation suppression, short-duration suppression, ping_monitor thresholds
- Per-host watch flag and role-filtered dashboards
- HTTP API & Web UI: hostname links in Live View, Host Overview with ZFS renderer, alert pie chart in nav bar, Settings threshold viewer
- hbc connection retry: indefinite retry for IPv4; IPv6 dropped after 3 early startup failures
- hbc daemon mode: logs routed to syslog after daemonizing
- hbc_mini: noted zfs_monitor and IPv6 early-fail protection not available
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
IPv4 connections are retried forever in heartbeat_sender if open() fails,
so a temporary network outage does not terminate the sender.
IPv6 connections that have never opened successfully are dropped after
IPV6_EARLY_FAIL_LIMIT (3) consecutive failures so that a network without
IPv6 support does not keep a dead sender running.
At startup all resolved connections are added to the list regardless of
whether the initial open() succeeds; the heartbeat_sender loop handles
the first real connection attempt.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
After daemonize() redirects stderr to /dev/null, the existing StreamHandler
writes to /dev/null. logging.basicConfig() is a no-op when handlers are
already configured, so log messages are silently lost.
Replace the daemon block to:
1. Call daemonize() first
2. Explicitly remove existing handlers (pointing to /dev/null)
3. Add SysLogHandler pointing to /dev/log with fallback to UDP localhost:514
4. Log startup message to the new syslog handler
Removes redundant syslog.openlog() call which is no longer needed.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When NagiosRunnerPlugin has no commands configured, set skip_reason before
returning False from initialize(). This allows PluginLoader to log INFO
(not WARNING) when the plugin is skipped.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
CLIENT_DEFAULTS seeds "plugins": {} so raw_config.get("plugins", raw_config)
always returned the empty subdict instead of falling back to the full config.
Plugins configured at top-level (e.g. nagios_runner: ...) were therefore
never found, resulting in "No Nagios commands configured".
Now checks the plugins subdict first, then top-level keys, so both
config layouts work correctly.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Threshold alerts (plugin metrics, RTT) were firing immediately on the
first breach. Now every state transition to WARNING/CRITICAL starts a
grace-period timer (grace_seconds from the 'grace' config key). The
notification is deferred until the next heartbeat after grace_seconds
have elapsed. If the metric recovers within the grace window, both the
alert and the recovery are suppressed — no spurious pages for transient
spikes.
Two helper methods added to ThresholdChecker:
- _apply_grace: handles the state-change path (defer or suppress)
- _check_pending_or_renotify: handles the stable-alert path (fire
deferred notification once grace expires, or fall through to reminders)
The overdue case is unchanged — on_overdue already fires only after
interval+grace seconds of silence, which is equivalent behaviour.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
threshold.py was emitting level="RECOVERED" for metric recoveries, which
failed the is_recover check in send_notification (which only matched "RECOVER"),
bypassing _alerted_channels routing and the min_level bypass added in the
previous commit. Changed to "RECOVER" so all recovery paths are consistent.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- AlertState.update() now resets last_notification when the alert level
changes, so a WARNING→CRITICAL escalation restarts the reminder interval
rather than inheriting a nearly-expired timer.
- _dispatch_to_channel() bypasses min_level for RECOVER, so recovery
notifications are delivered even after a server restart when
_alerted_channels is empty and the fallback dispatch path is used.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
andreas
