version 5.2.3

hbc/hbc_mini: log name and version at startup; ui: bump alert-metric font size
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-07 10:15:11 -04:00 · 2026-05-07 10:15:03 -04:00 · 2026-05-07 06:26:27 -04:00 · 2026-05-07 06:12:15 -04:00 · 2026-05-06 11:57:43 -04:00 · 2026-05-06 11:54:09 -04:00
24 changed files with 762 additions and 294 deletions
@@ -27,6 +27,7 @@ A lightweight daemon that listens for UDP heartbeat messages and acts on them: k
  - Configurable retention and backup management
 - **Plugin system for extensible monitoring** ✅
  - Collect system metrics (CPU, memory, disk, network)
+  - Monitor ZFS pool health, capacity, and I/O via `zpool(8)`
  - Execute existing Nagios monitoring plugins
  - Create custom plugins with simple Python classes
 - **Threshold alerting system** ✅
@@ -34,6 +35,8 @@ A lightweight daemon that listens for UDP heartbeat messages and acts on them: k
  - Hysteresis to prevent alert flapping
  - Automatic notifications on state changes
  - Re-notification for ongoing alerts
+- **Per-host watch flag** — set `watch: false` on any host to silence all notifications for that host without removing its configuration ✅
+- **Role-filtered dashboards** — Live Dashboard and Host Overview show only hosts where the logged-in user is owner or manager (admins see all) ✅
 - Modular codebase suitable for unit testing and CI ✅

 ---
@@ -55,21 +58,26 @@ Heartbeat includes a comprehensive plugin architecture that extends monitoring b
 ### Built-in Plugins

 - `os_info`: Collects OS, kernel, distribution, and architecture information
- `cpu_monitor`: Monitors CPU usage, load average, frequency, and process counts
- `memory_monitor`: Monitors RAM and swap usage, available memory
+- `cpu_monitor`: Monitors CPU usage, load average, frequency, process counts, and uptime
+- `memory_monitor`: Monitors RAM and swap usage, available memory (ZFS ARC-aware)
 - `disk_monitor`: Monitors disk usage, I/O statistics, and filesystem metrics
 - `network_monitor`: Monitors network interface statistics, bandwidth, and connections
+- `ping_monitor`: Measures round-trip latency to configured hosts
 - `filesystem_info`: Collects mounted filesystem information (physical filesystems only by default)
 - `nagios_runner`: Executes Nagios monitoring plugins (check_disk, check_load, check_http, etc.)
+- `zfs_monitor`: Monitors ZFS pool health, capacity, fragmentation, dedup ratio, and cumulative I/O via `zpool(8)`

 ### Nagios Integration

 The `nagios_runner` plugin provides seamless integration with the vast Nagios plugin ecosystem. You can run any Nagios-compatible plugin and have the results automatically parsed and stored:

- Executes plugins via subprocess with timeout protection
+- Executes plugins asynchronously (non-blocking) with timeout protection
+- Captures both stdout and stderr; if stdout is empty, stderr is used as the status message
+- Handles signal-killed processes (negative exit code → UNKNOWN status)
+- Validates absolute command paths at startup and warns on missing or non-executable files
 - Parses exit codes (OK/WARNING/CRITICAL/UNKNOWN)
 - Extracts performance data with thresholds
- Reports aggregated status across all configured checks
+- Reports per-check status, exit code, and output; no aggregate rollup field

 See [docs/NAGIOS_INTEGRATION.md](docs/NAGIOS_INTEGRATION.md) for complete integration guide including configuration examples and custom plugin development.

@@ -147,9 +155,11 @@ Heartbeat includes a sophisticated threshold alerting system that monitors plugi
 - **Multi-level alerts**: WARNING and CRITICAL severity levels
 - **Flexible operators**: Support for >, >=, <, <=, ==, != comparisons
 - **Hysteresis**: Prevents alert flapping with configurable recovery thresholds
- **Smart notifications**: Alerts only on state changes, not every check
+- **Smart notifications**: Alerts only on state changes, not every check; de-escalations (e.g. CRITICAL → WARNING) do not generate a notification
 - **Re-notifications**: Periodic reminders for ongoing alerts
+- **Short-duration suppression**: Recovery notifications are suppressed for down events under 4 seconds (avoids noise from transient blips)
 - **Journal integration**: All threshold events logged for audit trail
+- **`ping_monitor` thresholds**: Latency and packet-loss thresholds use the same format as all other plugin metrics

 ### Configuration

@@ -172,7 +182,8 @@ thresholds:
      warning: 80.0      # Warn when CPU > 80%
      critical: 90.0     # Critical when CPU > 90%
      operator: ">"
-      hysteresis: 0.1    # 10% hysteresis to prevent flapping
+      hysteresis: 0.02   # 2% hysteresis to prevent flapping
+      display: "(threshold: {op_symbol} {threshold_value}%)"  # optional
  
  memory_monitor:
    percent:
@@ -214,7 +225,7 @@ thresholds:
    <hostname>:
      warning: <milliseconds>   # Warn when RTT > this value
      critical: <milliseconds>  # Critical when RTT > this value
-      hysteresis: 0.1           # Optional: 10% hysteresis (default)
+      hysteresis: 0.02          # Optional: 2% hysteresis (default)
 ```

 **Example alerts:**
@@ -265,7 +276,59 @@ All plugin metrics can be thresholded:
 - **Memory**: percent, available_mb, swap_percent
 - **Disk**: Per-partition percent, free_gb, free_mb
 - **Network**: errors_total, dropped packets, connection counts
- **Nagios**: exit_code mapping (0=OK, 1=WARNING, 2=CRITICAL)
+- **Nagios**: Any field emitted by `nagios_runner` (`<name>_status_code`, `<name>_status`, `<name>_output`, performance data fields)
+
+### Display Format Templates
+
+Each threshold entry accepts an optional `display` field — a Python format string shown in notifications and on the Alerts dashboard:
+
+```yaml
+nagios_runner:
+  status_code:
+    warning: 1
+    critical: 2
+    operator: ">="
+    display: "{check_name}: exit {value} (expected < {threshold_value})"
+```
+
+Available variables:
+
+| Variable | Description |
+|---|---|
+| `{value}` | Current metric value |
+| `{threshold_value}` | Threshold that was crossed |
+| `{op_symbol}` | Comparison operator (`>`, `<`, `>=`, …); `"nagios"` for the nagios operator |
+| `{check_name}` | Prefix stripped by generic matching (see below) |
+| `{metric_name}` | Full field name within the plugin data |
+| `{output}` | For `nagios_runner` generic matches: the matched check's status text (alias for `{check_name}_output`) |
+| `{status}` | For `nagios_runner` generic matches: the matched check's status name — OK/WARNING/CRITICAL/UNKNOWN (alias for `{check_name}_status`) |
+| any plugin field | Any other field present in the plugin's data |
+
+### Generic Threshold Matching
+
+When a metric name has no exact threshold entry, the server progressively strips leading underscore-separated segments and re-tries the lookup. This lets a single generic entry cover an entire family of metrics.
+
+The classic use case is `nagios_runner`, which names each metric after the command that produced it:
+
+```
+nagios_runner.check_disk_root_status_code    → no exact match
+nagios_runner.disk_root_status_code          → no match
+nagios_runner.root_status_code               → no match
+nagios_runner.status_code                    → matched ✓
+```
+
+Configure the generic threshold once using the `nagios` operator, which maps exit codes directly to alert severity without requiring numeric warning/critical values:
+
+```yaml
+nagios_runner:
+  status_code:
+    operator: "nagios"   # 0=OK  1=WARNING  2=CRITICAL  3=UNKNOWN
+    display: "{check_name}: {output}"
+```
+
+The stripped prefix (`check_disk_root` in the example above) is available as `{check_name}` in the display template, so you can identify which check triggered the alert without writing a separate threshold entry per command.
+
+Exact matches always take priority. A generic entry only applies when no specific one is defined.

 ### Per-Host Threshold Profiles

@@ -363,9 +426,10 @@ Heartbeat includes a built-in HTTP/WebSocket server that provides both a REST AP
 ### Web Dashboards

 - **Login** (`/login`): Browser login form (shown automatically when auth is configured)
- **Live View** (`/live`): Real-time host connectivity, latency, and messages
- **Plugin Metrics** (`/plugins`): Browse and visualize metrics from all plugins
- **Alerts Dashboard** (`/alerts`): Monitor active alerts with severity filtering
+- **Live View** (`/live`): Real-time host connectivity, latency, and messages; hostnames link directly to the Host Overview page
+- **Host Overview** (`/plugins/<host>`): Per-host plugin metrics with ZFS pool visualization; filtered to hosts where the logged-in user is owner or manager (admins see all)
+- **Alerts Dashboard** (`/alerts`): Monitor active alerts with severity filtering; alert count pie chart shown in the navigation bar
+- **Settings** (`/settings`): Server configuration, user management, and threshold configuration viewer

 ### API Endpoints

@@ -443,6 +507,9 @@ hbc --boot your-server.example.com

 # Verbose output
 hbc -v your-server.example.com
+
+# Send 'boot' and 'shutdown' messages on start and exit 
+hbc -b your-server.example.com
 ```

 You can also run it via the module entrypoint:
@@ -451,12 +518,11 @@ You can also run it via the module entrypoint:
 python -m hbd.client.main your-server.example.com
 ```

-Client configuration can also be specified in YAML:
+Client configuration can also be specified in YAML (`~/.hbc.yaml`):

 ```yaml
-server: hbd.example.com
-port: 50003
-interval: 30
+hb_port: 50003        # Server port (default: 50003)
+interval: 30          # Heartbeat interval in seconds
 plugins:
  cpu_monitor:
    interval: 300      # Check every 5 minutes (default)
@@ -470,12 +536,20 @@ plugins:
  nagios_runner:
    interval: 300      # Check every 5 minutes (default)
    commands:
-      - /usr/lib/nagios/plugins/check_load -w 5,4,3 -c 10,8,6
-      - /usr/lib/nagios/plugins/check_disk -w 20% -c 10% -p /
+      - name: check_load
+        command: /usr/lib/nagios/plugins/check_load -w 5,4,3 -c 10,8,6
+      - name: check_disk
+        command: /usr/lib/nagios/plugins/check_disk -w 20% -c 10% -p /
 ```

+The server hostname is always passed as a positional command-line argument; there is no `server:` config key.
+
 All monitoring plugins default to 5-minute (300 second) intervals, but can be customized as needed.

+**Connection retry:** If a server is temporarily unreachable, `hbc` retries `open()` indefinitely on every heartbeat interval. IPv6 connections that never succeeded during early startup are dropped after 3 consecutive failures (to handle hosts without IPv6 routing), while IPv4 connections always retry.
+
+**Daemon logging:** When running with `-d`, `hbc` routes all log output to syslog (`LOG_DAEMON` facility) after daemonizing. Without `-d`, logs go to stderr as usual.
+
 ### hbc_mini — single-file client (no external dependencies)

 `scripts/hbc_mini.py` is a self-contained version of the heartbeat client that requires only Python 3.8+ and no external packages. Copy it to any host and run it directly — no virtualenv, no `pip install`.
@@ -531,8 +605,10 @@ python3 hbc_mini.py -m "maintenance starting" your-server.example.com

 - No YAML config (use JSON instead)
 - No `filesystem_info` plugin
+- No `zfs_monitor` plugin (requires `zpool(8)` and the full plugin loader)
 - `cpu_monitor` does not report per-core usage or CPU frequency (no psutil)
 - Plugins cannot be loaded from external `.py` files — all plugins are compiled in
+- No IPv6 early-fail protection — connections that fail to open at startup are silently skipped rather than retried

 Everything else — heartbeat protocol, ACK/CMD/UPD handling, `hb_install.sh`-based self-update, daemonize, syslog — is identical to the full client.

@@ -104,11 +104,6 @@ The `nagios_runner` plugin collects:
 - `{name}_{metric}_min` - Minimum value (if present)
 - `{name}_{metric}_max` - Maximum value (if present)

-**Overall:**
- `overall_status` - Worst status from all commands
- `overall_status_code` - Worst status code
- `plugin_count` - Number of Nagios plugins executed
-
 ## Configuration Options

 ```yaml
@@ -1110,33 +1110,6 @@ hosts:
  db-02:
    threshold_config: [tight_memory, db_disk]
 ```
-
-### Backward Compatibility
-
-The legacy single threshold configuration is fully supported:
-
-```yaml
-# Old format - still works
-thresholds:
-  cpu_monitor:
-    cpu_percent:
-      warning: 80.0
-      critical: 90.0
-```
-
-This is equivalent to:
-
-```yaml
-# New format
-threshold_configs:
-  default:
-    thresholds:
-      cpu_monitor:
-        cpu_percent:
-          warning: 80.0
-          critical: 90.0
-```
-
 ### Configuration Priority

 1. **Host `threshold_config` (list)**: Layer each named config's overrides left-to-right on top of the defaults
@@ -14,4 +14,4 @@ Install options:
 """

 __all__ = ["__version__"]
-__version__ = "5.1.15"
+__version__ = "5.2.3"
@@ -21,6 +21,7 @@ from typing import Dict, List, Optional
 # Import protocol and config
 from .config import load_config
 from ..common.proto import dicttos, stodict
+from .. import __version__

 # Import plugin system
 from .plugin import PluginRegistry, PluginLoader, InfoPlugin, MonitorPlugin
@@ -56,23 +57,26 @@ class AsyncConnection:
        self.transport: Optional[asyncio.DatagramTransport] = None
        self.protocol: Optional[asyncio.DatagramProtocol] = None
        self._dead = False
+        self._ever_opened = False
+        self._open_fail_count = 0   # consecutive failures before first success

        self.logger = logging.getLogger(f"hbc.conn.{addr}")
-    
+
    async def open(self) -> bool:
        """Open the UDP connection.
-        
+
        Returns:
            True if successful, False otherwise
        """
        try:
            loop = asyncio.get_event_loop()
-            
+
            # Create datagram endpoint
            self.transport, self.protocol = await loop.create_datagram_endpoint(
                lambda: HeartbeatProtocol(self),
                family=self.af
            )
+            self._ever_opened = True
            self.logger.debug(f"Opened connection to {self.addr}:{self.port}")
            return True
        except Exception as e:
@@ -169,9 +173,8 @@ class HeartbeatProtocol(asyncio.DatagramProtocol):
            self.logger.error(f"Error processing datagram: {e}", exc_info=True)
    
    def error_received(self, exc):
-        """Handle protocol errors."""
-        self.logger.warning(f"Protocol error on {self.connection.addr}: {exc} — dropping connection")
-        self.connection._dead = True
+        """Handle protocol errors — close transport so the heartbeat sender retries."""
+        self.logger.warning(f"Protocol error on {self.connection.addr}: {exc} — will retry")
        self.connection.close()


@@ -262,15 +265,51 @@ async def handle_update(conn: AsyncConnection, _msg: dict):  # pyright: ignore[r


 async def heartbeat_sender(conn: AsyncConnection, interval: int):
-    """Send periodic heartbeats.
-    
+    """Send periodic heartbeats, retrying the connection if it is not open.
+
+    IPv6 connections that fail to open before their first successful send are
+    dropped after IPV6_EARLY_FAIL_LIMIT attempts so that a network without IPv6
+    does not keep a dead sender alive.  IPv4 connections are retried indefinitely.
+
    Args:
        conn: Connection to send on
        interval: Heartbeat interval in seconds
    """
    logger = logging.getLogger("hbc.heartbeat")
-    
-    while running:
+    IPV6_EARLY_FAIL_LIMIT = 3
+
+    while running and not conn._dead:
+        # Ensure transport is open before attempting to send.
+        if not conn.transport:
+            opened = await conn.open()
+            if opened:
+                conn._open_fail_count = 0
+            else:
+                conn._open_fail_count += 1
+                # Drop an IPv6 connection that has never come up within the
+                # first few attempts — it is likely unavailable on this network.
+                if (not conn._ever_opened
+                        and conn.af == socket.AF_INET6
+                        and conn._open_fail_count >= IPV6_EARLY_FAIL_LIMIT):
+                    logger.warning(
+                        f"IPv6 connection to {conn.addr} unreachable after "
+                        f"{conn._open_fail_count} attempts, disabling"
+                    )
+                    conn._dead = True
+                    break
+                # Retry after the normal interval; IPv4 retries forever.
+                try:
+                    if shutdown_event:
+                        await asyncio.wait_for(shutdown_event.wait(), timeout=interval)
+                        break
+                    else:
+                        await asyncio.sleep(interval)
+                except asyncio.TimeoutError:
+                    pass
+                except asyncio.CancelledError:
+                    raise
+                continue
+
        try:
            msg = {
                "acks": conn.ackcount,
@@ -278,20 +317,17 @@ async def heartbeat_sender(conn: AsyncConnection, interval: int):
                "interval": interval
            }
            await conn.sendto(msg, "HTB")
-            
-        except Exception as e:
-            logger.error(f"Error sending heartbeat: {e}", exc_info=True)
+
        except asyncio.CancelledError:
            logger.debug("Heartbeat sender cancelled")
            raise
-        
+        except Exception as e:
+            logger.error(f"Error sending heartbeat: {e}", exc_info=True)
+
        # Wait for next interval or shutdown event
        try:
            if shutdown_event:
-                await asyncio.wait_for(
-                    shutdown_event.wait(), 
-                    timeout=interval
-                )
+                await asyncio.wait_for(shutdown_event.wait(), timeout=interval)
                break
            else:
                await asyncio.sleep(interval)
@@ -427,16 +463,13 @@ async def cleanup(connections: List[AsyncConnection]):
    logger = logging.getLogger("hbc.cleanup")
    logger.info("Cleaning up connections")
    
-    for conn in connections:
+    target = next((c for c in connections if c.transport), connections[0] if connections else None)
+    if target and send_shutdown:
        try:
-            msg = {
-                "shutdown": 1,
-                "acks": conn.ackcount
-            }
-            await conn.sendto(msg)
+            await target.sendto({"shutdown": 1, "acks": target.ackcount})
        except Exception as e:
            logger.error(f"Error sending shutdown: {e}")
-        
+    for conn in connections:
        conn.close()
    
    # Give messages time to send
@@ -445,7 +478,7 @@ async def cleanup(connections: List[AsyncConnection]):

 async def async_main(args, config):
    """Async main function."""
-    global running, shutdown_event, active_tasks
+    global running, shutdown_event, active_tasks, send_shutdown 
    
    # Create shutdown event
    shutdown_event = asyncio.Event()
@@ -462,6 +495,7 @@ async def async_main(args, config):
    hb_port = config.get("hb_port", PORT)
    interval = config.get("interval", INTERVAL)
    
+    logger.info(f"hbc {__version__} starting on {iam}")
    logger.info(f"Starting hbc for {iam} -> {hb_hosts}")
    logger.info(f"Port: {hb_port}, Interval: {interval}s")
    
@@ -479,30 +513,34 @@ async def async_main(args, config):
        for addr_info in addrs:
            af = addr_info[0]
            addr = addr_info[4][0]
-            
+
            conn = AsyncConnection(conn_id, addr, hb_port, af, iam)
-            if await conn.open():
-                connections.append(conn)
-                conn_id += 1
-    
+            if not await conn.open():
+                logger.warning(f"Initial open to {addr} failed, heartbeat sender will retry")
+            connections.append(conn)
+            conn_id += 1
+
    if not connections:
-        logger.error("No connections established")
+        logger.error("No connections established (DNS resolution failed for all hosts)")
        return 1
    
    logger.info(f"Created {len(connections)} connections")
    
    # Send boot/message if requested
+    send_shutdown = False
    if args.boot or args.message:
        boot_msg = {}
        if args.boot:
            boot_msg["boot"] = 1
+            args.boot = False  # Clear boot flag so we don't send it again in main loop
+            send_shutdown = True
        if args.message:
            boot_msg["service"] = "service"
            boot_msg["msg"] = args.message
        
        boot_msg["acks"] = 0
-        for conn in connections:
-            await conn.sendto(boot_msg)
+        target = next((c for c in connections if c.transport), connections[0])
+        await target.sendto(boot_msg)
        
        if args.message and not args.daemon:
            # Message-only mode
@@ -702,7 +740,7 @@ def main(argv=None):
    
    # Daemonize if requested
    if args.daemon:
-        print("Daemonizing...")
+        logging.info("Daemonizing...")
        daemonize()
        _reconfigure_logging_for_daemon(log_level)
        logging.info(f"hbc starting, sending heartbeat to {', '.join(args.hosts)}")
@@ -118,6 +118,13 @@ class CPUMonitorPlugin(MonitorPlugin):
                    data["cpu_iowait"] = round(cpu_times.iowait, 1)
            except Exception as e:
                self.logger.debug(f"Could not get CPU times: {e}")
+
+            # Uptime in seconds
+            try:
+                import time
+                data["uptime_seconds"] = int(time.time() - self.psutil.boot_time())
+            except Exception as e:
+                self.logger.debug(f"Could not get uptime: {e}")
            
            self.logger.debug(
                f"Collected CPU metrics: {data.get('cpu_percent', 'N/A')}% usage"
@@ -14,6 +14,24 @@ except ImportError:

 from hbd.client.plugin import MonitorPlugin

+
+def _zfs_arc_bytes() -> int:
+    """Return current ZFS ARC size in bytes, or 0 if ZFS is not present.
+
+    ZFS ARC is reclaimable but is not included in MemAvailable by the Linux
+    kernel (it is not in SReclaimable), so it would otherwise be counted as
+    used memory.
+    """
+    try:
+        with open("/proc/spl/kstat/zfs/arcstats") as fh:
+            for line in fh:
+                parts = line.split()
+                if len(parts) >= 3 and parts[0] == "size":
+                    return int(parts[2])
+    except (OSError, ValueError):
+        pass
+    return 0
+
 logger = logging.getLogger(__name__)


@@ -101,11 +119,21 @@ class MemoryMonitorPlugin(MonitorPlugin):
        
        # Virtual (physical) memory statistics
        vmem = psutil.virtual_memory()
+
+        # psutil's available already excludes page cache / file buffers
+        # (uses MemAvailable on Linux). Add ZFS ARC on top because the kernel
+        # does not include it in SReclaimable / MemAvailable even though it is
+        # reclaimable.
+        arc_bytes = _zfs_arc_bytes()
+        available = min(vmem.available + arc_bytes, vmem.total)
+        used = vmem.total - available
+        percent = round(used / vmem.total * 100, 1) if vmem.total else 0.0
+
        metrics['memory_total'] = vmem.total
-        metrics['memory_available'] = vmem.available
-        metrics['memory_used'] = vmem.used
+        metrics['memory_available'] = available
+        metrics['memory_used'] = used
        metrics['memory_free'] = vmem.free
-        metrics['memory_percent'] = vmem.percent
+        metrics['memory_percent'] = percent
        
        # Platform-specific memory details
        if hasattr(vmem, 'active'):
@@ -31,16 +31,13 @@ from hbd.client.plugin import MonitorPlugin


 # Nagios exit codes
-NAGIOS_OK = 0
-NAGIOS_WARNING = 1
-NAGIOS_CRITICAL = 2
 NAGIOS_UNKNOWN = 3

 STATUS_NAMES = {
-    NAGIOS_OK: "OK",
-    NAGIOS_WARNING: "WARNING",
-    NAGIOS_CRITICAL: "CRITICAL",
-    NAGIOS_UNKNOWN: "UNKNOWN"
+    0: "OK",
+    1: "WARNING",
+    2: "CRITICAL",
+    3: "UNKNOWN",
 }


@@ -128,52 +125,39 @@ class NagiosRunnerPlugin(MonitorPlugin):
            Dictionary with results from all plugins
        """
        results = {}
-        
-        # Track overall status (worst status wins)
-        worst_status = NAGIOS_OK
-        
+
        for cmd_config in self.commands:
            name = cmd_config.get("name")
            command = cmd_config.get("command")
-            
+
            if not name or not command:
                self.logger.warning("Skipping command with missing name or command")
                continue
-            
+
            # Execute plugin
            try:
                status_code, output, perfdata = await self._run_nagios_plugin(command)
-                
+
                # Store results
                results[f"{name}_status"] = STATUS_NAMES.get(status_code, "UNKNOWN")
                results[f"{name}_status_code"] = status_code
                results[f"{name}_output"] = output
-                
-                # Track worst status
-                if status_code > worst_status:
-                    worst_status = status_code
-                
+
                # Parse and add performance data
                if perfdata:
                    for metric_name, metric_value in perfdata.items():
                        results[f"{name}_{metric_name}"] = metric_value
-                
+
                self.logger.info(
                    f"Executed {name}: {STATUS_NAMES.get(status_code, 'UNKNOWN')} - {output[:50]}"
                )
-                
+
            except Exception as e:
                self.logger.error(f"Error running {name}: {e}", exc_info=True)
                results[f"{name}_status"] = "ERROR"
                results[f"{name}_status_code"] = NAGIOS_UNKNOWN
                results[f"{name}_output"] = str(e)
-                worst_status = NAGIOS_UNKNOWN
-        
-        # Add overall status
-        results["overall_status"] = STATUS_NAMES.get(worst_status, "UNKNOWN")
-        results["overall_status_code"] = worst_status
-        results["plugin_count"] = len(self.commands)
-        
+
        return results
    
    async def _run_nagios_plugin(
@@ -13,12 +13,8 @@ plugins:
    count: 3              # ICMP packets per ping run (default 3)
    timeout: 5            # seconds before a host is considered unreachable (default 5)
    hosts:
-      8.8.8.8:
-        warning: 20.0     # ms
-        critical: 100.0   # ms
-      192.168.1.1:
-        warning: 5.0
-        critical: 20.0
+      - 8.8.8.8
+      - 192.168.1.1
 ```

 Reported metrics per host (metric key uses the hostname with dots/colons replaced
@@ -95,6 +95,12 @@ THRESHOLD_DEFAULTS = {
                'warning': 200,
                'critical': 250.0,
                'count': 3  # Optional: number of consecutive breaches before alerting
+            },
+            'nagios_runner': {
+                'status_code': {
+                    'display': '{check_name} {output}',
+                    'operator': "nagios"
+                }   
            }
        }
    }
@@ -95,7 +95,7 @@ class Connection:
        if not Null:
            d["addr"] = self.addr
            if self.rtts[-1]:
-                d["rtt"] = "%0.1f" % self.rtts[-1]
+                d["rtt"] = "%d" % round(self.rtts[-1])
            elif self.state == Connection.UNKNOWN:
                d["rtt"] = ""
            else:
@@ -154,6 +154,25 @@ async def start(
        lst = [h.jsons() for h in hosts]
        return web.json_response(json.loads("[" + ",".join(lst) + "]"))

+    async def api_alert_summary(request):
+        """GET /api/0/alert_summary — counts of ok/warning/critical hosts visible to caller."""
+        user, err = _require_auth(request)
+        if err:
+            return err
+        from .threshold import AlertLevel
+        critical = warning = ok = 0
+        for host in hbdclass.Host.hosts.values():
+            if not _can_operate_host(user, host):
+                continue
+            levels = {s.level for s in host.alert_states.values()}
+            if AlertLevel.CRITICAL in levels:
+                critical += 1
+            elif AlertLevel.WARNING in levels:
+                warning += 1
+            else:
+                ok += 1
+        return web.json_response({"critical": critical, "warning": warning, "ok": ok})
+
    async def api_messages(request):
        lst = data.msgs[-30:]
        return web.json_response(lst)
@@ -518,6 +537,7 @@ async def start(
                hosts_with_plugins.append({
                    "name": hostname,
                    "plugins": list(host.plugin_data.keys()),
+                    "is_owner": _can_own_host(current_user, host),
                })

        tmpl = env.get_template("plugins.html")
@@ -870,7 +890,7 @@ async def start(
        tmpl = env.get_template("settings.html")
        body = tmpl.render(
            title="Settings - Heartbeat",
-            sections=settings_mod.get_settings_sections(config),
+            sections=settings_mod.get_settings_sections(config, threshold_checker=threshold_checker),
            current_user=current_user.to_dict() if current_user else None,
            active_page="settings",
        )
@@ -893,6 +913,7 @@ async def start(
            web.get("/api/0/users/{username}/avatar", api_user_avatar),
            # Hosts
            web.get("/api/0/hosts", api_hosts),
+            web.get("/api/0/alert_summary", api_alert_summary),
            web.get("/api/0/messages", api_messages),
            web.get("/api/0/hosts/{hostname}/plugins", api_host_plugins),
            web.get("/api/0/hosts/{hostname}/plugins/{plugin_name}", api_host_plugin_detail),
@@ -101,9 +101,10 @@ async def reload_configuration(config_obj, config_path, components):
            access = config_mod.get_host_access(new_config, hostname)
            host.apply_access(access["owner"], access["managers"], access["monitors"])

-        # Reload threshold checker
+        # Reload threshold checker and prune alerts orphaned by the new config
        if 'threshold_checker' in components:
            components['threshold_checker'].reload(new_config)
+            components['threshold_checker'].purge_stale_alerts(hbdclass)
        
        # Note: Changes to the following require restart:
        # - hb_port, hbd_port, ws_port (already bound)
@@ -241,6 +242,10 @@ async def _run_async(config, config_path=None):
    )
    udp.restore_connection_timers(hbdclass, restore_ctx)

+    # Drop alert states that no longer have a matching threshold (stale after
+    # upgrade or config change between runs).
+    threshold_checker.purge_stale_alerts(hbdclass)
+
    # HTTP server (asyncio-based via aiohttp)
    try:
        http_task = asyncio.create_task(
@@ -250,6 +255,7 @@ async def _run_async(config, config_path=None):
                config=config,
                hbdclass=hbdclass,
                tcss=None,
+                threshold_checker=threshold_checker,
                verbose=config.get("verbose", False),
                get_now=lambda: time.time(),
                VER="",
@@ -469,6 +475,8 @@ def run(config, config_path=None):
    if config.get("debug", 0) > 0:
        log_level = logging.DEBUG
    logging.basicConfig(level=log_level)
+    if not config.get("debug", 0):
+        logging.getLogger("aiohttp.access").propagate = False
    load_pickled_hosts(config, hbdclass)

    notify_mod.initlog(logfile=config.get("logfile", "messages.log"))
@@ -88,7 +88,7 @@ def _sanitize_channel(name, cfg):
 # Public API
 # ---------------------------------------------------------------------------

-def get_settings_sections(config: dict) -> list:
+def get_settings_sections(config: dict, threshold_checker=None) -> list:
    """Return ordered list of setting sections for the settings page.

    Each section:
@@ -182,46 +182,39 @@ def get_settings_sections(config: dict) -> list:
        })

    # ---- Threshold configurations -----------------------------------------
-    def _parse_metric_row(metric_path, metric_cfg):
-        if not isinstance(metric_cfg, dict):
-            return None
+    def _tc_to_row(tc):
        return {
-            "metric": metric_path,
-            "operator": metric_cfg.get("operator", ">"),
-            "warning": metric_cfg.get("warning"),
-            "critical": metric_cfg.get("critical"),
-            "hysteresis": metric_cfg.get("hysteresis"),
-            "count": metric_cfg.get("count", 1),
-            "enabled": metric_cfg.get("enabled", True),
+            "metric": tc.metric_path,
+            "operator": tc.operator.value,
+            "warning": tc.warning,
+            "critical": tc.critical,
+            "hysteresis": tc.hysteresis,
+            "count": tc.count,
+            "enabled": tc.enabled,
        }

    threshold_config_list = []
-    raw_tconfigs = config.get("threshold_configs") or {}
-    if raw_tconfigs:
-        for cfg_name, cfg_data in sorted(raw_tconfigs.items()):
-            if not isinstance(cfg_data, dict):
-                continue
-            metrics = [
-                r for r in (
-                    _parse_metric_row(mp, mc)
-                    for mp, mc in (cfg_data.get("thresholds") or {}).items()
-                ) if r
-            ]
-            threshold_config_list.append({
-                "name": cfg_name,
-                "metrics": sorted(metrics, key=lambda m: m["metric"]),
-            })
-    elif config.get("thresholds"):
-        metrics = [
-            r for r in (
-                _parse_metric_row(mp, mc)
-                for mp, mc in config["thresholds"].items()
-            ) if r
-        ]
-        threshold_config_list.append({
-            "name": "default",
-            "metrics": sorted(metrics, key=lambda m: m["metric"]),
-        })
+    if threshold_checker is not None:
+        if threshold_checker.threshold_configs:
+            for cfg_name, cfg_metrics in sorted(threshold_checker.threshold_configs.items()):
+                # For the default config use the merged effective set;
+                # for named overrides use only the explicitly defined metrics
+                # (threshold_raw_configs) so inherited defaults are not repeated.
+                if cfg_name == "default":
+                    display_metrics = cfg_metrics
+                else:
+                    display_metrics = threshold_checker.threshold_raw_configs.get(cfg_name, cfg_metrics)
+                metrics = sorted(
+                    [_tc_to_row(tc) for tc in display_metrics.values()],
+                    key=lambda m: m["metric"],
+                )
+                threshold_config_list.append({"name": cfg_name, "metrics": metrics})
+        elif threshold_checker.thresholds:
+            metrics = sorted(
+                [_tc_to_row(tc) for tc in threshold_checker.thresholds.values()],
+                key=lambda m: m["metric"],
+            )
+            threshold_config_list.append({"name": "default", "metrics": metrics})

    # ---- Hosts summary ----------------------------------------------------
    hosts_list = []
@@ -4,6 +4,11 @@

  <style>

+    html, body {
+      height: auto;
+      overflow-y: auto;
+    }
+
    .container {
      max-width: 1400px;
      margin: 0 auto;
@@ -170,14 +175,18 @@

    .alert-hostname {
      font-weight: bold;
-      color: #333;
+      color: #0066cc;
      font-size: 1.1em;
+      text-decoration: none;
+    }
+    .alert-hostname:hover {
+      text-decoration: underline;
    }

    .alert-metric {
-      color: #666;
-      font-family: 'Courier New', monospace;
-      font-size: 0.9em;
+      color: #0066cc;
+      font-size: 1.1em;
+      font-weight: normal;
    }

    .alert-details {
@@ -400,6 +409,10 @@
        } else if (alert.threshold_value !== undefined && alert.threshold_value !== null && alert.operator) {
          valueText += ` <span class="threshold-info">(threshold: ${alert.operator} ${formatValue(alert.threshold_value)})</span>`;
        }
+        if (alert.recovery_threshold !== undefined && alert.recovery_threshold !== null) {
+          const recOp = (alert.operator === '>' || alert.operator === '>=') ? '<' : '>';
+          valueText += ` <span class="threshold-info" style="color:#888">(recovers ${recOp} ${formatValue(alert.recovery_threshold)})</span>`;
+        }
        
        // Build actions section
        let actionsHtml = '';
@@ -424,9 +437,9 @@
            <div class="alert-main">
              <div class="alert-header">
                <span class="alert-level ${level}">${alert.level}</span>
-                <span class="alert-hostname">${alert.hostname}</span>
+                <a class="alert-hostname" href="/plugins#${alert.hostname}">${alert.hostname}</a>
+                <span class="alert-metric">${alert.metric_path.includes('.') ? alert.metric_path.slice(alert.metric_path.indexOf('.') + 1) : alert.metric_path}</span>
              </div>
-              <div class="alert-metric">${alert.metric_path}</div>
              <div class="alert-details">
                <span>${valueText}</span>
                <span class="alert-duration">Active for ${duration}</span>
@@ -126,11 +126,17 @@
      }

      /* Swiss railway clock — nav */
-      .nav-clock {
+      .nav-pie {
        flex-shrink: 0;
        line-height: 0;
        margin-left: auto;
        padding: 4px 4px 4px 0;
+      }
+      #alert-pie { display: block; cursor: default; }
+      .nav-clock {
+        flex-shrink: 0;
+        line-height: 0;
+        padding: 4px 4px 4px 0;
        cursor: pointer;
      }
      #swiss-clock { display: block; }
@@ -408,7 +408,7 @@
        );
        if (data.connections[i].state == "up") {
          state = '<span class="state-up">up</span>';
-          latency = Number.parseFloat(data.connections[i].rtts[0]).toFixed(2);
+          latency = String(Math.round(Number.parseFloat(data.connections[i].rtts[0])));
        } else {
          if (data.connections[i].state == "unknown") {
            state = "";
@@ -11,6 +11,9 @@
    {% endif %}
    <a href="/about"{% if active_page == "about" %} class="active"{% endif %}>About</a>
  </div>
+  <div class="nav-pie" title="Host alert status">
+    <canvas id="alert-pie" width="44" height="44"></canvas>
+  </div>
  <div class="nav-clock" title="Click for full-screen clock">
    <canvas id="swiss-clock" width="44" height="44"></canvas>
  </div>
@@ -42,4 +45,52 @@
      });
    }
  })();
+
+  function drawAlertPie(critical, warning, ok) {
+    var canvas = document.getElementById('alert-pie');
+    if (!canvas) return;
+    var ctx = canvas.getContext('2d');
+    var SIZE = canvas.width;
+    var R = SIZE / 2;
+    ctx.clearRect(0, 0, SIZE, SIZE);
+    var total = critical + warning + ok;
+    if (total === 0) {
+      ctx.beginPath();
+      ctx.arc(R, R, R - 1, 0, Math.PI * 2);
+      ctx.fillStyle = '#ccc';
+      ctx.fill();
+      return;
+    }
+    var slices = [
+      { value: critical, color: '#e53935' },
+      { value: warning,  color: '#ffb300' },
+      { value: ok,       color: '#43a047' }
+    ];
+    var start = -Math.PI / 2;
+    slices.forEach(function(s) {
+      if (s.value === 0) return;
+      var sweep = (s.value / total) * Math.PI * 2;
+      ctx.beginPath();
+      ctx.moveTo(R, R);
+      ctx.arc(R, R, R - 1, start, start + sweep);
+      ctx.closePath();
+      ctx.fillStyle = s.color;
+      ctx.fill();
+      start += sweep;
+    });
+  }
+
+  function updateAlertPie() {
+    fetch('/api/0/alert_summary').then(function(r) {
+      if (!r.ok) return;
+      return r.json();
+    }).then(function(d) {
+      if (d) drawAlertPie(d.critical || 0, d.warning || 0, d.ok || 0);
+    }).catch(function() {});
+  }
+
+  document.addEventListener('DOMContentLoaded', function() {
+    updateAlertPie();
+    setInterval(updateAlertPie, 30000);
+  });
 </script>
@@ -131,6 +131,52 @@
      text-overflow: ellipsis;
    }

+    .host-action-btn {
+      font-size: 0.75em;
+      font-weight: bold;
+      padding: 3px 10px;
+      border-radius: 4px;
+      border: none;
+      cursor: pointer;
+      text-decoration: none;
+      white-space: nowrap;
+    }
+    .host-action-btn.update-btn {
+      background: #e3f2fd;
+      color: #1565c0;
+    }
+    .host-action-btn.update-btn:hover { background: #bbdefb; }
+    .host-action-btn.delete-btn {
+      background: #ffebee;
+      color: #c62828;
+    }
+    .host-action-btn.delete-btn:hover { background: #ffcdd2; }
+
+    /* ── Action result toast ───────────────────────────────────── */
+    #action-toast {
+      position: fixed;
+      bottom: 24px;
+      left: 50%;
+      transform: translateX(-50%) translateY(20px);
+      background: #323232;
+      color: #fff;
+      padding: 12px 22px;
+      border-radius: 6px;
+      font-size: 0.9em;
+      max-width: 480px;
+      text-align: center;
+      opacity: 0;
+      pointer-events: none;
+      transition: opacity 0.25s, transform 0.25s;
+      z-index: 9000;
+      white-space: pre-wrap;
+    }
+    #action-toast.show {
+      opacity: 1;
+      transform: translateX(-50%) translateY(0);
+    }
+    #action-toast.error { background: #c62828; }
+
    /* ── Host body ──────────────────────────────────────────────── */

    .host-body {
@@ -379,6 +425,12 @@
              <span class="nagios-badge" id="nagios-badge-{{ host.name }}">—</span>
              {% endif %}
              <span class="os-label" id="os-label-{{ host.name }}"></span>
+              {% if host.is_owner %}
+              <button class="host-action-btn update-btn"
+                      onclick="event.stopPropagation(); hostAction(this, '/u?h={{ host.name }}')">Update</button>
+              <button class="host-action-btn delete-btn"
+                      onclick="event.stopPropagation(); hostDelete(this, '{{ host.name }}')">Delete</button>
+              {% endif %}
            </div>
          </div>

@@ -447,6 +499,17 @@
        return pluginCache[hostname]?.[pluginName] ?? null;
      }

+      // Return worst nagios exit code (0-3) found in a nagios_runner data object.
+      function nagiosWorstStatus(data) {
+        let worst = 0;
+        for (const [k, v] of Object.entries(data || {})) {
+          if (k.endsWith('_status_code') && typeof v === 'number' && v > worst) {
+            worst = v;
+          }
+        }
+        return worst;
+      }
+
      // ── Fetch helpers ───────────────────────────────────────────────────────

      async function fetchPlugin(hostname, pluginName) {
@@ -548,13 +611,13 @@
          ? chips.join('')
          : '<span class="glance-loading">—</span>';

-        // Nagios badge
+        // Nagios badge — derive worst status from individual check codes
        const nagios = getCache(hostname, 'nagios_runner');
        if (nagosBadge && nagios) {
-          const status = (nagios.data.overall_status || '—').toUpperCase();
-          const cls = status === 'OK' ? 'ok'
-            : status === 'WARNING' ? 'warning'
-            : status === 'CRITICAL' ? 'critical' : '';
+          const worst = nagiosWorstStatus(nagios.data);
+          const names = {0:'OK', 1:'WARNING', 2:'CRITICAL', 3:'UNKNOWN'};
+          const status = names[worst] || '—';
+          const cls = worst === 0 ? 'ok' : worst === 1 ? 'warning' : worst >= 2 ? 'critical' : '';
          nagosBadge.className = `nagios-badge ${cls}`;
          nagosBadge.textContent = status;
        }
@@ -663,9 +726,10 @@
            break;
          }
          case 'nagios_runner': {
-            const status = (d.overall_status || '?').toUpperCase();
-            const count = d.plugin_count;
-            text = status + (count != null ? ` — ${count} checks` : '');
+            const worst = nagiosWorstStatus(d);
+            const names = {0:'OK', 1:'WARNING', 2:'CRITICAL', 3:'UNKNOWN'};
+            const codes = Object.keys(d).filter(k => k.endsWith('_status_code'));
+            text = (names[worst] || '?') + (codes.length ? ` — ${codes.length} checks` : '');
            break;
          }
          case 'filesystem_info': {
@@ -1175,6 +1239,49 @@
          fetchHostGlance(first.dataset.hostname);
        }
      });
+      // ── Host action helpers ──────────────────────────────────────
+
+      let _toastTimer = null;
+      function showToast(msg, isError) {
+        const t = document.getElementById('action-toast');
+        t.textContent = msg;
+        t.classList.toggle('error', !!isError);
+        t.classList.add('show');
+        clearTimeout(_toastTimer);
+        _toastTimer = setTimeout(() => t.classList.remove('show'), 4000);
+      }
+
+      async function hostAction(btn, url) {
+        btn.disabled = true;
+        try {
+          const res = await fetch(url);
+          const text = await res.text();
+          showToast(text, !res.ok);
+        } catch (e) {
+          showToast('Request failed: ' + e.message, true);
+        } finally {
+          btn.disabled = false;
+        }
+      }
+
+      async function hostDelete(btn, hostname) {
+        if (!confirm('Delete host ' + hostname + '?')) return;
+        btn.disabled = true;
+        try {
+          const res = await fetch('/d?h=' + encodeURIComponent(hostname));
+          const text = await res.text();
+          showToast(text, !res.ok);
+          if (res.ok) {
+            const card = document.querySelector(`.host-card[data-hostname="${hostname}"]`);
+            if (card) card.remove();
+          }
+        } catch (e) {
+          showToast('Request failed: ' + e.message, true);
+          btn.disabled = false;
+        }
+      }
    </script>
+
+    <div id="action-toast"></div>
  </body>
 </html>
@@ -30,12 +30,13 @@ class AlertLevel(Enum):

 class ComparisonOperator(Enum):
    """Supported comparison operators for threshold checks."""
-    GT = ">"      # Greater than
-    GTE = ">="    # Greater than or equal
-    LT = "<"      # Less than
-    LTE = "<="    # Less than or equal
-    EQ = "=="     # Equal to
-    NEQ = "!="    # Not equal to
+    GT = ">"        # Greater than
+    GTE = ">="      # Greater than or equal
+    LT = "<"        # Less than
+    LTE = "<="      # Less than or equal
+    EQ = "=="       # Equal to
+    NEQ = "!="      # Not equal to
+    NAGIOS = "nagios"  # Nagios exit-code semantics: 0=OK 1=WARNING 2=CRITICAL 3=UNKNOWN


 class AlertState:
@@ -57,6 +58,7 @@ class AlertState:
        self.last_notification = None
        self.threshold_value = None  # The threshold value that triggered alert
        self.operator = None  # The comparison operator (>, <, >=, etc.)
+        self.hysteresis: Optional[float] = None  # Hysteresis fraction used for recovery
        self.formatted_message = None  # Formatted display message for UI
        self.acknowledged = False  # Whether alert has been acknowledged
        self.acknowledged_at = None  # Timestamp when acknowledged
@@ -151,7 +153,16 @@ class AlertState:
            result["operator"] = self.operator
        if self.formatted_message is not None:
            result["formatted_message"] = self.formatted_message
-        
+
+        # Compute and expose the recovery threshold so the UI can display it
+        if (self.hysteresis and self.threshold_value is not None
+                and self.operator is not None):
+            ha = abs(self.threshold_value * self.hysteresis)
+            if self.operator in ('>', '>='):
+                result["recovery_threshold"] = round(self.threshold_value - ha, 4)
+            elif self.operator in ('<', '<='):
+                result["recovery_threshold"] = round(self.threshold_value + ha, 4)
+
        return result
    
    def __setstate__(self, state):
@@ -159,6 +170,8 @@ class AlertState:
        self.__dict__.update(state)
        if not hasattr(self, 'consecutive_count'):
            self.consecutive_count = 0
+        if not hasattr(self, 'hysteresis'):
+            self.hysteresis = None

    def acknowledge(self):
        """Acknowledge this alert to stop reminder notifications."""
@@ -217,33 +230,43 @@ class ThresholdConfig:
    def evaluate(self, value: float) -> AlertLevel:
        """
        Evaluate a value against this threshold.
-        
+
        Args:
            value: Metric value to check
-            
+
        Returns:
            AlertLevel indicating the severity
        """
        if not self.enabled:
            return AlertLevel.OK
-        
+
+        # Nagios exit-code semantics: value IS the severity
+        if self.operator == ComparisonOperator.NAGIOS:
+            try:
+                code = int(value)
+            except (TypeError, ValueError):
+                return AlertLevel.UNKNOWN
+            return {0: AlertLevel.OK, 1: AlertLevel.WARNING, 2: AlertLevel.CRITICAL}.get(
+                code, AlertLevel.UNKNOWN
+            )
+
        try:
            # Convert value to float for comparison
            value = float(value)
        except (TypeError, ValueError):
            logger.warning("Cannot convert value %s to float for %s", value, self.metric_path)
            return AlertLevel.UNKNOWN
-        
+
        # Check critical threshold first
        if self.critical is not None:
            if self._compare(value, self.critical):
                return AlertLevel.CRITICAL
-        
+
        # Then check warning threshold
        if self.warning is not None:
            if self._compare(value, self.warning):
                return AlertLevel.WARNING
-        
+
        return AlertLevel.OK
    
    def evaluate_with_hysteresis(
@@ -262,7 +285,11 @@ class ThresholdConfig:
            New alert level considering hysteresis
        """
        new_level = self.evaluate(value)
-        
+
+        # Nagios exit codes are discrete integers — hysteresis doesn't apply
+        if self.operator == ComparisonOperator.NAGIOS:
+            return new_level
+
        # If no hysteresis, return new level
        if self.hysteresis == 0.0:
            return new_level
@@ -392,14 +419,28 @@ class ThresholdChecker:
    
    def _parse_config(self, config: Dict[str, Any]):
        """Parse threshold configuration from YAML structure.
-        
+
        Supports two formats:
        1. Legacy format with direct 'thresholds' section
        2. New format with 'threshold_configs' and 'host_threshold_mapping'
+
+        In all cases, THRESHOLD_DEFAULTS are seeded into threshold_configs["default"]
+        so the Settings page always shows the built-in defaults.
+        _parse_multi_config() overwrites this with the fully-merged effective defaults.
        """
+        # Always expose built-in defaults through threshold_configs["default"] so
+        # the Settings page has something to display even in legacy/no-config mode.
+        seed: Dict[str, ThresholdConfig] = {}
+        for plugin_name, plugin_thresholds in THRESHOLD_DEFAULTS.get("thresholds", {}).items():
+            if isinstance(plugin_thresholds, dict):
+                self._parse_plugin_thresholds(plugin_name, plugin_thresholds, target_dict=seed)
+        if seed:
+            self.threshold_configs["default"] = seed
+            self.threshold_raw_configs["default"] = {}
+
        # Check for new multi-config format
        if "threshold_configs" in config:
-            self._parse_multi_config(config)
+            self._parse_multi_config(config)  # overwrites threshold_configs["default"]
        elif "thresholds" in config:
            # Legacy single threshold configuration
            self._parse_legacy_config(config)
@@ -545,11 +586,14 @@ class ThresholdChecker:
            warning = threshold_config.get("warning")
            critical = threshold_config.get("critical")
            operator = threshold_config.get("operator", ">")
-            display = threshold_config.get("display", "(threshold: {op_symbol} {threshold_value})")
-            hysteresis = threshold_config.get("hysteresis", 0.1)  # 10% default
+            # Nagios operator maps exit codes directly; no numeric thresholds needed
+            is_nagios_op = (operator == "nagios")
+            default_display = "{check_name}: {output}" if is_nagios_op else "(threshold: {op_symbol} {threshold_value})"
+            display = threshold_config.get("display", default_display)
+            hysteresis = threshold_config.get("hysteresis", 0.0 if is_nagios_op else 0.02)
            enabled = threshold_config.get("enabled", True)
-            
-            if warning is None and critical is None:
+
+            if warning is None and critical is None and not is_nagios_op:
                logger.warning("No thresholds defined for %s, skipping", metric_path)
                continue
            
@@ -649,7 +693,7 @@ class ThresholdChecker:
        warning = rtt_thresholds.get("warning")
        critical = rtt_thresholds.get("critical")
        operator = rtt_thresholds.get("operator", ">")
-        hysteresis = rtt_thresholds.get("hysteresis", 0.1)  # 10% default
+        hysteresis = rtt_thresholds.get("hysteresis", 0.02)  # 2% default
        enabled = rtt_thresholds.get("enabled", True)
        display = rtt_thresholds.get("display")
        count = rtt_thresholds.get("count", 1)
@@ -794,6 +838,12 @@ class ThresholdChecker:
        elif new_level == AlertLevel.WARNING and threshold.warning is not None:
            threshold_value = threshold.warning

+        # Keep hysteresis on the state so the UI can show the recovery threshold
+        if new_level != AlertLevel.OK:
+            alert_state.hysteresis = threshold.hysteresis
+        else:
+            alert_state.hysteresis = None
+
        # Update state and check for changes
        old_level = alert_state.level
        if alert_state.update(new_level, value, threshold_value, threshold.operator.value):
@@ -803,6 +853,36 @@ class ThresholdChecker:
            self._check_pending_or_renotify(host_name, alert_state, metric_path, value, threshold, None)

        return None
+    def _find_threshold(
+        self, thresholds: Dict[str, "ThresholdConfig"], metric_path: str
+    ) -> Tuple[Optional["ThresholdConfig"], Optional[str]]:
+        """Return (threshold, check_name) for *metric_path*, falling back to suffix matches.
+
+        Allows generic thresholds like ``nagios_runner.status_code`` to match
+        fully-qualified paths like ``nagios_runner.check_disk_root_status_code``.
+        The exact match is always tried first; then successive leading
+        underscore-delimited segments are stripped from the field name until
+        a match is found or no segments remain.
+
+        Returns:
+            (ThresholdConfig, None) for an exact match.
+            (ThresholdConfig, "check_disk_root") for a suffix match — the second
+            element is the stripped prefix, available as ``{check_name}`` in
+            display format templates.
+            (None, None) when no threshold is found.
+        """
+        if metric_path in thresholds:
+            return thresholds[metric_path], None
+        plugin, sep, field = metric_path.partition(".")
+        if not sep:
+            return None, None
+        parts = field.split("_")
+        for i in range(1, len(parts)):
+            candidate = plugin + "." + "_".join(parts[i:])
+            if candidate in thresholds:
+                return thresholds[candidate], "_".join(parts[:i])
+        return None, None
+
    def check_plugin_data(
        self,
        host_name: str,
@@ -830,38 +910,39 @@ class ThresholdChecker:
        # Check flat metrics
        for metric_name, value in data.items():
            metric_path = f"{plugin_name}.{metric_name}"
-            
-            if metric_path not in thresholds:
+
+            threshold, check_name = self._find_threshold(thresholds, metric_path)
+            if threshold is None:
                continue
-            
-            threshold = thresholds[metric_path]
-            
+
            # Get or create alert state
            if metric_path not in alert_states:
                alert_states[metric_path] = AlertState(metric_path)
-            
+
            alert_state = alert_states[metric_path]
-            
+
            # Evaluate threshold with hysteresis
            new_level = threshold.evaluate_with_hysteresis(
                value,
                alert_state.level
            )
-            
+
            # Determine which threshold was exceeded
            threshold_value = None
            if new_level == AlertLevel.CRITICAL and threshold.critical is not None:
                threshold_value = threshold.critical
            elif new_level == AlertLevel.WARNING and threshold.warning is not None:
                threshold_value = threshold.warning
-            
+
+            alert_state.hysteresis = threshold.hysteresis if new_level != AlertLevel.OK else None
+
            # Update state and check for changes
            old_level = alert_state.level
            if alert_state.update(new_level, value, threshold_value, threshold.operator.value):
                state_changes.append((metric_path, old_level, new_level, value))
-                self._apply_grace(host_name, alert_state, metric_path, old_level, new_level, value, threshold, data)
+                self._apply_grace(host_name, alert_state, metric_path, old_level, new_level, value, threshold, data, check_name=check_name, metric_name=metric_name)
            elif new_level != AlertLevel.OK:
-                self._check_pending_or_renotify(host_name, alert_state, metric_path, value, threshold, data)
+                self._check_pending_or_renotify(host_name, alert_state, metric_path, value, threshold, data, check_name=check_name, metric_name=metric_name)

        # Check nested metrics (e.g., partition data in disk_monitor)
        self._check_nested_metrics(
@@ -920,7 +1001,9 @@ class ThresholdChecker:
                        threshold_value = threshold.critical
                    elif new_level == AlertLevel.WARNING and threshold.warning is not None:
                        threshold_value = threshold.warning
-                    
+
+                    alert_state.hysteresis = threshold.hysteresis if new_level != AlertLevel.OK else None
+
                    old_level = alert_state.level
                    if alert_state.update(new_level, value, threshold_value, threshold.operator.value):
                        state_changes.append((metric_path, old_level, new_level, value))
@@ -937,6 +1020,8 @@ class ThresholdChecker:
        value: Any,
        threshold: ThresholdConfig,
        plugin_data: Optional[Dict[str, Any]] = None,
+        check_name: Optional[str] = None,
+        metric_name: Optional[str] = None,
    ):
        """Trigger a notification for an alert state change.
        
@@ -958,56 +1043,54 @@ class ThresholdChecker:
        
        # Format operator symbol
        op_symbol = threshold.operator.value
-        
+
+        # Short metric label: strip the plugin-name prefix for readability
+        short_path = metric_path.partition(".")[2] or metric_path
+
        # Use a display-friendly value (inf is the sentinel for "overdue")
        import math
        display_value = "overdue" if isinstance(value, float) and math.isinf(value) else value

-        # Format message
-        if new_level == AlertLevel.OK:
-            lvl = "RECOVER"
-            message = f"{metric_path} = {display_value} ({old_level.name} -> OK)"
-        elif new_level == AlertLevel.WARNING:
-            lvl = "WARNING"
-            if threshold_value is not None:
-                threshold_info = self._format_display(
-                    threshold.display,
-                    value=display_value,
-                    threshold_value=threshold_value,
-                    op_symbol=op_symbol,
-                    plugin_data=plugin_data
-                )
-                message = f"{metric_path} = {display_value} {threshold_info}"
-            else:
-                message = f"{metric_path} = {display_value}"
-        elif new_level == AlertLevel.CRITICAL:
-            lvl = "CRITICAL"
-            if threshold_value is not None:
-                threshold_info = self._format_display(
-                    threshold.display,
-                    value=display_value,
-                    threshold_value=threshold_value,
-                    op_symbol=op_symbol,
-                    plugin_data=plugin_data
-                )
-                message = f"{metric_path} = {display_value} {threshold_info}"
-            else:
-                message = f"{metric_path} = {display_value}"
-        else:
-            lvl = "UNKNOWN"
-            message = f"{metric_path} = {display_value}"
-        
-        # Return the formatted threshold info for storing in AlertState
-        formatted_threshold_msg = None
-        if threshold_value is not None and new_level != AlertLevel.OK:
-            formatted_threshold_msg = self._format_display(
+        # Format message — for the nagios operator there is no numeric threshold_value;
+        # render the display template whenever one is available.
+        has_display = threshold_value is not None or threshold.operator == ComparisonOperator.NAGIOS
+
+        def _fmt():
+            return self._format_display(
                threshold.display,
                value=display_value,
                threshold_value=threshold_value,
                op_symbol=op_symbol,
-                plugin_data=plugin_data
+                plugin_data=plugin_data,
+                check_name=check_name,
+                metric_name=metric_name,
            )
-        
+
+        if new_level == AlertLevel.OK:
+            lvl = "RECOVER"
+            message = f"{short_path} = {display_value} ({old_level.name} -> OK)"
+        elif new_level == AlertLevel.WARNING:
+            lvl = "WARNING"
+            if has_display:
+                message = f"{short_path} = {display_value} {_fmt()}"
+            else:
+                message = f"{short_path} = {display_value}"
+        elif new_level == AlertLevel.CRITICAL:
+            lvl = "CRITICAL"
+            if has_display:
+                message = f"{short_path} = {display_value} {_fmt()}"
+            else:
+                message = f"{short_path} = {display_value}"
+        else:
+            lvl = "UNKNOWN"
+            if has_display:
+                message = f"{short_path} = {display_value} {_fmt()}"
+            else:
+                message = f"{short_path} = {display_value}"
+
+        # Formatted threshold info stored on AlertState for the UI
+        formatted_threshold_msg = _fmt() if has_display and new_level != AlertLevel.OK else None
+
        return lvl, message, formatted_threshold_msg
    
    def _send_notification(
@@ -1026,11 +1109,16 @@ class ThresholdChecker:
        if host is not None and not host.watched:
            eventlog(host_name, lvl, message, service="threshold")
            return
+        short_path = metric_path.partition(".")[2] or metric_path
+        title = f"[{lvl}] {host_name}  {short_path}"
+        # Strip the "metric = " prefix from message so body is just the value/detail
+        prefix = short_path + " = "
+        body = message[len(prefix):] if message.startswith(prefix) else message
        asyncio.get_event_loop().create_task(notify_mod.send_notification(
            host_name,
            notify_mod.Notification(
-                title=f"[{lvl}] {host_name}",
-                body=message,
+                title=title,
+                body=body,
                level=lvl,
            ),
        ))
@@ -1055,32 +1143,61 @@ class ThresholdChecker:
        self,
        display_format: str,
        value: Any,
-        threshold_value: float,
+        threshold_value: Optional[float],
        op_symbol: str,
        plugin_data: Optional[Dict[str, Any]] = None,
+        check_name: Optional[str] = None,
+        metric_name: Optional[str] = None,
    ) -> str:
        """Format the display string using available data.
-        
-        Args:
-            display_format: Format string from threshold config
-            value: Current metric value
-            threshold_value: Threshold value that was exceeded
-            op_symbol: Comparison operator symbol
-            plugin_data: Optional dictionary of plugin data fields
-            
+
+        Available template variables:
+            {value}           - current metric value
+            {threshold_value} - threshold that was exceeded
+            {op_symbol}       - comparison operator (>, <, >=, <=, ==, !=)
+            {check_name}      - prefix stripped for generic threshold match
+                                (e.g. "check_disk_root" when metric
+                                "check_disk_root_status_code" matched generic
+                                threshold "status_code")
+            {metric_name}     - field name within the plugin data dict
+            Any key from plugin_data is also available.
+
        Returns:
            Formatted display string
        """
+        if not display_format:
+            display_format = "(threshold: {op_symbol} {threshold_value})" if threshold_value is not None else ""
+
        # Build format context with standard variables
        format_context = {
            'value': value,
-            'threshold_value': threshold_value,
            'op_symbol': op_symbol,
        }
-        
+        if threshold_value is not None:
+            format_context['threshold_value'] = threshold_value
+
+        # Add generic-match context variables when available
+        if check_name is not None:
+            format_context['check_name'] = check_name
+        if metric_name is not None:
+            format_context['metric_name'] = metric_name
+
        # Add all plugin data fields if available
        if plugin_data:
            format_context.update(plugin_data)
+
+        # For nagios_runner generic matches, expose the matched check's output
+        # and status as short aliases {output} and {status} so display templates
+        # don't need to use the full {check_disk_root_output} form.
+        if check_name and plugin_data:
+            if 'output' not in format_context:
+                output = plugin_data.get(f"{check_name}_output")
+                if output is not None:
+                    format_context['output'] = output
+            if 'status' not in format_context:
+                status = plugin_data.get(f"{check_name}_status")
+                if status is not None:
+                    format_context['status'] = status
        
        try:
            # Format the display string
@@ -1111,6 +1228,8 @@ class ThresholdChecker:
        value: Any,
        threshold: ThresholdConfig,
        plugin_data: Optional[Dict[str, Any]],
+        check_name: Optional[str] = None,
+        metric_name: Optional[str] = None,
    ) -> None:
        """Handle a state-change transition with grace-period logic.

@@ -1123,7 +1242,8 @@ class ThresholdChecker:
          - Past grace: fires the RECOVER notification normally.
        """
        lvl, message, formatted_msg = self._trigger_notification(
-            host_name, metric_path, old_level, new_level, value, threshold, plugin_data
+            host_name, metric_path, old_level, new_level, value, threshold, plugin_data,
+            check_name=check_name, metric_name=metric_name,
        )
        alert_state.formatted_message = formatted_msg

@@ -1159,6 +1279,8 @@ class ThresholdChecker:
        value: Any,
        threshold: ThresholdConfig,
        plugin_data: Optional[Dict[str, Any]],
+        check_name: Optional[str] = None,
+        metric_name: Optional[str] = None,
    ) -> None:
        """Called when alert level is unchanged and non-OK.

@@ -1168,7 +1290,8 @@ class ThresholdChecker:
        if alert_state.pending_since is not None:
            if time.time() - alert_state.pending_since >= self.grace_seconds:
                lvl, message, formatted_msg = self._trigger_notification(
-                    host_name, metric_path, AlertLevel.OK, alert_state.level, value, threshold, plugin_data
+                    host_name, metric_path, AlertLevel.OK, alert_state.level, value, threshold, plugin_data,
+                    check_name=check_name, metric_name=metric_name,
                )
                alert_state.formatted_message = formatted_msg
                self._send_notification(
@@ -1177,7 +1300,7 @@ class ThresholdChecker:
                alert_state.pending_since = None
            # else: still within grace window, do nothing
        else:
-            self._check_renotify(host_name, alert_state, metric_path, value, threshold, plugin_data)
+            self._check_renotify(host_name, alert_state, metric_path, value, threshold, plugin_data, check_name=check_name, metric_name=metric_name)

    def _check_renotify(
        self,
@@ -1187,6 +1310,8 @@ class ThresholdChecker:
        value: Any,
        threshold: ThresholdConfig,
        plugin_data: Optional[Dict[str, Any]] = None,
+        check_name: Optional[str] = None,
+        metric_name: Optional[str] = None,
    ):
        """Check if we should send a repeat notification.
        
@@ -1224,7 +1349,8 @@ class ThresholdChecker:
            
            # Format operator symbol
            op_symbol = threshold.operator.value
-            
+            short_path = metric_path.partition(".")[2] or metric_path
+
            # Time to re-notify
            if threshold_value is not None:
                # Use display format string
@@ -1233,20 +1359,23 @@ class ThresholdChecker:
                    value=value,
                    threshold_value=threshold_value,
                    op_symbol=op_symbol,
-                    plugin_data=plugin_data
+                    plugin_data=plugin_data,
+                    check_name=check_name,
+                    metric_name=metric_name,
                )
-                message = f"REMINDER ({alert_state.level.name}): {host_name} - {metric_path} = {value} {threshold_info}, ongoing for {int(now - alert_state.since)}s"
+                body = f"{value} {threshold_info}, ongoing for {int(now - alert_state.since)}s"
            else:
-                message = f"REMINDER ({alert_state.level.name}): {host_name} - {metric_path} = {value} (ongoing for {int(now - alert_state.since)}s)"
-            
+                body = f"{value} (ongoing for {int(now - alert_state.since)}s)"
+            message = f"REMINDER ({alert_state.level.name}): {host_name} - {short_path} = {body}"
+
            from . import hbdclass
            host = hbdclass.Host.hosts.get(host_name)
            if host is None or host.watched:
                asyncio.get_event_loop().create_task(notify_mod.send_notification(
                    host_name,
                    notify_mod.Notification(
-                        title=f"[REMINDER/{alert_state.level.name}] {host_name}",
-                        body=message,
+                        title=f"[REMINDER/{alert_state.level.name}] {host_name}  {short_path}",
+                        body=body,
                        level=alert_state.level.name,
                    ),
                ))
@@ -1254,6 +1383,26 @@ class ThresholdChecker:
            alert_state.last_notification = now
            alert_state.notification_count += 1
    
+    def purge_stale_alerts(self, hbdclass) -> None:
+        """Remove alert states that have no matching threshold configuration.
+
+        Called after startup (pickle restore) and after each config reload so
+        that alerts orphaned by configuration changes do not linger forever.
+        Alerts whose metric_path is not present in the current threshold config
+        for that host are silently dropped.
+        """
+        for hostname, host in hbdclass.Host.hosts.items():
+            if not host.alert_states:
+                continue
+            configured = self.get_thresholds_for_host(hostname)
+            stale = [mp for mp in host.alert_states if self._find_threshold(configured, mp)[0] is None]
+            for mp in stale:
+                logger.info(
+                    "Purging stale alert state for %s / %s (no threshold configured)",
+                    hostname, mp,
+                )
+                del host.alert_states[mp]
+
    def get_active_alerts(self, alert_states: Dict[str, AlertState]) -> list:
        """
        Get all currently active (non-OK) alerts.
@@ -336,8 +336,7 @@ def handle_datagram(msg: dict, addr, transport, ctx: dict):
        # Apply user-access settings from config
        access = config_mod.get_host_access(cfg, uname)
        host.apply_access(access["owner"], access["managers"], access["monitors"])
-        if verbose:
-            print(("XX: New host, num now %s" % (len(hbdcls.Host.hosts))))
+        logger.info("New host signed on: %s (dyn=%s, access=%s)", uname, host.dyn, access)
        newh = True
    else:
        host = hbdcls.Host.hosts[uname]
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"

 [project]
 name = "hbd"
-version = "5.1.15"
+version = "5.2.3"
 description = "Heartbeat monitoring system — client (hbc) and server (hbd)"
 readme = "README.md"
 requires-python = ">=3.11"
@@ -41,7 +41,7 @@ from pathlib import Path
 from typing import Any, Dict, List, Optional, Tuple

 # updated by scripts/bumpminor.sh
-__version__ = "5.1.15"
+__version__ = "5.2.3"

 # ---------------------------------------------------------------------------
 # Protocol  (mirrors hbd/common/proto.py)
@@ -388,7 +388,6 @@ class NagiosRunnerPlugin(MonitorPlugin):

    async def _collect_metrics(self) -> Dict[str, Any]:
        results: Dict[str, Any] = {}
-        worst = 0
        for cmd_cfg in self.commands:
            name = cmd_cfg.get("name")
            command = cmd_cfg.get("command")
@@ -399,10 +398,6 @@ class NagiosRunnerPlugin(MonitorPlugin):
            results[f"{name}_status_code"] = rc
            results[f"{name}_output"] = msg
            results.update({f"{name}_{k}": v for k, v in perf.items()})
-            worst = max(worst, rc)
-        results["overall_status"] = _NAGIOS_STATUS.get(worst, "UNKNOWN")
-        results["overall_status_code"] = worst
-        results["plugin_count"] = len(self.commands)
        return results


@@ -487,6 +482,12 @@ class CPUMonitorPlugin(MonitorPlugin):
        except Exception:
            pass

+        try:
+            with open("/proc/uptime") as fh:
+                data["uptime_seconds"] = int(float(fh.read().split()[0]))
+        except Exception:
+            pass
+
        return data


@@ -535,6 +536,20 @@ class MemoryMonitorPlugin(MonitorPlugin):
        total = mi.get("MemTotal", 0)
        avail = mi.get("MemAvailable", mi.get("MemFree", 0))
        free = mi.get("MemFree", 0)
+
+        # ZFS ARC is reclaimable but not included in MemAvailable; add it.
+        arc_kb = 0
+        try:
+            with open("/proc/spl/kstat/zfs/arcstats") as _f:
+                for _line in _f:
+                    _p = _line.split()
+                    if len(_p) >= 3 and _p[0] == "size":
+                        arc_kb = int(_p[2]) // 1024
+                        break
+        except (OSError, ValueError):
+            pass
+
+        avail = min(avail + arc_kb, total)
        used = total - avail
        data: Dict[str, Any] = {
            "memory_total": total * 1024,
@@ -782,8 +797,7 @@ class _HeartbeatProtocol(asyncio.DatagramProtocol):
            self._log.error("datagram error: %s", e)

    def error_received(self, exc):
-        self._log.warning("protocol error on %s: %s — dropping connection", self._conn.addr, exc)
-        self._conn._dead = True
+        self._log.warning("protocol error on %s: %s — will retry", self._conn.addr, exc)
        self._conn.close()


@@ -1014,7 +1028,7 @@ def _reconfigure_syslog(level: int):
 # ---------------------------------------------------------------------------

 async def _async_main(args, cfg: Dict[str, Any]) -> int:
-    global _running, _shutdown_event, _active_tasks
+    global _running, _shutdown_event, _active_tasks, send_shutdown
    _running = True
    _shutdown_event = asyncio.Event()
    _active_tasks = []
@@ -1024,7 +1038,7 @@ async def _async_main(args, cfg: Dict[str, Any]) -> int:
    port = cfg.get("hb_port", PORT)
    interval = cfg.get("interval", INTERVAL)

-    log.info("starting: %s -> %s port=%d interval=%ds", iam, args.hosts, port, interval)
+    log.info("starting hbc_mini %s on %s -> %s port=%d interval=%ds",__version__, iam, args.hosts, port, interval)

    connections: List[AsyncConnection] = []
    conn_id = 1
@@ -1045,15 +1059,18 @@ async def _async_main(args, cfg: Dict[str, Any]) -> int:
        return 1

    # Boot / one-shot message
+    send_shutdown = False
    if args.boot or args.message:
        bmsg: Dict[str, Any] = {"acks": 0}
        if args.boot:
            bmsg["boot"] = 1
+            args.boot = False  # don't repeat on restart
+            send_shutdown = True
        if args.message:
            bmsg["service"] = "service"
            bmsg["msg"] = args.message
-        for c in connections:
-            await c.sendto(bmsg)
+        target = next((c for c in connections if c._transport), connections[0])
+        await target.sendto(bmsg)
        if args.message and not args.daemon:
            await asyncio.sleep(0.3)
            for c in connections:
@@ -1085,11 +1102,13 @@ async def _async_main(args, cfg: Dict[str, Any]) -> int:
        pass

    log.info("shutting down")
-    for conn in connections:
+    target = next((c for c in connections if c._transport), connections[0] if connections else None)
+    if target and send_shutdown:
        try:
-            await conn.sendto({"shutdown": 1, "acks": conn.ackcount})
+            await target.sendto({"shutdown": 1, "acks": target.ackcount})
        except Exception:
            pass
+    for conn in connections:
        conn.close()
    await asyncio.sleep(0.3)
    for plugin in plugins:
@@ -68,8 +68,7 @@ async def test_nagios_runner():
    print(f"   ✓ Collected {len(data)} data points")
    
    print(f"\n4. Results:")
-    print(f"   Overall Status: {data.get('overall_status')} (code: {data.get('overall_status_code')})")
-    print(f"   Plugins Executed: {data.get('plugin_count')}")
+    print(f"   Data points collected: {len(data)}")
    
    # Show individual plugin results
    print(f"\n5. Individual Plugin Results:")
Author	SHA1	Message	Date
andreas	54fbd8d73d	version 5.2.3 Release / release (push) Successful in 5s Details	2026-05-07 10:15:11 -04:00
andreas	7ab17e26e2	hbc/hbc_mini: log name and version at startup; ui: bump alert-metric font size Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-05-07 10:15:03 -04:00
andreas	28f5fa951c	ui: show metric name inline with hostname in alerts and notifications Alerts page: move metric name into the header row alongside hostname. Notifications: include metric name in title (hostname metric) and strip the metric prefix from the body so it contains only value/detail. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-05-07 06:26:27 -04:00
andreas	37f1c58969	docs: remove dead warning/critical keys from ping_monitor config example These fields were never read by the plugin; thresholds are configured server-side. Also document the -b flag in README. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-05-07 06:12:15 -04:00
andreas	f006077a71	send shutdown msg only if we sent a boot msg. Don't send eithe when restarting.	2026-05-06 11:57:43 -04:00
andreas	d9fc8d632f	send shutdown msg only if we sent a boot msg. Don't send eithe when restarting.	2026-05-06 11:54:09 -04:00
andreas	f640574e4f	version 5.2.2 Release / release (push) Successful in 5s Details	2026-05-06 09:57:43 -04:00
andreas	9a19424279	fix: retry connection on network error instead of permanently dropping it error_received() no longer sets _dead=True; it just closes the transport so the existing retry loop in heartbeat_sender (hbc) and sendto (hbc_mini) reopens the connection on the next interval. This allows hbc to recover when it starts before network connectivity is established. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-05-06 09:57:32 -04:00
andreas	ca8ba84e65	fix: silence aiohttp.access log and strip plugin prefix in alerts UI - main: disable aiohttp.access propagation unless --debug is active - alerts.html: strip plugin-name prefix from metric_path display (nagios_runner.check_disk_root_status_code → check_disk_root_status_code) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-05-06 07:39:55 -04:00
andreas	f3d08d1c9e	version 5.2.1 Release / release (push) Successful in 5s Details	2026-05-06 07:07:01 -04:00
andreas	1e4263b793	fix: threshold and logging improvements - threshold: fix crash when display is None (_format_display now falls back to default format string instead of calling None.format()) - threshold: shorten notification messages by stripping plugin-name prefix from metric_path (cpu_percent instead of cpu_monitor.cpu_percent) - main: demote aiohttp.access log records from INFO to DEBUG - udp: replace debug print with proper logger.info for new host sign-on Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-05-06 07:06:56 -04:00
andreas	e931acb9f5	version 5.2.0 Release / release (push) Successful in 5s Details	2026-05-05 13:47:46 -04:00
andreas	018409e71d	docs: correct README inaccuracies found during code audit - Add ping_monitor to built-in plugins list - Update cpu_monitor (uptime) and memory_monitor (ZFS ARC) descriptions - Replace "aggregated status" bullet with accurate per-check reporting note - Fix RTT hysteresis default: 0.1 → 0.02 - Fix client YAML config: remove non-existent server:/port: keys, use hb_port: - Fix nagios_runner commands format: plain strings → {name:, command:} dicts - Fix Supported Metrics: exit_code → actual <name>_status_code/<name>_status/<name>_output fields Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-05-05 13:45:43 -04:00
andreas	1824f637b4	fix: always show THRESHOLD_DEFAULTS in Settings threshold config Seed threshold_configs["default"] from THRESHOLD_DEFAULTS at the start of _parse_config() so the Settings page displays built-in defaults regardless of whether the server config uses the multi-config format, the legacy thresholds: format, or has no threshold config at all. _parse_multi_config() overwrites the seed with the fully-merged effective defaults when a threshold_configs section is present. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-05-05 13:02:28 -04:00
andreas	a534c06b26	feat: nagios operator for direct exit-code severity mapping Add ComparisonOperator.NAGIOS ("nagios") that maps Nagios exit codes directly to alert levels (0=OK 1=WARNING 2=CRITICAL 3=UNKNOWN) without requiring numeric warning/critical thresholds. Hysteresis is bypassed for discrete codes. Display template defaults to "{check_name}: {output}". _format_display() handles None threshold_value gracefully. Add nagios_runner.status_code as a built-in default threshold config so nagios checks alert out of the box. Also: fix alerts.html scrolling (override html,body), make hostname a link to /plugins#<hostname>, remove overall_status/overall_status_code/plugin_count from nagios_runner and hbc_mini, replace with computed worst-status in plugins.html via nagiosWorstStatus() helper. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-05-05 12:26:56 -04:00
andreas	d7b5c97a4e	version 5.1.21 Release / release (push) Successful in 6s Details	2026-05-05 11:05:48 -04:00
andreas	ae447ac4a6	feat: nagios_runner improvements and alerts page fixes - nagios_runner: remove overall_status/overall_status_code/plugin_count fields; each command still reports its own <name>_status and <name>_status_code - threshold: expose {output} and {status} aliases in display templates for nagios_runner generic matches (mapped from <check_name>_output/status) - alerts.html: fix scrolling by overriding html,body height/overflow (style.css sets both); make hostname a link to /plugins/<hostname> Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-05-05 11:05:45 -04:00
andreas	d44ce3d124	version 5.1.20 Release / release (push) Successful in 6s Details	2026-05-05 10:48:24 -04:00
andreas	b1985d0eb2	feat: generic threshold matching for nagios_runner with {check_name} display support _find_threshold() now returns the stripped prefix ("check_name") alongside the ThresholdConfig, enabling a single generic entry (e.g. nagios_runner.status_code) to cover all per-command metrics (check_disk_root_status_code, check_load_status_code, …). The prefix is threaded through to _format_display() as {check_name}, with {metric_name} also available in display templates. purge_stale_alerts() updated to use generic matching so it does not incorrectly drop alerts on generic-matched metrics. README updated with Display Format Templates and Generic Threshold Matching sections. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-05-05 10:48:17 -04:00
andreas	de778f680f	fix: reduce default hysteresis 10%→2%; show recovery threshold in alerts UI The 10% default hysteresis created an unreasonably wide recovery band: a 95% threshold would only clear once the value dropped below 85.5%, causing alerts to linger long after the metric was well below the trigger level. Change default hysteresis to 2% across all threshold parsers (plugin metrics, partitions, RTT). For a 95% threshold, recovery is now at 93.1% instead of 85.5%. Add AlertState.hysteresis field (set on every check, cleared on OK) and expose recovery_threshold in to_dict() so the Alerts dashboard can display "recovers < 93.1" alongside the trigger threshold, making the hysteresis band visible to the user. Pickle backward-compatible via __setstate__. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-05-04 14:47:50 -04:00
andreas	d7b368c7c6	version 5.1.19 Release / release (push) Successful in 5s Details	2026-05-04 12:10:01 -04:00
andreas	e790663f9f	feat: exclude ZFS ARC from memory_percent; add uptime_seconds to cpu_monitor memory_monitor / hbc_mini: ZFS ARC is reclaimable but not reflected in MemAvailable by the Linux kernel (not in SReclaimable). Read ARC size from /proc/spl/kstat/zfs/arcstats and add it to available memory before computing memory_percent and memory_used. No-op on systems without ZFS. cpu_monitor: report uptime_seconds via psutil.boot_time() (full client) and /proc/uptime (hbc_mini). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-05-04 12:09:58 -04:00
andreas	475319e248	fix: send boot/shutdown on first open connection, not blindly first in list Replace break-after-first-iteration with next(c for c in connections if c.transport) so the message goes to the first connection that actually has an open transport. Falls back to connections[0] if none are open yet (sendto will attempt reopen), avoiding silent message loss when the leading connection is still connecting. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-05-04 09:59:30 -04:00
andreas	ca5ef384a8	version 5.1.18 Release / release (push) Successful in 5s Details	2026-05-04 09:13:18 -04:00
andreas	c93dbdc0f4	fix: settings thresholds show correct per-config metrics; misc hbc fixes Settings page: pass threshold_checker to http.start so the Threshold Configurations section has data. Use threshold_checker's already-parsed ThresholdConfig objects instead of re-parsing the raw nested YAML. Named (non-default) configs now display only their explicit overrides via threshold_raw_configs, not the full merged set with defaults. hbc/hbc_mini: send boot and shutdown messages on first connection only to avoid duplicate packets when multiple servers are configured. Replace print("Daemonizing...") with logging.info so output goes to syslog in daemon mode. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-05-04 09:12:39 -04:00
andreas	3a546a1e5c	feat: fetch-based Update/Delete buttons with toast notification on Host Overview Replace href navigation with fetch() so the server response is captured and displayed in a slide-up toast at the bottom of the page. Delete also removes the host card from the DOM on success without a page reload. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-05-04 08:16:54 -04:00
andreas	74c89d098c	version 5.1.17 Release / release (push) Successful in 5s Details	2026-05-04 08:04:01 -04:00
andreas	3301dbfe34	feat: owner Update/Delete buttons on Host Overview; purge stale alerts on reload Host Overview (plugins.html): show Update and Delete buttons in the host-right zone when the logged-in user is the host owner (or admin / unauthenticated mode). Buttons link to /u?h=<host> and /d?h=<host> with stopPropagation so they don't toggle the accordion; Delete prompts for confirmation first. ThresholdChecker.purge_stale_alerts(): removes alert states whose metric_path has no matching threshold in the current config. Called after startup pickle restore and after every SIGHUP config reload so alerts orphaned by upgrades or config changes do not persist indefinitely. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-05-04 08:03:46 -04:00
andreas	d00d903e7d	fix: make Alerts page scrollable Override the global style.css body height/overflow that locks all pages to the viewport height (a remnant of the old drawer-menu layout). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-05-04 13:33:08 +02:00
andreas	babb5d61aa	docs: update README with changes since `917d6a4` - ZFS monitor plugin (zfs_monitor) added to plugin list and features - nagios_runner: async execution, stderr capture, signal handling, path validation - Threshold alerting: de-escalation suppression, short-duration suppression, ping_monitor thresholds - Per-host watch flag and role-filtered dashboards - HTTP API & Web UI: hostname links in Live View, Host Overview with ZFS renderer, alert pie chart in nav bar, Settings threshold viewer - hbc connection retry: indefinite retry for IPv4; IPv6 dropped after 3 early startup failures - hbc daemon mode: logs routed to syslog after daemonizing - hbc_mini: noted zfs_monitor and IPv6 early-fail protection not available Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-05-04 12:46:35 +02:00
andreas	11d1c718b3	feat: retry AsyncConnection.open() indefinitely; drop IPv6 only on early startup failure IPv4 connections are retried forever in heartbeat_sender if open() fails, so a temporary network outage does not terminate the sender. IPv6 connections that have never opened successfully are dropped after IPV6_EARLY_FAIL_LIMIT (3) consecutive failures so that a network without IPv6 support does not keep a dead sender running. At startup all resolved connections are added to the list regardless of whether the initial open() succeeds; the heartbeat_sender loop handles the first real connection attempt. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-05-04 12:29:35 +02:00
Andreas Wrede	a99b6b54c7	feat: add alert pie chart to nav bar Show a colour-coded pie chart (red=critical, yellow=warning, green=ok) to the left of the clock in the nav bar. Backed by a new GET /api/0/alert_summary endpoint that counts hosts per alert level for the current user's visible hosts. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-05-03 13:45:15 -04:00
Andreas Wrede	8da3d550eb	version 5.1.16 Release / release (push) Successful in 5s Details	2026-05-03 06:08:14 -04:00
Andreas Wrede	a76d0fc840	feat: generic ping_monitor thresholds; round RTT to nearest ms - threshold.py: add _find_threshold() with suffix fallback so thresholds like ping_monitor.rtt_avg match ping_monitor.8_8_8_8_rtt_avg etc.; each pinged host keeps its own alert state - hbdclass.py: format RTT as integer ms (round()) - live.html: JS RTT display rounded to nearest ms (Math.round) Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>	2026-05-03 06:08:11 -04:00