Compare commits

..

17 Commits

Author SHA1 Message Date
Andreas Wrede 46f8c32c0b version 5.1.13
Release / release (push) Successful in 5s
2026-05-02 12:43:06 -04:00
Andreas Wrede 691f62aa69 feat: host-level watch flag suppresses notifications; filter dashboard/overview by owner/manager; add ZFS monitor plugin
- watch: true (default) per host; watch: false suppresses all notifications
  for that host in udp.py and threshold.py
- Live Dashboard and Host Overview now show only hosts where the logged-in
  user is owner or manager (admins see all); WebSocket broadcasts filtered
  per-connection by the same rule
- Add hbd/client/plugins/zfs_monitor.py: collects per-pool health, capacity,
  fragmentation, dedup ratio, and cumulative I/O ops/bandwidth via zpool(8)

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
2026-05-02 12:42:35 -04:00
Andreas Wrede cffc9805f9 fix: mask api_password and access_token in settings page; add List to threshold imports
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
2026-05-02 11:51:55 -04:00
Andreas Wrede 917d6a401b feat: composable threshold_config list for per-host threshold layering
threshold_config in the hosts section now accepts a list of named
configs applied left-to-right on top of the defaults, so focused
override profiles can be mixed without duplication. Single-string
and legacy host_threshold_mapping forms are unchanged.

- Add threshold_raw_configs to store per-config overrides separately
- Normalise threshold_config to list on parse (string or list)
- get_thresholds_for_host folds the list over the default base
- Update README and docs/THRESHOLD_ALERTING.md with examples

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
2026-05-02 10:35:23 -04:00
Andreas Wrede 2bd3a9beb6 feat: restart on SIGHUP in hbc and hbc_mini
Sets dorestart and triggers a clean shutdown; os.execv re-execs
the process with the original arguments after cleanup.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
2026-05-02 10:06:26 -04:00
Andreas Wrede 5523c60866 version 5.1.12
Release / release (push) Successful in 5s
2026-05-02 08:56:04 -04:00
Andreas Wrede ab37ac7194 undo last
Release / release (push) Failing after 5s
2026-05-02 08:51:12 -04:00
Andreas Wrede f811a19d80 Merge branch 'master' of git.wrede.ca:andreas/heartbeat 2026-05-02 08:50:40 -04:00
Andreas Wrede 6239825f43 allow manual release workflow 2026-05-02 08:50:37 -04:00
Andreas Wrede b56245bb23 Specify tag for workflow 2026-05-02 08:46:12 -04:00
Andreas Wrede 331c4e804d allow manual release workflow 2026-05-02 08:36:33 -04:00
Andreas Wrede 9fd945a481 fix install under docker 2026-05-02 08:32:14 -04:00
Andreas Wrede 26df08eeff version 5.1.11
Release / release (push) Failing after 5s
2026-05-02 07:55:27 -04:00
Andreas Wrede 5819dd6b25 cleanup install script 2026-05-02 07:55:18 -04:00
Andreas Wrede 6fb67f8615 version 5.1.10
Release / release (push) Successful in 5s
2026-05-01 13:50:15 -04:00
Andreas Wrede e70ae6f176 fix: change version in hbc_mini as well 2026-05-01 13:50:04 -04:00
Andreas Wrede a77f6d380c fix: install script should not copy over itself 2026-05-01 12:48:29 -04:00
16 changed files with 596 additions and 193 deletions
+35
View File
@@ -267,6 +267,41 @@ All plugin metrics can be thresholded:
- **Network**: errors_total, dropped packets, connection counts
- **Nagios**: exit_code mapping (0=OK, 1=WARNING, 2=CRITICAL)
### Per-Host Threshold Profiles
Named threshold configurations let different hosts use different limits. A host's `threshold_config` can be a single name or a **list** — lists are applied left-to-right so profiles compose without duplication:
```yaml
threshold_configs:
default:
thresholds:
cpu_monitor:
cpu_percent: {warning: 80, critical: 90}
memory_monitor:
memory_percent: {warning: 85, critical: 95}
tight_cpu: # override CPU limits only
thresholds:
cpu_monitor:
cpu_percent: {warning: 60, critical: 75}
db_disk: # add a database partition check
thresholds:
disk_monitor:
partitions:
/var/lib/postgresql:
percent: {warning: 75, critical: 88}
hosts:
web-01:
threshold_config: default # single profile
db-01:
threshold_config: [tight_cpu, db_disk] # layered: CPU override + extra disk check
```
Each named config's overrides are applied in order on top of the defaults. Metrics not mentioned in a profile are inherited unchanged.
See [docs/THRESHOLD_ALERTING.md](docs/THRESHOLD_ALERTING.md) for comprehensive documentation including best practices, troubleshooting, and advanced configuration.
---
+183 -47
View File
@@ -814,34 +814,32 @@ Planned features:
## Multi-Threshold Configuration
**New in version 2.0**: Support for multiple named threshold configurations with per-host mapping.
Support for multiple named threshold configurations with per-host mapping and composable layering.
### Overview
The multi-threshold feature allows you to:
- Define multiple sets of threshold configurations
- Map different hosts to different threshold sets
- Define multiple named threshold configurations
- Assign one or more configurations to each host
- Compose configurations by layering — each named config's overrides are applied in order on top of the defaults
- Use different sensitivity levels for different environments
- Maintain a default configuration for unmapped hosts
### Configuration Structure
Named configurations are defined under `threshold_configs`. Each host selects which ones to use via `threshold_config` in the `hosts` section (a string for a single config, or a list to layer multiple):
```yaml
# Optional: Set the default configuration name (defaults to "default")
# Optional: set the default configuration name (defaults to "default")
default_threshold_config: "default"
# Define multiple named threshold configurations
threshold_configs:
# Configuration name 1
default:
thresholds:
# Standard threshold definitions
cpu_monitor:
cpu_percent:
warning: 80.0
critical: 90.0
# Configuration name 2
high_sensitivity:
thresholds:
cpu_monitor:
@@ -849,7 +847,6 @@ threshold_configs:
warning: 60.0
critical: 75.0
# Configuration name 3
low_sensitivity:
thresholds:
cpu_monitor:
@@ -857,14 +854,77 @@ threshold_configs:
warning: 90.0
critical: 95.0
# Map specific hosts to specific configurations
host_threshold_mapping:
prod-web-01: high_sensitivity
prod-web-02: high_sensitivity
dev-server-01: low_sensitivity
# Unmapped hosts use default_threshold_config
hosts:
prod-web-01:
threshold_config: high_sensitivity # single config
dev-server-01:
threshold_config: low_sensitivity
# Hosts with no threshold_config use default_threshold_config
```
### Composable Configurations (list form)
`threshold_config` can be a list. Configs are applied **left to right**: the defaults are the base, then each named config's overrides are layered on top. Later entries in the list win on any metric they define.
```yaml
threshold_configs:
default:
thresholds:
cpu_monitor:
cpu_percent: {warning: 80, critical: 90}
memory_monitor:
memory_percent: {warning: 85, critical: 95}
disk_monitor:
partitions:
/:
percent: {warning: 80, critical: 90}
# Tighter CPU limits for busy servers
high_cpu_load:
thresholds:
cpu_monitor:
cpu_percent: {warning: 60, critical: 75}
# Tighter disk limits for data-heavy servers
busy_disk:
thresholds:
disk_monitor:
partitions:
/:
percent: {warning: 70, critical: 85}
hosts:
# Gets default thresholds only
web-01:
threshold_config: default
# Gets tighter CPU limits, default memory and disk
build-server:
threshold_config: high_cpu_load
# Layers both: tighter CPU AND tighter disk, default memory
db-01:
threshold_config: [high_cpu_load, busy_disk]
# Three layers: busy_disk overrides high_cpu_load if they conflict
storage-01:
threshold_config: [default, high_cpu_load, busy_disk]
```
**How layering works:**
Starting from the `default` thresholds:
| Layer | Applied config | Effect |
|-------|---------------|--------|
| Base | `default` | all default thresholds |
| +1 | `high_cpu_load` | cpu_percent overridden to 60/75 |
| +2 | `busy_disk` | disk percent overridden to 70/85; cpu_percent stays at 60/75 |
Each named config only overrides the metrics it explicitly defines. Metrics not mentioned in a config inherit from the layers beneath.
### Use Cases
#### 1. Environment-Based Thresholds
@@ -887,11 +947,15 @@ threshold_configs:
warning: 90.0 # More relaxed for dev
critical: 98.0
host_threshold_mapping:
prod-web-01: production
prod-web-02: production
dev-web-01: development
dev-web-02: development
hosts:
prod-web-01:
threshold_config: production
prod-web-02:
threshold_config: production
dev-web-01:
threshold_config: development
dev-web-02:
threshold_config: development
```
#### 2. Server Role-Based Thresholds
@@ -914,7 +978,7 @@ threshold_configs:
warning: 70.0
critical: 85.0
memory_monitor:
percent:
memory_percent:
warning: 90.0 # Databases can use high memory
critical: 97.0
disk_monitor:
@@ -927,17 +991,23 @@ threshold_configs:
cache:
thresholds:
memory_monitor:
percent:
memory_percent:
warning: 95.0 # Redis/Memcached can use very high memory
critical: 99.0
host_threshold_mapping:
web-01: webserver
web-02: webserver
db-01: database
db-02: database
redis-01: cache
memcached-01: cache
hosts:
web-01:
threshold_config: webserver
web-02:
threshold_config: webserver
db-01:
threshold_config: database
db-02:
threshold_config: database
redis-01:
threshold_config: cache
memcached-01:
threshold_config: cache
```
#### 3. Sensitivity Levels
@@ -952,7 +1022,7 @@ threshold_configs:
partitions:
/:
percent:
warning: 70.0 # Very sensitive
warning: 70.0
critical: 80.0
hysteresis: 0.15
@@ -976,12 +1046,69 @@ threshold_configs:
critical: 98.0
hysteresis: 0.05
host_threshold_mapping:
payment-gateway: critical
auth-server: critical
web-01: standard
web-02: standard
test-server: relaxed
hosts:
payment-gateway:
threshold_config: critical
auth-server:
threshold_config: critical
web-01:
threshold_config: standard
web-02:
threshold_config: standard
test-server:
threshold_config: relaxed
```
#### 4. Composable Profiles
Build host-specific thresholds by combining small, focused configs:
```yaml
threshold_configs:
# Baseline — everything at default levels
default:
thresholds:
cpu_monitor:
cpu_percent: {warning: 80, critical: 90}
memory_monitor:
memory_percent: {warning: 85, critical: 95}
# Overlay: tighter CPU only
tight_cpu:
thresholds:
cpu_monitor:
cpu_percent: {warning: 60, critical: 75}
# Overlay: tighter memory only
tight_memory:
thresholds:
memory_monitor:
memory_percent: {warning: 70, critical: 85}
# Overlay: extra disk partition for database servers
db_disk:
thresholds:
disk_monitor:
partitions:
/var/lib/postgresql:
percent: {warning: 75, critical: 88}
hosts:
# Plain web server
web-01:
threshold_config: default
# Build server: tight CPU, default memory and disk
build-01:
threshold_config: tight_cpu
# Database: tight CPU + tight memory + extra disk partition
db-01:
threshold_config: [tight_cpu, tight_memory, db_disk]
# Replica database: tight memory + extra disk, normal CPU
db-02:
threshold_config: [tight_memory, db_disk]
```
### Backward Compatibility
@@ -1012,16 +1139,25 @@ threshold_configs:
### Configuration Priority
1. **Host-specific mapping**: If host is in `host_threshold_mapping`, use that config
2. **Default config**: Use `default_threshold_config`
3. **First alphabetically**: If default not found, use first config alphabetically
4. **Legacy fallback**: If `threshold_configs` not present, use `thresholds`
1. **Host `threshold_config` (list)**: Layer each named config's overrides left-to-right on top of the defaults
2. **Host `threshold_config` (string)**: Use that single named config directly
3. **`host_threshold_mapping`** (legacy): Same as above, string only
4. **`default_threshold_config`**: Used for hosts with no mapping
5. **First alphabetically**: If the default config is not found, use the first config alphabetically
6. **Legacy `thresholds` section**: Used when `threshold_configs` is absent entirely
### Example: Complete Multi-Threshold Setup
### Backward Compatibility
See `hbd/config_multi_threshold_example.yaml` for a complete example with:
- 4 named configurations (default, high_sensitivity, low_sensitivity, database)
- Host-to-config mappings for production, development, and test systems
- Specialized database server thresholds
- Custom display messages with plugin data
The legacy `host_threshold_mapping` top-level key and the flat `thresholds` section are still fully supported:
```yaml
# Still works — equivalent to hosts: {prod-web-01: {threshold_config: high_sensitivity}}
host_threshold_mapping:
prod-web-01: high_sensitivity
# Still works — equivalent to threshold_configs: {default: {thresholds: ...}}
thresholds:
cpu_monitor:
cpu_percent: {warning: 80, critical: 90}
```
+1 -1
View File
@@ -14,4 +14,4 @@ Install options:
"""
__all__ = ["__version__"]
__version__ = "5.1.9"
__version__ = "5.1.13"
+7
View File
@@ -525,6 +525,13 @@ async def async_main(args, config):
for sig in (signal.SIGTERM, signal.SIGINT):
loop.add_signal_handler(sig, stop)
def _sighup():
global dorestart
dorestart = True
stop()
loop.add_signal_handler(signal.SIGHUP, _sighup)
# Start async tasks
# Heartbeat senders (one per connection)
for conn in connections:
+130
View File
@@ -0,0 +1,130 @@
"""
ZFS pool monitoring plugin for Heartbeat.
Collects per-pool health, capacity, and cumulative I/O statistics via zpool(8).
"""
import asyncio
import logging
import shutil
from typing import Any, Dict, List, Optional
from hbd.client.plugin import MonitorPlugin
logger = logging.getLogger(__name__)
def _int(s: str) -> Optional[int]:
try:
return int(s.strip().rstrip("KMGTkBkmgt%x"))
except (ValueError, AttributeError):
return None
def _float(s: str) -> Optional[float]:
try:
return float(s.strip().rstrip("%x"))
except (ValueError, AttributeError):
return None
class ZFSMonitorPlugin(MonitorPlugin):
"""Monitor ZFS pool health, capacity, and I/O statistics.
Collects per pool:
- health: ONLINE, DEGRADED, FAULTED, etc.
- size / alloc / free: total, allocated and free bytes
- capacity: percentage used (0-100)
- frag: fragmentation percentage
- dedup: deduplication ratio
- read_ops / write_ops: cumulative I/O operations since last boot/clear
- read_bw / write_bw: cumulative bytes transferred since last boot/clear
Configuration:
interval: collection interval in seconds (default: 300)
pools: list of pool names to monitor (default: all)
"""
name = "zfs_monitor"
description = "ZFS pool health, capacity, and I/O statistics"
interval = 300
def __init__(self, config: Optional[Dict[str, Any]] = None):
super().__init__(config)
self.interval = self.config.get("interval", 300)
self._pools_filter: Optional[List[str]] = self.config.get("pools", None)
async def initialize(self) -> bool:
if not shutil.which("zpool"):
self.skip_reason = "zpool not found"
return False
logger.info("ZFS monitor initialized (interval: %ds)", self.interval)
return True
async def _run(self, *args: str) -> List[str]:
"""Run a command and return its stdout lines, or [] on error."""
try:
proc = await asyncio.create_subprocess_exec(
*args,
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.DEVNULL,
)
stdout, _ = await asyncio.wait_for(proc.communicate(), timeout=15)
return stdout.decode(errors="replace").splitlines()
except (FileNotFoundError, asyncio.TimeoutError) as exc:
logger.warning("zfs_monitor: %s: %s", args[0], exc)
return []
async def _zpool_list(self) -> Dict[str, Dict]:
"""Return per-pool health and capacity from `zpool list`."""
lines = await self._run(
"zpool", "list", "-H", "-p",
"-o", "name,health,size,alloc,free,cap,frag,dedup",
)
pools: Dict[str, Dict] = {}
for line in lines:
parts = line.split("\t")
if len(parts) < 8:
continue
name = parts[0].strip()
if self._pools_filter and name not in self._pools_filter:
continue
pools[name] = {
"health": parts[1].strip(),
"size": _int(parts[2]),
"alloc": _int(parts[3]),
"free": _int(parts[4]),
"capacity": _float(parts[5]),
"frag": _float(parts[6]),
"dedup": _float(parts[7]),
}
return pools
async def _zpool_iostat(self) -> Dict[str, Dict]:
"""Return per-pool cumulative I/O counters from `zpool iostat`."""
lines = await self._run("zpool", "iostat", "-H", "-p")
io: Dict[str, Dict] = {}
for line in lines:
parts = line.split("\t")
if len(parts) < 7:
continue
name = parts[0].strip()
if not name or name.startswith(" "):
continue
io[name] = {
"read_ops": _int(parts[3]),
"write_ops": _int(parts[4]),
"read_bw": _int(parts[5]),
"write_bw": _int(parts[6]),
}
return io
async def _collect_metrics(self) -> Dict[str, Any]:
pools, io = await asyncio.gather(self._zpool_list(), self._zpool_iostat())
for name, stats in io.items():
if name in pools:
pools[name].update(stats)
return {"pools": pools}
plugin = ZFSMonitorPlugin
+1 -1
View File
@@ -225,7 +225,7 @@ def get_watchhosts(config):
hosts_config = config.get("hosts", {})
if isinstance(hosts_config, dict):
for host_name, host_attrs in hosts_config.items():
if isinstance(host_attrs, dict) and host_attrs.get("watch", False):
if isinstance(host_attrs, dict) and host_attrs.get("watch", True):
watchhosts.append(host_name)
return watchhosts
+2 -1
View File
@@ -286,7 +286,7 @@ class Host:
Host.hosts[name] = self
self.num = num
self.dyn = False
self.watched = False
self.watched = True
self.upcount = 0
self.interval = 0
self.doesack = -1
@@ -304,6 +304,7 @@ class Host:
def statedict(self):
d = {}
d["raw_name"] = self.name
d["name"] = self.name
if self.dyn:
d["name"] += "*"
+4 -2
View File
@@ -258,7 +258,9 @@ async def start(
extra_scripts=extra_scripts,
hbd_version=hbd_version,
hosts=[
hbdclass.Host.hosts[h].stateinfo() for h in sorted(hbdclass.Host.hosts)
hbdclass.Host.hosts[h].stateinfo()
for h in sorted(hbdclass.Host.hosts)
if _can_operate_host(current_user, hbdclass.Host.hosts[h])
],
messages=data.msgs[-30:],
current_user=current_user.to_dict() if current_user else None,
@@ -510,7 +512,7 @@ async def start(
hosts_with_plugins = []
for hostname in sorted(hbdclass.Host.hosts.keys()):
host = hbdclass.Host.hosts[hostname]
if not _can_view_host(current_user, host):
if not _can_operate_host(current_user, host):
continue
if host.plugin_data:
hosts_with_plugins.append({
+2 -2
View File
@@ -24,7 +24,7 @@ sensitive bool True when the raw value must never be shown
# Credential field names that should always be masked.
_SECRET_KEYS = frozenset({
"password", "token", "user_key", "api_key", "secret",
"smtp_password", "smtp_user",
"smtp_password", "smtp_user", "api_password", "access_token",
})
_CHANNEL_TYPE_LABELS = {
@@ -188,7 +188,7 @@ def get_settings_sections(config: dict) -> list:
continue
hosts_list.append({
"name": hname,
"watch": bool(hcfg.get("watch", False)),
"watch": bool(hcfg.get("watch", True)),
"dyndns": bool(hcfg.get("dyndns", False)),
"owner": hcfg.get("owner", ""),
"managers": hcfg.get("managers", []),
+84 -43
View File
@@ -9,10 +9,11 @@ This module provides a flexible threshold checking system that:
- Supports multiple comparison operators
"""
import asyncio
import logging
import time
from enum import Enum
from typing import Dict, Any, Optional, Tuple, Callable
from typing import Dict, List, Any, Optional, Tuple, Callable
from . import notify as notify_mod
from .config import THRESHOLD_DEFAULTS
@@ -328,14 +329,17 @@ class ThresholdChecker:
renotify_interval: Seconds between repeat notifications (default: 1 hour)
journal: Optional MessageJournal instance for logging threshold events
"""
# Named threshold configurations: {config_name: {metric_path: ThresholdConfig}}
# Named threshold configurations (pre-merged: defaults + overrides): {config_name: {metric_path: ThresholdConfig}}
self.threshold_configs = {}
# Raw overrides only for each named config (no defaults baked in): {config_name: {metric_path: ThresholdConfig}}
self.threshold_raw_configs: Dict[str, Dict[str, ThresholdConfig]] = {}
# Single threshold set for backward compatibility: {metric_path: ThresholdConfig}
self.thresholds = {}
# Host to config name mapping: {host_name: config_name}
self.host_config_mapping = {}
# Host to ordered list of config names: {host_name: [config_name, ...]}
self.host_config_mapping: Dict[str, List[str]] = {}
# Default config name to use when no mapping exists
self.default_config = "default"
@@ -372,6 +376,7 @@ class ThresholdChecker:
# Clear old configuration
self.threshold_configs.clear()
self.threshold_raw_configs.clear()
self.thresholds.clear()
self.host_config_mapping.clear()
self.grace_seconds = float(config.get("grace", 2))
@@ -424,9 +429,10 @@ class ThresholdChecker:
self._parse_plugin_thresholds(plugin_name, plugin_thresholds, target_dict=effective_defaults)
self.threshold_configs["default"] = dict(effective_defaults)
self.threshold_raw_configs["default"] = {}
logger.info("Registered 'default' threshold config with %d metrics", len(effective_defaults))
# Parse each named configuration, seeding it with effective_defaults first
# Parse each named configuration
for config_name, config_data in threshold_configs.items():
if config_name == "default":
continue # already handled above
@@ -440,33 +446,41 @@ class ThresholdChecker:
continue
logger.info("Parsing threshold configuration: %s", config_name)
self.threshold_configs[config_name] = dict(effective_defaults)
# Raw overrides only (used for multi-config layering)
raw_overrides: Dict[str, ThresholdConfig] = {}
thresholds_config = config_data["thresholds"]
for plugin_name, plugin_thresholds in thresholds_config.items():
if not isinstance(plugin_thresholds, dict):
continue
if isinstance(plugin_thresholds, dict):
self._parse_plugin_thresholds(plugin_name, plugin_thresholds, target_dict=raw_overrides)
self.threshold_raw_configs[config_name] = raw_overrides
self._parse_plugin_thresholds(
plugin_name,
plugin_thresholds,
target_dict=self.threshold_configs[config_name]
)
# Pre-merged version (defaults + overrides) for single-config fast path
self.threshold_configs[config_name] = dict(effective_defaults)
self.threshold_configs[config_name].update(raw_overrides)
# Parse host to config mapping from two possible sources
# 1. New format: hosts section with threshold_config attribute
# Parse host config list mapping from two possible sources
def _normalise(value) -> List[str]:
"""Accept a string or list; always return a list."""
if isinstance(value, list):
return [str(v) for v in value]
return [str(value)]
# 1. hosts section with threshold_config attribute (string or list)
if "hosts" in config:
hosts_config = config["hosts"]
if isinstance(hosts_config, dict):
for host_name, host_attrs in hosts_config.items():
if isinstance(host_attrs, dict) and "threshold_config" in host_attrs:
self.host_config_mapping[host_name] = host_attrs["threshold_config"]
self.host_config_mapping[host_name] = _normalise(host_attrs["threshold_config"])
# 2. Legacy format: host_threshold_mapping section (for backward compatibility)
# 2. Legacy host_threshold_mapping section (string values only)
if "host_threshold_mapping" in config:
legacy_mapping = config.get("host_threshold_mapping", {})
if isinstance(legacy_mapping, dict):
self.host_config_mapping.update(legacy_mapping)
for host_name, value in legacy_mapping.items():
self.host_config_mapping[host_name] = _normalise(value)
# Set default config (first one alphabetically or explicitly set)
self.default_config = config.get("default_threshold_config", "default")
@@ -664,7 +678,10 @@ class ThresholdChecker:
)
def get_thresholds_for_host(self, host_name: str) -> Dict[str, ThresholdConfig]:
"""Get the appropriate threshold configuration for a host.
"""Get the effective threshold configuration for a host.
When threshold_config is a list, configs are applied left-to-right on top
of the default thresholds so earlier entries can be overridden by later ones.
Args:
host_name: Name of the host
@@ -676,23 +693,40 @@ class ThresholdChecker:
if self.thresholds and not self.threshold_configs:
return self.thresholds
# Multi-config mode: look up host-specific configuration
if self.threshold_configs:
config_name = self.host_config_mapping.get(host_name, self.default_config)
if not self.threshold_configs:
return {}
if config_name in self.threshold_configs:
return self.threshold_configs[config_name]
else:
config_names = self.host_config_mapping.get(host_name)
# No host-specific mapping → return pre-merged default
if not config_names:
return self.threshold_configs.get(self.default_config, {})
# Single config → fast path using pre-merged copy
if len(config_names) == 1:
name = config_names[0]
if name in self.threshold_configs:
return self.threshold_configs[name]
logger.warning(
"Threshold config '%s' not found for host '%s', using default '%s'",
name, host_name, self.default_config,
)
return self.threshold_configs.get(self.default_config, {})
# Multiple configs → start from defaults, layer raw overrides in order
result = dict(self.threshold_configs.get(self.default_config, {}))
for name in config_names:
if name == self.default_config:
continue # defaults already the base
raw = self.threshold_raw_configs.get(name)
if raw is None:
logger.warning(
"Threshold config '%s' not found for host '%s', using default '%s'",
config_name,
host_name,
self.default_config
"Threshold config '%s' not found for host '%s', skipping",
name, host_name,
)
return self.threshold_configs.get(self.default_config, {})
# No thresholds configured
return {}
else:
result.update(raw)
return result
def check_value(
self,
@@ -987,6 +1021,11 @@ class ThresholdChecker:
value: Any,
):
"""Send notification and log to journal/eventlog."""
from . import hbdclass
host = hbdclass.Host.hosts.get(host_name)
if host is not None and not host.watched:
eventlog(host_name, lvl, message, service="threshold")
return
asyncio.get_event_loop().create_task(notify_mod.send_notification(
host_name,
notify_mod.Notification(
@@ -999,7 +1038,6 @@ class ThresholdChecker:
# Log to journal
if self.journal is not None:
try:
import asyncio
loop = asyncio.get_event_loop()
loop.create_task(self.journal.log_threshold_event(
host_name=host_name,
@@ -1191,17 +1229,20 @@ class ThresholdChecker:
else:
message = f"REMINDER ({alert_state.level.name}): {host_name} - {metric_path} = {value} (ongoing for {int(now - alert_state.since)}s)"
asyncio.get_event_loop().create_task(notify_mod.send_notification(
host_name,
notify_mod.Notification(
title=f"[REMINDER/{alert_state.level.name}] {host_name}",
body=message,
level=alert_state.level.name,
),
))
from . import hbdclass
host = hbdclass.Host.hosts.get(host_name)
if host is None or host.watched:
asyncio.get_event_loop().create_task(notify_mod.send_notification(
host_name,
notify_mod.Notification(
title=f"[REMINDER/{alert_state.level.name}] {host_name}",
body=message,
level=alert_state.level.name,
),
))
logger.info("Re-notification sent: %s", message)
alert_state.last_notification = now
alert_state.notification_count += 1
logger.info("Re-notification sent: %s", message)
def get_active_alerts(self, alert_states: Dict[str, AlertState]) -> list:
"""
+25 -20
View File
@@ -211,10 +211,11 @@ def _make_timer_callbacks(uname, host, ctx):
connection.newstate(connection.__class__.OVERDUE, now, cfg.get("grace", 2))
msg = f"{connection.afam} overdue"
eventlog(uname, "CRITICAL", msg)
asyncio.create_task(notify_mod.send_notification(
uname,
notify_mod.Notification(title=f"[CRITICAL] {uname}", body=msg, level="CRITICAL"),
))
if host.watched:
asyncio.create_task(notify_mod.send_notification(
uname,
notify_mod.Notification(title=f"[CRITICAL] {uname}", body=msg, level="CRITICAL"),
))
# Track in alert_states so the Alerts Dashboard shows this
_set_connectivity_alert(host, connection.afam, "CRITICAL")
if threshold_checker:
@@ -407,10 +408,11 @@ def handle_datagram(msg: dict, addr, transport, ctx: dict):
if res:
eventlog(uname, "WARNING", res)
asyncio.create_task(notify_mod.send_notification(
uname,
notify_mod.Notification(title=f"[WARNING] {uname}", body=res, level="WARNING"),
))
if host.watched:
asyncio.create_task(notify_mod.send_notification(
uname,
notify_mod.Notification(title=f"[WARNING] {uname}", body=res, level="WARNING"),
))
interval = int(msg.get("interval", 0) or 0)
shutdown = msg.get("shutdown", 0)
@@ -420,10 +422,11 @@ def handle_datagram(msg: dict, addr, transport, ctx: dict):
if boot:
eventlog(uname, "INFO", "booted")
asyncio.create_task(notify_mod.send_notification(
uname,
notify_mod.Notification(title=f"[INFO] {uname}", body=f"{host.name} booted", level="INFO"),
))
if host.watched:
asyncio.create_task(notify_mod.send_notification(
uname,
notify_mod.Notification(title=f"[INFO] {uname}", body=f"{host.name} booted", level="INFO"),
))
if message:
eventlog(uname, "INFO", "msg: %s" % message, service=service)
@@ -440,10 +443,11 @@ def handle_datagram(msg: dict, addr, transport, ctx: dict):
else:
m = "%s back after being %s for %s" % (conn.afam, lasts, dur(d))
eventlog(uname, "RECOVER", m)
asyncio.create_task(notify_mod.send_notification(
uname,
notify_mod.Notification(title=f"[RECOVER] {uname}", body=m, level="RECOVER"),
))
if host.watched:
asyncio.create_task(notify_mod.send_notification(
uname,
notify_mod.Notification(title=f"[RECOVER] {uname}", body=m, level="RECOVER"),
))
if boot or newh:
host.upcount = host.doesack
@@ -453,10 +457,11 @@ def handle_datagram(msg: dict, addr, transport, ctx: dict):
if shutdown:
m = "%s shutdown" % conn.afam
eventlog(uname, "INFO", m)
asyncio.create_task(notify_mod.send_notification(
uname,
notify_mod.Notification(title=f"[INFO] {uname}", body=m, level="INFO"),
))
if host.watched:
asyncio.create_task(notify_mod.send_notification(
uname,
notify_mod.Notification(title=f"[INFO] {uname}", body=m, level="INFO"),
))
conn.newstate(hbdcls.Connection.DOWN, now)
_set_connectivity_alert(host, conn.afam, "CRITICAL")
+53 -10
View File
@@ -13,7 +13,8 @@ from . import data
logger = logging.getLogger(__name__)
_connections: set = set()
# Map of WebSocket → User object (or None when auth is disabled)
_connections: dict = {}
_loop: Optional[asyncio.AbstractEventLoop] = None
_get_hosts: Optional[Callable[[], Iterable]] = None
_verbose: bool = False
@@ -34,23 +35,53 @@ def setup(
_verbose = verbose
def _user_can_see_host(user, host_name: str) -> bool:
"""Return True if *user* may see updates for *host_name* (manager or higher)."""
from . import hbdclass, users as users_mod
if user is None or not users_mod.users_enabled():
return True
if user.admin:
return True
host = hbdclass.Host.hosts.get(host_name)
if host is None:
return False
return host.is_manager(user.username)
def _get_token(request) -> str:
"""Extract session token from request (mirrors logic in http.py)."""
auth = request.headers.get("Authorization", "")
if auth.startswith("Bearer "):
return auth[7:].strip()
token = request.headers.get("X-Auth-Token", "")
if token:
return token
return request.cookies.get("hbd_session", "")
async def handler(request):
"""aiohttp WebSocket upgrade handler — register as GET /ws."""
from aiohttp import web
from . import users as users_mod
ws = web.WebSocketResponse()
await ws.prepare(request)
_connections.add(ws)
token = _get_token(request)
user = users_mod.get_session_user(token) if token else None
_connections[ws] = user
remote = request.remote
logger.info("WebSocket connected from %s", remote)
try:
# Send current host state to the new client
# Send current host state, filtered to hosts this user may see
if _get_hosts:
try:
for h in list(_get_hosts()):
await ws.send_str(json.dumps({"type": "host", "data": h}))
host_name = h.get("raw_name") or h.get("name", "")
if _user_can_see_host(user, host_name):
await ws.send_str(json.dumps({"type": "host", "data": h}))
except Exception as e:
logger.error("Error sending initial hosts: %s", e)
@@ -74,7 +105,7 @@ async def handler(request):
except Exception as e:
logger.exception("WebSocket handler error from %s: %s", remote, e)
finally:
_connections.discard(ws)
_connections.pop(ws, None)
logger.info("WebSocket disconnected from %s", remote)
return ws
@@ -83,25 +114,37 @@ async def handler(request):
def broadcast(typ: str, payload) -> bool:
"""Thread-safe broadcast to all connected WebSocket clients.
For host and plugin updates, only sends to clients whose user has
manager-or-higher access to that host. Other message types are
broadcast to all clients.
Can be called from any thread; schedules sends on the event loop.
Returns False if the loop is not running yet.
"""
if not _loop:
return False
# Determine the host name for access-filtered message types
host_name: Optional[str] = None
if typ in ("host", "plugin"):
host_name = payload.get("raw_name") or payload.get("host") or payload.get("name")
jmsg = json.dumps({"type": typ, "data": payload})
async def _send_all():
dead = set()
for ws in list(_connections):
for ws, user in list(_connections.items()):
try:
if not ws.closed:
await ws.send_str(jmsg)
else:
if ws.closed:
dead.add(ws)
continue
if host_name is not None and not _user_can_see_host(user, host_name):
continue
await ws.send_str(jmsg)
except Exception:
dead.add(ws)
for ws in dead:
_connections.discard(ws)
_connections.pop(ws, None)
asyncio.run_coroutine_threadsafe(_send_all(), _loop)
return True
+1 -1
View File
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
[project]
name = "hbd"
version = "5.1.9"
version = "5.1.13"
description = "Heartbeat monitoring system — client (hbc) and server (hbd)"
readme = "README.md"
requires-python = ">=3.11"
+3 -1
View File
@@ -4,12 +4,14 @@ set -e
uv version --bump patch
VER=$(uv version --short)
sed -i".bak" "s/__version__ = \"[0-9.]*\"\(.*\)$/__version__ = \"$VER\"\1/" hbd/__init__.py
sed -i".bak" "s/__version__ = \"[0-9.]*\"\(.*\)$/__version__ = \"$VER\"\1/" scripts/hbc_mini.py
# commit pyproject.toml
git commit -m "version $VER" pyproject.toml hbd/__init__.py
git commit -m "version $VER" pyproject.toml hbd/__init__.py scripts/hbc_mini.py
git push
# tag version
git tag -a v$VER -m "Version $VER"
git push --tags
rm hbd/__init__.py.bak
rm scripts/hbc_mini.py.bak
+37 -46
View File
@@ -17,9 +17,9 @@ venv=""
[ "$2" = "HA" ] && on_ha=1
[ -z "$what" ] && what="client"
if [ -d /homeassistant ]; then
echo "HA, running \"docker exec homeassistant $0 $@\""
docker exec homeassistant $0 $@ HA
if [ -d /homeassistant ]; then # if running from HA command line
echo "HA, running \"docker exec homeassistant /config/bin/hb_install.sh $@\""
docker exec homeassistant /config/bin/hb_install.sh $@ HA
rc=$?
if [ $rc -ne 0 ]; then
echo "Failed to install heartbeat in HA, please check the logs for more details"
@@ -28,8 +28,9 @@ if [ -d /homeassistant ]; then
exit 0
fi
if [ $on_ha -eq 1 ]; then
echo "Installing under docker on Home Assistant OS, using /config/bin for executables and /config/venvs for virtual environments "
if [ $on_ha -eq 1 ] || [ -r /.dockerenv ] && [ -d /config/bin ]; then
# Installing under docker on Home Assistant OS, using /config/bin for executables and /config/venvs for virtual environments
echo "Home Assistant OS detected, installing under docker"
where="/config/bin"
venv="/config/venvs"
else
@@ -52,23 +53,26 @@ else
venv="$HOME/venvs"
fi
fi
echo "Installing heartbeat $what"
echo "Installing $what to $where"
if [ ! -z "$venv" ]; then
echo "Using virtual environment at $venv/hbd"
fi
if [ "$venv" != "" ] && [ ! -d $venv/hbd ]; then
set +e
python3 -m pip --version > /dev/null 2>&1
rc=$?
set -e
arg=""
if [ $rc -ne 0 ]; then
have_pip=$(python3 -c "import pip" 2>/dev/null &> /dev/null && echo "Installed" || echo "Not Installed")
if [ "$have_pip" = "Not Installed" ]; then
# some systems do not have pip installed by default, so we need to fetch get-pip.py and install pip
echo "pip is not installed, fetching get-pip.py and installing pip"
arg="--without-pip"
fi
mkdir -p $venv
have_venv=$(python3 -c "import venv" &> /dev/null && echo "Installed" || echo "Not Installed")
have_venv=$(python3 -c "import venv" 2>/dev/null &> /dev/null && echo "Installed" || echo "Not Installed")
if [ "$have_venv" = "Not Installed" ]; then
if [ "$have_pip" = "Not Installed" ]; then
echo "python has no venv, and no pip to install virtualenv, cannot continue"
exit 1
fi
echo "python venv module not found, installing virtualenv"
python3 -m pip install --user virtualenv
python3 -m virtualenv $venv/hbd --system-site-packages $arg
@@ -82,43 +86,30 @@ if [ "$venv" != "" ] && [ ! -d $venv/hbd ]; then
deactivate
fi
if [ -z "$venv" ]; then
echo "Installing heartbeat $what globally"
else
echo "Installing heartbeat $what in virtual environment $venv/hbd"
if [ ! -z "$venv" ]; then
. $venv/hbd/bin/activate
fi
python3 -mpip install --upgrade --index-url https://git.wrede.ca/api/packages/andreas/pypi/simple/ --extra-index-url https://pypi.org/simple hbd[$what]
if [ "$what" = "mini" ]; then
curl -s -o $where/hbc_mini https://git.wrede.ca/andreas/heartbeat/raw/branch/master/scripts/hbc_mini.py
chmod +x $where/hbc_mini
else
python3 -mpip install --upgrade --index-url https://git.wrede.ca/api/packages/andreas/pypi/simple/ --extra-index-url https://pypi.org/simple hbd[$what]
fi
if [ "$what" = "server" ]; then
rm -f $where/hbd
ln -sf $(which hbd) $where/hbd
echo "hbd installed, you can run it with \"$where/hbd\" or \"hbd\" if $where is in your PATH"
elif [ "$what" = "client" ]; then
hbc_path=$(which hbc)
if [ -z "$hbc_path" ]; then
echo "hbc not found in PATH, installation failed"
exit 1
fi
if [ "$hbc_path" != "$where/hbc" ]; then
if [ ! -z "$venv" ]; then
echo "linking executables to $where"
if [ "$what" = "server" ]; then
rm -f $where/hbd
ln -sf $(which hbd) $where/hbd
elif [ "$what" = "client" ]; then
rm -f $where/hbc
ln -sf $(which hbc) $where/hbc
fi
if [ "$0" != "$where/hb_install.sh" ]; then
cp "$0" $where/hb_install.sh
chmod +x $where/hb_install.sh
fi
if [ $on_ha -eq 1 ]; then
echo "restarting hbc "
job=$(grep run_hbc configuration.yaml | sed 's/run_hbc://')
$job
else
echo "hbc installed, you can run it with \"$where/hbc\" or \"hbc\" if $where is in your PATH"
fi
elif [ "$what" = "mini" ]; then
hbc_path=$(which hbc_mini)
if [ "$hbc_path" != "$where/hbc_mini" ]; then
ln -sf $(which hbc) $where/hbc_mini
fi
echo "hbc mini installed, you can run it with \"$where/hbc_mini\" or \"hbc_mini\" if $where is in your PATH"
rm -f $where/hb_install.sh
ln -sf $(which hb_install.sh) $where/hb_install.sh
fi
echo "Installation complete. To upgrade, run the following:"
echo " $where/hb_install.sh $what"
echo "To install on another machine, run the following obtain the install script and run it:"
echo "from https://git.wrede.ca/andreas/heartbeat/raw/branch/master/scripts/hb_install.sh"
echo "and then run sh hb_install.sh [mini|client]"
+12 -2
View File
@@ -40,6 +40,9 @@ from logging.handlers import SysLogHandler
from pathlib import Path
from typing import Any, Dict, List, Optional, Tuple
# updated by scripts/bumpminor.sh
__version__ = "5.1.13"
# ---------------------------------------------------------------------------
# Protocol (mirrors hbd/common/proto.py)
# ---------------------------------------------------------------------------
@@ -233,7 +236,7 @@ class OSInfoPlugin(InfoPlugin):
"machine": platform.machine(),
"architecture": platform.architecture()[0],
"python_version": platform.python_version(),
"hbc_version": "5.1.8",
"hbc_version": __version__,
"hbc_type": "mini",
}
if platform.system() == "Linux":
@@ -875,7 +878,7 @@ async def _handle_update(conn: AsyncConnection):
log.info("running installer: %s", installer)
try:
proc = await asyncio.create_subprocess_exec(
installer, "miniclient",
installer, "mini",
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.STDOUT,
)
@@ -1063,6 +1066,13 @@ async def _async_main(args, cfg: Dict[str, Any]) -> int:
for sig in (signal.SIGTERM, signal.SIGINT):
loop.add_signal_handler(sig, _stop)
def _sighup():
global _dorestart
_dorestart = True
_stop()
loop.add_signal_handler(signal.SIGHUP, _sighup)
for conn in connections:
_active_tasks.append(asyncio.create_task(_heartbeat_sender(conn, interval)))