Compare commits

...

81 Commits

Author SHA1 Message Date
Andreas Wrede f50acca509 version 5.1.14
Release / release (push) Successful in 5s
2026-05-02 13:21:40 -04:00
Andreas Wrede 72fc82b91f feat: add ZFS pool renderer to Host Overview
Add renderZfsTables() to plugins.html with health/capacity/frag/dedup
table and cumulative I/O table; colour-code health and capacity thresholds;
add zfs_monitor to plugin_order and summary/render dispatch.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
2026-05-02 13:21:28 -04:00
Andreas Wrede 46f8c32c0b version 5.1.13
Release / release (push) Successful in 5s
2026-05-02 12:43:06 -04:00
Andreas Wrede 691f62aa69 feat: host-level watch flag suppresses notifications; filter dashboard/overview by owner/manager; add ZFS monitor plugin
- watch: true (default) per host; watch: false suppresses all notifications
  for that host in udp.py and threshold.py
- Live Dashboard and Host Overview now show only hosts where the logged-in
  user is owner or manager (admins see all); WebSocket broadcasts filtered
  per-connection by the same rule
- Add hbd/client/plugins/zfs_monitor.py: collects per-pool health, capacity,
  fragmentation, dedup ratio, and cumulative I/O ops/bandwidth via zpool(8)

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
2026-05-02 12:42:35 -04:00
Andreas Wrede cffc9805f9 fix: mask api_password and access_token in settings page; add List to threshold imports
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
2026-05-02 11:51:55 -04:00
Andreas Wrede 917d6a401b feat: composable threshold_config list for per-host threshold layering
threshold_config in the hosts section now accepts a list of named
configs applied left-to-right on top of the defaults, so focused
override profiles can be mixed without duplication. Single-string
and legacy host_threshold_mapping forms are unchanged.

- Add threshold_raw_configs to store per-config overrides separately
- Normalise threshold_config to list on parse (string or list)
- get_thresholds_for_host folds the list over the default base
- Update README and docs/THRESHOLD_ALERTING.md with examples

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
2026-05-02 10:35:23 -04:00
Andreas Wrede 2bd3a9beb6 feat: restart on SIGHUP in hbc and hbc_mini
Sets dorestart and triggers a clean shutdown; os.execv re-execs
the process with the original arguments after cleanup.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
2026-05-02 10:06:26 -04:00
Andreas Wrede 5523c60866 version 5.1.12
Release / release (push) Successful in 5s
2026-05-02 08:56:04 -04:00
Andreas Wrede ab37ac7194 undo last
Release / release (push) Failing after 5s
2026-05-02 08:51:12 -04:00
Andreas Wrede f811a19d80 Merge branch 'master' of git.wrede.ca:andreas/heartbeat 2026-05-02 08:50:40 -04:00
Andreas Wrede 6239825f43 allow manual release workflow 2026-05-02 08:50:37 -04:00
Andreas Wrede b56245bb23 Specify tag for workflow 2026-05-02 08:46:12 -04:00
Andreas Wrede 331c4e804d allow manual release workflow 2026-05-02 08:36:33 -04:00
Andreas Wrede 9fd945a481 fix install under docker 2026-05-02 08:32:14 -04:00
Andreas Wrede 26df08eeff version 5.1.11
Release / release (push) Failing after 5s
2026-05-02 07:55:27 -04:00
Andreas Wrede 5819dd6b25 cleanup install script 2026-05-02 07:55:18 -04:00
Andreas Wrede 6fb67f8615 version 5.1.10
Release / release (push) Successful in 5s
2026-05-01 13:50:15 -04:00
Andreas Wrede e70ae6f176 fix: change version in hbc_mini as well 2026-05-01 13:50:04 -04:00
Andreas Wrede a77f6d380c fix: install script should not copy over itself 2026-05-01 12:48:29 -04:00
Andreas Wrede 6aae2a1dab version 5.1.9
Release / release (push) Successful in 6s
2026-05-01 11:13:51 -04:00
Andreas Wrede 85ee0e1040 install hbc_mini via package or script 2026-05-01 11:13:33 -04:00
Andreas Wrede c4f09e9ced version 5.1.8
Release / release (push) Successful in 5s
- fix: matrix/sms_voipms notifications blocked the event loop on timeout;
  make send_notification async, dispatch all channel drivers as non-blocking
  tasks (asyncio.to_thread for sync drivers, asyncio.wait_for for async);
  update all call sites to fire-and-forget via create_task
- feat: add /about page with version, runtime, uptime counter, and repo link
- fix: hbc_mini plugin data format now matches full hbc client so Host
  Overview displays memory, disk, and network metrics correctly

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
2026-05-01 05:33:27 -04:00
Andreas Wrede 64710fd4cd tweak h1 margins 2026-05-01 04:51:11 -04:00
Andreas Wrede 1f5e7465a3 fix nav bar position 2026-05-01 04:32:04 -04:00
Andreas Wrede b290b21e23 track hbc type and version 2026-04-30 18:22:35 -04:00
Andreas Wrede 65c4267847 version 5.1.7
Release / release (push) Successful in 5s
2026-04-30 17:50:46 -04:00
Andreas Wrede 462a445235 feat: add hbc_mini single-file client; drop dead connections on protocol error
- scripts/hbc_mini.py: self-contained hbc with no external deps; uses
  /proc for CPU/memory/network on Linux, df for disk, JSON config
- hbc + hbc_mini: mark connection _dead and stop sending on protocol error
- README: document hbc_mini usage, config, and plugin availability
- pyproject.toml: include hbc_mini.py in script-files

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
2026-04-30 17:50:19 -04:00
Andreas Wrede 368e178f93 install the hb_install.sh script 2026-04-30 17:03:37 -04:00
Andreas Wrede 6905bf266a version 5.1.6
Release / release (push) Successful in 5s
2026-04-30 15:39:11 -04:00
Andreas Wrede b6dcce4f35 simplify eventlog usage, fix arguments 2026-04-30 15:38:46 -04:00
Andreas Wrede e6436fc236 version 5.1.5
Release / release (push) Successful in 5s
2026-04-30 13:55:21 -04:00
Andreas Wrede c5ce41762e feat: update hbc via hb_install.sh instead of code patching
Server now sends a bare UPD command; client runs hb_install.sh to
reinstall from the package registry, then restarts. hb_install.sh
also copies itself alongside hbc on client installs.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
2026-04-30 13:55:15 -04:00
Andreas Wrede 26ca0c095f install.sh --> hb_innstall.sh 2026-04-30 09:54:48 -04:00
Andreas Wrede 1eecd67594 update docu 2026-04-30 09:19:11 -04:00
Andreas Wrede caf3c2c0ac don't error exit on pip insttalled test 2026-04-30 09:16:22 -04:00
Andreas Wrede 9af4006097 version 5.1.4
Release / release (push) Successful in 6s
2026-04-30 08:12:15 -04:00
Andreas Wrede ddf7067d13 feat: redesign Plugin Metrics page as Host Overview
Replace pill-tab plugin view with an accordion layout that shows key
metrics (CPU%, MEM%, top disk%, net delta, nagios status) at a glance
in each host card header. Plugin sections expand as structured tables.

- Rename page to "Host Overview" (URL /plugins unchanged)
- Three-wave parallel data loading: glance plugins on host expand,
  on-demand fetch for filesystem_info and extras
- Per-plugin table renderers with inline percent bars and threshold
  colour coding
- Add escHtml() for XSS-safe rendering of all field values
- Remove stale planning docs (REFACTORING.md, hbd/Plan.md)

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
2026-04-30 08:12:07 -04:00
andreas 505353a8a8 Update CLAUDE. md 2026-04-29 21:20:28 -04:00
andreas 0402d33c71 Add CLAUDE. md 2026-04-29 21:18:21 -04:00
andreas 7d8ca5d8db version 5.1.3
Release / release (push) Successful in 4s
2026-04-25 16:52:56 +02:00
andreas 56037a036d fix: remove unused pytest import in test_nagios_runner 2026-04-25 16:39:56 +02:00
andreas 65ceb31d8d fix: use os.path.exists check for /dev/log instead of dead-code OSError catch 2026-04-25 16:36:00 +02:00
andreas 1c9b6c1ca9 fix: reconfigure logging to syslog after daemonize() instead of no-op basicConfig
After daemonize() redirects stderr to /dev/null, the existing StreamHandler
writes to /dev/null. logging.basicConfig() is a no-op when handlers are
already configured, so log messages are silently lost.

Replace the daemon block to:
1. Call daemonize() first
2. Explicitly remove existing handlers (pointing to /dev/null)
3. Add SysLogHandler pointing to /dev/log with fallback to UDP localhost:514
4. Log startup message to the new syslog handler

Removes redundant syslog.openlog() call which is no longer needed.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-25 16:29:54 +02:00
andreas d7e6b478e1 fix: use shlex.split() in nagios_runner path validation to handle quoted paths 2026-04-25 16:28:32 +02:00
andreas 535dbda47d feat: validate absolute command paths at nagios_runner init 2026-04-25 16:24:33 +02:00
andreas c9567dddae fix: remove stale shell config key from NagiosRunnerPlugin docstring 2026-04-25 16:23:03 +02:00
andreas b5963badd6 feat: async subprocess in nagios_runner with stderr capture and signal handling
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-25 16:18:09 +02:00
andreas a76a39b4a0 fix: remove redundant no-commands log lines; fix skip_reason docstring style
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-25 16:15:44 +02:00
andreas 94e1597978 feat: set skip_reason on nagios_runner when no commands configured
When NagiosRunnerPlugin has no commands configured, set skip_reason before
returning False from initialize(). This allows PluginLoader to log INFO
(not WARNING) when the plugin is skipped.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-25 16:13:03 +02:00
andreas c9c2ed772f fix: document skip_reason in Plugin docstring; remove unused import in test 2026-04-25 16:10:35 +02:00
andreas aeb78dcb8e feat: add skip_reason to Plugin; improve PluginLoader init messaging 2026-04-25 16:08:07 +02:00
andreas 77b337e4dd Add implementation plan for plugin error checking and daemon logging fixes
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-25 16:04:13 +02:00
andreas 293461f3f6 Add design spec for plugin error checking and daemon logging fixes
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-25 15:49:09 +02:00
andreas c70a4807dc version 5.1.2
Release / release (push) Successful in 6s
2026-04-25 07:25:06 +02:00
andreas 1a470e7cfa Fix plugin config lookup shadowed by CLIENT_DEFAULTS plugins key
CLIENT_DEFAULTS seeds "plugins": {} so raw_config.get("plugins", raw_config)
always returned the empty subdict instead of falling back to the full config.
Plugins configured at top-level (e.g. nagios_runner: ...) were therefore
never found, resulting in "No Nagios commands configured".

Now checks the plugins subdict first, then top-level keys, so both
config layouts work correctly.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-24 12:58:42 +02:00
andreas 990c658e65 Apply grace period to all threshold alerts before logging/notifying
Threshold alerts (plugin metrics, RTT) were firing immediately on the
first breach. Now every state transition to WARNING/CRITICAL starts a
grace-period timer (grace_seconds from the 'grace' config key). The
notification is deferred until the next heartbeat after grace_seconds
have elapsed. If the metric recovers within the grace window, both the
alert and the recovery are suppressed — no spurious pages for transient
spikes.

Two helper methods added to ThresholdChecker:
- _apply_grace: handles the state-change path (defer or suppress)
- _check_pending_or_renotify: handles the stable-alert path (fire
  deferred notification once grace expires, or fall through to reminders)

The overdue case is unchanged — on_overdue already fires only after
interval+grace seconds of silence, which is equivalent behaviour.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-24 12:00:40 +02:00
andreas b78d6ac0fe Fix RECOVER routing: use consistent level name and route via alerted channel
threshold.py was emitting level="RECOVERED" for metric recoveries, which
failed the is_recover check in send_notification (which only matched "RECOVER"),
bypassing _alerted_channels routing and the min_level bypass added in the
previous commit. Changed to "RECOVER" so all recovery paths are consistent.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-24 11:29:04 +02:00
andreas afd5060f59 Fix early reminder notifications and lost recovery notifications
- AlertState.update() now resets last_notification when the alert level
  changes, so a WARNING→CRITICAL escalation restarts the reminder interval
  rather than inheriting a nearly-expired timer.
- _dispatch_to_channel() bypasses min_level for RECOVER, so recovery
  notifications are delivered even after a server restart when
  _alerted_channels is empty and the fallback dispatch path is used.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-22 18:11:22 +02:00
andreas f61f7aebc2 Use python3 consistently 2026-04-19 09:49:30 +02:00
Andreas Wrede 5c382d2b8d One more nit 2026-04-13 09:31:35 -04:00
Andreas Wrede 35bba451f5 Various formating nits 2026-04-13 09:27:51 -04:00
Andreas Wrede 80edfba0c0 fix inconsistencies in page layout, add swiss clock 2026-04-13 08:45:50 -04:00
Andreas Wrede 6bc8de192e fix non-alerting of overdue hosts 2026-04-12 18:44:36 -04:00
Andreas Wrede 2d8166d04a unse python3 -mpip instead of plain pip 2026-04-12 18:44:11 -04:00
Andreas Wrede ab33d81b30 catch syntax wanring when parsing version string 2026-04-12 16:39:51 -04:00
Andreas Wrede 2c0328f36d update install.sh to handle missing venv module 2026-04-12 16:39:14 -04:00
Andreas Wrede fb8e27825d make install.sh work on systems withou pip 2026-04-12 14:16:44 -04:00
Andreas Wrede 1366c69cdc version 5.1.1
Release / release (push) Successful in 5s
2026-04-12 13:06:30 -04:00
Andreas Wrede d0c8c186f4 Fix typo 2026-04-12 13:04:17 -04:00
Andreas Wrede 19f7c8312e Mkae columns sortabel agian, check hbc version, provide modile html pages 2026-04-12 12:53:00 -04:00
Andreas Wrede 24b0e362fb provide cli function stop, restart and reload for hbd
Thought for 1s
2026-04-12 12:06:07 -04:00
Andreas Wrede 3a030548c0 Fix profile not updating 2026-04-12 11:57:12 -04:00
Andreas Wrede 094cb7ed9d Merge branch 'master' of git.wrede.ca:andreas/heartbeat 2026-04-12 11:23:28 -04:00
Andreas Wrede 0199ca4693 re-factor notifications, add sms and matrix as channels 2026-04-12 11:21:21 -04:00
Andreas Wrede 75344ebbbd re-factor notifications, add sms and matrix as channels 2026-04-12 11:04:00 -04:00
Andreas Wrede 7f049a4e26 accept websocket connection on http:.../ws 2026-04-12 06:44:32 -04:00
Andreas Wrede 6559f5462c Merge branch 'master' of git.wrede.ca:andreas/heartbeat 2026-04-12 06:34:28 -04:00
Andreas Wrede 6556d35f97 Merge branch 'master' of git.wrede.ca:andreas/heartbeat 2026-04-12 06:32:52 -04:00
Andreas Wrede dec96a0da6 Merge branch 'master' of git.wrede.ca:andreas/heartbeat 2026-04-11 16:40:02 -04:00
Andreas Wrede 8d3de01117 Update install script 2026-04-11 16:36:20 -04:00
Andreas Wrede 5bedf026b1 Update install script 2026-04-11 16:19:41 -04:00
44 changed files with 5843 additions and 2555 deletions
+4 -4
View File
@@ -24,11 +24,11 @@ jobs:
- name: Install build tools - name: Install build tools
run: | run: |
python -m pip install --upgrade pip python3 -m pip install --upgrade pip
pip install build twine python3 -m pip install build twine
- name: Build package - name: Build package
run: python -m build run: python3 -m build
- name: Extract version from tag - name: Extract version from tag
id: get_version id: get_version
@@ -39,7 +39,7 @@ jobs:
TWINE_USERNAME: ${{ secrets.PYPI_USERNAME }} TWINE_USERNAME: ${{ secrets.PYPI_USERNAME }}
TWINE_PASSWORD: ${{ secrets.PYPI_TOKEN }} TWINE_PASSWORD: ${{ secrets.PYPI_TOKEN }}
run: | run: |
python -m twine upload --repository-url https://git.wrede.ca/api/packages/andreas/pypi dist/* python3 -m twine upload --repository-url https://git.wrede.ca/api/packages/andreas/pypi dist/*
- name: Create release - name: Create release
uses: actions/gitea-release-action@v1 uses: actions/gitea-release-action@v1
+4
View File
@@ -0,0 +1,4 @@
1. Don't assume. Don't hide confusion. Surface tradeoffs.
2. Minimum code that solves the problem. Nothing speculative.
3. Touch only what you must. Clean up only your own mess.
4. Define success criteria. Loop until verified.
+98 -1
View File
@@ -267,6 +267,41 @@ All plugin metrics can be thresholded:
- **Network**: errors_total, dropped packets, connection counts - **Network**: errors_total, dropped packets, connection counts
- **Nagios**: exit_code mapping (0=OK, 1=WARNING, 2=CRITICAL) - **Nagios**: exit_code mapping (0=OK, 1=WARNING, 2=CRITICAL)
### Per-Host Threshold Profiles
Named threshold configurations let different hosts use different limits. A host's `threshold_config` can be a single name or a **list** — lists are applied left-to-right so profiles compose without duplication:
```yaml
threshold_configs:
default:
thresholds:
cpu_monitor:
cpu_percent: {warning: 80, critical: 90}
memory_monitor:
memory_percent: {warning: 85, critical: 95}
tight_cpu: # override CPU limits only
thresholds:
cpu_monitor:
cpu_percent: {warning: 60, critical: 75}
db_disk: # add a database partition check
thresholds:
disk_monitor:
partitions:
/var/lib/postgresql:
percent: {warning: 75, critical: 88}
hosts:
web-01:
threshold_config: default # single profile
db-01:
threshold_config: [tight_cpu, db_disk] # layered: CPU override + extra disk check
```
Each named config's overrides are applied in order on top of the defaults. Metrics not mentioned in a profile are inherited unchanged.
See [docs/THRESHOLD_ALERTING.md](docs/THRESHOLD_ALERTING.md) for comprehensive documentation including best practices, troubleshooting, and advanced configuration. See [docs/THRESHOLD_ALERTING.md](docs/THRESHOLD_ALERTING.md) for comprehensive documentation including best practices, troubleshooting, and advanced configuration.
--- ---
@@ -377,7 +412,7 @@ This project now declares its dependencies in `pyproject.toml`. Instead
of the old `requirements.txt` flow, install the package into a virtualenv of the old `requirements.txt` flow, install the package into a virtualenv
using `pip`: using `pip`:
See `scripts/install.sh` for a way to install. See `scripts/hb_install.sh` for a way to install.
Run the daemon (example): Run the daemon (example):
@@ -441,6 +476,68 @@ plugins:
All monitoring plugins default to 5-minute (300 second) intervals, but can be customized as needed. All monitoring plugins default to 5-minute (300 second) intervals, but can be customized as needed.
### hbc_mini — single-file client (no external dependencies)
`scripts/hbc_mini.py` is a self-contained version of the heartbeat client that requires only Python 3.8+ and no external packages. Copy it to any host and run it directly — no virtualenv, no `pip install`.
```bash
# Basic usage
python3 hbc_mini.py your-server.example.com
# Run as daemon
python3 hbc_mini.py -d your-server.example.com
# Send a boot message
python3 hbc_mini.py -b your-server.example.com
# Send a one-off message
python3 hbc_mini.py -m "maintenance starting" your-server.example.com
```
**Config:** `~/.hbc.json` (same keys as `~/.hbc.yaml`, JSON format). Example:
```json
{
"hb_port": 50003,
"interval": 30,
"plugins": {
"ping_monitor": {
"interval": 60,
"hosts": ["8.8.8.8", "192.168.1.1"]
},
"nagios_runner": {
"interval": 300,
"commands": [
{"name": "check_load", "command": "/usr/lib/nagios/plugins/check_load -w 5,4,3 -c 10,8,6"}
]
}
}
}
```
**Plugin availability:**
| Plugin | Platform | Data source |
|---|---|---|
| `os_info` | all | `platform` stdlib |
| `ping_monitor` | all | `ping` subprocess |
| `nagios_runner` | all (not Windows) | subprocess |
| `cpu_monitor` | Linux | `/proc/stat` |
| `memory_monitor` | Linux | `/proc/meminfo` |
| `disk_monitor` | Linux, macOS, BSD | `df -P` subprocess |
| `network_monitor` | Linux | `/proc/net/dev` |
**What is not available compared to the full `hbc`:**
- No YAML config (use JSON instead)
- No `filesystem_info` plugin
- `cpu_monitor` does not report per-core usage or CPU frequency (no psutil)
- Plugins cannot be loaded from external `.py` files — all plugins are compiled in
Everything else — heartbeat protocol, ACK/CMD/UPD handling, `hb_install.sh`-based self-update, daemonize, syslog — is identical to the full client.
---
## 🐞 Debugging in VS Code ## 🐞 Debugging in VS Code
This repository includes a ready-to-use `.vscode/launch.json` with configurations to run or attach the VS Code debugger to `hbd`. This repository includes a ready-to-use `.vscode/launch.json` with configurations to run or attach the VS Code debugger to `hbd`.
-234
View File
@@ -1,234 +0,0 @@
# HBD/HBC Separation Refactoring
## Overview
The heartbeat monitoring system has been refactored into a modular package structure with separate client and server components. This allows users to install only what they need and provides clear separation of concerns.
## New Package Structure
```
hbd/
├── __init__.py # Main package (minimal)
├── client/ # HBC - System monitoring client
│ ├── __init__.py
│ ├── main.py # Entry point (was hbc.py)
│ ├── config.py # Client-specific configuration
│ ├── plugin.py # Plugin framework
│ ├── threshold.py # Threshold checking
│ └── plugins/ # Monitoring plugins
│ ├── cpu_monitor.py
│ ├── disk_monitor.py
│ ├── memory_monitor.py
│ ├── network_monitor.py
│ ├── filesystem_info.py
│ ├── os_info.py
│ └── nagios_runner.py
├── server/ # HBD - Heartbeat daemon/server
│ ├── __init__.py
│ ├── main.py # Server runtime (was server.py)
│ ├── cli.py # Command-line interface
│ ├── config.py # Server-specific configuration
│ ├── http.py # HTTP/REST API
│ ├── ws.py # WebSocket server
│ ├── udp.py # UDP heartbeat listener
│ ├── dns.py # DNS update functionality
│ ├── notify.py # Notification handlers
│ ├── monitor.py # Host monitoring
│ ├── hbdclass.py # Host class definitions
│ ├── journal.py # Message journaling
│ ├── templates/ # Jinja2 web templates
│ └── static/ # Web UI assets
└── common/ # Shared utilities
├── __init__.py
├── proto.py # Protocol encoding/decoding
└── utils.py # Common utilities
## Configuration Files
### Client Configuration (hbd/client/config.py)
Client-specific defaults:
- `hb_port`: Port where hbd servers listen (default: 50003)
- `interval`: Heartbeat interval in seconds (default: 10)
- `plugins`: Per-plugin configuration
- `thresholds`: Threshold configuration for monitoring
### Server Configuration (hbd/server/config.py)
Server-specific defaults:
- `hb_port`: Port to listen for heartbeats (default: 50003)
- `hbd_port`: HTTP API port (default: 50004)
- `ws_port`: WebSocket port (default: 50005)
- `logfile`: Log file path
- `pushsrv`, `pushover_token`, etc.: Notification settings
- `watchhosts`, `dyndnshosts`: Host monitoring
- `smtpserver`, etc.: Email settings
- `journal_*`: Message journaling settings
## Installation Options
### Install Core Only (minimal, PyYAML only)
```bash
pip install hbd
```
### Install Client Only (for monitoring)
```bash
pip install hbd[client]
# Installs: PyYAML, psutil
```
### Install Server Only (for daemon)
```bash
pip install hbd[server]
# Installs: PyYAML, websockets, mattermostdriver, aiohttp, Jinja2
```
### Install Everything
```bash
pip install hbd[all]
# Installs all dependencies for both client and server
```
### Development Installation
```bash
pip install -e ".[dev]"
# Includes all dependencies plus testing/linting tools
```
## Command-Line Interfaces
### HBC (Client)
```bash
hbc [options] host1 [host2 ...]
# Entry point: hbd.client.main:main
# Location: hbd/client/main.py
```
### HBD (Server)
```bash
hbd [options]
# Entry point: hbd.server.cli:main
# Location: hbd/server/cli.py → hbd/server/main.py
```
## Import Changes
### Client Code
```python
# Old imports
from .config import load_config
from .proto import dicttos, stodict
from .plugin import PluginRegistry
# New imports
from .config import load_config # Still in client/
from ..common.proto import dicttos # Moved to common/
from .plugin import PluginRegistry # Still in client/
```
### Server Code
```python
# Old imports
from .config import load_config
from .proto import stodict
from .threshold import AlertLevel
# New imports
from .config import load_config # Server-specific config
from ..common.proto import stodict # Moved to common/
from ..client.threshold import AlertLevel # Client module
```
### Plugin Code
```python
# Old import
from hbd.plugin import MonitorPlugin
# New import
from hbd.client.plugin import MonitorPlugin
```
## Benefits
1. **Modular Installation**: Install only what you need
- Client-only systems don't need web server dependencies
- Server-only systems don't need psutil
2. **Clearer Architecture**: Explicit separation of concerns
- Client: System monitoring and data collection
- Server: Heartbeat reception, web UI, notifications
- Common: Shared protocol and utilities
3. **Independent Evolution**: Client and server can evolve separately
- Different release cycles possible
- Clear API boundaries via common/
4. **Smaller Footprint**: Reduced dependency installation
- Client: ~1 dependency (psutil)
- Server: ~4 dependencies (websockets, aiohttp, Jinja2, mattermostdriver)
## Migration Guide
### For Existing Installations
1. **Reinstall the package**:
```bash
pip install -e ".[all]" # For development
# or
pip install hbd[all] # For production
```
2. **Configuration files remain unchanged**:
- Both client and server read from `~/.hb.yaml`
- All existing config keys are supported in both configs
- Server has additional keys (journal, websocket, email, etc.)
- Client has minimal keys (interval, plugins, thresholds)
3. **Commands remain the same**:
- `hbc` command works identically
- `hbd` command works identically
### For New Deployments
1. **Client-only system** (monitoring host):
```bash
pip install hbd[client]
hbc server1.example.com server2.example.com
```
2. **Server-only system** (monitoring daemon):
```bash
pip install hbd[server]
hbd -c /etc/hbd.yaml -f
```
3. **Combined system** (dev/test):
```bash
pip install hbd[all]
```
## Testing
All imports and entry points have been tested and validated:
- ✅ Package imports work correctly
- ✅ `hbc` command entry point functional
- ✅ `hbd` command entry point functional
- ✅ Optional dependencies properly configured
- ✅ All internal imports updated
## Files Archived
The following files were renamed to avoid conflicts:
- `hbd/config.py` → `hbd/config.py.old` (split into client/server configs)
- `hbd/hbc_old.py` → `hbd/hbc_old.py.bak` (backup file)
## Next Steps
1. Test client functionality with a monitoring host
2. Test server functionality with web UI and notifications
3. Update documentation (README.md) with new structure
4. Consider publishing to PyPI with new structure
5. Update any deployment scripts/Dockerfiles to use optional dependencies
+40
View File
@@ -0,0 +1,40 @@
async def send_sms(hass, user, password, sender_did, call):
"""Send SMS message using multipart form-data like MMS."""
_LOGGER = logging.getLogger(__name__)
recipient = call.data.get("recipient")
message = call.data.get("message")
if not recipient or not message:
_LOGGER.error("Recipient or message missing.")
return
# Build form data dictionary
form_data = {
'api_username': str(user),
'api_password': str(password),
'did': str(sender_did),
'dst': str(recipient),
'message': str(message),
'method': 'sendSMS'
}
async with aiohttp.ClientSession() as session:
with aiohttp.MultipartWriter("form-data") as mp:
for key, value in form_data.items():
part = mp.append(value)
part.set_content_disposition('form-data', name=key)
_LOGGER.error("voipms_sms: sending SMS: %s", mp)
async with session.post(REST_ENDPOINT, data=mp) as response:
response_text = await response.text()
if response.status == 200:
response_json = json.loads(response_text)
if response_json['status'] == "success":
_LOGGER.info("voipms_sms: SMS sent successfully: %s", response_text)
else:
_LOGGER.error("voipms_sms: SMS not sent: %s", response_text)
else:
_LOGGER.error("voipms_sms: Failed to send SMS. Status: %s, Response: %s", response.status, response_text)
+235 -473
View File
@@ -2,532 +2,294 @@
## Overview ## Overview
The Heartbeat Monitoring System includes a flexible notification system that can send alerts through multiple channels including Email, Pushover, Signal, and Mattermost. The system supports centralized channel definitions with per-host routing, allowing fine-grained control over notification delivery. Notifications are dispatched to the **owner and managers** of a host, each via their own configured notification channels. Channel definitions are global; users reference them by name. No users configured → no notifications sent.
## Architecture ## Architecture
### Components ```
Alert event (udp.py / threshold.py)
└─ notify.send_notification(host_name, Notification)
├─ look up host.owner + host.managers
├─ for each user → user.notification_channels
└─ for each channel → _dispatch_to_channel (filtered by min_level)
```
1. **Notification Channels** (`notification_channels` in config) Every notification carries:
- Centralized definitions of notification providers - **title** — `[LEVEL] hostname` (e.g. `[CRITICAL] webserver01`)
- Each channel has a type and type-specific credentials - **body** — detail message (metric value, threshold, duration)
- Reusable across multiple hosts - **url** — link to the plugin metrics page (`{base_url}/plugins#{hostname}`)
- **level** — `RECOVER | WARNING | CRITICAL | INFO`
2. **Channel Dispatcher** (`hbd/server/notify.py`)
- `pushmsg_for_host(hostname, message)`: Main entry point for host-specific notifications
- `_dispatch_to_channel(channel_name, channel_config, message)`: Routes to specific provider
- Provider functions: `pushover()`, `pushsignal()`, `pushmattermost()`, `send_email()`
3. **Configuration Utilities** (`hbd/server/config.py`)
- `get_notification_channels_for_host(config, hostname)`: Retrieves channel names for a host
- `get_notification_channels_config(config, hostname)`: Retrieves full channel configurations
- `get_channel_config(config, channel_name)`: Gets configuration for a specific channel
4. **Integration Points**
- **Threshold alerts**: `threshold.py` calls `notify_mod.pushmsg_for_host()`
- **Heartbeat events**: `udp.py` calls `notify_mod.pushmsg_for_host()` for boot/shutdown/overdue
- **Custom alerts**: Any code can call `notify_mod.pushmsg_for_host(hostname, message)`
## Configuration ## Configuration
### Centralized Channel Definitions ### Base URL
Define notification channels once in your configuration file: Set `base_url` so notification links point to your hbd instance:
```yaml
base_url: https://hbd.example.com
```
### Global channel definitions
Define channels once; reference them by name from user configs:
```yaml ```yaml
notification_channels: notification_channels:
# Signal notifications
signal_ops:
type: signal
cli_path: /usr/local/bin/signal-cli
user: +1234567890 # Your Signal number
recipient: +1234567890 # Recipient number
signal_oncall: pushover_ops:
type: signal type: pushover
cli_path: /usr/local/bin/signal-cli token: your-app-token
user: +1234567890 user: your-user-key
recipient: +0987654321 # Different recipient min_level: WARNING # optional, default: WARNING
# Email notifications
email_ops: email_ops:
type: email type: email
recipients: recipients: [ops@example.com]
- ops@example.com sender: hbd@example.com
- alerts@example.com
sender: heartbeat@example.com
smtp_server: smtp.example.com smtp_server: smtp.example.com
smtp_port: 587 smtp_port: 587
smtp_user: heartbeat@example.com smtp_user: hbd@example.com
smtp_password: your-smtp-password smtp_password: secret
min_level: WARNING
email_devteam: matrix_oncall:
type: email type: matrix
recipients: [dev-alerts@example.com] homeserver: https://matrix.example.org
sender: heartbeat-dev@example.com access_token: syt_xxx
smtp_server: smtp.example.com room_id: "!abc:matrix.example.org"
smtp_port: 587 min_level: CRITICAL # only send critical alerts to this room
smtp_user: heartbeat-dev@example.com
smtp_password: your-smtp-password
# Pushover notifications sms_oncall:
pushover_urgent: type: sms_voipms
type: pushover api_user: me@example.com
token: your-pushover-app-token api_password: secret
user: your-pushover-user-key did: "5551234567" # your voip.ms DID number
dst: "5559876543" # destination number
min_level: CRITICAL
pushover_normal: signal_ops:
type: pushover
token: your-pushover-app-token
user: another-user-key
# Mattermost notifications
mattermost_devops:
type: mattermost
host: mattermost.example.com
token: your-webhook-token
channel: devops-alerts
username: heartbeat-bot
icon: https://example.com/heartbeat-icon.png
```
### Default Notification Channels
Specify default channels for hosts that don't have specific channel assignments:
```yaml
default_notification_channels:
- email_ops
- mattermost_devops
```
Hosts without `notification_channels` defined will use these defaults.
### Per-Host Channel Assignment
Assign specific channels to each host in the `hosts` section:
```yaml
hosts:
# Critical production web server - multiple channels for redundancy
prod-web-01:
threshold_config: high_sensitivity
watch: true
notification_channels:
- signal_oncall # Immediate mobile notification
- pushover_urgent # Secondary mobile notification
- email_ops # Email for record keeping
dyndns: false
# Database server - ops team notifications only
prod-db-01:
threshold_config: database
watch: true
notification_channels:
- signal_ops
- email_ops
dyndns: false
# Development server - email only, no urgent notifications
dev-server-01:
threshold_config: low_sensitivity
watch: false
notification_channels:
- email_devteam
dyndns: false
# Test server - uses default_notification_channels
test-server-01:
threshold_config: default
watch: false
dyndns: false
# No notification_channels specified = uses default_notification_channels
```
## Channel Types
### Email
Sends notifications via SMTP.
**Configuration fields:**
```yaml
type: email
recipients: [email1@example.com, email2@example.com] # Required: List of recipients
sender: heartbeat@example.com # Required: From address
smtp_server: smtp.example.com # Required: SMTP server hostname
smtp_port: 587 # Optional: Default 587
smtp_user: heartbeat@example.com # Optional: For authenticated SMTP
smtp_password: your-password # Optional: For authenticated SMTP
```
**Features:**
- Supports multiple recipients
- TLS/STARTTLS support on port 587
- Authenticated and unauthenticated SMTP
**Example:**
```yaml
notification_channels:
email_critical:
type: email
recipients: [admin@example.com, oncall@example.com]
sender: alerts@example.com
smtp_server: smtp.fastmail.com
smtp_port: 587
smtp_user: alerts@example.com
smtp_password: app-specific-password
```
### Pushover
Sends push notifications to mobile devices via Pushover API.
**Configuration fields:**
```yaml
type: pushover
token: your-application-token # Required: Your Pushover app token
user: your-user-key # Required: Recipient's user key
```
**Features:**
- Instant mobile push notifications
- Works on iOS and Android
- Supports delivery confirmations
**Setup:**
1. Create a Pushover account at https://pushover.net
2. Create an application to get your app token
3. Note your user key from your account dashboard
**Example:**
```yaml
notification_channels:
pushover_admin:
type: pushover
token: azGDORePK8gMaC0QOYAMyEEuzJnyUi
user: uQiRzpo4DXghDmr9QzzfQu27cmVRsG
```
### Signal
Sends notifications via Signal messenger using signal-cli.
**Configuration fields:**
```yaml
type: signal
cli_path: /usr/local/bin/signal-cli # Optional: Path to signal-cli binary
user: +1234567890 # Required: Your Signal phone number
recipient: +0987654321 # Required: Recipient phone number
```
**Prerequisites:**
1. Install signal-cli: https://github.com/AsamK/signal-cli
2. Register signal-cli with your phone number:
```bash
signal-cli -u +1234567890 register
signal-cli -u +1234567890 verify CODE
```
3. Ensure signal-cli is in PATH or specify full path in config
**Features:**
- End-to-end encrypted messaging
- Works without phone being online
- No API fees or rate limits
**Example:**
```yaml
notification_channels:
signal_admin:
type: signal type: signal
cli_path: /usr/local/bin/signal-cli cli_path: /usr/local/bin/signal-cli
user: +12025551234 user: +12025551234
recipient: +12025559999 recipient: +12025559999
```
### Mattermost mattermost_devops:
Sends notifications to Mattermost team chat via incoming webhooks.
**Configuration fields:**
```yaml
type: mattermost type: mattermost
host: mattermost.example.com # Required: Mattermost server hostname host: mattermost.example.com
token: your-webhook-token # Required: Incoming webhook token token: webhook-token
channel: channel-name # Required: Target channel name channel: devops-alerts
username: heartbeat-bot # Optional: Bot display name username: heartbeat-bot
icon: https://example.com/icon.png # Optional: Bot icon URL
``` ```
**Prerequisites:** ### Users with notification channels
1. Enable incoming webhooks in Mattermost
2. Create an incoming webhook for your team
3. Note the webhook token from the webhook URL
**Features:** Each user lists which global channels they receive notifications on:
- Team-wide visibility
- Rich formatting support
- Message threading
**Example:**
```yaml ```yaml
notification_channels: users:
mattermost_ops: alice:
type: mattermost full_name: Alice Smith
host: chat.example.com password: pbkdf2:sha256:...
token: abc123def456ghi789 admin: true
channel: infrastructure-alerts notification_channels: [pushover_ops, email_ops]
username: heartbeat-monitor
icon: https://example.com/heartbeat-icon.png bob:
full_name: Bob Jones
password: pbkdf2:sha256:...
notification_channels: [sms_oncall, matrix_oncall]
``` ```
## Notification Events ### Host access — owner and managers
The system sends notifications for various events: Notifications for a host go to its owner and all managers:
### Threshold Alerts
When monitored metrics exceed configured thresholds:
- **State changes**: OK → WARNING, WARNING → CRITICAL, CRITICAL → OK
- **Format**: `{LEVEL}: {hostname} - {metric_path} = {value} {threshold_info}`
- **Example**: `CRITICAL: prod-web-01 - cpu_monitor.cpu_percent = 95.2 (threshold: > 90.0)`
- **Re-notifications**: Periodic reminders for ongoing alerts (default: hourly)
### Heartbeat Events
Host lifecycle events:
- **Host boot**: `{hostname} booted`
- **Host shutdown**: `{hostname} {connection_type} shutdown`
- **Host recovery**: `{hostname} {connection_type} is back`
- **Connection issues**: `{hostname} {message}`
- **Host overdue**: `{hostname} {connection_type} overdue`
Only hosts with `watch: true` send heartbeat event notifications.
### Custom Alerts
Application code can send custom notifications:
```python
from hbd.server import notify as notify_mod
# Send to host-specific channels
notify_mod.pushmsg_for_host("prod-web-01", "Custom alert message")
# Send using global config
notify_mod.pushmsg_from_config("Global notification")
# Send to specific config
notify_mod.pushmsg(custom_config_dict, "Targeted notification")
```
## Design Principles
The notification system follows these core principles:
- **Centralization**: Define notification providers once, reference them by name
- **Flexibility**: Each host can use different channels for different notification needs
- **Redundancy**: Critical hosts can specify multiple channels for failover
- **Clarity**: Clean separation between channel definition and channel assignment
- **Type Safety**: Provider-specific validation at configuration time
## Best Practices
### Channel Organization
- **Create purpose-specific channels**: `email_ops`, `signal_oncall`, `pushover_urgent`
- **Separate by team/role**: `email_devteam`, `signal_dbateam`, `mattermost_security`
- **Use descriptive names**: Channel names appear in logs and debugging
### Redundancy
For critical hosts, use multiple notification channels:
```yaml ```yaml
hosts: hosts:
critical-db: webserver01:
notification_channels: owner: alice # receives all notifications for this host
- signal_oncall # Primary: Mobile alert managers: [bob] # also receives notifications
- pushover_urgent # Backup: Different mobile platform threshold_config: default
- email_ops # Tertiary: Email for record-keeping watch: true # bold in dashboard (cosmetic only)
dyndns: false
dbserver01:
owner: alice
managers: [bob]
threshold_config: database
dyndns: false
``` ```
### Notification Fatigue Prevention `watch: true` only affects display (bold name in the live dashboard). Notifications are now controlled entirely by owner/managers.
- **Use `watch: false`** for non-critical hosts ## Channel Types
- **Configure appropriate thresholds** to avoid false positives
- **Set different channels for different severities**
- **Use `default_notification_channels`** for baseline, add more for critical systems
### Security ### `min_level` filtering
- **Protect credentials**: Use file permissions to protect config files with passwords/tokens Every channel accepts an optional `min_level` field:
- **Rotate tokens**: Periodically rotate API tokens and passwords
- **Use app-specific passwords**: For email, use app-specific passwords instead of main account password
- **Separate accounts**: Consider separate notification accounts for different environments (prod vs dev)
### Testing | Value | Channels receive |
|---|---|
| `WARNING` (default) | WARNING, CRITICAL, RECOVER |
| `CRITICAL` | CRITICAL only (and RECOVER) |
Test notification channels before relying on them: `RECOVER` is always passed through — you don't want to miss a recovery.
### pushover
Sends push notifications via [Pushover](https://pushover.net). Includes title, body, and a clickable URL.
```yaml
type: pushover
token: your-app-token # Required: Pushover application token
user: your-user-key # Required: Recipient's user key
min_level: WARNING
```
### email
Sends via SMTP. Subject = title, body = message + URL on final line.
```yaml
type: email
recipients: [ops@example.com, oncall@example.com]
sender: hbd@example.com
smtp_server: smtp.example.com
smtp_port: 587 # 587 = STARTTLS (default), 465 = SSL
smtp_user: hbd@example.com
smtp_password: secret
min_level: WARNING
```
### matrix
Sends a formatted HTML message to a Matrix room via [matrix-nio](https://github.com/poljar/matrix-nio).
```yaml
type: matrix
homeserver: https://matrix.example.org
access_token: syt_xxx # Bot account access token
room_id: "!abc:matrix.example.org"
min_level: WARNING
```
**Setup:**
1. Create a bot Matrix account
2. Obtain its access token (Element → Settings → Help & About → Access Token)
3. Invite the bot to the target room and note the room ID
### sms_voipms
Sends SMS via the [voip.ms REST API](https://voip.ms/api/v1/rest.php). Message is truncated to 160 characters.
```yaml
type: sms_voipms
api_user: me@example.com # voip.ms account email
api_password: secret # voip.ms API password
did: "5551234567" # Your voip.ms DID (sending number)
dst: "5559876543" # Destination number
min_level: CRITICAL
```
### signal
Sends via [signal-cli](https://github.com/AsamK/signal-cli).
```yaml
type: signal
cli_path: /usr/local/bin/signal-cli
user: +12025551234 # Your registered Signal number
recipient: +12025559999 # Recipient number
min_level: WARNING
```
**Setup:**
```bash ```bash
# Test signal-cli directly signal-cli -u +12025551234 register
signal-cli -u +1234567890 send -m "Test message" +0987654321 signal-cli -u +12025551234 verify CODE
# Test SMTP
echo "Test" | mail -s "Test Subject" admin@example.com
# Test through heartbeat system (Python REPL)
from hbd.server import notify as notify_mod, config as config_mod
cfg = config_mod.load_config(".hb.yaml")
notify_mod.setup(cfg)
notify_mod.pushmsg_for_host("test-host", "Test notification")
``` ```
### mattermost
Sends via Mattermost incoming webhook. Message is formatted as Markdown.
```yaml
type: mattermost
host: mattermost.example.com
token: your-webhook-token
channel: devops-alerts
username: heartbeat-bot # Optional: display name
icon: https://…/icon.png # Optional: bot icon URL
min_level: WARNING
```
## Notification events
| Source | Level | Title example | Body example |
|---|---|---|---|
| Host overdue | CRITICAL | `[CRITICAL] webserver01` | `IPv4 overdue` |
| Host recover | RECOVER | `[RECOVER] webserver01` | `IPv4 back after being overdue for 5:23` |
| Host boot | INFO | `[INFO] webserver01` | `webserver01 booted` |
| Host shutdown | INFO | `[INFO] webserver01` | `IPv4 shutdown` |
| Threshold breach | WARNING/CRITICAL | `[CRITICAL] webserver01` | `cpu_percent = 95.2 (threshold: > 90.0)` |
| Threshold reminder | CRITICAL | `[REMINDER/CRITICAL] webserver01` | `REMINDER (CRITICAL): … ongoing for 3600s` |
| Connection issue | WARNING | `[WARNING] webserver01` | `new address detected …` |
Reminder notifications (re-notify) are sent only for CRITICAL level alerts.
## API reference
### `send_notification(host_name, notif) -> dict`
Main entry point. Dispatches to owner + managers.
```python
from hbd.server.notify import send_notification, Notification
send_notification(
"webserver01",
Notification(
title="[CRITICAL] webserver01",
body="cpu_percent = 95.2 (threshold: > 90.0)",
level="CRITICAL",
url="https://hbd.example.com/plugins#webserver01",
),
)
```
Returns `{channel_name: bool}` for each channel dispatched.
### `setup(cfg, loop=None)`
Called once at startup from `main.py`. Pass the running asyncio event loop so Matrix sends work correctly.
## Troubleshooting ## Troubleshooting
### Notifications Not Sending **No notifications sent:**
- Check that users are configured (`users:` section in yaml)
- Check that the host has an `owner` or `managers` set
- Check that users have `notification_channels` listed
- Check that the channel names in user config match keys under `notification_channels:`
1. **Check logs**: Look for "Failed to send notification" errors **min_level filtering too aggressive:**
2. **Verify host is watched**: Ensure `watch: true` in host definition - Default is `WARNING` — both WARNING and CRITICAL are sent
3. **Check channel configuration**: Verify credentials and settings - Set `min_level: WARNING` explicitly if you were expecting warnings but set CRITICAL
4. **Test channel directly**: Use command-line tools to test provider
5. **Check network**: Ensure server can reach notification endpoints
### Signal Issues **Matrix sends time out:**
- Verify the access token is valid and the bot is in the room
- `matrix-nio` must be installed: `pip install matrix-nio`
- **signal-cli not found**: Specify full path in `cli_path` **voip.ms SMS fails:**
- **Not registered**: Run `signal-cli -u +NUMBER register` and verify - Enable the API in your voip.ms account (Account → API)
- **Trust issues**: Run `signal-cli -u +NUMBER receive` to sync trust store - Verify the DID is SMS-capable in your voip.ms account
- **Recipient not found**: Ensure recipient is in your Signal contacts
### Email Issues **Signal not found:**
- Specify full `cli_path`
- Run `signal-cli -u +NUMBER receive` to sync trust store
- **Authentication failed**: Check SMTP username/password **Email authentication failed:**
- **TLS errors**: Verify SMTP port (587 for STARTTLS, 465 for SSL) - Use app-specific passwords for Gmail/Fastmail
- **Relay denied**: Ensure SMTP server allows relay from your IP - Verify port: 587 for STARTTLS, 465 for SSL
- **Timeout**: Check firewall rules for SMTP ports
### Pushover Issues **Pushover `400` errors:**
- Double-check `token` (app) and `user` (user key) — they are different values
- **Invalid token/user**: Verify token and user key from Pushover dashboard
- **API rate limits**: Pushover has monthly message limits on free tier
- **HTTP errors**: Check Pushover API status page
### Mattermost Issues
- **Webhook not found**: Verify webhook token and ensure webhook is enabled
- **Channel not found**: Check channel name spelling and permissions
- **Driver import error**: Install mattermostdriver: `pip install mattermostdriver`
## API Reference
### Main Functions
#### `pushmsg_for_host(hostname: str, msg: str, debug: int = 0) -> dict`
Send notification to host-specific channels.
**Parameters:**
- `hostname`: Name of the host (used to look up notification channels)
- `msg`: Message to send
- `debug`: Debug level (0=no debug, 1+=debug output)
**Returns:** Dictionary of results per channel: `{"signal_ops": True, "email_ops": False}`
**Example:**
```python
from hbd.server import notify as notify_mod
notify_mod.pushmsg_for_host("prod-web-01", "Server CPU at 95%")
```
**Behavior:**
1. Looks up notification channels configured for the host
2. If no host-specific channels, uses `default_notification_channels`
3. Dispatches to each channel in parallel
4. Returns dict of results keyed by channel name
5. Logs success/failure for each channel
## Examples
### Complete Configuration Example
```yaml
# Notification channel definitions
notification_channels:
signal_oncall:
type: signal
cli_path: /usr/local/bin/signal-cli
user: +12025551234
recipient: +12025555678
email_ops:
type: email
recipients: [ops@example.com, alerts@example.com]
sender: heartbeat@example.com
smtp_server: smtp.fastmail.com
smtp_port: 587
smtp_user: heartbeat@example.com
smtp_password: app-password-here
# Default channels
default_notification_channels: [email_ops]
# Host definitions with channel assignments
hosts:
prod-web-01:
threshold_config: high_sensitivity
watch: true
notification_channels: [signal_oncall, email_ops]
dyndns: false
dev-server-01:
threshold_config: low_sensitivity
watch: false
notification_channels: [email_ops]
dyndns: false
```
### Multiple Environments Example
```yaml
notification_channels:
# Production channels
signal_prod_oncall:
type: signal
user: +12025551234
recipient: +12025551111 # On-call phone
email_prod_ops:
type: email
recipients: [prod-ops@example.com]
sender: prod-heartbeat@example.com
smtp_server: smtp.example.com
# Staging channels
email_staging:
type: email
recipients: [staging-alerts@example.com]
sender: staging-heartbeat@example.com
smtp_server: smtp.example.com
# Development channels
mattermost_dev:
type: mattermost
host: chat.example.com
token: dev-webhook-token
channel: dev-alerts
hosts:
prod-api-01:
notification_channels: [signal_prod_oncall, email_prod_ops]
staging-api-01:
notification_channels: [email_staging]
dev-api-01:
notification_channels: [mattermost_dev]
```
+183 -47
View File
@@ -814,34 +814,32 @@ Planned features:
## Multi-Threshold Configuration ## Multi-Threshold Configuration
**New in version 2.0**: Support for multiple named threshold configurations with per-host mapping. Support for multiple named threshold configurations with per-host mapping and composable layering.
### Overview ### Overview
The multi-threshold feature allows you to: The multi-threshold feature allows you to:
- Define multiple sets of threshold configurations - Define multiple named threshold configurations
- Map different hosts to different threshold sets - Assign one or more configurations to each host
- Compose configurations by layering — each named config's overrides are applied in order on top of the defaults
- Use different sensitivity levels for different environments - Use different sensitivity levels for different environments
- Maintain a default configuration for unmapped hosts
### Configuration Structure ### Configuration Structure
Named configurations are defined under `threshold_configs`. Each host selects which ones to use via `threshold_config` in the `hosts` section (a string for a single config, or a list to layer multiple):
```yaml ```yaml
# Optional: Set the default configuration name (defaults to "default") # Optional: set the default configuration name (defaults to "default")
default_threshold_config: "default" default_threshold_config: "default"
# Define multiple named threshold configurations
threshold_configs: threshold_configs:
# Configuration name 1
default: default:
thresholds: thresholds:
# Standard threshold definitions
cpu_monitor: cpu_monitor:
cpu_percent: cpu_percent:
warning: 80.0 warning: 80.0
critical: 90.0 critical: 90.0
# Configuration name 2
high_sensitivity: high_sensitivity:
thresholds: thresholds:
cpu_monitor: cpu_monitor:
@@ -849,7 +847,6 @@ threshold_configs:
warning: 60.0 warning: 60.0
critical: 75.0 critical: 75.0
# Configuration name 3
low_sensitivity: low_sensitivity:
thresholds: thresholds:
cpu_monitor: cpu_monitor:
@@ -857,14 +854,77 @@ threshold_configs:
warning: 90.0 warning: 90.0
critical: 95.0 critical: 95.0
# Map specific hosts to specific configurations hosts:
host_threshold_mapping: prod-web-01:
prod-web-01: high_sensitivity threshold_config: high_sensitivity # single config
prod-web-02: high_sensitivity
dev-server-01: low_sensitivity dev-server-01:
# Unmapped hosts use default_threshold_config threshold_config: low_sensitivity
# Hosts with no threshold_config use default_threshold_config
``` ```
### Composable Configurations (list form)
`threshold_config` can be a list. Configs are applied **left to right**: the defaults are the base, then each named config's overrides are layered on top. Later entries in the list win on any metric they define.
```yaml
threshold_configs:
default:
thresholds:
cpu_monitor:
cpu_percent: {warning: 80, critical: 90}
memory_monitor:
memory_percent: {warning: 85, critical: 95}
disk_monitor:
partitions:
/:
percent: {warning: 80, critical: 90}
# Tighter CPU limits for busy servers
high_cpu_load:
thresholds:
cpu_monitor:
cpu_percent: {warning: 60, critical: 75}
# Tighter disk limits for data-heavy servers
busy_disk:
thresholds:
disk_monitor:
partitions:
/:
percent: {warning: 70, critical: 85}
hosts:
# Gets default thresholds only
web-01:
threshold_config: default
# Gets tighter CPU limits, default memory and disk
build-server:
threshold_config: high_cpu_load
# Layers both: tighter CPU AND tighter disk, default memory
db-01:
threshold_config: [high_cpu_load, busy_disk]
# Three layers: busy_disk overrides high_cpu_load if they conflict
storage-01:
threshold_config: [default, high_cpu_load, busy_disk]
```
**How layering works:**
Starting from the `default` thresholds:
| Layer | Applied config | Effect |
|-------|---------------|--------|
| Base | `default` | all default thresholds |
| +1 | `high_cpu_load` | cpu_percent overridden to 60/75 |
| +2 | `busy_disk` | disk percent overridden to 70/85; cpu_percent stays at 60/75 |
Each named config only overrides the metrics it explicitly defines. Metrics not mentioned in a config inherit from the layers beneath.
### Use Cases ### Use Cases
#### 1. Environment-Based Thresholds #### 1. Environment-Based Thresholds
@@ -887,11 +947,15 @@ threshold_configs:
warning: 90.0 # More relaxed for dev warning: 90.0 # More relaxed for dev
critical: 98.0 critical: 98.0
host_threshold_mapping: hosts:
prod-web-01: production prod-web-01:
prod-web-02: production threshold_config: production
dev-web-01: development prod-web-02:
dev-web-02: development threshold_config: production
dev-web-01:
threshold_config: development
dev-web-02:
threshold_config: development
``` ```
#### 2. Server Role-Based Thresholds #### 2. Server Role-Based Thresholds
@@ -914,7 +978,7 @@ threshold_configs:
warning: 70.0 warning: 70.0
critical: 85.0 critical: 85.0
memory_monitor: memory_monitor:
percent: memory_percent:
warning: 90.0 # Databases can use high memory warning: 90.0 # Databases can use high memory
critical: 97.0 critical: 97.0
disk_monitor: disk_monitor:
@@ -927,17 +991,23 @@ threshold_configs:
cache: cache:
thresholds: thresholds:
memory_monitor: memory_monitor:
percent: memory_percent:
warning: 95.0 # Redis/Memcached can use very high memory warning: 95.0 # Redis/Memcached can use very high memory
critical: 99.0 critical: 99.0
host_threshold_mapping: hosts:
web-01: webserver web-01:
web-02: webserver threshold_config: webserver
db-01: database web-02:
db-02: database threshold_config: webserver
redis-01: cache db-01:
memcached-01: cache threshold_config: database
db-02:
threshold_config: database
redis-01:
threshold_config: cache
memcached-01:
threshold_config: cache
``` ```
#### 3. Sensitivity Levels #### 3. Sensitivity Levels
@@ -952,7 +1022,7 @@ threshold_configs:
partitions: partitions:
/: /:
percent: percent:
warning: 70.0 # Very sensitive warning: 70.0
critical: 80.0 critical: 80.0
hysteresis: 0.15 hysteresis: 0.15
@@ -976,12 +1046,69 @@ threshold_configs:
critical: 98.0 critical: 98.0
hysteresis: 0.05 hysteresis: 0.05
host_threshold_mapping: hosts:
payment-gateway: critical payment-gateway:
auth-server: critical threshold_config: critical
web-01: standard auth-server:
web-02: standard threshold_config: critical
test-server: relaxed web-01:
threshold_config: standard
web-02:
threshold_config: standard
test-server:
threshold_config: relaxed
```
#### 4. Composable Profiles
Build host-specific thresholds by combining small, focused configs:
```yaml
threshold_configs:
# Baseline — everything at default levels
default:
thresholds:
cpu_monitor:
cpu_percent: {warning: 80, critical: 90}
memory_monitor:
memory_percent: {warning: 85, critical: 95}
# Overlay: tighter CPU only
tight_cpu:
thresholds:
cpu_monitor:
cpu_percent: {warning: 60, critical: 75}
# Overlay: tighter memory only
tight_memory:
thresholds:
memory_monitor:
memory_percent: {warning: 70, critical: 85}
# Overlay: extra disk partition for database servers
db_disk:
thresholds:
disk_monitor:
partitions:
/var/lib/postgresql:
percent: {warning: 75, critical: 88}
hosts:
# Plain web server
web-01:
threshold_config: default
# Build server: tight CPU, default memory and disk
build-01:
threshold_config: tight_cpu
# Database: tight CPU + tight memory + extra disk partition
db-01:
threshold_config: [tight_cpu, tight_memory, db_disk]
# Replica database: tight memory + extra disk, normal CPU
db-02:
threshold_config: [tight_memory, db_disk]
``` ```
### Backward Compatibility ### Backward Compatibility
@@ -1012,16 +1139,25 @@ threshold_configs:
### Configuration Priority ### Configuration Priority
1. **Host-specific mapping**: If host is in `host_threshold_mapping`, use that config 1. **Host `threshold_config` (list)**: Layer each named config's overrides left-to-right on top of the defaults
2. **Default config**: Use `default_threshold_config` 2. **Host `threshold_config` (string)**: Use that single named config directly
3. **First alphabetically**: If default not found, use first config alphabetically 3. **`host_threshold_mapping`** (legacy): Same as above, string only
4. **Legacy fallback**: If `threshold_configs` not present, use `thresholds` 4. **`default_threshold_config`**: Used for hosts with no mapping
5. **First alphabetically**: If the default config is not found, use the first config alphabetically
6. **Legacy `thresholds` section**: Used when `threshold_configs` is absent entirely
### Example: Complete Multi-Threshold Setup ### Backward Compatibility
See `hbd/config_multi_threshold_example.yaml` for a complete example with: The legacy `host_threshold_mapping` top-level key and the flat `thresholds` section are still fully supported:
- 4 named configurations (default, high_sensitivity, low_sensitivity, database)
- Host-to-config mappings for production, development, and test systems ```yaml
- Specialized database server thresholds # Still works — equivalent to hosts: {prod-web-01: {threshold_config: high_sensitivity}}
- Custom display messages with plugin data host_threshold_mapping:
prod-web-01: high_sensitivity
# Still works — equivalent to threshold_configs: {default: {thresholds: ...}}
thresholds:
cpu_monitor:
cpu_percent: {warning: 80, critical: 90}
```
@@ -0,0 +1,602 @@
# Plugin Error Checking Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Improve plugin error checking in hbc, especially for nagios_runner, and fix logger messages silently discarded in daemon mode.
**Architecture:** Three focused changes across three files: (1) `hbd/client/plugin.py` gains a `skip_reason` attribute on Plugin and updated PluginLoader messaging; (2) `hbd/client/plugins/nagios_runner.py` gains async subprocess execution, stderr capture, signal-killed process handling, and init-time command path validation; (3) `hbd/client/main.py` gains proper post-fork logging reconfiguration to syslog.
**Tech Stack:** Python 3.11+, asyncio, `logging.handlers.SysLogHandler`, pytest
---
## File Map
| Action | Path | What changes |
|---|---|---|
| Modify | `hbd/client/plugin.py` | `Plugin.__init__` gains `skip_reason`; `PluginLoader` checks it |
| Modify | `hbd/client/plugins/nagios_runner.py` | async subprocess, stderr, signal codes, init validation, `skip_reason` |
| Modify | `hbd/client/main.py` | `_reconfigure_logging_for_daemon()` helper; remove redundant syslog calls |
| Create | `tests/test_plugin.py` | PluginLoader messaging tests |
| Create | `tests/test_nagios_runner.py` | NagiosRunnerPlugin behaviour tests |
Run tests throughout with:
```bash
python -m pytest tests/test_plugin.py tests/test_nagios_runner.py -v
```
---
## Task 1: Plugin.skip_reason + PluginLoader messaging
**Files:**
- Modify: `hbd/client/plugin.py:40-48` (Plugin.__init__)
- Modify: `hbd/client/plugin.py:369-381` (PluginLoader.load_from_directory)
- Create: `tests/test_plugin.py`
- [ ] **Step 1: Write failing tests**
Create `tests/test_plugin.py`:
```python
import asyncio
import logging
import textwrap
from hbd.client.plugin import Plugin, PluginLoader, PluginRegistry
def test_plugin_skip_reason_defaults_none(tmp_path):
plugin_code = textwrap.dedent("""
from hbd.client.plugin import MonitorPlugin
class MinimalPlugin(MonitorPlugin):
name = "minimal"
version = "1.0.0"
interval = 60
async def initialize(self):
return True
async def _collect_metrics(self):
return {}
""")
(tmp_path / "minimal.py").write_text(plugin_code)
registry = PluginRegistry()
loader = PluginLoader(registry)
asyncio.run(loader.load_from_directory(tmp_path))
plugin = registry.get("minimal")
assert plugin is not None
assert plugin.skip_reason is None
def test_loader_logs_info_when_skip_reason_set(tmp_path, caplog):
plugin_code = textwrap.dedent("""
from hbd.client.plugin import MonitorPlugin
class SkippablePlugin(MonitorPlugin):
name = "skippable"
version = "1.0.0"
interval = 60
async def initialize(self):
self.skip_reason = "not configured in yaml"
return False
async def _collect_metrics(self):
return {}
""")
(tmp_path / "skippable.py").write_text(plugin_code)
registry = PluginRegistry()
loader = PluginLoader(registry)
with caplog.at_level(logging.INFO, logger="plugin.loader"):
count = asyncio.run(loader.load_from_directory(tmp_path))
assert count == 0
assert any("skipped: not configured in yaml" in r.message for r in caplog.records)
assert not any("failed initialization" in r.message for r in caplog.records)
def test_loader_logs_warning_when_no_skip_reason(tmp_path, caplog):
plugin_code = textwrap.dedent("""
from hbd.client.plugin import MonitorPlugin
class FailPlugin(MonitorPlugin):
name = "fail"
version = "1.0.0"
interval = 60
async def initialize(self):
return False
async def _collect_metrics(self):
return {}
""")
(tmp_path / "fail_plugin.py").write_text(plugin_code)
registry = PluginRegistry()
loader = PluginLoader(registry)
with caplog.at_level(logging.WARNING, logger="plugin.loader"):
count = asyncio.run(loader.load_from_directory(tmp_path))
assert count == 0
assert any("failed initialization" in r.message for r in caplog.records)
```
- [ ] **Step 2: Run tests to verify they fail**
```bash
python -m pytest tests/test_plugin.py -v
```
Expected: `test_plugin_skip_reason_defaults_none` FAILS (attribute missing), others may error.
- [ ] **Step 3: Add `skip_reason` to `Plugin.__init__`**
In `hbd/client/plugin.py`, in `Plugin.__init__` (around line 46), add one line:
```python
def __init__(self, config: Optional[Dict[str, Any]] = None):
self.config = config or {}
self.logger = logging.getLogger(f"plugin.{self.name}")
self._initialized = False
self.skip_reason: Optional[str] = None
```
- [ ] **Step 4: Update PluginLoader messaging**
In `hbd/client/plugin.py`, replace the `if not initialized:` block (around line 372):
```python
if not initialized:
if plugin.skip_reason:
self.logger.info(
f"Plugin {plugin.name} skipped: {plugin.skip_reason}"
)
else:
self.logger.warning(
f"Plugin {plugin.name} failed initialization, skipping"
)
continue
```
- [ ] **Step 5: Run tests to verify they pass**
```bash
python -m pytest tests/test_plugin.py -v
```
Expected: all 3 tests PASS.
- [ ] **Step 6: Commit**
```bash
git add hbd/client/plugin.py tests/test_plugin.py
git commit -m "feat: add skip_reason to Plugin; improve PluginLoader init messaging"
```
---
## Task 2: NagiosRunnerPlugin — skip_reason when no commands
**Files:**
- Modify: `hbd/client/plugins/nagios_runner.py:88-105` (initialize)
- Modify: `tests/test_nagios_runner.py` (create)
- [ ] **Step 1: Write failing test**
Create `tests/test_nagios_runner.py`:
```python
import asyncio
import logging
import os
import stat
import pytest
from hbd.client.plugins.nagios_runner import (
NagiosRunnerPlugin,
NAGIOS_OK,
NAGIOS_WARNING,
NAGIOS_CRITICAL,
NAGIOS_UNKNOWN,
)
def test_no_commands_sets_skip_reason():
plugin = NagiosRunnerPlugin(config={"commands": []})
result = asyncio.run(plugin.initialize())
assert result is False
assert plugin.skip_reason is not None
assert "nagios_runner.commands" in plugin.skip_reason
```
- [ ] **Step 2: Run test to verify it fails**
```bash
python -m pytest tests/test_nagios_runner.py::test_no_commands_sets_skip_reason -v
```
Expected: FAIL — `plugin.skip_reason` is `None`.
- [ ] **Step 3: Set skip_reason in NagiosRunnerPlugin.initialize()**
In `hbd/client/plugins/nagios_runner.py`, replace the early-return block in `initialize()` (around line 96):
```python
if not self.commands:
self.skip_reason = "no commands configured (add nagios_runner.commands to config)"
self.logger.info("No Nagios commands configured")
return False
```
- [ ] **Step 4: Run test to verify it passes**
```bash
python -m pytest tests/test_nagios_runner.py::test_no_commands_sets_skip_reason -v
```
Expected: PASS.
- [ ] **Step 5: Commit**
```bash
git add hbd/client/plugins/nagios_runner.py tests/test_nagios_runner.py
git commit -m "feat: set skip_reason on nagios_runner when no commands configured"
```
---
## Task 3: NagiosRunnerPlugin — async subprocess, stderr capture, negative return codes
**Files:**
- Modify: `hbd/client/plugins/nagios_runner.py` (imports + `_run_nagios_plugin`)
- Modify: `tests/test_nagios_runner.py`
- [ ] **Step 1: Write failing tests**
Append to `tests/test_nagios_runner.py`:
```python
def test_stderr_used_when_stdout_empty(tmp_path):
script = tmp_path / "check_err.sh"
script.write_text("#!/bin/sh\necho 'error from stderr' >&2\nexit 2\n")
script.chmod(script.stat().st_mode | stat.S_IEXEC)
config = {"commands": [{"name": "t", "command": str(script)}], "timeout": 5}
plugin = NagiosRunnerPlugin(config=config)
asyncio.run(plugin.initialize())
data = asyncio.run(plugin._collect_metrics())
assert "error from stderr" in data["t_output"]
assert data["t_status_code"] == NAGIOS_CRITICAL
def test_stderr_appended_when_both_present(tmp_path):
script = tmp_path / "check_both.sh"
script.write_text("#!/bin/sh\necho 'OK - all good'\necho 'extra detail' >&2\nexit 0\n")
script.chmod(script.stat().st_mode | stat.S_IEXEC)
config = {"commands": [{"name": "t", "command": str(script)}], "timeout": 5}
plugin = NagiosRunnerPlugin(config=config)
asyncio.run(plugin.initialize())
data = asyncio.run(plugin._collect_metrics())
assert "OK - all good" in data["t_output"]
assert "extra detail" in data["t_output"]
assert data["t_status_code"] == NAGIOS_OK
def test_negative_returncode_maps_to_unknown():
# kill -9 $$ kills the shell itself; asyncio sees returncode -9
config = {"commands": [{"name": "t", "command": "kill -9 $$"}], "timeout": 5}
plugin = NagiosRunnerPlugin(config=config)
asyncio.run(plugin.initialize())
data = asyncio.run(plugin._collect_metrics())
assert data["t_status_code"] == NAGIOS_UNKNOWN
assert "signal" in data["t_output"].lower()
```
- [ ] **Step 2: Run tests to verify they fail**
```bash
python -m pytest tests/test_nagios_runner.py::test_stderr_used_when_stdout_empty \
tests/test_nagios_runner.py::test_stderr_appended_when_both_present \
tests/test_nagios_runner.py::test_negative_returncode_maps_to_unknown -v
```
Expected: all FAIL — current implementation ignores stderr and doesn't handle negative codes.
- [ ] **Step 3: Update imports in nagios_runner.py**
Replace the import block at the top of `hbd/client/plugins/nagios_runner.py`:
```python
import asyncio
import os
import re
from typing import Any, Dict, List, Optional, Tuple
from hbd.client.plugin import MonitorPlugin
```
(Remove `import subprocess`; add `import asyncio` and `import os`.)
- [ ] **Step 4: Upgrade collection log level from DEBUG to INFO**
In `hbd/client/plugins/nagios_runner.py`, in `_collect_metrics()`, change the debug log (around line 144) so results are visible at INFO level:
```python
self.logger.info(
f"Executed {name}: {STATUS_NAMES.get(status_code, 'UNKNOWN')} - {output[:50]}"
)
```
- [ ] **Step 5: Replace `_run_nagios_plugin` with async implementation**
Replace the entire `_run_nagios_plugin` method in `hbd/client/plugins/nagios_runner.py`:
```python
async def _run_nagios_plugin(
self,
command: str
) -> Tuple[int, str, Dict[str, Any]]:
"""Execute a Nagios plugin and parse its output."""
try:
proc = await asyncio.create_subprocess_shell(
command,
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.PIPE,
)
try:
stdout_bytes, stderr_bytes = await asyncio.wait_for(
proc.communicate(), timeout=self.timeout
)
except asyncio.TimeoutError:
proc.kill()
await proc.communicate()
self.logger.error(f"Command timed out: {command}")
return NAGIOS_UNKNOWN, f"Command timed out after {self.timeout}s", {}
status_code = proc.returncode
if status_code < 0:
return NAGIOS_UNKNOWN, f"Process killed by signal {-status_code}", {}
if status_code > 3:
status_code = NAGIOS_UNKNOWN
stdout = stdout_bytes.decode(errors="replace").strip()
stderr = stderr_bytes.decode(errors="replace").strip()
# Parse perfdata from stdout before mixing in stderr
perfdata = self._parse_perfdata(stdout)
# Build status message
status_part = stdout.split('|')[0].strip() if '|' in stdout else stdout
if not stdout and stderr:
output_msg = stderr
elif stdout and stderr:
output_msg = f"{status_part} [stderr: {stderr}]"
else:
output_msg = status_part
return status_code, output_msg, perfdata
except Exception as e:
self.logger.error(f"Error executing command: {e}")
return NAGIOS_UNKNOWN, f"Execution error: {str(e)}", {}
```
Also remove the now-unused `self.shell` line from `__init__` (the `shell` config key is no longer used since `create_subprocess_shell` always uses a shell):
In `NagiosRunnerPlugin.__init__`, remove:
```python
self.shell: bool = config.get("shell", True) if config else True
```
- [ ] **Step 6: Run tests to verify they pass**
```bash
python -m pytest tests/test_nagios_runner.py -v
```
Expected: all tests PASS including the 3 new ones.
- [ ] **Step 7: Commit**
```bash
git add hbd/client/plugins/nagios_runner.py tests/test_nagios_runner.py
git commit -m "feat: async subprocess in nagios_runner with stderr capture and signal handling"
```
---
## Task 4: NagiosRunnerPlugin — command path validation at init
**Files:**
- Modify: `hbd/client/plugins/nagios_runner.py` (initialize)
- Modify: `tests/test_nagios_runner.py`
- [ ] **Step 1: Write failing tests**
Append to `tests/test_nagios_runner.py`:
```python
def test_absolute_path_not_found_warns(caplog):
fake_cmd = "/nonexistent_hbc_test_path/check_something"
config = {"commands": [{"name": "t", "command": fake_cmd}]}
plugin = NagiosRunnerPlugin(config=config)
with caplog.at_level(logging.WARNING, logger="plugin.nagios_runner"):
asyncio.run(plugin.initialize())
assert any("not found" in r.message for r in caplog.records)
def test_absolute_path_not_executable_warns(caplog, tmp_path):
non_exec = tmp_path / "check_test"
non_exec.write_text("#!/bin/sh\necho OK\n")
non_exec.chmod(0o644) # readable but not executable
config = {"commands": [{"name": "t", "command": str(non_exec)}]}
plugin = NagiosRunnerPlugin(config=config)
with caplog.at_level(logging.WARNING, logger="plugin.nagios_runner"):
asyncio.run(plugin.initialize())
assert any("not executable" in r.message for r in caplog.records)
def test_relative_path_not_checked(caplog):
# Relative paths (resolved via PATH) must not generate warnings
config = {"commands": [{"name": "t", "command": "echo OK"}]}
plugin = NagiosRunnerPlugin(config=config)
with caplog.at_level(logging.WARNING, logger="plugin.nagios_runner"):
asyncio.run(plugin.initialize())
assert not any(
"not found" in r.message or "not executable" in r.message
for r in caplog.records
)
```
- [ ] **Step 2: Run tests to verify they fail**
```bash
python -m pytest tests/test_nagios_runner.py::test_absolute_path_not_found_warns \
tests/test_nagios_runner.py::test_absolute_path_not_executable_warns \
tests/test_nagios_runner.py::test_relative_path_not_checked -v
```
Expected: `test_absolute_path_not_found_warns` and `test_absolute_path_not_executable_warns` FAIL (no warnings logged); `test_relative_path_not_checked` may pass.
- [ ] **Step 3: Add command path validation to `initialize()`**
In `hbd/client/plugins/nagios_runner.py`, extend `initialize()` by adding validation after the existing "log each command" loop (after line 103, before `return True`):
```python
# Validate absolute command paths early
for cmd_config in self.commands:
name = cmd_config.get("name", "unnamed")
command = cmd_config.get("command", "")
if not command:
continue
exe = command.split()[0]
if os.path.isabs(exe):
if not os.path.isfile(exe):
self.logger.warning(
f"Command '{name}': executable not found: {exe}"
)
elif not os.access(exe, os.X_OK):
self.logger.warning(
f"Command '{name}': executable not executable: {exe}"
)
```
- [ ] **Step 4: Run full test suite to verify all pass**
```bash
python -m pytest tests/test_plugin.py tests/test_nagios_runner.py -v
```
Expected: all tests PASS.
- [ ] **Step 5: Commit**
```bash
git add hbd/client/plugins/nagios_runner.py tests/test_nagios_runner.py
git commit -m "feat: validate absolute command paths at nagios_runner init"
```
---
## Task 5: Daemon mode logging — route to syslog after fork
**Files:**
- Modify: `hbd/client/main.py` (new helper + updated daemon block)
No automated test for daemonization itself (fork behaviour is hard to unit-test). Manual verification steps are provided below.
- [ ] **Step 1: Add `_reconfigure_logging_for_daemon` helper**
In `hbd/client/main.py`, add this function just before `def build_parser()` (around line 589):
```python
def _reconfigure_logging_for_daemon(log_level: int) -> None:
"""Replace StreamHandlers (now writing to /dev/null) with a SysLogHandler."""
from logging.handlers import SysLogHandler
root = logging.getLogger()
for handler in root.handlers[:]:
root.removeHandler(handler)
handler.close()
try:
syslog_handler = SysLogHandler(
address="/dev/log",
facility=SysLogHandler.LOG_DAEMON,
)
except OSError:
syslog_handler = SysLogHandler(
address=("localhost", 514),
facility=SysLogHandler.LOG_DAEMON,
)
# Attach the fallback first so the warning reaches syslog
syslog_handler.setFormatter(
logging.Formatter("hbc[%(process)d]: %(name)s %(levelname)s: %(message)s")
)
root.addHandler(syslog_handler)
root.setLevel(log_level)
logging.warning("/dev/log not found, using syslog UDP localhost:514")
return
syslog_handler.setFormatter(
logging.Formatter("hbc[%(process)d]: %(name)s %(levelname)s: %(message)s")
)
root.addHandler(syslog_handler)
root.setLevel(log_level)
```
- [ ] **Step 2: Update the daemon block in `main()`**
In `hbd/client/main.py`, replace the entire `if args.daemon:` block (lines 664675):
```python
if args.daemon:
print("Daemonizing...")
daemonize()
_reconfigure_logging_for_daemon(log_level)
logging.info(f"hbc starting, sending heartbeat to {', '.join(args.hosts)}")
```
This removes the `import syslog`, `syslog.openlog()`, and `syslog.syslog()` calls (now handled by the logging system) and removes the no-op second `logging.basicConfig()` call.
- [ ] **Step 3: Run existing test suite to confirm no regressions**
```bash
python -m pytest tests/test_plugin.py tests/test_nagios_runner.py -v
```
Expected: all tests still PASS.
- [ ] **Step 4: Manual smoke test — verify syslog output in daemon mode**
```bash
# In one terminal, tail syslog
sudo journalctl -f -t hbc
# In another terminal, start hbc in daemon mode (replace HOST with a real or dummy host)
python -m hbd.client.main -d -v localhost
# Expected in journalctl output:
# hbc[<pid>]: hbc.main INFO: Starting hbc for <hostname> -> ['localhost']
# hbc[<pid>]: hbc.main INFO: hbc starting, sending heartbeat to localhost
# hbc[<pid>]: plugin.loader INFO: ...
# Stop the daemon
pkill -f "hbd.client.main"
```
- [ ] **Step 5: Commit**
```bash
git add hbd/client/main.py
git commit -m "fix: reconfigure logging to syslog after daemonize() instead of no-op basicConfig"
```
@@ -0,0 +1,92 @@
# Plugin Error Checking & Daemon Logging — Design Spec
**Date:** 2026-04-25
**Scope:** hbc client — daemon mode logging, nagios_runner plugin robustness, PluginLoader messaging
**Files affected:** `hbd/client/main.py`, `hbd/client/plugins/nagios_runner.py`, `hbd/client/plugin.py`
---
## 1. Daemon Mode Logging
### Problem
In `main()`, `logging.basicConfig()` is called before `daemonize()` (establishing a StreamHandler to stderr), then called again after `daemonize()`. The second call is a no-op — Python ignores `basicConfig()` when handlers are already configured. After daemonization, stderr is redirected to `/dev/null`, so all subsequent log output is silently discarded.
The existing `syslog.openlog()` / `syslog.syslog()` calls (lines 666668) write a single startup message but do not integrate with the `logging` system, so plugin and connection log messages never reach syslog.
### Fix
After `daemonize()`, explicitly reconfigure the root logger:
1. Remove all existing handlers (they now write to `/dev/null`).
2. Add `logging.handlers.SysLogHandler(address='/dev/log', facility=LOG_DAEMON)`.
3. Set formatter: `hbc[%(process)d]: %(name)s %(levelname)s: %(message)s`
4. Preserve the `log_level` already determined from `-v`/`-x` CLI flags.
Remove the redundant `syslog.openlog()` / `syslog.syslog()` calls — the logging system handles routing.
**Fallback:** If `/dev/log` does not exist (containers, some BSDs), fall back to `SysLogHandler(address=('localhost', 514))`. Log one warning (to stderr, before handlers are replaced) so the operator knows.
---
## 2. Nagios Runner Improvements
### 2a — Async Subprocess
`_run_nagios_plugin()` is declared `async def` but calls `subprocess.run()` synchronously, blocking the event loop for the full command duration.
**Fix:** Replace with `asyncio.create_subprocess_shell()` + `await proc.communicate()`. Enforce timeout with `asyncio.wait_for(..., timeout=self.timeout)` and catch `asyncio.TimeoutError`.
### 2b — Stderr Capture
Subprocess stderr is currently discarded (`capture_output=True` only captures stdout in the sync call; stderr content is lost).
**Fix:** Pass `stderr=asyncio.subprocess.PIPE` to `create_subprocess_shell`. After `communicate()`, if stdout is empty but stderr has content, use stderr as the output message. If both have content, append stderr to the output for visibility.
### 2c — Negative Return Codes
A negative `returncode` means the process was killed by a signal (SIGKILL, OOM, etc.). The current code treats these as-is, which may produce unexpected status values.
**Fix:** If `returncode < 0`, map to `NAGIOS_UNKNOWN` with message `"Process killed by signal {-returncode}"`.
### 2d — Command Path Validation at Init
`initialize()` currently only checks that the commands list is non-empty.
**Fix:** For each command entry during `initialize()`:
- Warn and skip the entry if `name` or `command` is missing.
- Extract the executable (first whitespace-delimited token of the command string).
- If the executable is an absolute path, check `os.path.isfile()` and `os.access(..., os.X_OK)`. Log a `WARNING` if either check fails.
- Commands with relative paths or shell builtins are not checked (they may be on PATH) — just noted.
- Validation warns only; all original entries in `self.commands` are retained and still attempted at collection time (where the existing missing-name/command guard already skips them). The plugin initializes successfully as long as the commands list is non-empty.
---
## 3. PluginLoader Messaging
### Problem
When `initialize()` returns `False`, the loader always logs:
> `WARNING: Plugin X failed initialization, skipping`
This is alarming when the real reason is simply "no commands configured". There is no API to distinguish "not configured" from "genuinely broken".
### Fix
Add an optional `skip_reason` attribute to `Plugin.__init__()` (defaults to `None`).
In `PluginLoader.load_from_directory()`, after `initialize()` returns `False`:
- If `plugin.skip_reason` is set → `logger.info(f"Plugin {plugin.name} skipped: {plugin.skip_reason}")`
- If `plugin.skip_reason` is `None``logger.warning(f"Plugin {plugin.name} failed initialization, skipping")` (existing behaviour)
In `NagiosRunnerPlugin.initialize()`, when no commands are configured:
```python
self.skip_reason = "no commands configured (add nagios_runner.commands to config)"
return False
```
Genuine failures (exceptions) continue to go through the existing `except` block in the loader, logging at `ERROR` with traceback — unchanged.
---
## Decisions
| Topic | Decision |
|---|---|
| Daemon log destination | syslog only (LOG_DAEMON facility) |
| Syslog fallback | localhost:514 UDP if `/dev/log` absent |
| Nagios result log level | INFO for all statuses (OK/WARNING/CRITICAL/UNKNOWN) |
| Invalid command handling at init | Warn and continue; still attempt at collection time |
| PluginLoader API change | `skip_reason` attribute on Plugin base class, checked by loader |
-9
View File
@@ -1,9 +0,0 @@
Plan
Heartbeat is a client/server based network monitor and host observer. hbd, the server portion receives heartbeat and state messages from clients and maintaines state and hisgtory of the informations it receives.
hbc, the client portion gathers information on various aspects of the
system it is running on, and sends it to hbd. Initially this info is basic, like OS make and version, hardware info (CPU type, memory and disks), fileystem info and some resource info. hbc/hbd support a plugin system to extend the info gathered and stored.
hbd also can send notification based on missed hbc updates, and on violation of pre-set limits for various state paramaters.
+1 -1
View File
@@ -14,4 +14,4 @@ Install options:
""" """
__all__ = ["__version__"] __all__ = ["__version__"]
__version__ = "5.1.0" __version__ = "5.1.14"
+74 -41
View File
@@ -14,7 +14,7 @@ import signal
import socket import socket
import sys import sys
import time import time
from hashlib import md5 from logging.handlers import SysLogHandler
from pathlib import Path from pathlib import Path
from typing import Dict, List, Optional from typing import Dict, List, Optional
@@ -55,6 +55,7 @@ class AsyncConnection:
self.transport: Optional[asyncio.DatagramTransport] = None self.transport: Optional[asyncio.DatagramTransport] = None
self.protocol: Optional[asyncio.DatagramProtocol] = None self.protocol: Optional[asyncio.DatagramProtocol] = None
self._dead = False
self.logger = logging.getLogger(f"hbc.conn.{addr}") self.logger = logging.getLogger(f"hbc.conn.{addr}")
@@ -92,6 +93,9 @@ class AsyncConnection:
msg: Message dictionary msg: Message dictionary
msg_id: Message ID (HTB, PLG, etc.) msg_id: Message ID (HTB, PLG, etc.)
""" """
if self._dead:
return
if not self.transport: if not self.transport:
await self.open() await self.open()
@@ -166,7 +170,9 @@ class HeartbeatProtocol(asyncio.DatagramProtocol):
def error_received(self, exc): def error_received(self, exc):
"""Handle protocol errors.""" """Handle protocol errors."""
self.logger.error(f"Protocol error: {exc}") self.logger.warning(f"Protocol error on {self.connection.addr}: {exc} — dropping connection")
self.connection._dead = True
self.connection.close()
async def handle_command(conn: AsyncConnection, msg: dict): async def handle_command(conn: AsyncConnection, msg: dict):
@@ -203,48 +209,45 @@ async def handle_command(conn: AsyncConnection, msg: dict):
await conn.sendto(response) await conn.sendto(response)
async def handle_update(conn: AsyncConnection, msg: dict): async def handle_update(conn: AsyncConnection, _msg: dict): # pyright: ignore[reportUnusedParameter]
"""Handle self-update from server.""" """Handle self-update by running hb_install.sh."""
import codecs
import shutil import shutil
logger = logging.getLogger("hbc.update") logger = logging.getLogger("hbc.update")
try: installer = shutil.which("hb_install.sh")
code = codecs.decode(msg["code"], "base64").decode() if installer is None:
csum = msg["csum"] candidate = Path(sys.argv[0]).parent / "hb_install.sh"
except Exception as e: if candidate.exists():
error = f"Missing code/csum: {e}" installer = str(candidate)
if installer is None:
error = "hb_install.sh not found in PATH or alongside hbc"
logger.error(error) logger.error(error)
await conn.sendto({"service": "update", "msg": error}) await conn.sendto({"service": "update", "msg": error})
return return
# Verify checksum logger.info(f"Running installer: {installer}")
m = md5() try:
m.update(code.encode()) proc = await asyncio.create_subprocess_exec(
if m.hexdigest() != csum: installer, "client",
error = "Checksum mismatch" stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.STDOUT,
)
out, _ = await asyncio.wait_for(proc.communicate(), timeout=120)
except asyncio.TimeoutError:
error = "Installer timed out"
logger.error(error)
await conn.sendto({"service": "update", "msg": error})
return
except Exception as e:
error = f"Installer failed: {e}"
logger.error(error) logger.error(error)
await conn.sendto({"service": "update", "msg": error}) await conn.sendto({"service": "update", "msg": error})
return return
# Backup current file if proc.returncode != 0:
fn = sys.argv[0] error = f"Installer exited {proc.returncode}: {out.decode().strip()}"
ofn = f"{fn}.sav"
try:
shutil.copy2(fn, ofn)
except Exception as e:
error = f"Backup failed: {e}"
logger.error(error)
await conn.sendto({"service": "update", "msg": error})
return
# Write new code
try:
with open(fn, "w") as fh:
fh.write(code)
except Exception as e:
error = f"Write failed: {e}"
logger.error(error) logger.error(error)
await conn.sendto({"service": "update", "msg": error}) await conn.sendto({"service": "update", "msg": error})
return return
@@ -522,6 +525,13 @@ async def async_main(args, config):
for sig in (signal.SIGTERM, signal.SIGINT): for sig in (signal.SIGTERM, signal.SIGINT):
loop.add_signal_handler(sig, stop) loop.add_signal_handler(sig, stop)
def _sighup():
global dorestart
dorestart = True
stop()
loop.add_signal_handler(signal.SIGHUP, _sighup)
# Start async tasks # Start async tasks
# Heartbeat senders (one per connection) # Heartbeat senders (one per connection)
for conn in connections: for conn in connections:
@@ -586,6 +596,36 @@ def daemonize(
os.dup2(se.fileno(), sys.stderr.fileno()) os.dup2(se.fileno(), sys.stderr.fileno())
def _reconfigure_logging_for_daemon(log_level: int) -> None:
"""Replace StreamHandlers (now writing to /dev/null) with a SysLogHandler."""
root = logging.getLogger()
for handler in root.handlers[:]:
root.removeHandler(handler)
handler.close()
use_udp_fallback = not os.path.exists("/dev/log")
if use_udp_fallback:
syslog_handler = SysLogHandler(
address=("localhost", 514),
facility=SysLogHandler.LOG_DAEMON,
)
else:
syslog_handler = SysLogHandler(
address="/dev/log",
facility=SysLogHandler.LOG_DAEMON,
)
syslog_handler.setFormatter(
logging.Formatter("hbc[%(process)d]: %(name)s %(levelname)s: %(message)s")
)
root.addHandler(syslog_handler)
root.setLevel(log_level)
if use_udp_fallback:
logging.warning("/dev/log not found, using syslog UDP localhost:514")
def build_parser(): def build_parser():
"""Build argument parser.""" """Build argument parser."""
parser = argparse.ArgumentParser( parser = argparse.ArgumentParser(
@@ -663,16 +703,9 @@ def main(argv=None):
# Daemonize if requested # Daemonize if requested
if args.daemon: if args.daemon:
print("Daemonizing...") print("Daemonizing...")
import syslog
syslog.openlog("hbc", syslog.LOG_PID, syslog.LOG_DAEMON)
syslog.syslog(syslog.LOG_INFO, f"Starting heartbeat to {', '.join(args.hosts)}")
daemonize() daemonize()
_reconfigure_logging_for_daemon(log_level)
# Reconfigure logging for syslog logging.info(f"hbc starting, sending heartbeat to {', '.join(args.hosts)}")
logging.basicConfig(
level=log_level,
format="hbc[%(process)d]: %(name)s %(levelname)s: %(message)s"
)
# Run async main # Run async main
try: try:
+14 -5
View File
@@ -29,6 +29,7 @@ class Plugin(ABC):
description: Human-readable description description: Human-readable description
interval: Collection interval in seconds (0 for InfoPlugin = collect once) interval: Collection interval in seconds (0 for InfoPlugin = collect once)
enabled: Whether plugin is active (can be disabled via config) enabled: Whether plugin is active (can be disabled via config)
skip_reason: Set by plugin before returning False from initialize(); causes loader to log INFO instead of WARNING.
""" """
name: str = "" name: str = ""
@@ -46,6 +47,7 @@ class Plugin(ABC):
self.config = config or {} self.config = config or {}
self.logger = logging.getLogger(f"plugin.{self.name}") self.logger = logging.getLogger(f"plugin.{self.name}")
self._initialized = False self._initialized = False
self.skip_reason: Optional[str] = None
@abstractmethod @abstractmethod
async def initialize(self) -> bool: async def initialize(self) -> bool:
@@ -312,9 +314,10 @@ class PluginLoader:
loaded_count = 0 loaded_count = 0
raw_config = config or {} raw_config = config or {}
# Per-plugin config lives under the 'plugins' key; fall back to top-level # Per-plugin config lives under the 'plugins' key or at top-level.
# for backwards compatibility. # CLIENT_DEFAULTS seeds "plugins": {} so the key always exists; check
plugin_config = raw_config.get("plugins", raw_config) # both the subdict and top-level so that either layout in .hbc.yaml works.
plugins_subconfig = raw_config.get("plugins", {})
# Scan for Python files # Scan for Python files
for plugin_file in directory.glob("*.py"): for plugin_file in directory.glob("*.py"):
@@ -359,14 +362,20 @@ class PluginLoader:
self.logger.debug(f"Found plugin class: {name}") self.logger.debug(f"Found plugin class: {name}")
# Instantiate plugin with config # Instantiate plugin with config — check plugins subdict first,
plugin_instance_config = plugin_config.get(obj.name, {}) # then top-level keys (e.g. nagios_runner: ... at root of config).
plugin_instance_config = plugins_subconfig.get(obj.name) or raw_config.get(obj.name, {})
plugin = obj(config=plugin_instance_config) plugin = obj(config=plugin_instance_config)
# Initialize plugin # Initialize plugin
try: try:
initialized = await plugin.initialize() initialized = await plugin.initialize()
if not initialized: if not initialized:
if plugin.skip_reason:
self.logger.info(
f"Plugin {plugin.name} skipped: {plugin.skip_reason}"
)
else:
self.logger.warning( self.logger.warning(
f"Plugin {plugin.name} failed initialization, skipping" f"Plugin {plugin.name} failed initialization, skipping"
) )
+58 -38
View File
@@ -21,8 +21,10 @@ nagios_runner:
``` ```
""" """
import asyncio
import os
import re import re
import subprocess import shlex
from typing import Any, Dict, List, Optional, Tuple from typing import Any, Dict, List, Optional, Tuple
from hbd.client.plugin import MonitorPlugin from hbd.client.plugin import MonitorPlugin
@@ -52,7 +54,6 @@ class NagiosRunnerPlugin(MonitorPlugin):
interval: Collection interval in seconds (default: 300) interval: Collection interval in seconds (default: 300)
commands: List of command definitions with 'name' and 'command' keys commands: List of command definitions with 'name' and 'command' keys
timeout: Command execution timeout in seconds (default: 30) timeout: Command execution timeout in seconds (default: 30)
shell: Whether to execute commands via shell (default: True)
Example: Example:
nagios_runner: nagios_runner:
@@ -76,15 +77,8 @@ class NagiosRunnerPlugin(MonitorPlugin):
# Extract configuration # Extract configuration
self.commands: List[Dict[str, str]] = config.get("commands", []) if config else [] self.commands: List[Dict[str, str]] = config.get("commands", []) if config else []
self.timeout: int = config.get("timeout", 30) if config else 30 self.timeout: int = config.get("timeout", 30) if config else 30
self.shell: bool = config.get("shell", True) if config else True
self.interval = config.get("interval", 300) if config else 300 self.interval = config.get("interval", 300) if config else 300
# Validate commands
if not self.commands:
self.logger.info(
"No Nagios commands configured. Add 'nagios_runner.commands' to config."
)
async def initialize(self) -> bool: async def initialize(self) -> bool:
"""Initialize the Nagios runner plugin. """Initialize the Nagios runner plugin.
@@ -94,7 +88,7 @@ class NagiosRunnerPlugin(MonitorPlugin):
self.logger.info(f"Initializing {self.name} plugin") self.logger.info(f"Initializing {self.name} plugin")
if not self.commands: if not self.commands:
self.logger.info("No Nagios commands configured") self.skip_reason = "no commands configured (add nagios_runner.commands to config)"
return False return False
self.logger.info(f"Configured to run {len(self.commands)} Nagios plugin(s)") self.logger.info(f"Configured to run {len(self.commands)} Nagios plugin(s)")
@@ -102,6 +96,29 @@ class NagiosRunnerPlugin(MonitorPlugin):
name = cmd_config.get("name", "unnamed") name = cmd_config.get("name", "unnamed")
self.logger.info(f" - {name}: {cmd_config.get('command', 'N/A')}") self.logger.info(f" - {name}: {cmd_config.get('command', 'N/A')}")
# Validate absolute command paths early
for cmd_config in self.commands:
name = cmd_config.get("name", "unnamed")
command = cmd_config.get("command", "")
if not command:
continue
try:
tokens = shlex.split(command)
except ValueError:
continue # malformed command string; skip validation
if not tokens:
continue
exe = tokens[0]
if os.path.isabs(exe):
if not os.path.isfile(exe):
self.logger.warning(
f"Command '{name}': executable not found: {exe}"
)
elif not os.access(exe, os.X_OK):
self.logger.warning(
f"Command '{name}': executable not executable: {exe}"
)
return True return True
async def _collect_metrics(self) -> Dict[str, Any]: async def _collect_metrics(self) -> Dict[str, Any]:
@@ -141,7 +158,7 @@ class NagiosRunnerPlugin(MonitorPlugin):
for metric_name, metric_value in perfdata.items(): for metric_name, metric_value in perfdata.items():
results[f"{name}_{metric_name}"] = metric_value results[f"{name}_{metric_name}"] = metric_value
self.logger.debug( self.logger.info(
f"Executed {name}: {STATUS_NAMES.get(status_code, 'UNKNOWN')} - {output[:50]}" f"Executed {name}: {STATUS_NAMES.get(status_code, 'UNKNOWN')} - {output[:50]}"
) )
@@ -163,46 +180,49 @@ class NagiosRunnerPlugin(MonitorPlugin):
self, self,
command: str command: str
) -> Tuple[int, str, Dict[str, Any]]: ) -> Tuple[int, str, Dict[str, Any]]:
"""Execute a Nagios plugin and parse its output. """Execute a Nagios plugin and parse its output."""
Args:
command: Command string to execute
Returns:
Tuple of (status_code, output_message, performance_data_dict)
"""
try: try:
# Run command proc = await asyncio.create_subprocess_shell(
result = subprocess.run(
command, command,
shell=self.shell, stdout=asyncio.subprocess.PIPE,
capture_output=True, stderr=asyncio.subprocess.PIPE,
timeout=self.timeout,
text=True
) )
try:
stdout_bytes, stderr_bytes = await asyncio.wait_for(
proc.communicate(), timeout=self.timeout
)
except asyncio.TimeoutError:
proc.kill()
await proc.communicate()
self.logger.error(f"Command timed out: {command}")
return NAGIOS_UNKNOWN, f"Command timed out after {self.timeout}s", {}
status_code = result.returncode status_code = proc.returncode
output = result.stdout.strip()
if status_code < 0:
return NAGIOS_UNKNOWN, f"Process killed by signal {-status_code}", {}
# Nagios plugins can return codes > 3, treat as UNKNOWN
if status_code > 3: if status_code > 3:
status_code = NAGIOS_UNKNOWN status_code = NAGIOS_UNKNOWN
# Parse performance data stdout = stdout_bytes.decode(errors="replace").strip()
perfdata = self._parse_perfdata(output) stderr = stderr_bytes.decode(errors="replace").strip()
# Extract just the status message (before the pipe if present) # Parse perfdata from stdout before mixing in stderr
if '|' in output: perfdata = self._parse_perfdata(stdout)
output_msg = output.split('|')[0].strip()
# Build status message
status_part = stdout.split('|')[0].strip() if '|' in stdout else stdout
if not stdout and stderr:
output_msg = stderr
elif stdout and stderr:
output_msg = f"{status_part} [stderr: {stderr}]"
else: else:
output_msg = output output_msg = status_part
return status_code, output_msg, perfdata return status_code, output_msg, perfdata
except subprocess.TimeoutExpired:
self.logger.error(f"Command timed out: {command}")
return NAGIOS_UNKNOWN, f"Command timed out after {self.timeout}s", {}
except Exception as e: except Exception as e:
self.logger.error(f"Error executing command: {e}") self.logger.error(f"Error executing command: {e}")
return NAGIOS_UNKNOWN, f"Execution error: {str(e)}", {} return NAGIOS_UNKNOWN, f"Execution error: {str(e)}", {}
+3
View File
@@ -48,6 +48,7 @@ class OSInfoPlugin(InfoPlugin):
Dictionary with OS details Dictionary with OS details
""" """
try: try:
from hbd import __version__ as hbc_version
data = { data = {
"system": platform.system(), # e.g., "Linux", "Darwin", "Windows" "system": platform.system(), # e.g., "Linux", "Darwin", "Windows"
"node": platform.node(), # hostname "node": platform.node(), # hostname
@@ -58,6 +59,8 @@ class OSInfoPlugin(InfoPlugin):
"architecture": platform.architecture()[0], # e.g., "64bit" "architecture": platform.architecture()[0], # e.g., "64bit"
"python_version": platform.python_version(), "python_version": platform.python_version(),
"python_implementation": platform.python_implementation(), "python_implementation": platform.python_implementation(),
"hbc_version": hbc_version,
"hbc_type": "full",
} }
# Add Linux-specific distribution info # Add Linux-specific distribution info
+130
View File
@@ -0,0 +1,130 @@
"""
ZFS pool monitoring plugin for Heartbeat.
Collects per-pool health, capacity, and cumulative I/O statistics via zpool(8).
"""
import asyncio
import logging
import shutil
from typing import Any, Dict, List, Optional
from hbd.client.plugin import MonitorPlugin
logger = logging.getLogger(__name__)
def _int(s: str) -> Optional[int]:
try:
return int(s.strip().rstrip("KMGTkBkmgt%x"))
except (ValueError, AttributeError):
return None
def _float(s: str) -> Optional[float]:
try:
return float(s.strip().rstrip("%x"))
except (ValueError, AttributeError):
return None
class ZFSMonitorPlugin(MonitorPlugin):
"""Monitor ZFS pool health, capacity, and I/O statistics.
Collects per pool:
- health: ONLINE, DEGRADED, FAULTED, etc.
- size / alloc / free: total, allocated and free bytes
- capacity: percentage used (0-100)
- frag: fragmentation percentage
- dedup: deduplication ratio
- read_ops / write_ops: cumulative I/O operations since last boot/clear
- read_bw / write_bw: cumulative bytes transferred since last boot/clear
Configuration:
interval: collection interval in seconds (default: 300)
pools: list of pool names to monitor (default: all)
"""
name = "zfs_monitor"
description = "ZFS pool health, capacity, and I/O statistics"
interval = 300
def __init__(self, config: Optional[Dict[str, Any]] = None):
super().__init__(config)
self.interval = self.config.get("interval", 300)
self._pools_filter: Optional[List[str]] = self.config.get("pools", None)
async def initialize(self) -> bool:
if not shutil.which("zpool"):
self.skip_reason = "zpool not found"
return False
logger.info("ZFS monitor initialized (interval: %ds)", self.interval)
return True
async def _run(self, *args: str) -> List[str]:
"""Run a command and return its stdout lines, or [] on error."""
try:
proc = await asyncio.create_subprocess_exec(
*args,
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.DEVNULL,
)
stdout, _ = await asyncio.wait_for(proc.communicate(), timeout=15)
return stdout.decode(errors="replace").splitlines()
except (FileNotFoundError, asyncio.TimeoutError) as exc:
logger.warning("zfs_monitor: %s: %s", args[0], exc)
return []
async def _zpool_list(self) -> Dict[str, Dict]:
"""Return per-pool health and capacity from `zpool list`."""
lines = await self._run(
"zpool", "list", "-H", "-p",
"-o", "name,health,size,alloc,free,cap,frag,dedup",
)
pools: Dict[str, Dict] = {}
for line in lines:
parts = line.split("\t")
if len(parts) < 8:
continue
name = parts[0].strip()
if self._pools_filter and name not in self._pools_filter:
continue
pools[name] = {
"health": parts[1].strip(),
"size": _int(parts[2]),
"alloc": _int(parts[3]),
"free": _int(parts[4]),
"capacity": _float(parts[5]),
"frag": _float(parts[6]),
"dedup": _float(parts[7]),
}
return pools
async def _zpool_iostat(self) -> Dict[str, Dict]:
"""Return per-pool cumulative I/O counters from `zpool iostat`."""
lines = await self._run("zpool", "iostat", "-H", "-p")
io: Dict[str, Dict] = {}
for line in lines:
parts = line.split("\t")
if len(parts) < 7:
continue
name = parts[0].strip()
if not name or name.startswith(" "):
continue
io[name] = {
"read_ops": _int(parts[3]),
"write_ops": _int(parts[4]),
"read_bw": _int(parts[5]),
"write_bw": _int(parts[6]),
}
return io
async def _collect_metrics(self) -> Dict[str, Any]:
pools, io = await asyncio.gather(self._zpool_list(), self._zpool_iostat())
for name, stats in io.items():
if name in pools:
pools[name].update(stats)
return {"pools": pools}
plugin = ZFSMonitorPlugin
+8 -3
View File
@@ -52,11 +52,16 @@ def decode_value(val: str) -> Any:
except Exception: except Exception:
return val[1:] # Return as string without @ return val[1:] # Return as string without @
# Try numeric evaluation (original behavior) # Try numeric conversion (avoid eval to prevent SyntaxWarnings on version strings)
if val[0].isdigit() or (val[0] == '-' and len(val) > 1 and val[1].isdigit()): if val[0].isdigit() or (val[0] == '-' and len(val) > 1 and val[1].isdigit()):
try: try:
return eval(val) return int(val)
except Exception: except ValueError:
pass
try:
return float(val)
except ValueError:
pass
return val return val
return val return val
+198
View File
@@ -47,6 +47,48 @@ def build_parser():
help="Username (informational only, for display)", help="Username (informational only, for display)",
) )
# --- notify ---
notify_p = subparsers.add_parser(
"notify",
help="Send a test message via a configured notification channel",
)
notify_p.add_argument("-c", "--config", dest="configfile", help="Config file path (YAML)")
notify_p.add_argument(
"channel",
help="Channel name as defined in notification_channels",
)
notify_p.add_argument(
"message",
nargs="?",
default="Test notification from hbd",
help="Message body (default: 'Test notification from hbd')",
)
notify_p.add_argument(
"--level",
default="WARNING",
choices=["INFO", "WARNING", "CRITICAL", "RECOVER"],
help="Notification level (default: WARNING)",
)
notify_p.add_argument(
"--title",
default=None,
help="Notification title (default: '[LEVEL] test')",
)
# --- stop ---
stop_p = subparsers.add_parser("stop", help="Stop the running hbd instance")
stop_p.add_argument("-c", "--config", dest="configfile", help="Config file path (YAML)")
# --- reload ---
reload_p = subparsers.add_parser("reload", help="Reload configuration (SIGHUP)")
reload_p.add_argument("-c", "--config", dest="configfile", help="Config file path (YAML)")
# --- restart ---
restart_p = subparsers.add_parser("restart", help="Restart the running hbd instance")
restart_p.add_argument("-c", "--config", dest="configfile", help="Config file path (YAML)")
restart_p.add_argument("-f", "--foreground", action="store_true", help="Run in foreground after restart")
restart_p.add_argument("-v", "--verbose", action="store_true", help="Verbose output after restart")
return parser return parser
@@ -75,6 +117,146 @@ def cmd_passwd(args):
print(f" password: {hashed}") print(f" password: {hashed}")
def cmd_notify(args):
"""Send a test message via a single notification channel."""
from .config import load_config
from .notify import Notification, _dispatch_to_channel, setup
config = load_config(args.configfile)
setup(config)
channels = config.get("notification_channels", {})
if args.channel not in channels:
available = ", ".join(channels.keys()) if channels else "(none)"
print(f"Error: channel '{args.channel}' not found in notification_channels.", file=sys.stderr)
print(f"Available channels: {available}", file=sys.stderr)
sys.exit(1)
channel_cfg = channels[args.channel]
level = args.level.upper()
title = args.title or f"[{level}] test"
base_url = config.get("base_url", "").rstrip("/")
notif = Notification(
title=title,
body=args.message,
level=level,
url=f"{base_url}/plugins" if base_url else "",
)
import asyncio
from .notify import _send_matrix_async, _send_sms_voipms_async, _DRIVERS
ch_type = channel_cfg.get("type", "")
print(f"Sending via {args.channel} ({ch_type}): {title}{args.message}")
if ch_type == "matrix":
ok = asyncio.run(_send_matrix_async(channel_cfg, notif))
elif ch_type == "sms_voipms":
ok = asyncio.run(_send_sms_voipms_async(channel_cfg, notif))
else:
driver = _DRIVERS.get(ch_type)
if driver is None:
print(f"Error: unknown channel type '{ch_type}'", file=sys.stderr)
sys.exit(1)
ok = driver(channel_cfg, notif)
if ok:
print("OK")
else:
print("FAILED — check logs for details", file=sys.stderr)
sys.exit(1)
def _read_pid(configfile) -> int | None:
"""Return the PID from the pidfile, or None if not found / not running."""
import os
config = load_config(configfile)
pidfile = config.get("pidfile", "")
if not pidfile:
print("Error: no pidfile configured.", file=sys.stderr)
return None
try:
with open(pidfile) as f:
pid = int(f.read().strip())
# Verify process is actually running
os.kill(pid, 0)
return pid
except FileNotFoundError:
print(f"PID file not found ({pidfile}). Is hbd running?", file=sys.stderr)
return None
except ProcessLookupError:
print(f"PID file exists but process {pid} is not running.", file=sys.stderr)
return None
except Exception as e:
print(f"Error reading pidfile: {e}", file=sys.stderr)
return None
def cmd_stop(args):
import os, signal as _signal, time
pid = _read_pid(args.configfile)
if pid is None:
sys.exit(1)
print(f"Stopping hbd (pid {pid})...")
os.kill(pid, _signal.SIGTERM)
# Wait up to 10 s for the process to exit
for _ in range(20):
time.sleep(0.5)
try:
os.kill(pid, 0)
except ProcessLookupError:
print("hbd stopped.")
return
print("Warning: hbd did not stop within 10 seconds.", file=sys.stderr)
sys.exit(1)
def cmd_reload(args):
import os, signal as _signal
pid = _read_pid(args.configfile)
if pid is None:
sys.exit(1)
print(f"Sending SIGHUP to hbd (pid {pid})...")
os.kill(pid, _signal.SIGHUP)
print("Reload signal sent.")
def cmd_restart(args):
import os, signal as _signal, time, subprocess
pid = _read_pid(args.configfile)
if pid is not None:
print(f"Stopping hbd (pid {pid})...")
os.kill(pid, _signal.SIGTERM)
for _ in range(20):
time.sleep(0.5)
try:
os.kill(pid, 0)
except ProcessLookupError:
print("hbd stopped.")
break
else:
print("Warning: hbd did not stop within 10 seconds.", file=sys.stderr)
sys.exit(1)
else:
print("hbd does not appear to be running — starting fresh.")
# Re-launch hbd with the same config
cmd = [sys.executable, "-m", "hbd.server.cli", "serve"]
if args.configfile:
cmd += ["-c", args.configfile]
if getattr(args, "foreground", False):
cmd += ["-f"]
if getattr(args, "verbose", False):
cmd += ["-v"]
if getattr(args, "foreground", False):
# Run in foreground — replace current process
os.execv(sys.executable, cmd)
else:
subprocess.Popen(cmd, start_new_session=True)
print("hbd restarted.")
def main(argv=None): def main(argv=None):
parser = build_parser() parser = build_parser()
args = parser.parse_args(argv) args = parser.parse_args(argv)
@@ -83,6 +265,22 @@ def main(argv=None):
cmd_passwd(args) cmd_passwd(args)
return return
if args.command == "notify":
cmd_notify(args)
return
if args.command == "stop":
cmd_stop(args)
return
if args.command == "reload":
cmd_reload(args)
return
if args.command == "restart":
cmd_restart(args)
return
# Default: run the server (supports both `hbd serve ...` and `hbd ...`) # Default: run the server (supports both `hbd serve ...` and `hbd ...`)
config = load_config(args.configfile) config = load_config(args.configfile)
+10 -113
View File
@@ -17,12 +17,13 @@ SERVER_DEFAULTS = {
# Persistence # Persistence
"pickfile": os.path.join(os.path.expanduser("~"), ".hb.pick"), # File to store host state between restarts "pickfile": os.path.join(os.path.expanduser("~"), ".hb.pick"), # File to store host state between restarts
"pidfile": os.path.join(os.path.expanduser("~"), ".hb.pid"), # PID file for stop/restart/reload
# Logging # Logging
"logfile": os.path.join(os.path.expanduser("~"), ".hb.log"), "logfile": os.path.join(os.path.expanduser("~"), ".hb.log"),
# Notification channels # Notification channels
"notification_channels": {}, # Named channels with type and credentials "notification_channels": {}, # Named channels with type and credentials
"default_notification_channels": [], # Default channels if host doesn't specify "base_url": "", # Base URL for notification links (e.g. https://hbd.example.com)
# Monitoring settings # Monitoring settings
"interval": 20, # Expected heartbeat interval (for server checks) "interval": 20, # Expected heartbeat interval (for server checks)
@@ -34,8 +35,7 @@ SERVER_DEFAULTS = {
"default_owner": None, # Username that owns hosts with no explicit owner "default_owner": None, # Username that owns hosts with no explicit owner
# Host management # Host management
"hosts": {}, # New unified host definitions (optional) "hosts": {}, # Unified host definitions
"watchhosts": [], # Hosts to monitor and notify about (legacy)
"dyndnshosts": [], # Hosts with dynamic DNS (legacy) "dyndnshosts": [], # Hosts with dynamic DNS (legacy)
"drophosts": [], # Hosts to ignore "drophosts": [], # Hosts to ignore
"dyndomains": ["wrede.org"], "dyndomains": ["wrede.org"],
@@ -216,34 +216,18 @@ class ReloadableConfig:
def get_watchhosts(config): def get_watchhosts(config):
"""Extract watchhosts from config, supporting both new and legacy formats. """Extract watched hostnames from config (hosts with watch: true).
Args:
config: Configuration dictionary
Returns: Returns:
List of hostnames to watch List of hostnames to watch
""" """
watchhosts = [] watchhosts = []
hosts_config = config.get("hosts", {})
# New format: hosts section with watch attribute
if "hosts" in config:
hosts_config = config["hosts"]
if isinstance(hosts_config, dict): if isinstance(hosts_config, dict):
for host_name, host_attrs in hosts_config.items(): for host_name, host_attrs in hosts_config.items():
if isinstance(host_attrs, dict) and host_attrs.get("watch", False): if isinstance(host_attrs, dict) and host_attrs.get("watch", True):
watchhosts.append(host_name) watchhosts.append(host_name)
return watchhosts
# Legacy format: watchhosts list
if "watchhosts" in config:
legacy_watchhosts = config.get("watchhosts", [])
if isinstance(legacy_watchhosts, (list, set)):
watchhosts.extend(legacy_watchhosts)
elif isinstance(legacy_watchhosts, dict):
# Old dict format: {"host1": {attrs}, "host2": {attrs}}
watchhosts.extend(legacy_watchhosts.keys())
return list(set(watchhosts)) # Remove duplicates
def get_dyndnshosts(config): def get_dyndnshosts(config):
@@ -275,105 +259,18 @@ def get_dyndnshosts(config):
def get_host_config(config, hostname): def get_host_config(config, hostname):
"""Get configuration for a specific host. """Get configuration for a specific host from the hosts section.
Args:
config: Configuration dictionary
hostname: Host name
Returns: Returns:
Dictionary with host attributes or empty dict Dictionary with host attributes or empty dict
""" """
if "hosts" in config:
hosts_config = config.get("hosts", {}) hosts_config = config.get("hosts", {})
if isinstance(hosts_config, dict) and hostname in hosts_config: if isinstance(hosts_config, dict) and hostname in hosts_config:
return hosts_config[hostname] if isinstance(hosts_config[hostname], dict) else {} val = hosts_config[hostname]
return val if isinstance(val, dict) else {}
# Check legacy watchhosts for notification settings
if "watchhosts" in config:
watchhosts = config.get("watchhosts", {})
if isinstance(watchhosts, dict) and hostname in watchhosts:
legacy_attrs = watchhosts[hostname]
if isinstance(legacy_attrs, dict):
# Convert legacy format to new format
return {
"watch": True,
"notify": legacy_attrs.get("notify"),
"notify_src": legacy_attrs.get("src"),
}
return {} return {}
def get_notification_channels_for_host(config, hostname):
"""Get notification channels configured for a specific host.
Args:
config: Configuration dictionary
hostname: Host name
Returns:
List of channel names to use for this host
"""
host_config = get_host_config(config, hostname)
# Check if host specifies notification channels
channels = host_config.get("notification_channels", [])
if channels:
if isinstance(channels, str):
return [channels]
elif isinstance(channels, list):
return channels
# Fall back to default channels
default_channels = config.get("default_notification_channels", [])
if default_channels:
if isinstance(default_channels, str):
return [default_channels]
elif isinstance(default_channels, list):
return default_channels
# No channels configured, return empty list (will use legacy global config)
return []
def get_channel_config(config, channel_name):
"""Get configuration for a specific notification channel.
Args:
config: Configuration dictionary
channel_name: Name of the notification channel
Returns:
Dictionary with channel configuration or None if not found
"""
channels = config.get("notification_channels", {})
if isinstance(channels, dict) and channel_name in channels:
return channels[channel_name]
return None
def get_notification_channels_config(config, hostname):
"""Get list of notification channel configurations for a host.
Args:
config: Configuration dictionary
hostname: Host name
Returns:
List of (channel_name, channel_config) tuples
"""
channel_names = get_notification_channels_for_host(config, hostname)
channels = []
for channel_name in channel_names:
channel_config = get_channel_config(config, channel_name)
if channel_config and channel_config.get("type"):
channels.append((channel_name, channel_config))
return channels
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# User / host-access helpers # User / host-access helpers
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
+10 -1
View File
@@ -286,7 +286,7 @@ class Host:
Host.hosts[name] = self Host.hosts[name] = self
self.num = num self.num = num
self.dyn = False self.dyn = False
self.watched = False self.watched = True
self.upcount = 0 self.upcount = 0
self.interval = 0 self.interval = 0
self.doesack = -1 self.doesack = -1
@@ -304,6 +304,7 @@ class Host:
def statedict(self): def statedict(self):
d = {} d = {}
d["raw_name"] = self.name
d["name"] = self.name d["name"] = self.name
if self.dyn: if self.dyn:
d["name"] += "*" d["name"] += "*"
@@ -422,6 +423,14 @@ class Host:
ddict["managers"] = list(getattr(self, "managers", [])) ddict["managers"] = list(getattr(self, "managers", []))
ddict["monitors"] = list(getattr(self, "monitors", [])) ddict["monitors"] = list(getattr(self, "monitors", []))
# hbc version from latest os_info plugin data
hbc_version = None
latest_os = self.get_latest_plugin_data("os_info")
if latest_os:
_, os_data = latest_os
hbc_version = os_data.get("hbc_version")
ddict["hbc_version"] = hbc_version
return ddict return ddict
def jsons(self): def jsons(self):
+94 -23
View File
@@ -1,7 +1,11 @@
"""HTTP server implementation using aiohttp and jinja2.""" """HTTP server implementation using aiohttp and jinja2."""
import asyncio import asyncio
import datetime
import json import json
import platform
import socket
import sys
import time import time
import urllib.parse import urllib.parse
import os import os
@@ -12,6 +16,7 @@ from . import data
from . import notify as notify_mod from . import notify as notify_mod
from . import settings as settings_mod from . import settings as settings_mod
from . import users as users_mod from . import users as users_mod
from . import ws as ws_mod
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
@@ -110,6 +115,7 @@ async def start(
This function is intended to be awaited inside the main asyncio event loop. This function is intended to be awaited inside the main asyncio event loop.
""" """
get_now = get_now or (lambda: time.time()) get_now = get_now or (lambda: time.time())
_start_epoch = time.time()
async def old_index(request): async def old_index(request):
_require_auth_redirect(request) _require_auth_redirect(request)
@@ -209,15 +215,11 @@ async def start(
return err return err
qa = request.rel_url.query qa = request.rel_url.query
uname = urllib.parse.unquote(qa.get("h", "")) uname = urllib.parse.unquote(qa.get("h", ""))
ucode = qa.get("c") if not uname:
if not ucode or not uname: return web.Response(status=400, text="need h= argument")
return web.Response(status=400, text="need h= and c= arguments")
if uname != "All" and uname not in hbdclass.Host.hosts: if uname != "All" and uname not in hbdclass.Host.hosts:
return web.Response(status=400, text=f"h={uname} not found") return web.Response(status=400, text=f"h={uname} not found")
if uname != "All": names = [uname] if uname != "All" else list(hbdclass.Host.hosts)
names = [uname]
else:
names = [n for n in hbdclass.Host.hosts]
out = [] out = []
for n in names: for n in names:
host = hbdclass.Host.hosts[n] host = hbdclass.Host.hosts[n]
@@ -226,8 +228,7 @@ async def start(
continue continue
op_err = None op_err = None
try: try:
r = {"csum": None, "code": ucode} host.cmds.append(("UPD", {}))
host.cmds.append(("UPD", r))
except Exception as e: except Exception as e:
op_err = str(e) op_err = str(e)
out.append(f"update started for {n}: {op_err if op_err else 'OK'}") out.append(f"update started for {n}: {op_err if op_err else 'OK'}")
@@ -242,11 +243,12 @@ async def start(
env = jinja2.Environment(loader=jinja2.FileSystemLoader(templates_dir)) env = jinja2.Environment(loader=jinja2.FileSystemLoader(templates_dir))
host = config.get("hb_host", "localhost") host = config.get("hb_host", "localhost")
extra_scripts = config.get("http_extra_scripts", "") extra_scripts = config.get("http_extra_scripts", "")
host = request.host.split(":")[0] host = request.host # includes port if non-standard
if config.get("wss_port"): forwarded_proto = request.headers.get("X-Forwarded-Proto", "")
heartbeat_ws_url = f"wss://{host}:{config['wss_port']}/hbd" is_secure = request.secure or forwarded_proto.lower() == "https"
else: scheme = "wss" if is_secure else "ws"
heartbeat_ws_url = f"ws://{host}:{config.get('ws_port', 50005)}/hbd" heartbeat_ws_url = f"{scheme}://{host}/ws"
from hbd import __version__ as hbd_version
tmpl = env.get_template("live.html") tmpl = env.get_template("live.html")
body = tmpl.render( body = tmpl.render(
title="Heartbeat", title="Heartbeat",
@@ -254,8 +256,11 @@ async def start(
request=request, request=request,
heartbeat_ws_url=heartbeat_ws_url, heartbeat_ws_url=heartbeat_ws_url,
extra_scripts=extra_scripts, extra_scripts=extra_scripts,
hbd_version=hbd_version,
hosts=[ hosts=[
hbdclass.Host.hosts[h].stateinfo() for h in sorted(hbdclass.Host.hosts) hbdclass.Host.hosts[h].stateinfo()
for h in sorted(hbdclass.Host.hosts)
if _can_operate_host(current_user, hbdclass.Host.hosts[h])
], ],
messages=data.msgs[-30:], messages=data.msgs[-30:],
current_user=current_user.to_dict() if current_user else None, current_user=current_user.to_dict() if current_user else None,
@@ -507,7 +512,7 @@ async def start(
hosts_with_plugins = [] hosts_with_plugins = []
for hostname in sorted(hbdclass.Host.hosts.keys()): for hostname in sorted(hbdclass.Host.hosts.keys()):
host = hbdclass.Host.hosts[hostname] host = hbdclass.Host.hosts[hostname]
if not _can_view_host(current_user, host): if not _can_operate_host(current_user, host):
continue continue
if host.plugin_data: if host.plugin_data:
hosts_with_plugins.append({ hosts_with_plugins.append({
@@ -517,8 +522,8 @@ async def start(
tmpl = env.get_template("plugins.html") tmpl = env.get_template("plugins.html")
body = tmpl.render( body = tmpl.render(
title="Plugin Metrics - Heartbeat", title="Host Overview - Heartbeat",
header="Plugin Metrics", header="Host Overview",
hosts=hosts_with_plugins, hosts=hosts_with_plugins,
current_user=current_user.to_dict() if current_user else None, current_user=current_user.to_dict() if current_user else None,
active_page="plugins", active_page="plugins",
@@ -755,17 +760,39 @@ async def start(
templates_dir = config.get("templates_dir", os.path.join(pkg_dir, "templates")) templates_dir = config.get("templates_dir", os.path.join(pkg_dir, "templates"))
env = jinja2.Environment(loader=jinja2.FileSystemLoader(templates_dir)) env = jinja2.Environment(loader=jinja2.FileSystemLoader(templates_dir))
# Build host access summary for this user # Build host access summary for this user.
# Merge live hosts with config-only hosts (not yet seen) so the profile
# reflects the config file immediately after a reload.
from . import config as config_mod
owned, managed, monitored = [], [], [] owned, managed, monitored = [], [], []
if current_user: if current_user:
for hostname, host in sorted(hbdclass.Host.hosts.items()): # Collect all known hostnames: live + configured
if host.is_owner(current_user.username): cfg_hostnames = set(config.get("hosts", {}).keys())
live_hostnames = set(hbdclass.Host.hosts.keys())
all_hostnames = sorted(cfg_hostnames | live_hostnames)
for hostname in all_hostnames:
live_host = hbdclass.Host.hosts.get(hostname)
if live_host is not None:
# Use live object — it has apply_access already called
is_own = live_host.is_owner(current_user.username)
is_mgr = not is_own and live_host.is_manager(current_user.username)
is_mon = not is_own and not is_mgr and live_host.is_monitor(current_user.username)
else:
# Config-only host — read access directly from config
access = config_mod.get_host_access(config, hostname)
is_own = access["owner"] == current_user.username
is_mgr = current_user.username in access["managers"]
is_mon = current_user.username in access["monitors"]
if is_own:
owned.append(hostname) owned.append(hostname)
elif host.is_manager(current_user.username): elif is_mgr:
managed.append(hostname) managed.append(hostname)
elif host.is_monitor(current_user.username): elif is_mon:
monitored.append(hostname) monitored.append(hostname)
# Resolve notification channel configs for display # Resolve notification channel configs for display
notif_channels = [] notif_channels = []
if current_user: if current_user:
@@ -786,6 +813,48 @@ async def start(
) )
return web.Response(text=body, content_type="text/html") return web.Response(text=body, content_type="text/html")
# -------------------------------------------------------------------------
# About page
# -------------------------------------------------------------------------
async def about_page(request):
"""GET /about — version, runtime, and project information."""
current_user, _ = _require_auth_redirect(request)
pkg_dir = os.path.dirname(__file__)
templates_dir = config.get("templates_dir", os.path.join(pkg_dir, "templates"))
env = jinja2.Environment(loader=jinja2.FileSystemLoader(templates_dir))
from hbd import __version__ as hbd_version
uptime_secs = int(time.time() - _start_epoch)
days, rem = divmod(uptime_secs, 86400)
hours, rem = divmod(rem, 3600)
mins, secs = divmod(rem, 60)
if days:
uptime_str = f"{days}d {hours}h {mins}m"
elif hours:
uptime_str = f"{hours}h {mins}m {secs}s"
else:
uptime_str = f"{mins}m {secs}s"
start_dt = datetime.datetime.fromtimestamp(_start_epoch)
start_time_str = start_dt.strftime("%Y-%m-%d %H:%M:%S")
tmpl = env.get_template("about.html")
body = tmpl.render(
title="About - Heartbeat",
header="About",
hbd_version=hbd_version,
python_version=f"{sys.version_info.major}.{sys.version_info.minor}.{sys.version_info.micro} ({platform.python_implementation()})",
server_hostname=socket.gethostname(),
start_epoch=int(_start_epoch),
start_time_str=start_time_str,
uptime_str=uptime_str,
host_count=len(hbdclass.Host.hosts),
current_user=current_user.to_dict() if current_user else None,
active_page="about",
)
return web.Response(text=body, content_type="text/html")
# ------------------------------------------------------------------------- # -------------------------------------------------------------------------
# Settings page (admin only) # Settings page (admin only)
# ------------------------------------------------------------------------- # -------------------------------------------------------------------------
@@ -839,10 +908,12 @@ async def start(
web.get("/live", live), web.get("/live", live),
web.get("/plugins", plugins_page), web.get("/plugins", plugins_page),
web.get("/alerts", alerts_page), web.get("/alerts", alerts_page),
web.get("/about", about_page),
web.get("/profile", profile_page), web.get("/profile", profile_page),
web.get("/settings", settings_page), web.get("/settings", settings_page),
web.get("/static/{path:.*}", static), web.get("/static/{path:.*}", static),
web.get("/favicon.ico", favicon), web.get("/favicon.ico", favicon),
web.get("/ws", ws_mod.handler),
] ]
) )
+24 -38
View File
@@ -162,7 +162,7 @@ async def _run_async(config, config_path=None):
from . import journal as journal_mod from . import journal as journal_mod
from . import threshold as threshold_mod from . import threshold as threshold_mod
notify_mod.setup(config) notify_mod.setup(config, loop=loop)
# Initialize message journal # Initialize message journal
msg_journal = journal_mod.get_journal(config) msg_journal = journal_mod.get_journal(config)
@@ -210,7 +210,6 @@ async def _run_async(config, config_path=None):
ctx = dict( ctx = dict(
config=config, config=config,
hbdclass=hbdclass, hbdclass=hbdclass,
log=eventlog,
msg_to_websockets=msg_to_websockets, msg_to_websockets=msg_to_websockets,
msg_journal=msg_journal, msg_journal=msg_journal,
threshold_checker=threshold_checker, threshold_checker=threshold_checker,
@@ -237,7 +236,6 @@ async def _run_async(config, config_path=None):
restore_ctx = dict( restore_ctx = dict(
config=config, config=config,
hbdclass=hbdclass, hbdclass=hbdclass,
log=eventlog,
msg_to_websockets=msg_to_websockets, msg_to_websockets=msg_to_websockets,
threshold_checker=threshold_checker, threshold_checker=threshold_checker,
) )
@@ -275,45 +273,17 @@ async def _run_async(config, config_path=None):
except Exception as e: except Exception as e:
logger.exception("dns worker failed to start: %s", e) logger.exception("dns worker failed to start: %s", e)
# Start the websocket servers as a background task # Register WebSocket state — connections are now served through /ws on the HTTP port
if config.get("wss_port", None): ws_task = None
ssl_context = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER) ws_mod.setup(
ssl_path = config.get("cert_path", "") loop=loop,
wss_pem = ssl_path + config.get("wss_pem", "")
wss_key = ssl_path + config.get("wss_key", "")
try:
ssl_context.load_cert_chain(wss_pem, keyfile=wss_key)
except FileNotFoundError:
logger.error("error: missing SSL keys %s or %s", wss_pem, wss_key)
sys.exit(1)
logger.info(
"Starting secure WebSocket server on port %s with cert %s",
config.get("wss_port", None),
wss_pem,
)
else:
ssl_context = None
try:
ws_port = config.get("ws_port", 50005)
logger.info("Starting WebSocket server on port %s", ws_port)
ws_task = asyncio.create_task(
ws_mod.start(
host=config.get("hbd_host", ""),
ws_port=ws_port,
wss_port=config.get("wss_port", None),
ssl_context=ssl_context,
get_hosts=lambda: [ get_hosts=lambda: [
hbdclass.Host.hosts[h].stateinfo() hbdclass.Host.hosts[h].stateinfo()
for h in sorted(hbdclass.Host.hosts) for h in sorted(hbdclass.Host.hosts)
], ],
# get_msgs=lambda: msgs, verbose=config.get("verbose", False),
config=config,
) )
) logger.info("WebSocket handler registered on /ws (HTTP port %s)", config.get("hbd_port", 50004))
logger.info("WebSocket task started")
except Exception as e:
logger.exception("websocket server failed to start: %s", e)
# Periodic autosave task # Periodic autosave task
autosave_interval = config.get("autosave_interval", 300) # default: 5 minutes autosave_interval = config.get("autosave_interval", 300) # default: 5 minutes
@@ -375,7 +345,7 @@ async def _run_async(config, config_path=None):
except Exception as e: except Exception as e:
logger.warning("Error closing UDP transport: %s", e) logger.warning("Error closing UDP transport: %s", e)
tasks_to_cancel = [http_task, ws_task, autosave] tasks_to_cancel = [http_task, autosave]
for task in tasks_to_cancel: for task in tasks_to_cancel:
if task: if task:
try: try:
@@ -503,6 +473,16 @@ def run(config, config_path=None):
notify_mod.initlog(logfile=config.get("logfile", "messages.log")) notify_mod.initlog(logfile=config.get("logfile", "messages.log"))
users_mod.load_users(config) users_mod.load_users(config)
# Write pidfile
pidfile = config.get("pidfile", "")
if pidfile:
try:
with open(pidfile, "w") as f:
f.write(str(os.getpid()))
except Exception as e:
logger.warning("Failed to write pidfile %s: %s", pidfile, e)
eventlog(None, "INFO", f"hbd version {__version__} starting up") eventlog(None, "INFO", f"hbd version {__version__} starting up")
if config_path: if config_path:
@@ -525,6 +505,12 @@ def run(config, config_path=None):
logger.info("hbd shutdown complete") logger.info("hbd shutdown complete")
eventlog(None, "INFO", f"hbd version {__version__} shutdown") eventlog(None, "INFO", f"hbd version {__version__} shutdown")
notify_mod.closelog() notify_mod.closelog()
# Remove pidfile
if pidfile:
try:
os.unlink(pidfile)
except Exception:
pass
# Explicitly close the loop # Explicitly close the loop
try: try:
# Cancel all remaining tasks # Cancel all remaining tasks
+377 -220
View File
@@ -1,37 +1,100 @@
"""Notification helpers: email, pushover, mattermost, signal and dispatcher.""" """Notification helpers: email, pushover, matrix, mattermost, signal, sms and dispatcher.
Channel types supported:
pushover - Pushover app notifications
email - SMTP email
matrix - Matrix (via matrix-nio)
mattermost - Mattermost webhook
signal - Signal via signal-cli subprocess
sms_voipms - SMS via voip.ms REST API
Each channel can specify ``min_level: WARNING|CRITICAL`` (default: WARNING).
Notifications are dispatched to the owner + managers of the host, each via
their own ``notification_channels`` list. When no users are configured the
server runs silently (no notifications sent).
"""
import asyncio
import logging import logging
from typing import Optional
import http.client
import urllib.parse
import subprocess
import smtplib import smtplib
import subprocess
import time import time
import sys import sys
from dataclasses import dataclass, field
from typing import Optional
from . import data from . import data
from . import ws as ws_mod from . import ws as ws_mod
from . import main as main_mod
DEFAULT_PUSHPROVIDERS = ["all", "pushover", "mattermost", "signal"]
msg_to_websockets = ws_mod.broadcast
# module-level configuration set via setup()
_config = {}
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
msg_to_websockets = ws_mod.broadcast
# Module-level state set via setup()
_config: dict = {}
# Tracks which channels fired a WARNING/CRITICAL per host.
# {host_name: set of channel_names} — used to route RECOVER to the same channels.
_alerted_channels: dict = {}
logf = None logf = None
# ---------------------------------------------------------------------------
# Level ordering
# ---------------------------------------------------------------------------
_LEVEL_ORDER = {"RECOVER": 0, "INFO": 0, "WARNING": 1, "CRITICAL": 2}
def _level_value(level: str) -> int:
return _LEVEL_ORDER.get(level.upper(), 0)
# ---------------------------------------------------------------------------
# Notification dataclass
# ---------------------------------------------------------------------------
@dataclass
class Notification:
"""Structured notification payload."""
title: str # e.g. "[CRITICAL] webserver01"
body: str # detail message
level: str # RECOVER | WARNING | CRITICAL | INFO
url: str = "" # link to plugin metrics page
# ---------------------------------------------------------------------------
# Module setup
# ---------------------------------------------------------------------------
def setup(cfg: dict, loop: Optional[asyncio.AbstractEventLoop] = None):
"""Initialize notifier from configuration dict."""
global _config
_config = dict(cfg)
def reload_config(cfg: dict):
"""Reload notification configuration on SIGHUP."""
global _config
_config = dict(cfg)
logger.info("Notification configuration reloaded")
# ---------------------------------------------------------------------------
# Event log (websocket + file + in-memory)
# ---------------------------------------------------------------------------
def initlog(logfile): def initlog(logfile):
global logf global logf
try: try:
logf = open(logfile, "a+") logf = open(logfile, "a+")
except Exception as e: except Exception as e:
import sys
print("cannot open logfile %s, using STDERR: %s" % (logfile, e)) print("cannot open logfile %s, using STDERR: %s" % (logfile, e))
logf = sys.stderr logf = sys.stderr
return logf return logf
def closelog(): def closelog():
global logf global logf
if logf and logf != sys.stderr: if logf and logf != sys.stderr:
@@ -40,6 +103,7 @@ def closelog():
except Exception: except Exception:
pass pass
def eventlog(host, lvl, m, service=None): def eventlog(host, lvl, m, service=None):
ts = time.time() ts = time.time()
s = f"{time.strftime('%Y-%m-%d %H:%M:%S', time.localtime(ts))} {lvl} " s = f"{time.strftime('%Y-%m-%d %H:%M:%S', time.localtime(ts))} {lvl} "
@@ -56,91 +120,29 @@ def eventlog(host, lvl, m, service=None):
logger.warning("failed to write to logfile: %s", e) logger.warning("failed to write to logfile: %s", e)
msg_to_websockets("message", s) msg_to_websockets("message", s)
def setup(cfg: dict):
"""Initialize notifier defaults from a configuration dict."""
global _config
_config = dict(cfg)
# ---------------------------------------------------------------------------
# Low-level channel drivers
# ---------------------------------------------------------------------------
def reload_config(cfg: dict): def _send_pushover(channel_cfg: dict, notif: Notification) -> bool:
"""Reload notification configuration. import http.client
import urllib.parse
This function updates the module-level notification configuration token = channel_cfg.get("token", "")
during runtime config reloads. user = channel_cfg.get("user", "")
if not token or not user:
Args: logger.warning("pushover: missing token or user")
cfg: New configuration dictionary
"""
global _config
_config = dict(cfg)
logger.info("Notification configuration reloaded")
def send_email(toaddrs, smtpserver, sender, subject, body, debug=0):
"""Send a plain email via SMTP. Returns True on success."""
try:
smtpport = _config.get("smtpport", 587)
server = smtplib.SMTP(smtpserver, smtpport)
if debug > 0:
server.set_debuglevel(1)
if smtpport == 587:
server.starttls()
server.ehlo()
smtpuser = _config.get("smtpuser", None)
smtppassword = _config.get("smtppassword", None)
if smtpuser and smtppassword:
server.login(smtpuser, smtppassword)
server.sendmail(sender, toaddrs, body)
except Exception as e:
logger.warning("email send failed: %s", e)
try:
server.quit()
except Exception:
pass
return False return False
try: params: dict = {"token": token, "user": user, "title": notif.title, "message": notif.body}
server.quit() if notif.url:
except Exception: params["url"] = notif.url
pass params["url_title"] = "Plugin metrics"
return True
def email(subject: str, msg: str, debug: int = 0) -> bool:
"""Convenience wrapper exposed to the rest of the application.
Uses module-level configuration to supply recipient list, smtp server
and sender address.
"""
toaddrs = _config.get("toemail")
fromemail = _config.get("fromemail")
smtpserver = _config.get("smtpserver")
if not toaddrs or not fromemail or not smtpserver:
logger.warning(
"email config incomplete: toemail=%s, fromemail=%s, smtpserver=%s",
toaddrs,
fromemail,
smtpserver,
)
return False
date = time.strftime("%a, %d %b %Y %H:%M:%S %z", time.localtime())
body = "To: %s\nFrom: %s\nSubject: %s\nDate: %s\n\n%s" % (
toaddrs[0] if toaddrs else "",
fromemail,
subject,
date,
msg,
)
return send_email(toaddrs, smtpserver, fromemail, subject, body, debug=debug)
def pushover(token: str, user: str, msg: str, debug: int = 0) -> bool:
"""Send message via Pushover API."""
conn = http.client.HTTPSConnection("api.pushover.net:443") conn = http.client.HTTPSConnection("api.pushover.net:443")
try: try:
conn.request( conn.request(
"POST", "POST",
"/1/messages.json", "/1/messages.json",
urllib.parse.urlencode({"token": token, "user": user, "message": msg}), urllib.parse.urlencode(params),
{"Content-type": "application/x-www-form-urlencoded"}, {"Content-type": "application/x-www-form-urlencoded"},
) )
r = conn.getresponse() r = conn.getresponse()
@@ -151,176 +153,331 @@ def pushover(token: str, user: str, msg: str, debug: int = 0) -> bool:
return False return False
def pushmattermost( def _send_email(channel_cfg: dict, notif: Notification) -> bool:
host: str, recipients = channel_cfg.get("recipients", [])
token: str, sender = channel_cfg.get("sender", "")
channel: str, smtp_server = channel_cfg.get("smtp_server", "")
msg: str, smtp_port = channel_cfg.get("smtp_port", 587)
username: str = "hbd", smtp_user = channel_cfg.get("smtp_user")
icon: Optional[str] = None, smtp_password = channel_cfg.get("smtp_password")
debug: int = 0,
) -> bool:
"""Send a message to Mattermost via simple webhook driver if available.
This helper tries to import mattermostdriver.Driver and uses webhooks if present. if not recipients or not sender or not smtp_server:
If the import fails it returns False. logger.warning("email: missing recipients, sender, or smtp_server")
""" return False
date = time.strftime("%a, %d %b %Y %H:%M:%S %z", time.localtime())
body_text = notif.body
if notif.url:
body_text += f"\n\n{notif.url}"
raw = "To: %s\nFrom: %s\nSubject: %s\nDate: %s\n\n%s" % (
recipients[0] if isinstance(recipients, list) else recipients,
sender,
notif.title,
date,
body_text,
)
try:
server = smtplib.SMTP(smtp_server, smtp_port)
if smtp_port == 587:
server.starttls()
server.ehlo()
if smtp_user and smtp_password:
server.login(smtp_user, smtp_password)
server.sendmail(sender, recipients, raw)
server.quit()
return True
except Exception as e:
logger.warning("email send failed: %s", e)
try:
server.quit()
except Exception:
pass
return False
def _send_mattermost(channel_cfg: dict, notif: Notification) -> bool:
try: try:
from mattermostdriver import Driver from mattermostdriver import Driver
except Exception: except ImportError:
logger.error("mattermostdriver not installed")
return False return False
host = channel_cfg.get("host", "")
token = channel_cfg.get("token", "")
channel = channel_cfg.get("channel", "")
if not host or not token or not channel:
logger.warning("mattermost: missing host, token, or channel")
return False
text = f"**{notif.title}**\n{notif.body}"
if notif.url:
text += f"\n[Plugin metrics]({notif.url})"
ses = {"url": host, "scheme": "http", "basepath": "/api/v4", "port": 8065} ses = {"url": host, "scheme": "http", "basepath": "/api/v4", "port": 8065}
mm = Driver(ses) mm = Driver(ses)
payload = {"text": msg, "channel": channel, "username": username} payload: dict = {"text": text, "channel": channel, "username": channel_cfg.get("username", "hbd")}
icon = channel_cfg.get("icon")
if icon: if icon:
payload["icon_url"] = icon payload["icon_url"] = icon
try: try:
rc = mm.webhooks.call_webhook(token, payload) rc = mm.webhooks.call_webhook(token, payload)
logger.debug("mattermost rc: %s", rc)
return bool(rc is None or rc == "") return bool(rc is None or rc == "")
except Exception as e: except Exception as e:
logger.error("mattermost error: %s", e) logger.error("mattermost error: %s", e)
return False return False
def pushsignal( def _send_signal(channel_cfg: dict, notif: Notification) -> bool:
signal_cli_bin: str, user: str, recipient: str, msg: str, debug: int = 0 cli = channel_cfg.get("cli_path", "/usr/local/bin/signal-cli")
) -> bool: user = channel_cfg.get("user", "")
"""Send a message via signal-cli (requires local installation). recipient = channel_cfg.get("recipient", "")
if not user or not recipient:
Uses subprocess to call signal-cli. Returns True if the command succeeded. logger.warning("signal: missing user or recipient")
""" return False
CLI = [signal_cli_bin, "-u", user, "send", "-m", msg, recipient] msg = f"{notif.title}\n{notif.body}"
logger.debug("signal cli: %s", CLI) if notif.url:
try: msg += f"\n{notif.url}"
res = subprocess.run(CLI, capture_output=True) try:
if res.returncode != 0: res = subprocess.run([cli, "-u", user, "send", "-m", msg, recipient], capture_output=True)
logger.error("signal failed: %s".res.stderr.decode()) if res.returncode != 0:
logger.error("signal failed: %s", res.stderr.decode())
return False return False
logger.debug("signal sent: %s", res.stdout.decode())
return True return True
except Exception as e: except Exception as e:
logger.exception("signal exception: %s", e) logger.exception("signal exception: %s", e)
return False return False
def _dispatch_to_channel(channel_name: str, channel_config: dict, msg: str, debug: int = 0) -> bool: async def _send_sms_voipms_async(channel_cfg: dict, notif: Notification) -> bool:
"""Dispatch a message to a specific notification channel. """Send SMS via voip.ms REST API using multipart form-data POST."""
import json
import aiohttp
Args: api_user = channel_cfg.get("api_user", "")
channel_name: Name of the channel (for logging) api_password = channel_cfg.get("api_password", "")
channel_config: Channel configuration dictionary with 'type' and type-specific fields did = channel_cfg.get("did", "")
msg: Message to send dst = channel_cfg.get("dst", "")
debug: Debug level if not api_user or not api_password or not did or not dst:
logger.warning("sms_voipms: missing api_user, api_password, did, or dst")
Returns:
True if notification sent successfully, False otherwise
"""
channel_type = channel_config.get("type")
if channel_type == "pushover":
return pushover(
channel_config.get("token", ""),
channel_config.get("user", ""),
msg,
debug=debug
)
elif channel_type == "email":
# Build email from channel config
recipients = channel_config.get("recipients", [])
sender = channel_config.get("sender", "")
smtp_server = channel_config.get("smtp_server", "")
smtp_port = channel_config.get("smtp_port", 587)
smtp_user = channel_config.get("smtp_user")
smtp_password = channel_config.get("smtp_password")
if not recipients or not sender or not smtp_server:
logger.warning(
"Email channel '%s' missing required fields: recipients=%s, sender=%s, smtp_server=%s",
channel_name, recipients, sender, smtp_server
)
return False return False
# Temporarily update _config for email() function # SMS body: title + body, truncated to 160 chars
old_config = dict(_config) text = f"{notif.title}: {notif.body}"
_config["toemail"] = recipients if len(text) > 160:
_config["fromemail"] = sender text = text[:157] + "..."
_config["smtpserver"] = smtp_server
_config["smtpport"] = smtp_port
if smtp_user:
_config["smtpuser"] = smtp_user
if smtp_password:
_config["smtppassword"] = smtp_password
result = email("Heartbeat notification", msg, debug=debug) form_data = {
"api_username": api_user,
"api_password": api_password,
"method": "sendSMS",
"did": did,
"dst": dst,
"message": text,
}
# Restore config try:
_config.clear() async with aiohttp.ClientSession() as session:
_config.update(old_config) with aiohttp.MultipartWriter("form-data") as mp:
for key, value in form_data.items():
return result part = mp.append(value)
part.set_content_disposition("form-data", name=key)
elif channel_type == "signal": async with session.post("https://voip.ms/api/v1/rest.php", data=mp) as resp:
return pushsignal( body = await resp.text()
channel_config.get("cli_path", "/usr/local/bin/signal-cli"), if resp.status != 200:
channel_config.get("user", ""), logger.error("sms_voipms HTTP %s: %s", resp.status, body)
channel_config.get("recipient", ""), return False
msg, result = json.loads(body)
debug=debug if result.get("status") == "success":
) return True
logger.error("sms_voipms error: %s", result.get("status"))
elif channel_type == "mattermost": return False
return pushmattermost( except Exception as e:
channel_config.get("host", ""), logger.error("sms_voipms exception: %s", e)
channel_config.get("token", ""),
channel_config.get("channel", ""),
msg,
username=channel_config.get("username", "hbd"),
icon=channel_config.get("icon"),
debug=debug
)
else:
logger.warning("Unknown channel type '%s' for channel '%s'", channel_type, channel_name)
return False return False
def pushmsg_for_host(hostname: str, msg: str, debug: int = 0) -> dict:
"""Send notification for a specific host using its configured channels.
This function looks up the host's notification channels from the config
and sends the message to those channels.
Args: async def _send_matrix_async(channel_cfg: dict, notif: Notification) -> bool:
hostname: Name of the host to send notification for """Send a Matrix message using matrix-nio."""
msg: Message to send try:
debug: Debug level from nio import AsyncClient, RoomMessageText # noqa: F401
except ImportError:
logger.error("matrix-nio not installed; pip install matrix-nio")
return False
Returns: from nio import AsyncClient
Dictionary of results per channel: {"channel_name": True/False} homeserver = channel_cfg.get("homeserver", "")
access_token = channel_cfg.get("access_token", "")
room_id = channel_cfg.get("room_id", "")
if not homeserver or not access_token or not room_id:
logger.warning("matrix: missing homeserver, access_token, or room_id")
return False
text = f"{notif.title}\n{notif.body}"
if notif.url:
text += f"\n{notif.url}"
html = f"<strong>{notif.title}</strong><br>{notif.body}"
if notif.url:
html += f'<br><a href="{notif.url}">Plugin metrics</a>'
client = AsyncClient(homeserver)
client.access_token = access_token
try:
from nio import RoomSendResponse
content = {
"msgtype": "m.text",
"body": text,
"format": "org.matrix.custom.html",
"formatted_body": html,
}
resp = await client.room_send(room_id, "m.room.message", content)
if hasattr(resp, "event_id"):
return True
logger.error("matrix send failed: %s", resp)
return False
except Exception as e:
logger.error("matrix exception: %s", e)
return False
finally:
await client.close()
# ---------------------------------------------------------------------------
# Channel dispatcher (all async — sync drivers run in a thread executor)
# ---------------------------------------------------------------------------
# Sync drivers kept for `hbd notify` CLI usage (asyncio.run wraps them there).
_DRIVERS = {
"pushover": _send_pushover,
"email": _send_email,
"mattermost": _send_mattermost,
"signal": _send_signal,
}
_TIMEOUT = 15 # seconds per channel send
async def _dispatch_to_channel(channel_name: str, channel_cfg: dict, notif: Notification) -> bool:
"""Send *notif* to a single named channel, honouring min_level."""
level = notif.level.upper()
if level != "RECOVER":
min_level = channel_cfg.get("min_level", "WARNING").upper()
if _level_value(level) < _level_value(min_level):
logger.debug(
"channel '%s': skipping level %s (min_level=%s)", channel_name, level, min_level
)
return True # filtered intentionally
ch_type = channel_cfg.get("type", "")
try:
if ch_type == "matrix":
return await asyncio.wait_for(_send_matrix_async(channel_cfg, notif), timeout=_TIMEOUT)
if ch_type == "sms_voipms":
return await asyncio.wait_for(_send_sms_voipms_async(channel_cfg, notif), timeout=_TIMEOUT)
sync_driver = _DRIVERS.get(ch_type)
if sync_driver is None:
logger.warning("unknown channel type '%s' for channel '%s'", ch_type, channel_name)
return False
return await asyncio.wait_for(
asyncio.to_thread(sync_driver, channel_cfg, notif), timeout=_TIMEOUT
)
except asyncio.TimeoutError:
logger.error("channel '%s' timed out after %ds", channel_name, _TIMEOUT)
return False
# ---------------------------------------------------------------------------
# Central dispatch function
# ---------------------------------------------------------------------------
def _build_url(host_name: str) -> str:
base_url = _config.get("base_url", "").rstrip("/")
if not base_url:
return ""
return f"{base_url}/plugins#{host_name}"
async def send_notification(host_name: str, notif: Notification) -> dict:
"""Dispatch *notif* to all managers/owner of *host_name*.
Looks up the host's owner + managers, resolves each user's
notification_channels, and dispatches. Silently does nothing if
no users are configured.
Returns a dict of {channel_name: bool} results.
""" """
from . import config as config_mod from . import users as users_mod
from . import hbdclass
# Get notification channels for this host if not users_mod.users_enabled():
channels = config_mod.get_notification_channels_config(_config, hostname)
if not channels:
logger.warning("No notification channels configured for host '%s'", hostname)
return {} return {}
# Dispatch to each channel # Collect recipient usernames: owner + managers
results = {} host = hbdclass.Host.hosts.get(host_name)
for channel_name, channel_config in channels: if host is None:
logger.debug("send_notification: host '%s' not found", host_name)
return {}
recipients: set[str] = set()
owner = getattr(host, "owner", None)
if owner:
recipients.add(owner)
for m in getattr(host, "managers", []):
recipients.add(m)
if not recipients:
logger.debug("send_notification: no owner/managers for '%s'", host_name)
return {}
# Fill url if not already set
if not notif.url:
notif.url = _build_url(host_name)
global_channels: dict = _config.get("notification_channels", {})
results: dict = {}
level = notif.level.upper()
is_alert = level in ("WARNING", "CRITICAL")
is_recover = level in ("RECOVER",)
# For RECOVER: send to every channel that previously fired an alert for this host,
# regardless of that channel's min_level.
if is_recover and host_name in _alerted_channels:
for channel_name in list(_alerted_channels[host_name]):
channel_cfg = global_channels.get(channel_name)
if not channel_cfg:
continue
try: try:
success = _dispatch_to_channel(channel_name, channel_config, msg, debug=debug) ok = await _dispatch_to_channel(channel_name, channel_cfg, notif)
results[channel_name] = success results[channel_name] = ok
if success: if ok:
logger.info("Notification sent to channel '%s': %s", channel_name, msg) logger.info("recover sent to channel '%s': %s", channel_name, notif.title)
else:
logger.warning("Failed to send notification to channel '%s'", channel_name)
except Exception as e: except Exception as e:
logger.error("Error sending to channel '%s': %s", channel_name, e) logger.error("error sending recover to channel '%s': %s", channel_name, e)
del _alerted_channels[host_name]
return results
for username in recipients:
user = users_mod.get_user(username)
if user is None:
logger.debug("send_notification: user '%s' not found", username)
continue
for channel_name in user.notification_channels:
if channel_name in results:
continue
channel_cfg = global_channels.get(channel_name)
if not channel_cfg:
logger.warning("channel '%s' not defined in notification_channels", channel_name)
results[channel_name] = False
continue
try:
ok = await _dispatch_to_channel(channel_name, channel_cfg, notif)
results[channel_name] = ok
if ok:
logger.info("notification sent to channel '%s': %s", channel_name, notif.title)
if is_alert:
_alerted_channels.setdefault(host_name, set()).add(channel_name)
else:
logger.warning("failed to send notification to channel '%s'", channel_name)
except Exception as e:
logger.error("error sending to channel '%s': %s", channel_name, e)
results[channel_name] = False results[channel_name] = False
return results return results
+2 -2
View File
@@ -24,7 +24,7 @@ sensitive bool True when the raw value must never be shown
# Credential field names that should always be masked. # Credential field names that should always be masked.
_SECRET_KEYS = frozenset({ _SECRET_KEYS = frozenset({
"password", "token", "user_key", "api_key", "secret", "password", "token", "user_key", "api_key", "secret",
"smtp_password", "smtp_user", "smtp_password", "smtp_user", "api_password", "access_token",
}) })
_CHANNEL_TYPE_LABELS = { _CHANNEL_TYPE_LABELS = {
@@ -188,7 +188,7 @@ def get_settings_sections(config: dict) -> list:
continue continue
hosts_list.append({ hosts_list.append({
"name": hname, "name": hname,
"watch": bool(hcfg.get("watch", False)), "watch": bool(hcfg.get("watch", True)),
"dyndns": bool(hcfg.get("dyndns", False)), "dyndns": bool(hcfg.get("dyndns", False)),
"owner": hcfg.get("owner", ""), "owner": hcfg.get("owner", ""),
"managers": hcfg.get("managers", []), "managers": hcfg.get("managers", []),
+64
View File
@@ -140,4 +140,68 @@
float: left; float: left;
} }
/* ── Responsive / mobile ── */
/* Suppress the global transition on mobile to avoid sluggish feel */
@media (max-width: 640px) {
* { transition: none !important; }
html, body {
overflow: auto;
height: auto;
font-size: 16px; /* prevent iOS auto-zoom on inputs */
}
/* Pages that use flex-column full-viewport layout need to relax on mobile */
body[style*="height: 100vh"],
body {
height: auto !important;
min-height: 100vh;
}
/* Containers: full width, no fixed heights */
.container {
max-width: 100% !important;
max-height: none !important;
overflow: visible !important;
padding: 8px !important;
}
/* Log section: fixed reasonable height instead of flex-grow */
.log-section {
flex: none !important;
max-height: 40vh !important;
overflow-y: auto !important;
}
/* Table section: allow vertical scroll, cap height */
.table-section {
max-height: 55vh !important;
overflow-y: auto !important;
overflow-x: auto !important;
padding: 8px !important;
}
/* Slightly larger tap targets in tables */
#ntable td, #ntable th {
padding: 4px 6px !important;
font-size: 0.82em !important;
}
/* Cards on plugin/alerts pages */
.host-card, .alert-card, .card {
padding: 10px !important;
margin-bottom: 8px !important;
}
/* Settings page tables */
table { width: 100%; }
h1 { font-size: 1.2em !important; }
h2 { font-size: 1em !important; }
}
/* Suppress nav-username text on very narrow screens — avatar/initials is enough */
@media (max-width: 400px) {
.nav-username { display: none; }
}
+199
View File
@@ -0,0 +1,199 @@
<!DOCTYPE html>
<html>
{% include 'head.html' %}
<style>
html, body { overflow: visible; }
.container {
max-width: 700px;
margin: 0 auto;
}
h1 {
color: #333;
margin-bottom: 4px;
font-size: 1.5em;
}
.subtitle {
color: #666;
margin-bottom: 24px;
font-size: 0.9em;
}
.section {
background: #fff;
border-radius: 8px;
box-shadow: 0 1px 6px rgba(0,0,0,0.1);
padding: 20px 24px;
margin-bottom: 20px;
}
.section h2 {
font-size: 1em;
font-weight: 700;
color: #333;
margin: 0 0 16px;
padding-bottom: 10px;
border-bottom: 1px solid #eee;
text-transform: uppercase;
letter-spacing: 0.5px;
}
.info-row {
display: flex;
align-items: baseline;
padding: 8px 0;
border-bottom: 1px solid #f5f5f5;
font-size: 0.9em;
}
.info-row:last-child { border-bottom: none; }
.info-label {
width: 160px;
flex-shrink: 0;
color: #666;
font-size: 0.88em;
}
.info-value {
color: #222;
word-break: break-all;
}
.info-value a {
color: #0066cc;
text-decoration: none;
}
.info-value a:hover { text-decoration: underline; }
.version-badge {
display: inline-block;
padding: 3px 12px;
background: #e8f0fe;
color: #1a73e8;
border-radius: 12px;
font-size: 0.85em;
font-weight: 600;
font-family: monospace;
}
.hb-logo {
font-size: 2.5em;
font-weight: 700;
color: #0066cc;
letter-spacing: -1px;
margin-bottom: 6px;
}
.hb-tagline {
color: #555;
font-size: 0.95em;
}
.logo-section {
display: flex;
align-items: center;
gap: 20px;
padding: 8px 0 4px;
}
.logo-text { flex: 1; }
</style>
<body>
{% include 'nav.html' %}
<div class="container">
<h1>{{ header }}</h1>
<p class="subtitle">Heartbeat monitoring system</p>
<div class="section">
<div class="logo-section">
<div class="logo-text">
<div class="hb-logo">Heartbeat</div>
<div class="hb-tagline">Lightweight host monitoring over UDP</div>
</div>
<span class="version-badge">v{{ hbd_version }}</span>
</div>
</div>
<div class="section">
<h2>Version</h2>
<div class="info-row">
<span class="info-label">Server version</span>
<span class="info-value">{{ hbd_version }}</span>
</div>
<div class="info-row">
<span class="info-label">Python</span>
<span class="info-value">{{ python_version }}</span>
</div>
<div class="info-row">
<span class="info-label">License</span>
<span class="info-value">MIT</span>
</div>
</div>
<div class="section">
<h2>Runtime</h2>
<div class="info-row">
<span class="info-label">Host</span>
<span class="info-value">{{ server_hostname }}</span>
</div>
<div class="info-row">
<span class="info-label">Started</span>
<span class="info-value">{{ start_time_str }}</span>
</div>
<div class="info-row">
<span class="info-label">Uptime</span>
<span class="info-value" id="uptime-value">{{ uptime_str }}</span>
</div>
<div class="info-row">
<span class="info-label">Hosts monitored</span>
<span class="info-value">{{ host_count }}</span>
</div>
</div>
<div class="section">
<h2>Contact &amp; Source</h2>
<div class="info-row">
<span class="info-label">Author</span>
<span class="info-value">Andreas Wrede</span>
</div>
<div class="info-row">
<span class="info-label">Email</span>
<span class="info-value"><a href="mailto:aew@wrede.ca">aew@wrede.ca</a></span>
</div>
<div class="info-row">
<span class="info-label">Repository</span>
<span class="info-value"><a href="https://git.wrede.ca/andreas/heartbeat" target="_blank" rel="noopener">git.wrede.ca/andreas/heartbeat</a></span>
</div>
</div>
</div>
<script>
(function() {
var startEpoch = {{ start_epoch }};
var el = document.getElementById('uptime-value');
if (!el) return;
function fmt(s) {
var d = Math.floor(s / 86400);
var h = Math.floor((s % 86400) / 3600);
var m = Math.floor((s % 3600) / 60);
var sec = s % 60;
if (d > 0) return d + 'd ' + h + 'h ' + m + 'm';
if (h > 0) return h + 'h ' + m + 'm ' + sec + 's';
return m + 'm ' + sec + 's';
}
function tick() {
var up = Math.floor(Date.now() / 1000 - startEpoch);
el.textContent = fmt(up);
}
tick();
setInterval(tick, 1000);
})();
</script>
</body>
</html>
+22 -44
View File
@@ -3,20 +3,13 @@
{% include 'head.html' %} {% include 'head.html' %}
<style> <style>
body {
margin: 20px;
background: #f5f5f5;
}
.container { .container {
max-width: 1400px; max-width: 1400px;
margin: 0 auto; margin: 0 auto;
} }
h1 { h1 { color: #333; margin-bottom: 5px; margin-top: 15px; font-size: 1.5em; }
color: #333;
margin-bottom: 10px;
}
.subtitle { .subtitle {
color: #666; color: #666;
@@ -24,55 +17,40 @@
} }
.summary-cards { .summary-cards {
display: grid; display: flex;
grid-template-columns: repeat(auto-fit, minmax(200px, 1fr)); flex-wrap: wrap;
gap: 20px; gap: 10px;
margin-bottom: 30px; margin-bottom: 16px;
} }
.summary-card { .summary-card {
background: white; background: white;
border-radius: 8px; border-radius: 6px;
padding: 20px; padding: 6px 14px;
box-shadow: 0 2px 8px rgba(0,0,0,0.1); box-shadow: 0 1px 4px rgba(0,0,0,0.1);
text-align: center; display: flex;
align-items: center;
gap: 8px;
border-left: 4px solid #ddd;
} }
.summary-card.critical { .summary-card.critical { border-left-color: #ea1e0f; }
border-left: 5px solid #f44336; .summary-card.warning { border-left-color: #ff9800; }
} .summary-card.ok { border-left-color: #4caf50; }
.summary-card.warning {
border-left: 5px solid #ff9800;
}
.summary-card.ok {
border-left: 5px solid #4caf50;
}
.summary-number { .summary-number {
font-size: 3em; font-size: 1.4em;
font-weight: bold; font-weight: bold;
margin: 10px 0; line-height: 1;
} }
.summary-number.critical { .summary-number.critical { color: #ea1e0f; }
color: #f44336; .summary-number.warning { color: #ff9800; }
} .summary-number.ok { color: #4caf50; }
.summary-number.warning {
color: #ff9800;
}
.summary-number.ok {
color: #4caf50;
}
.summary-label { .summary-label {
color: #666; color: #666;
text-transform: uppercase; font-size: 0.85em;
font-size: 0.9em;
letter-spacing: 1px;
} }
.filters { .filters {
@@ -131,7 +109,7 @@
} }
.alert-item.acknowledged { .alert-item.acknowledged {
opacity: 0.6; opacity: 0.8;
background: #f0f0f0; background: #f0f0f0;
} }
+224 -4
View File
@@ -1,22 +1,44 @@
<head> <head>
<meta http-equiv="content-type" content="text/html; charset=utf-8" /> <meta http-equiv="content-type" content="text/html; charset=utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<link rel="stylesheet" href="/static/style.css" type="text/css" /> <link rel="stylesheet" href="/static/style.css" type="text/css" />
<link rel="icon" href="/static/images/favicon.ico" sizes="32x32" /> <link rel="icon" href="/static/images/favicon.ico" sizes="32x32" />
<title>{{ title }}</title> <title>{{ title }}</title>
{% if extra_scripts %}<script src="{{ extra_scripts }}"></script>{% endif %} {% if extra_scripts %}<script src="{{ extra_scripts }}"></script>{% endif %}
<style> <style>
/* ── Reset / shared baseline ── */
*, *::before, *::after { box-sizing: border-box; }
html {
font-family: 'Segoe UI', system-ui, -apple-system, sans-serif;
font-size: 14px;
}
body {
margin: 0;
padding: 10px;
padding-top: 60px;
background: #f5f5f5;
}
h1 { font-size: 1.5em; color: #333; margin: 0 0 5px; }
h2 { font-size: 1.1em; color: #333; margin: 0 0 8px; }
p { margin: 0; }
/* Navigation bar — shared across all pages */ /* Navigation bar — shared across all pages */
.nav { .nav {
position: fixed;
top: 0;
left: 0;
right: 0;
z-index: 200;
background: #fff; background: #fff;
padding: 10px 15px; padding: 6px 12px;
margin-bottom: 10px;
box-shadow: 0 2px 4px rgba(0,0,0,.1); box-shadow: 0 2px 4px rgba(0,0,0,.1);
border-radius: 4px;
display: flex; display: flex;
align-items: center; align-items: center;
justify-content: space-between; justify-content: space-between;
flex-wrap: wrap;
gap: 8px;
} }
.nav-links { display: flex; align-items: center; } .nav-links { display: flex; align-items: center; flex-wrap: wrap; gap: 4px; }
.nav a { .nav a {
margin-right: 20px; margin-right: 20px;
text-decoration: none; text-decoration: none;
@@ -39,6 +61,17 @@
transition: background 0.15s; transition: background 0.15s;
} }
.nav-user:hover { background: #f0f4ff; text-decoration: none; } .nav-user:hover { background: #f0f4ff; text-decoration: none; }
.nav-username {
max-width: 0;
overflow: hidden;
white-space: nowrap;
opacity: 0;
transition: max-width 0.2s ease, opacity 0.2s ease;
}
.nav-user:hover .nav-username {
max-width: 160px;
opacity: 1;
}
.nav-avatar { .nav-avatar {
width: 28px; height: 28px; width: 28px; height: 28px;
border-radius: 50%; border-radius: 50%;
@@ -57,5 +90,192 @@
font-weight: 700; font-weight: 700;
flex-shrink: 0; flex-shrink: 0;
} }
/* ── Mobile nav: hamburger toggle ── */
.nav-hamburger {
display: none;
flex-direction: column;
justify-content: space-between;
width: 26px; height: 20px;
cursor: pointer;
flex-shrink: 0;
background: none;
border: none;
padding: 0;
}
.nav-hamburger span {
display: block;
height: 3px;
background: #555;
border-radius: 2px;
}
@media (max-width: 640px) {
.nav-hamburger { display: flex; }
.nav-links {
display: none;
width: 100%;
flex-direction: column;
align-items: flex-start;
padding-top: 8px;
border-top: 1px solid #eee;
order: 3;
}
.nav-links.nav-open { display: flex; }
.nav-links a { margin-right: 0; padding: 6px 0; font-size: 1em; }
}
/* Swiss railway clock — nav */
.nav-clock {
flex-shrink: 0;
line-height: 0;
margin-left: auto;
padding: 4px 4px 4px 0;
cursor: pointer;
}
#swiss-clock { display: block; }
/* Swiss railway clock — full-page overlay */
#clock-overlay {
display: none;
position: fixed;
inset: 0;
z-index: 9999;
background: #1a1a1a;
align-items: center;
justify-content: center;
cursor: pointer;
}
#clock-overlay.visible { display: flex; }
#swiss-clock-overlay { display: block; }
</style> </style>
<script>
/* ── Swiss Federal Railway (SBB) clock ── */
/* Draw one frame of the clock onto any canvas element. */
function drawSwissClock(canvas) {
var SIZE = canvas.width;
var R = SIZE / 2;
var ctx = canvas.getContext('2d');
var now = new Date();
var h = now.getHours() % 12;
var m = now.getMinutes();
var s = now.getSeconds();
var ms = now.getMilliseconds();
/* Seconds hand idles ~1.5 s at 12 before advancing (SBB behaviour) */
var sFrac = s + ms / 1000;
var sAngle = sFrac >= 58.5 ? 0 : (sFrac / 58.5) * Math.PI * 2;
ctx.clearRect(0, 0, SIZE, SIZE);
/* face */
ctx.beginPath();
ctx.arc(R, R, R - 1, 0, Math.PI * 2);
ctx.fillStyle = '#fff';
ctx.fill();
ctx.strokeStyle = '#333';
ctx.lineWidth = SIZE * 0.018;
ctx.stroke();
/* tick marks */
for (var i = 0; i < 60; i++) {
var a = (i / 60) * Math.PI * 2 - Math.PI / 2;
var isHour = (i % 5 === 0);
ctx.beginPath();
ctx.moveTo(R + Math.cos(a) * (isHour ? R * 0.72 : R * 0.88),
R + Math.sin(a) * (isHour ? R * 0.72 : R * 0.88));
ctx.lineTo(R + Math.cos(a) * R * 0.94,
R + Math.sin(a) * R * 0.94);
ctx.strokeStyle = '#222';
ctx.lineWidth = isHour ? SIZE * 0.027 : SIZE * 0.011;
ctx.lineCap = 'butt';
ctx.stroke();
}
/* hands */
function hand(angle, tip, tail, width, color) {
ctx.save();
ctx.translate(R, R);
ctx.rotate(angle);
ctx.beginPath();
ctx.moveTo(tail, 0);
ctx.lineTo(tip, 0);
ctx.strokeStyle = color;
ctx.lineWidth = width;
ctx.lineCap = 'square';
ctx.stroke();
ctx.restore();
}
hand((m + s / 60) / 60 * Math.PI * 2 - Math.PI / 2,
R * 0.88, -R * 0.12, SIZE * 0.027, '#222'); /* minute */
hand((h + m / 60) / 12 * Math.PI * 2 - Math.PI / 2,
R * 0.58, -R * 0.12, SIZE * 0.039, '#222'); /* hour */
hand(sAngle - Math.PI / 2, R * 0.78, -R * 0.22,
SIZE * 0.013, '#e00'); /* second tail+tip */
/* round dot at tip of second hand */
var dotR = SIZE * 0.028;
ctx.save();
ctx.translate(R, R);
ctx.rotate(sAngle - Math.PI / 2);
ctx.beginPath();
ctx.arc(R * 0.78, 0, dotR, 0, Math.PI * 2);
ctx.fillStyle = '#e00';
ctx.fill();
ctx.restore();
/* centre cap */
ctx.beginPath();
ctx.arc(R, R, R * 0.04, 0, Math.PI * 2);
ctx.fillStyle = '#222';
ctx.fill();
}
/* Resize the overlay canvas to fit the viewport, keeping it square. */
function resizeOverlayClock() {
var oc = document.getElementById('swiss-clock-overlay');
if (!oc) return;
var size = Math.min(window.innerWidth, window.innerHeight) * 0.88;
size = Math.floor(size);
oc.width = size;
oc.height = size;
}
/* Main tick — redraws both nav clock and (if visible) overlay clock. */
function clockTick() {
var nav = document.getElementById('swiss-clock');
if (nav) drawSwissClock(nav);
var overlay = document.getElementById('clock-overlay');
if (overlay && overlay.classList.contains('visible')) {
var oc = document.getElementById('swiss-clock-overlay');
if (oc) drawSwissClock(oc);
}
var delay = 100 - (Date.now() % 100);
setTimeout(clockTick, delay);
}
document.addEventListener('DOMContentLoaded', function() {
/* Start the shared tick loop */
clockTick();
/* Overlay toggle — clicking the nav clock opens it */
var navClock = document.querySelector('.nav-clock');
var overlay = document.getElementById('clock-overlay');
if (navClock && overlay) {
navClock.addEventListener('click', function() {
resizeOverlayClock();
overlay.classList.add('visible');
});
overlay.addEventListener('click', function() {
overlay.classList.remove('visible');
});
window.addEventListener('resize', function() {
if (overlay.classList.contains('visible')) resizeOverlayClock();
});
}
});
</script>
<script src="static/sorttable.js"></script>
</head> </head>
+102 -19
View File
@@ -7,13 +7,29 @@
display: flex; display: flex;
flex-direction: column; flex-direction: column;
height: 100vh; height: 100vh;
box-sizing: border-box;
padding: 10px;
margin: 0;
background: #f5f5f5;
overflow: hidden; overflow: hidden;
} }
@media (max-width: 640px) {
body {
height: auto;
min-height: 100vh;
overflow: auto;
flex-direction: column;
}
.container {
max-height: none;
overflow: visible;
}
.table-section {
max-height: 55vh;
}
.log-section {
flex: none;
max-height: 40vh;
}
}
.container { .container {
flex: 1; flex: 1;
min-height: 0; min-height: 0;
@@ -29,6 +45,7 @@
h1 { h1 {
color: #333; color: #333;
margin-bottom: 5px; margin-bottom: 5px;
margin-top: 15px;
font-size: 1.5em; font-size: 1.5em;
} }
@@ -59,6 +76,9 @@
border-radius: 6px; border-radius: 6px;
padding: 15px; padding: 15px;
box-shadow: 0 1px 4px rgba(0,0,0,0.1); box-shadow: 0 1px 4px rgba(0,0,0,0.1);
overflow-x: auto;
overflow-y: auto;
max-height: 60vh;
} }
.log-section { .log-section {
@@ -81,7 +101,8 @@
#ntable th { #ntable th {
border: 1px solid #e0e0e0; border: 1px solid #e0e0e0;
text-align: left; text-align: left;
padding: 8px 10px; padding: 2px 4px;
white-space: nowrap;
} }
#ntable tr:nth-child(even) { #ntable tr:nth-child(even) {
@@ -92,8 +113,24 @@
background-color: #e3f2fd; background-color: #e3f2fd;
} }
#ntable tbody tr.row-warning {
background-color: #fff8c5;
}
#ntable tbody tr.row-critical {
background-color: #fde8e8;
}
#ntable tbody tr.row-warning:hover {
background-color: #fff0a0;
}
#ntable tbody tr.row-critical:hover {
background-color: #f9c8c8;
}
#ntable th { #ntable th {
padding: 12px 10px; padding: 6px 8px;
background-color: #2196f3; background-color: #2196f3;
color: white; color: white;
font-weight: 600; font-weight: 600;
@@ -143,7 +180,7 @@
/* Message styling */ /* Message styling */
#messages { #messages {
font-size: 0.85em; font-size: 0.85em;
line-height: 1.6; line-height: 1.0;
} }
#messages div { #messages div {
@@ -205,15 +242,37 @@
var nTable = document; var nTable = document;
var name_idx = {}; var name_idx = {};
var c = 0; var c = 0;
var HBD_VERSION = "{{ hbd_version }}";
function hostNameHtml(data) {
var nameHtml = data.name;
if (!data.hbc_version || data.hbc_version !== HBD_VERSION) {
nameHtml += ' 🥀';
}
return data.dyn ? '<b>' + nameHtml + '</b>' : nameHtml;
}
function setup() { function setup() {
name_idx = {}; name_idx = {};
nTable = document.getElementById("ntable"); nTable = document.getElementById("ntable");
for (var i = 0, row; (row = nTable.rows[i]); i++) { for (var i = 0, row; (row = nTable.rows[i]); i++) {
if (i == 0) continue; if (i == 0) continue;
name = nTable.rows[i].cells[0].innerText; var cell = nTable.rows[i].cells[0];
var name = cell.dataset.name || cell.innerText.replace(/\s*🥀\s*$/, '').trim();
name_idx[name] = nTable.rows[i]; name_idx[name] = nTable.rows[i];
/* console.log("name_Id[" + name + "]: " + name_idx[name].innerText); */ }
}
function updateRowAlert(row, data) {
var criticalUnacked = data.alert_critical_unacked || 0;
var criticalAcked = data.alert_critical_acked || 0;
var warningUnacked = data.alert_warning_unacked || 0;
var warningAcked = data.alert_warning_acked || 0;
row.classList.remove('row-warning', 'row-critical');
if (criticalUnacked > 0 || criticalAcked > 0) {
row.classList.add('row-critical');
} else if (warningUnacked > 0 || warningAcked > 0) {
row.classList.add('row-warning');
} }
} }
@@ -251,11 +310,8 @@
row.appendChild(c_ipv6state); row.appendChild(c_ipv6state);
row.appendChild(c_ipv6latency); row.appendChild(c_ipv6latency);
row.appendChild(c_ipv6statets); row.appendChild(c_ipv6statets);
if (data.dyn) { c_name.dataset.name = data.name;
c_name.innerHTML = "<b>" + data.name + "</b>"; c_name.innerHTML = hostNameHtml(data);
} else {
c_name.innerHTML = data.name;
}
// Set alert counts in "x/y" format (unacked/acked) // Set alert counts in "x/y" format (unacked/acked)
var warningUnacked = data.alert_warning_unacked || 0; var warningUnacked = data.alert_warning_unacked || 0;
@@ -284,12 +340,31 @@
var table = document.getElementById("ntablebody"); // find table to append to var table = document.getElementById("ntablebody"); // find table to append to
table.appendChild(row); // append row to table table.appendChild(row); // append row to table
name_idx[c_name] = row; name_idx[c_name] = row;
updateRowAlert(row, data);
} }
function formatTS(ts) { function formatTS(ts) {
const milliseconds = ts * 1000; const now = new Date();
const dateObject = new Date(milliseconds); const d = new Date(ts * 1000);
return dateObject.toLocaleString("de-DE");
const pad = n => String(n).padStart(2, '0');
const timeStr = `${pad(d.getHours())}:${pad(d.getMinutes())}:${pad(d.getSeconds())}`;
// Same calendar day → show time only
if (d.toDateString() === now.toDateString()) {
return timeStr;
}
// Within 8 days → show "-X d hh:mm:ss"
const todayStart = new Date(now.getFullYear(), now.getMonth(), now.getDate());
const dStart = new Date(d.getFullYear(), d.getMonth(), d.getDate());
const diffDays = Math.round((todayStart - dStart) / 86400000);
if (diffDays < 8) {
return `-${diffDays}d ${timeStr}`;
}
// Older → date only
return `${d.getFullYear()}-${pad(d.getMonth() + 1)}-${pad(d.getDate())}`;
} }
function update_table(data) { function update_table(data) {
@@ -298,6 +373,11 @@
setup(); setup();
} }
// Update name cell (version indicator)
var nameCell = name_idx[data.name].cells[0];
nameCell.dataset.name = data.name;
nameCell.innerHTML = hostNameHtml(data);
// Update warning and critical counts in "x/y" format (unacked/acked) // Update warning and critical counts in "x/y" format (unacked/acked)
var warningUnacked = data.alert_warning_unacked || 0; var warningUnacked = data.alert_warning_unacked || 0;
var warningAcked = data.alert_warning_acked || 0; var warningAcked = data.alert_warning_acked || 0;
@@ -345,6 +425,7 @@
name_idx[data.name].cells[4 + i * 4].innerHTML = state; name_idx[data.name].cells[4 + i * 4].innerHTML = state;
name_idx[data.name].cells[5 + i * 4].innerHTML = latency; name_idx[data.name].cells[5 + i * 4].innerHTML = latency;
} }
updateRowAlert(name_idx[data.name], data);
} }
function WS_Connect() { function WS_Connect() {
@@ -405,8 +486,10 @@
{% include 'menu.html' %} {% include 'menu.html' %}
<div class="container"> <div class="container">
<div>
<h1>{{ header }}</h1> <h1>{{ header }}</h1>
<p class="subtitle">Real-time host monitoring and event log</p> <p class="subtitle">Real-time host monitoring and event log</p>
</div>
<div class="table-section"> <div class="table-section">
<table id="ntable" class="sortable"> <table id="ntable" class="sortable">
@@ -427,8 +510,8 @@
</thead> </thead>
<tbody id="ntablebody"> <tbody id="ntablebody">
{% for host in hosts %} {% for host in hosts %}
<tr> <tr class="{% if host.alert_critical_unacked > 0 or host.alert_critical_acked > 0 %}row-critical{% elif host.alert_warning_unacked > 0 or host.alert_warning_acked > 0 %}row-warning{% endif %}">
<td>{{ host.name }}</td> <td data-name="{{ host.name }}">{{ host.name }}{% if not host.hbc_version or host.hbc_version != hbd_version %} 🥀{% endif %}</td>
<td style="text-align: center; color: #ff9800; font-weight: bold;"> <td style="text-align: center; color: #ff9800; font-weight: bold;">
{%- set warning_unacked = host.alert_warning_unacked -%} {%- set warning_unacked = host.alert_warning_unacked -%}
{%- set warning_acked = host.alert_warning_acked -%} {%- set warning_acked = host.alert_warning_acked -%}
+28 -2
View File
@@ -1,11 +1,18 @@
<div class="nav"> <div class="nav">
<div class="nav-links"> <button class="nav-hamburger" id="nav-hamburger-btn" aria-label="Menu" aria-expanded="false">
<span></span><span></span><span></span>
</button>
<div class="nav-links" id="nav-links">
<a href="/live"{% if active_page == "live" %} class="active"{% endif %}>Live Dashboard</a> <a href="/live"{% if active_page == "live" %} class="active"{% endif %}>Live Dashboard</a>
<a href="/plugins"{% if active_page == "plugins" %} class="active"{% endif %}>Plugin Metrics</a> <a href="/plugins"{% if active_page == "plugins" %} class="active"{% endif %}>Host Overview</a>
<a href="/alerts"{% if active_page == "alerts" %} class="active"{% endif %}>Alerts</a> <a href="/alerts"{% if active_page == "alerts" %} class="active"{% endif %}>Alerts</a>
{% if current_user and current_user.admin %} {% if current_user and current_user.admin %}
<a href="/settings"{% if active_page == "settings" %} class="active"{% endif %}>Settings</a> <a href="/settings"{% if active_page == "settings" %} class="active"{% endif %}>Settings</a>
{% endif %} {% endif %}
<a href="/about"{% if active_page == "about" %} class="active"{% endif %}>About</a>
</div>
<div class="nav-clock" title="Click for full-screen clock">
<canvas id="swiss-clock" width="44" height="44"></canvas>
</div> </div>
{% if current_user %} {% if current_user %}
<a href="/profile" class="nav-user{% if active_page == 'profile' %} active{% endif %}" title="{{ current_user.full_name or current_user.username }}"> <a href="/profile" class="nav-user{% if active_page == 'profile' %} active{% endif %}" title="{{ current_user.full_name or current_user.username }}">
@@ -14,6 +21,25 @@
{% else %} {% else %}
<span class="nav-initials">{{ (current_user.full_name or current_user.username)[:1] | upper }}</span> <span class="nav-initials">{{ (current_user.full_name or current_user.username)[:1] | upper }}</span>
{% endif %} {% endif %}
<span class="nav-username">{{ current_user.full_name or current_user.username }}</span>
</a> </a>
{% endif %} {% endif %}
</div> </div>
<!-- Full-page clock overlay (click anywhere to dismiss) -->
<div id="clock-overlay">
<canvas id="swiss-clock-overlay" width="400" height="400"></canvas>
</div>
<script>
(function() {
var btn = document.getElementById('nav-hamburger-btn');
var links = document.getElementById('nav-links');
if (btn && links) {
btn.addEventListener('click', function() {
var open = links.classList.toggle('nav-open');
btn.setAttribute('aria-expanded', open ? 'true' : 'false');
});
}
})();
</script>
File diff suppressed because it is too large Load Diff
+1 -5
View File
@@ -3,11 +3,7 @@
{% include 'head.html' %} {% include 'head.html' %}
<style> <style>
body { html, body { overflow: visible; }
margin: 20px;
background: #f5f5f5;
font-family: 'Segoe UI', system-ui, sans-serif;
}
.container { .container {
max-width: 900px; max-width: 900px;
+70 -13
View File
@@ -3,22 +3,13 @@
{% include 'head.html' %} {% include 'head.html' %}
<style> <style>
html, body { html, body { overflow: visible; }
overflow: visible;
}
body {
margin: 20px;
background: #f5f5f5;
font-family: 'Segoe UI', system-ui, sans-serif;
}
.container { .container {
max-width: 960px; max-width: 960px;
margin: 0 auto;
} }
h1 { color: #333; margin-bottom: 4px; font-size: 1.5em; } h1 { color: #333; margin-bottom: 5px; margin-top: 15px; font-size: 1.5em; }
.subtitle { color: #666; margin-bottom: 24px; font-size: 0.9em; } .subtitle { color: #666; margin-bottom: 24px; font-size: 0.9em; }
/* ---- Sidebar + content layout ---- */ /* ---- Sidebar + content layout ---- */
@@ -32,7 +23,7 @@
width: 180px; width: 180px;
flex-shrink: 0; flex-shrink: 0;
position: sticky; position: sticky;
top: 20px; top: 60px;
} }
.sidebar-nav a { .sidebar-nav a {
@@ -217,6 +208,49 @@
.channel-field-value { color: #333; word-break: break-all; } .channel-field-value { color: #333; word-break: break-all; }
/* ---- Hosts table ---- */ /* ---- Hosts table ---- */
/* ---- Mobile: collapsible sidebar ---- */
.sidebar-toggle {
display: none;
width: 100%;
padding: 8px 12px;
background: #e8eaf6;
border: none;
border-radius: 6px;
font-size: 0.9em;
font-weight: 600;
color: #283593;
cursor: pointer;
text-align: left;
margin-bottom: 16px;
}
.sidebar-toggle::after { content: ' ▾'; float: right; }
.sidebar-toggle.open::after { content: ' ▴'; }
@media (max-width: 640px) {
.sidebar-toggle { display: block; }
.settings-layout { flex-direction: column; gap: 0; }
.settings-sidebar {
width: 100%;
position: static;
margin-bottom: 0;
}
.sidebar-nav {
display: none;
background: white;
border-radius: 6px;
box-shadow: 0 1px 4px rgba(0,0,0,.1);
margin-bottom: 16px;
padding: 4px 0;
}
.sidebar-nav.open { display: block; }
.sidebar-nav a { padding: 10px 16px; font-size: 1em; }
.field-row { flex-direction: column; gap: 4px; }
.field-label { width: 100%; font-size: 0.82em; color: #888; }
}
.host-bool { text-align: center; } .host-bool { text-align: center; }
.dot-yes { color: #2e7d32; font-size: 1.1em; } .dot-yes { color: #2e7d32; font-size: 1.1em; }
.dot-no { color: #ddd; font-size: 1.1em; } .dot-no { color: #ddd; font-size: 1.1em; }
@@ -233,9 +267,10 @@
<!-- Sidebar navigation --> <!-- Sidebar navigation -->
<nav class="settings-sidebar"> <nav class="settings-sidebar">
<button class="sidebar-toggle" id="sidebar-toggle" aria-expanded="false">Sections</button>
<div class="sidebar-nav" id="sidebar-nav"> <div class="sidebar-nav" id="sidebar-nav">
{% for section in sections %} {% for section in sections %}
<a href="#{{ section.id }}">{{ section.title }}</a> <a href="#{{ section.id }}" onclick="closeSidebar()">{{ section.title }}</a>
{% endfor %} {% endfor %}
</div> </div>
</nav> </nav>
@@ -428,6 +463,28 @@
}, { threshold: 0.25 }); }, { threshold: 0.25 });
sections.forEach(s => observer.observe(s)); sections.forEach(s => observer.observe(s));
// Collapsible sidebar on mobile
var sidebarToggle = document.getElementById('sidebar-toggle');
var sidebarNav = document.getElementById('sidebar-nav');
if (sidebarToggle && sidebarNav) {
sidebarToggle.addEventListener('click', function() {
var open = sidebarNav.classList.toggle('open');
sidebarToggle.classList.toggle('open', open);
sidebarToggle.setAttribute('aria-expanded', open ? 'true' : 'false');
});
}
</script>
<script>
function closeSidebar() {
var sidebarNav = document.getElementById('sidebar-nav');
var sidebarToggle = document.getElementById('sidebar-toggle');
if (sidebarNav) { sidebarNav.classList.remove('open'); }
if (sidebarToggle) {
sidebarToggle.classList.remove('open');
sidebarToggle.setAttribute('aria-expanded', 'false');
}
}
</script> </script>
</body> </body>
</html> </html>
+168 -70
View File
@@ -9,10 +9,11 @@ This module provides a flexible threshold checking system that:
- Supports multiple comparison operators - Supports multiple comparison operators
""" """
import asyncio
import logging import logging
import time import time
from enum import Enum from enum import Enum
from typing import Dict, Any, Optional, Tuple, Callable from typing import Dict, List, Any, Optional, Tuple, Callable
from . import notify as notify_mod from . import notify as notify_mod
from .config import THRESHOLD_DEFAULTS from .config import THRESHOLD_DEFAULTS
@@ -60,6 +61,7 @@ class AlertState:
self.acknowledged = False # Whether alert has been acknowledged self.acknowledged = False # Whether alert has been acknowledged
self.acknowledged_at = None # Timestamp when acknowledged self.acknowledged_at = None # Timestamp when acknowledged
self.consecutive_count = 0 # Consecutive exceedances while still OK (for count gating) self.consecutive_count = 0 # Consecutive exceedances while still OK (for count gating)
self.pending_since: Optional[float] = None # non-None while waiting out grace period before notifying
def update( def update(
self, self,
@@ -105,6 +107,7 @@ class AlertState:
self.level = level self.level = level
self.since = now self.since = now
self.notification_count = 0 self.notification_count = 0
self.last_notification = None # restart reminder interval on level change
# Reset acknowledgment on state change # Reset acknowledgment on state change
if level != AlertLevel.OK: if level != AlertLevel.OK:
# Only reset if changing to a different alert level # Only reset if changing to a different alert level
@@ -326,19 +329,23 @@ class ThresholdChecker:
renotify_interval: Seconds between repeat notifications (default: 1 hour) renotify_interval: Seconds between repeat notifications (default: 1 hour)
journal: Optional MessageJournal instance for logging threshold events journal: Optional MessageJournal instance for logging threshold events
""" """
# Named threshold configurations: {config_name: {metric_path: ThresholdConfig}} # Named threshold configurations (pre-merged: defaults + overrides): {config_name: {metric_path: ThresholdConfig}}
self.threshold_configs = {} self.threshold_configs = {}
# Raw overrides only for each named config (no defaults baked in): {config_name: {metric_path: ThresholdConfig}}
self.threshold_raw_configs: Dict[str, Dict[str, ThresholdConfig]] = {}
# Single threshold set for backward compatibility: {metric_path: ThresholdConfig} # Single threshold set for backward compatibility: {metric_path: ThresholdConfig}
self.thresholds = {} self.thresholds = {}
# Host to config name mapping: {host_name: config_name} # Host to ordered list of config names: {host_name: [config_name, ...]}
self.host_config_mapping = {} self.host_config_mapping: Dict[str, List[str]] = {}
# Default config name to use when no mapping exists # Default config name to use when no mapping exists
self.default_config = "default" self.default_config = "default"
self.renotify_interval = renotify_interval self.renotify_interval = renotify_interval
self.grace_seconds: float = float(config.get("grace", 2))
self.journal = journal self.journal = journal
# Parse configuration # Parse configuration
@@ -369,8 +376,10 @@ class ThresholdChecker:
# Clear old configuration # Clear old configuration
self.threshold_configs.clear() self.threshold_configs.clear()
self.threshold_raw_configs.clear()
self.thresholds.clear() self.thresholds.clear()
self.host_config_mapping.clear() self.host_config_mapping.clear()
self.grace_seconds = float(config.get("grace", 2))
# Parse new configuration # Parse new configuration
self._parse_config(config) self._parse_config(config)
@@ -420,9 +429,10 @@ class ThresholdChecker:
self._parse_plugin_thresholds(plugin_name, plugin_thresholds, target_dict=effective_defaults) self._parse_plugin_thresholds(plugin_name, plugin_thresholds, target_dict=effective_defaults)
self.threshold_configs["default"] = dict(effective_defaults) self.threshold_configs["default"] = dict(effective_defaults)
self.threshold_raw_configs["default"] = {}
logger.info("Registered 'default' threshold config with %d metrics", len(effective_defaults)) logger.info("Registered 'default' threshold config with %d metrics", len(effective_defaults))
# Parse each named configuration, seeding it with effective_defaults first # Parse each named configuration
for config_name, config_data in threshold_configs.items(): for config_name, config_data in threshold_configs.items():
if config_name == "default": if config_name == "default":
continue # already handled above continue # already handled above
@@ -436,33 +446,41 @@ class ThresholdChecker:
continue continue
logger.info("Parsing threshold configuration: %s", config_name) logger.info("Parsing threshold configuration: %s", config_name)
self.threshold_configs[config_name] = dict(effective_defaults)
# Raw overrides only (used for multi-config layering)
raw_overrides: Dict[str, ThresholdConfig] = {}
thresholds_config = config_data["thresholds"] thresholds_config = config_data["thresholds"]
for plugin_name, plugin_thresholds in thresholds_config.items(): for plugin_name, plugin_thresholds in thresholds_config.items():
if not isinstance(plugin_thresholds, dict): if isinstance(plugin_thresholds, dict):
continue self._parse_plugin_thresholds(plugin_name, plugin_thresholds, target_dict=raw_overrides)
self.threshold_raw_configs[config_name] = raw_overrides
self._parse_plugin_thresholds( # Pre-merged version (defaults + overrides) for single-config fast path
plugin_name, self.threshold_configs[config_name] = dict(effective_defaults)
plugin_thresholds, self.threshold_configs[config_name].update(raw_overrides)
target_dict=self.threshold_configs[config_name]
)
# Parse host to config mapping from two possible sources # Parse host config list mapping from two possible sources
# 1. New format: hosts section with threshold_config attribute
def _normalise(value) -> List[str]:
"""Accept a string or list; always return a list."""
if isinstance(value, list):
return [str(v) for v in value]
return [str(value)]
# 1. hosts section with threshold_config attribute (string or list)
if "hosts" in config: if "hosts" in config:
hosts_config = config["hosts"] hosts_config = config["hosts"]
if isinstance(hosts_config, dict): if isinstance(hosts_config, dict):
for host_name, host_attrs in hosts_config.items(): for host_name, host_attrs in hosts_config.items():
if isinstance(host_attrs, dict) and "threshold_config" in host_attrs: if isinstance(host_attrs, dict) and "threshold_config" in host_attrs:
self.host_config_mapping[host_name] = host_attrs["threshold_config"] self.host_config_mapping[host_name] = _normalise(host_attrs["threshold_config"])
# 2. Legacy format: host_threshold_mapping section (for backward compatibility) # 2. Legacy host_threshold_mapping section (string values only)
if "host_threshold_mapping" in config: if "host_threshold_mapping" in config:
legacy_mapping = config.get("host_threshold_mapping", {}) legacy_mapping = config.get("host_threshold_mapping", {})
if isinstance(legacy_mapping, dict): if isinstance(legacy_mapping, dict):
self.host_config_mapping.update(legacy_mapping) for host_name, value in legacy_mapping.items():
self.host_config_mapping[host_name] = _normalise(value)
# Set default config (first one alphabetically or explicitly set) # Set default config (first one alphabetically or explicitly set)
self.default_config = config.get("default_threshold_config", "default") self.default_config = config.get("default_threshold_config", "default")
@@ -660,7 +678,10 @@ class ThresholdChecker:
) )
def get_thresholds_for_host(self, host_name: str) -> Dict[str, ThresholdConfig]: def get_thresholds_for_host(self, host_name: str) -> Dict[str, ThresholdConfig]:
"""Get the appropriate threshold configuration for a host. """Get the effective threshold configuration for a host.
When threshold_config is a list, configs are applied left-to-right on top
of the default thresholds so earlier entries can be overridden by later ones.
Args: Args:
host_name: Name of the host host_name: Name of the host
@@ -672,23 +693,40 @@ class ThresholdChecker:
if self.thresholds and not self.threshold_configs: if self.thresholds and not self.threshold_configs:
return self.thresholds return self.thresholds
# Multi-config mode: look up host-specific configuration if not self.threshold_configs:
if self.threshold_configs: return {}
config_name = self.host_config_mapping.get(host_name, self.default_config)
if config_name in self.threshold_configs: config_names = self.host_config_mapping.get(host_name)
return self.threshold_configs[config_name]
else: # No host-specific mapping → return pre-merged default
if not config_names:
return self.threshold_configs.get(self.default_config, {})
# Single config → fast path using pre-merged copy
if len(config_names) == 1:
name = config_names[0]
if name in self.threshold_configs:
return self.threshold_configs[name]
logger.warning( logger.warning(
"Threshold config '%s' not found for host '%s', using default '%s'", "Threshold config '%s' not found for host '%s', using default '%s'",
config_name, name, host_name, self.default_config,
host_name,
self.default_config
) )
return self.threshold_configs.get(self.default_config, {}) return self.threshold_configs.get(self.default_config, {})
# No thresholds configured # Multiple configs → start from defaults, layer raw overrides in order
return {} result = dict(self.threshold_configs.get(self.default_config, {}))
for name in config_names:
if name == self.default_config:
continue # defaults already the base
raw = self.threshold_raw_configs.get(name)
if raw is None:
logger.warning(
"Threshold config '%s' not found for host '%s', skipping",
name, host_name,
)
else:
result.update(raw)
return result
def check_value( def check_value(
self, self,
@@ -759,15 +797,10 @@ class ThresholdChecker:
# Update state and check for changes # Update state and check for changes
old_level = alert_state.level old_level = alert_state.level
if alert_state.update(new_level, value, threshold_value, threshold.operator.value): if alert_state.update(new_level, value, threshold_value, threshold.operator.value):
# For check_value, we don't have full plugin data, pass None self._apply_grace(host_name, alert_state, metric_path, old_level, new_level, value, threshold, None)
lvl, message, formatted_msg = self._trigger_notification(host_name, metric_path, old_level, new_level, value, threshold, None)
# Update alert state with formatted message
alert_state.formatted_message = formatted_msg
self._send_notification(host_name, lvl, message, metric_path, old_level, new_level, value)
return (old_level, new_level) return (old_level, new_level)
elif new_level != AlertLevel.OK: elif new_level != AlertLevel.OK:
# Check if we should re-notify self._check_pending_or_renotify(host_name, alert_state, metric_path, value, threshold, None)
self._check_renotify(host_name, alert_state, metric_path, value, threshold, None)
return None return None
def check_plugin_data( def check_plugin_data(
@@ -826,13 +859,9 @@ class ThresholdChecker:
old_level = alert_state.level old_level = alert_state.level
if alert_state.update(new_level, value, threshold_value, threshold.operator.value): if alert_state.update(new_level, value, threshold_value, threshold.operator.value):
state_changes.append((metric_path, old_level, new_level, value)) state_changes.append((metric_path, old_level, new_level, value))
lvl, message, formatted_msg = self._trigger_notification(host_name, metric_path, old_level, new_level, value, threshold, data) self._apply_grace(host_name, alert_state, metric_path, old_level, new_level, value, threshold, data)
# Update alert state with formatted message
alert_state.formatted_message = formatted_msg
self._send_notification(host_name, lvl, message, metric_path, old_level, new_level, value)
elif new_level != AlertLevel.OK: elif new_level != AlertLevel.OK:
# Check if we should re-notify self._check_pending_or_renotify(host_name, alert_state, metric_path, value, threshold, data)
self._check_renotify(host_name, alert_state, metric_path, value, threshold, data)
# Check nested metrics (e.g., partition data in disk_monitor) # Check nested metrics (e.g., partition data in disk_monitor)
self._check_nested_metrics( self._check_nested_metrics(
@@ -895,20 +924,9 @@ class ThresholdChecker:
old_level = alert_state.level old_level = alert_state.level
if alert_state.update(new_level, value, threshold_value, threshold.operator.value): if alert_state.update(new_level, value, threshold_value, threshold.operator.value):
state_changes.append((metric_path, old_level, new_level, value)) state_changes.append((metric_path, old_level, new_level, value))
lvl, message, formatted_msg = self._trigger_notification( self._apply_grace(host_name, alert_state, metric_path, old_level, new_level, value, threshold, data)
host_name,
metric_path,
old_level,
new_level,
value,
threshold,
data # Pass full plugin data for format string
)
# Update alert state with formatted message
alert_state.formatted_message = formatted_msg
self._send_notification(host_name, lvl, message, metric_path, old_level, new_level, value)
elif new_level != AlertLevel.OK: elif new_level != AlertLevel.OK:
self._check_renotify(host_name, alert_state, metric_path, value, threshold, data) self._check_pending_or_renotify(host_name, alert_state, metric_path, value, threshold, data)
def _trigger_notification( def _trigger_notification(
self, self,
@@ -947,7 +965,7 @@ class ThresholdChecker:
# Format message # Format message
if new_level == AlertLevel.OK: if new_level == AlertLevel.OK:
lvl = "RECOVERED" lvl = "RECOVER"
message = f"{metric_path} = {display_value} ({old_level.name} -> OK)" message = f"{metric_path} = {display_value} ({old_level.name} -> OK)"
elif new_level == AlertLevel.WARNING: elif new_level == AlertLevel.WARNING:
lvl = "WARNING" lvl = "WARNING"
@@ -1003,17 +1021,23 @@ class ThresholdChecker:
value: Any, value: Any,
): ):
"""Send notification and log to journal/eventlog.""" """Send notification and log to journal/eventlog."""
# Send notification using host-specific channels from . import hbdclass
try: host = hbdclass.Host.hosts.get(host_name)
notify_mod.pushmsg_for_host(host_name, f"{lvl}: {host_name} - {message}") if host is not None and not host.watched:
logger.info("Notification sent: %s", message) eventlog(host_name, lvl, message, service="threshold")
except Exception as e: return
logger.error("Failed to send notification: %s", e) asyncio.get_event_loop().create_task(notify_mod.send_notification(
host_name,
notify_mod.Notification(
title=f"[{lvl}] {host_name}",
body=message,
level=lvl,
),
))
# Log to journal # Log to journal
if self.journal is not None: if self.journal is not None:
try: try:
import asyncio
loop = asyncio.get_event_loop() loop = asyncio.get_event_loop()
loop.create_task(self.journal.log_threshold_event( loop.create_task(self.journal.log_threshold_event(
host_name=host_name, host_name=host_name,
@@ -1077,6 +1101,74 @@ class ThresholdChecker:
) )
return f"(threshold: {op_symbol} {threshold_value})" return f"(threshold: {op_symbol} {threshold_value})"
def _apply_grace(
self,
host_name: str,
alert_state: AlertState,
metric_path: str,
old_level: AlertLevel,
new_level: AlertLevel,
value: Any,
threshold: ThresholdConfig,
plugin_data: Optional[Dict[str, Any]],
) -> None:
"""Handle a state-change transition with grace-period logic.
Transitioning INTO alert: defers the notification for grace_seconds.
Transitioning TO OK:
- Still in grace window (pending_since set): suppresses both the alert
and the recovery — the spike never warranted a page.
- Past grace: fires the RECOVER notification normally.
"""
lvl, message, formatted_msg = self._trigger_notification(
host_name, metric_path, old_level, new_level, value, threshold, plugin_data
)
alert_state.formatted_message = formatted_msg
if new_level == AlertLevel.OK:
if alert_state.pending_since is not None:
logger.info(
"Alert suppressed (recovered within %.0fs grace): %s on %s",
self.grace_seconds, metric_path, host_name,
)
alert_state.pending_since = None
else:
self._send_notification(host_name, lvl, message, metric_path, old_level, new_level, value)
else:
alert_state.pending_since = time.time()
logger.debug(
"Alert deferred (%.0fs grace): %s on %s = %s",
self.grace_seconds, metric_path, host_name, value,
)
def _check_pending_or_renotify(
self,
host_name: str,
alert_state: AlertState,
metric_path: str,
value: Any,
threshold: ThresholdConfig,
plugin_data: Optional[Dict[str, Any]],
) -> None:
"""Called when alert level is unchanged and non-OK.
If a deferred notification is pending and grace_seconds have elapsed,
fires it now. Otherwise falls through to normal reminder logic.
"""
if alert_state.pending_since is not None:
if time.time() - alert_state.pending_since >= self.grace_seconds:
lvl, message, formatted_msg = self._trigger_notification(
host_name, metric_path, AlertLevel.OK, alert_state.level, value, threshold, plugin_data
)
alert_state.formatted_message = formatted_msg
self._send_notification(
host_name, lvl, message, metric_path, AlertLevel.OK, alert_state.level, value
)
alert_state.pending_since = None
# else: still within grace window, do nothing
else:
self._check_renotify(host_name, alert_state, metric_path, value, threshold, plugin_data)
def _check_renotify( def _check_renotify(
self, self,
host_name: str, host_name: str,
@@ -1137,14 +1229,20 @@ class ThresholdChecker:
else: else:
message = f"REMINDER ({alert_state.level.name}): {host_name} - {metric_path} = {value} (ongoing for {int(now - alert_state.since)}s)" message = f"REMINDER ({alert_state.level.name}): {host_name} - {metric_path} = {value} (ongoing for {int(now - alert_state.since)}s)"
# Send re-notification using host-specific channels from . import hbdclass
try: host = hbdclass.Host.hosts.get(host_name)
notify_mod.pushmsg_for_host(host_name, message) if host is None or host.watched:
asyncio.get_event_loop().create_task(notify_mod.send_notification(
host_name,
notify_mod.Notification(
title=f"[REMINDER/{alert_state.level.name}] {host_name}",
body=message,
level=alert_state.level.name,
),
))
logger.info("Re-notification sent: %s", message)
alert_state.last_notification = now alert_state.last_notification = now
alert_state.notification_count += 1 alert_state.notification_count += 1
logger.info("Re-notification sent: %s", message)
except Exception as e:
logger.error("Failed to send re-notification: %s", e)
def get_active_alerts(self, alert_states: Dict[str, AlertState]) -> list: def get_active_alerts(self, alert_states: Dict[str, AlertState]) -> list:
""" """
+57 -28
View File
@@ -171,7 +171,25 @@ def dicttos(ID, d):
DROPOVERDUE = 7 * 24 * 3600 # seconds before an overdue host becomes UNKNOWN DROPOVERDUE = 7 * 24 * 3600 # seconds before an overdue host becomes UNKNOWN
def _make_timer_callbacks(uname, host, watchhosts, ctx): def _set_connectivity_alert(host, afam, level_name):
"""Update (or clear) a connectivity alert_state entry for a host/address-family.
level_name is "CRITICAL", "WARNING", or "OK". "OK" removes the entry so
that recovered hosts don't clutter the Alerts Dashboard.
"""
from .threshold import AlertState, AlertLevel
metric_path = f"connectivity.{afam}"
level = getattr(AlertLevel, level_name, AlertLevel.OK)
if level == AlertLevel.OK:
host.alert_states.pop(metric_path, None)
return
if metric_path not in host.alert_states:
host.alert_states[metric_path] = AlertState(metric_path)
state = host.alert_states[metric_path]
state.update(level, level_name)
def _make_timer_callbacks(uname, host, ctx):
"""Return (on_overdue, on_unknown) async callbacks for connection timer logic. """Return (on_overdue, on_unknown) async callbacks for connection timer logic.
Captured values are bound at call time so callbacks are safe to use in loops. Captured values are bound at call time so callbacks are safe to use in loops.
@@ -182,6 +200,7 @@ def _make_timer_callbacks(uname, host, watchhosts, ctx):
async def on_unknown(connection): async def on_unknown(connection):
connection.newstate(connection.__class__.UNKNOWN, connection.lastbeat) connection.newstate(connection.__class__.UNKNOWN, connection.lastbeat)
# Keep connectivity alert active when host transitions to unknown
if msg_to_websockets: if msg_to_websockets:
msg_to_websockets("host", host.stateinfo()) msg_to_websockets("host", host.stateinfo())
@@ -191,9 +210,14 @@ def _make_timer_callbacks(uname, host, watchhosts, ctx):
now = time.time() now = time.time()
connection.newstate(connection.__class__.OVERDUE, now, cfg.get("grace", 2)) connection.newstate(connection.__class__.OVERDUE, now, cfg.get("grace", 2))
msg = f"{connection.afam} overdue" msg = f"{connection.afam} overdue"
eventlog(uname, "CRITICAL" if uname in watchhosts else "WARNING", msg) eventlog(uname, "CRITICAL", msg)
if uname in watchhosts: if host.watched:
notify_mod.pushmsg_for_host(uname, f"{uname} {msg}") asyncio.create_task(notify_mod.send_notification(
uname,
notify_mod.Notification(title=f"[CRITICAL] {uname}", body=msg, level="CRITICAL"),
))
# Track in alert_states so the Alerts Dashboard shows this
_set_connectivity_alert(host, connection.afam, "CRITICAL")
if threshold_checker: if threshold_checker:
threshold_checker.check_value( threshold_checker.check_value(
host_name=uname, host_name=uname,
@@ -218,8 +242,6 @@ def restore_connection_timers(hbdclass, ctx):
now = time.time() now = time.time()
cfg = ctx.get("config", {}) cfg = ctx.get("config", {})
grace = cfg.get("grace", 2) grace = cfg.get("grace", 2)
from . import config as config_mod
watchhosts = config_mod.get_watchhosts(cfg)
restored = 0 restored = 0
for uname, host in list(hbdclass.Host.hosts.items()): for uname, host in list(hbdclass.Host.hosts.items()):
@@ -229,7 +251,7 @@ def restore_connection_timers(hbdclass, ctx):
if state == hbdclass.Connection.DOWN: if state == hbdclass.Connection.DOWN:
continue continue
on_overdue, on_unknown = _make_timer_callbacks(uname, host, watchhosts, ctx) on_overdue, on_unknown = _make_timer_callbacks(uname, host, ctx)
if state == hbdclass.Connection.UP and interval > 0: if state == hbdclass.Connection.UP and interval > 0:
elapsed = now - conn.lastbeat elapsed = now - conn.lastbeat
@@ -294,7 +316,6 @@ def handle_datagram(msg: dict, addr, transport, ctx: dict):
cfg = ctx.get("config", {}) cfg = ctx.get("config", {})
hbdcls = ctx.get("hbdclass") hbdcls = ctx.get("hbdclass")
log = ctx.get("log")
msg_to_websockets = ctx.get("msg_to_websockets") msg_to_websockets = ctx.get("msg_to_websockets")
DEBUG = ctx.get("DEBUG", 0) DEBUG = ctx.get("DEBUG", 0)
verbose = ctx.get("verbose", False) verbose = ctx.get("verbose", False)
@@ -322,9 +343,6 @@ def handle_datagram(msg: dict, addr, transport, ctx: dict):
host = hbdcls.Host.hosts[uname] host = hbdcls.Host.hosts[uname]
newh = False newh = False
# Get watchhosts once for use throughout message handling
watchhosts = config_mod.get_watchhosts(cfg)
cid = msg.get("id", 0) cid = msg.get("id", 0)
try: try:
rtt = float(msg.get("rtt")) rtt = float(msg.get("rtt"))
@@ -390,8 +408,11 @@ def handle_datagram(msg: dict, addr, transport, ctx: dict):
if res: if res:
eventlog(uname, "WARNING", res) eventlog(uname, "WARNING", res)
if uname in watchhosts: if host.watched:
notify_mod.pushmsg_for_host(uname, "%s %s" % (host.name, res)) asyncio.create_task(notify_mod.send_notification(
uname,
notify_mod.Notification(title=f"[WARNING] {uname}", body=res, level="WARNING"),
))
interval = int(msg.get("interval", 0) or 0) interval = int(msg.get("interval", 0) or 0)
shutdown = msg.get("shutdown", 0) shutdown = msg.get("shutdown", 0)
@@ -401,17 +422,19 @@ def handle_datagram(msg: dict, addr, transport, ctx: dict):
if boot: if boot:
eventlog(uname, "INFO", "booted") eventlog(uname, "INFO", "booted")
if uname in watchhosts: if host.watched:
m = "%s booted" % (host.name) asyncio.create_task(notify_mod.send_notification(
notify_mod.pushmsg_for_host(uname, m) uname,
notify_mod.Notification(title=f"[INFO] {uname}", body=f"{host.name} booted", level="INFO"),
))
if message: if message:
eventlog(uname, "INFO", "msg: %s" % message, service=service) eventlog(uname, "INFO", "msg: %s" % message, service=service)
if uname in watchhosts:
notify_mod.pushmsg_for_host(uname, message)
if conn.getstate() != hbdcls.Connection.UP: if conn.getstate() != hbdcls.Connection.UP:
lasts = conn.state lasts = conn.state
d = conn.newstate(hbdcls.Connection.UP, now) d = conn.newstate(hbdcls.Connection.UP, now)
# Clear connectivity alert now that the host is back up
_set_connectivity_alert(host, conn.afam, "OK")
# Don't log/notify RECOVER for a brand-new host seen for the first time — # Don't log/notify RECOVER for a brand-new host seen for the first time —
# it was never down, it just hasn't been seen before. # it was never down, it just hasn't been seen before.
if not newh: if not newh:
@@ -420,8 +443,11 @@ def handle_datagram(msg: dict, addr, transport, ctx: dict):
else: else:
m = "%s back after being %s for %s" % (conn.afam, lasts, dur(d)) m = "%s back after being %s for %s" % (conn.afam, lasts, dur(d))
eventlog(uname, "RECOVER", m) eventlog(uname, "RECOVER", m)
if uname in watchhosts: if host.watched:
notify_mod.pushmsg_for_host(uname, "%s %s is back" % (uname, conn.afam)) asyncio.create_task(notify_mod.send_notification(
uname,
notify_mod.Notification(title=f"[RECOVER] {uname}", body=m, level="RECOVER"),
))
if boot or newh: if boot or newh:
host.upcount = host.doesack host.upcount = host.doesack
@@ -429,10 +455,15 @@ def handle_datagram(msg: dict, addr, transport, ctx: dict):
host.upcount += 1 host.upcount += 1
if shutdown: if shutdown:
eventlog(uname, "INFO", "%s shutdown" % conn.afam) m = "%s shutdown" % conn.afam
if uname in watchhosts: eventlog(uname, "INFO", m)
notify_mod.pushmsg_for_host(uname, "%s %s shutdown" % (uname, conn.afam)) if host.watched:
asyncio.create_task(notify_mod.send_notification(
uname,
notify_mod.Notification(title=f"[INFO] {uname}", body=m, level="INFO"),
))
conn.newstate(hbdcls.Connection.DOWN, now) conn.newstate(hbdcls.Connection.DOWN, now)
_set_connectivity_alert(host, conn.afam, "CRITICAL")
if interval > 0: if interval > 0:
host.interval = interval host.interval = interval
@@ -442,7 +473,7 @@ def handle_datagram(msg: dict, addr, transport, ctx: dict):
if interval > 0 and conn.getstate() != hbdcls.Connection.DOWN: if interval > 0 and conn.getstate() != hbdcls.Connection.DOWN:
grace = cfg.get("grace", 2) grace = cfg.get("grace", 2)
timeout_seconds = interval + grace timeout_seconds = interval + grace
on_overdue, _ = _make_timer_callbacks(uname, host, watchhosts, ctx) on_overdue, _ = _make_timer_callbacks(uname, host, ctx)
conn.reset_overdue_timer(timeout_seconds, on_overdue) conn.reset_overdue_timer(timeout_seconds, on_overdue)
# Check RTT thresholds using the threshold checker # Check RTT thresholds using the threshold checker
@@ -464,12 +495,10 @@ def handle_datagram(msg: dict, addr, transport, ctx: dict):
op, rmsg = host.cmds[0] op, rmsg = host.cmds[0]
if op == "CMD": if op == "CMD":
del host.cmds[0] del host.cmds[0]
if log: eventlog(uname, "INFO", "command sent")
log(uname, "command sent")
elif op == "UPD": elif op == "UPD":
del host.cmds[0] del host.cmds[0]
if log: eventlog(uname, "INFO", "update initiated")
log(uname, "update initiated")
opkt = dicttos(op, rmsg) opkt = dicttos(op, rmsg)
try: try:
transport.sendto(opkt, addr) transport.sendto(opkt, addr)
+117 -121
View File
@@ -1,7 +1,8 @@
"""WebSocket server and broadcast helpers for hbd. """WebSocket handler and broadcast helpers for hbd.
Provides an asyncio-based WebSocket server and a thread-safe broadcast WebSocket connections are served through the regular HTTP port via the
function that other threads or synchronous code can call. /ws route registered in http.py (aiohttp WebSocketResponse upgrade).
The separate standalone WebSocket server on ws_port is no longer used.
""" """
import asyncio import asyncio
@@ -10,147 +11,142 @@ import logging
from typing import Callable, Iterable, Optional from typing import Callable, Iterable, Optional
from . import data from . import data
import websockets
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)
_connections = set() # Map of WebSocket → User object (or None when auth is disabled)
_connections: dict = {}
_loop: Optional[asyncio.AbstractEventLoop] = None _loop: Optional[asyncio.AbstractEventLoop] = None
_get_hosts: Optional[Callable[[], Iterable]] = None _get_hosts: Optional[Callable[[], Iterable]] = None
#_get_msgs: Optional[Callable[[], Iterable]] = None _verbose: bool = False
_verbose = False
async def _handler(websocket, path=None): def setup(
_connections.add(websocket) loop: asyncio.AbstractEventLoop,
remote_address = websocket.remote_address get_hosts: Optional[Callable[[], Iterable]] = None,
if path is None: verbose: bool = False,
path = getattr(websocket, "path", None)
logger.info("WebSocket connection from %s: %s", remote_address, path)
try:
# send initial hosts
if _get_hosts:
try:
hosts = list(_get_hosts())
logger.debug("Sending %d hosts to new WebSocket client", len(hosts))
for h in hosts:
jmsg = json.dumps({"type": "host", "data": h})
await websocket.send(jmsg)
except Exception as e:
logger.error("Error sending initial hosts: %s", e, exc_info=True)
# send recent messages
if data.msgs:
try:
# msgs = list(_get_msgs())[-100:]
logger.debug("Sending %d recent messages to new WebSocket client", len(data.msgs))
for m in data.msgs:
jmsg = json.dumps({"type": "message", "data": m})
await websocket.send(jmsg)
except Exception as e:
logger.error("Error sending initial messages: %s", e, exc_info=True)
# keep connection open until client disconnects
async for _ in websocket:
# we don't expect meaningful incoming messages besides the initial
# client 'hello' that some clients send; ignore for now
if _verbose:
logger.debug("received ws data: %s", _)
except (
websockets.exceptions.ConnectionClosedOK,
websockets.exceptions.ConnectionClosedError,
) as e:
logger.info("WebSocket closed from %s: %r", remote_address, e)
except Exception as e:
logger.exception("WebSocket handler exception from %s: %s", remote_address, e)
finally:
logger.debug("Removing WebSocket connection from %s", remote_address)
_connections.discard(websocket)
async def start(
host: str,
ws_port: int,
wss_port: Optional[int] = None,
ssl_context=None,
get_hosts: Optional[Callable] = None,
# get_msgs: Optional[Callable] = None,
config: dict = {},
): ):
"""Start WebSocket servers and block until cancelled. """Register the running loop and initial-state callback.
This is intended to be awaited inside the main asyncio event loop. Call this once from _run_async before starting the HTTP server.
If `wss_port` and `ssl_context` are provided, a WSS server will also be
started.
""" """
global _loop, _get_hosts, _verbose global _loop, _get_hosts, _verbose
_loop = asyncio.get_running_loop() _loop = loop
_get_hosts = get_hosts _get_hosts = get_hosts
_verbose = config.get("verbose", False), _verbose = verbose
_debug = config.get("debug", 0),
# Start servers and keep the server objects for clean shutdown
running_servers = []
ws_server = await websockets.serve(_handler, host, ws_port)
running_servers.append(ws_server)
if wss_port and ssl_context:
wss_server = await websockets.serve(_handler, host, wss_port, ssl=ssl_context)
running_servers.append(wss_server)
logger.info( def _user_can_see_host(user, host_name: str) -> bool:
"WebSocket server(s) started on port %s (wss %s)", ws_port, wss_port """Return True if *user* may see updates for *host_name* (manager or higher)."""
) from . import hbdclass, users as users_mod
if user is None or not users_mod.users_enabled():
return True
if user.admin:
return True
host = hbdclass.Host.hosts.get(host_name)
if host is None:
return False
return host.is_manager(user.username)
def _get_token(request) -> str:
"""Extract session token from request (mirrors logic in http.py)."""
auth = request.headers.get("Authorization", "")
if auth.startswith("Bearer "):
return auth[7:].strip()
token = request.headers.get("X-Auth-Token", "")
if token:
return token
return request.cookies.get("hbd_session", "")
async def handler(request):
"""aiohttp WebSocket upgrade handler — register as GET /ws."""
from aiohttp import web
from . import users as users_mod
ws = web.WebSocketResponse()
await ws.prepare(request)
token = _get_token(request)
user = users_mod.get_session_user(token) if token else None
_connections[ws] = user
remote = request.remote
logger.info("WebSocket connected from %s", remote)
try: try:
# Block until cancelled # Send current host state, filtered to hosts this user may see
await asyncio.Future() if _get_hosts:
except asyncio.CancelledError: try:
pass for h in list(_get_hosts()):
host_name = h.get("raw_name") or h.get("name", "")
if _user_can_see_host(user, host_name):
await ws.send_str(json.dumps({"type": "host", "data": h}))
except Exception as e:
logger.error("Error sending initial hosts: %s", e)
# Send recent messages
if data.msgs:
try:
for m in data.msgs:
await ws.send_str(json.dumps({"type": "message", "data": m}))
except Exception as e:
logger.error("Error sending initial messages: %s", e)
# Keep connection open, ignore incoming frames
async for msg in ws:
from aiohttp import WSMsgType
if msg.type == WSMsgType.TEXT:
if _verbose:
logger.debug("ws recv from %s: %s", remote, msg.data)
elif msg.type in (WSMsgType.ERROR, WSMsgType.CLOSE):
break
except Exception as e:
logger.exception("WebSocket handler error from %s: %s", remote, e)
finally: finally:
# Close all active browser connections so their handler coroutines exit _connections.pop(ws, None)
active = list(_connections) logger.info("WebSocket disconnected from %s", remote)
if active:
logger.info("Closing %d active WebSocket connection(s)...", len(active)) return ws
await asyncio.gather(
*[ws.close() for ws in active],
return_exceptions=True,
)
# Stop the listening servers and wait for all handlers to finish
for srv in running_servers:
srv.close()
await asyncio.gather(
*[srv.wait_closed() for srv in running_servers],
return_exceptions=True,
)
logger.info("WebSocket server(s) stopped")
def broadcast(typ: str, data) -> bool: def broadcast(typ: str, payload) -> bool:
"""Thread-safe broadcast helper. """Thread-safe broadcast to all connected WebSocket clients.
Schedules coroutine(s) on the running loop to send message to all For host and plugin updates, only sends to clients whose user has
connected websockets. Returns False if server was not running. manager-or-higher access to that host. Other message types are
broadcast to all clients.
Can be called from any thread; schedules sends on the event loop.
Returns False if the loop is not running yet.
""" """
if not _loop: if not _loop:
return False return False
jmsg = json.dumps({"type": typ, "data": data})
to_close = [] # Determine the host name for access-filtered message types
for ws in list(_connections): host_name: Optional[str] = None
if ws.state != websockets.protocol.State.OPEN: if typ in ("host", "plugin"):
to_close.append(ws) host_name = payload.get("raw_name") or payload.get("host") or payload.get("name")
jmsg = json.dumps({"type": typ, "data": payload})
async def _send_all():
dead = set()
for ws, user in list(_connections.items()):
try:
if ws.closed:
dead.add(ws)
continue continue
try: if host_name is not None and not _user_can_see_host(user, host_name):
asyncio.run_coroutine_threadsafe(ws.send(jmsg), _loop) continue
await ws.send_str(jmsg)
except Exception: except Exception:
to_close.append(ws) dead.add(ws)
logger.debug("ws.send exception: closed") for ws in dead:
for ws in to_close: _connections.pop(ws, None)
try:
asyncio.run_coroutine_threadsafe(ws.wait_closed(), _loop) asyncio.run_coroutine_threadsafe(_send_all(), _loop)
except Exception:
pass
if ws in _connections:
_connections.remove(ws)
return True return True
+8 -1
View File
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
[project] [project]
name = "hbd" name = "hbd"
version = "5.1.0" version = "5.1.14"
description = "Heartbeat monitoring system — client (hbc) and server (hbd)" description = "Heartbeat monitoring system — client (hbc) and server (hbd)"
readme = "README.md" readme = "README.md"
requires-python = ">=3.11" requires-python = ">=3.11"
@@ -31,8 +31,12 @@ server = [
"mattermostdriver>=7.3.0", "mattermostdriver>=7.3.0",
"aiohttp>=3.11", "aiohttp>=3.11",
"Jinja2>=3.1.6", "Jinja2>=3.1.6",
"matrix-nio>=0.24",
] ]
# Minimal client — hbc_mini only, no external dependencies
mini = []
# Install both client and server # Install both client and server
all = [ all = [
"hbd[client,server]", "hbd[client,server]",
@@ -53,6 +57,9 @@ dev = [
hbd = "hbd.server.cli:main" hbd = "hbd.server.cli:main"
hbc = "hbd.client.main:main" hbc = "hbd.client.main:main"
[tool.setuptools]
script-files = ["scripts/hb_install.sh", "scripts/hbc_mini.py"]
[tool.setuptools.packages.find] [tool.setuptools.packages.find]
where = ["."] where = ["."]
include = ["hbd*"] include = ["hbd*"]
+3 -1
View File
@@ -4,12 +4,14 @@ set -e
uv version --bump patch uv version --bump patch
VER=$(uv version --short) VER=$(uv version --short)
sed -i".bak" "s/__version__ = \"[0-9.]*\"\(.*\)$/__version__ = \"$VER\"\1/" hbd/__init__.py sed -i".bak" "s/__version__ = \"[0-9.]*\"\(.*\)$/__version__ = \"$VER\"\1/" hbd/__init__.py
sed -i".bak" "s/__version__ = \"[0-9.]*\"\(.*\)$/__version__ = \"$VER\"\1/" scripts/hbc_mini.py
# commit pyproject.toml # commit pyproject.toml
git commit -m "version $VER" pyproject.toml hbd/__init__.py git commit -m "version $VER" pyproject.toml hbd/__init__.py scripts/hbc_mini.py
git push git push
# tag version # tag version
git tag -a v$VER -m "Version $VER" git tag -a v$VER -m "Version $VER"
git push --tags git push --tags
rm hbd/__init__.py.bak rm hbd/__init__.py.bak
rm scripts/hbc_mini.py.bak
+115
View File
@@ -0,0 +1,115 @@
#!/bin/sh
# Helper script to install the heartbeat tools. By default, it will only
# install the heartbeat client, hbc. The server is installed when the arg 'server' is passed
# to the script. The script will install the heartbeat tools in a python
# virtual environment in ~/venvs/hbd. The hbd and hbc commands will be
# installed from the wheel and symlinked to ~/bin/hbd and ~/bin/hbc,
# respectively. If the virtual environment already exists, it will be
# reused. The script will also remove any existing symlinks for hbd and hbc
# in ~/bin before creating new ones.
set -e
what=$1
on_ha=0
where=""
venv=""
[ "$2" = "HA" ] && on_ha=1
[ -z "$what" ] && what="client"
if [ -d /homeassistant ]; then # if running from HA command line
echo "HA, running \"docker exec homeassistant /config/bin/hb_install.sh $@\""
docker exec homeassistant /config/bin/hb_install.sh $@ HA
rc=$?
if [ $rc -ne 0 ]; then
echo "Failed to install heartbeat in HA, please check the logs for more details"
exit 1
fi
exit 0
fi
if [ $on_ha -eq 1 ] || [ -r /.dockerenv ] && [ -d /config/bin ]; then
# Installing under docker on Home Assistant OS, using /config/bin for executables and /config/venvs for virtual environments
echo "Home Assistant OS detected, installing under docker"
where="/config/bin"
venv="/config/venvs"
else
if [ ! -d $HOME/.local/bin ] && [ ! -d $HOME/bin ]; then
echo "No suitable bin directory found in PATH, please add either $HOME/.local/bin or $HOME/bin to your PATH"
exit 1
fi
for where in $HOME/bin $HOME/.local/bin notset ; do
if echo ":$PATH:" | grep -q ":$where:" ; then
break
fi
done
if [ "$where" = "notset" ]; then
echo "No suitable bin directory found in PATH, please add either $HOME/.local/bin or $HOME/bin to your PATH"
exit 1
fi
if [ "$what" = "mini" ]; then
venv=""
else
venv="$HOME/venvs"
fi
fi
echo "Installing $what to $where"
if [ ! -z "$venv" ]; then
echo "Using virtual environment at $venv/hbd"
fi
if [ "$venv" != "" ] && [ ! -d $venv/hbd ]; then
arg=""
have_pip=$(python3 -c "import pip" 2>/dev/null &> /dev/null && echo "Installed" || echo "Not Installed")
if [ "$have_pip" = "Not Installed" ]; then
# some systems do not have pip installed by default, so we need to fetch get-pip.py and install pip
echo "pip is not installed, fetching get-pip.py and installing pip"
arg="--without-pip"
fi
mkdir -p $venv
have_venv=$(python3 -c "import venv" 2>/dev/null &> /dev/null && echo "Installed" || echo "Not Installed")
if [ "$have_venv" = "Not Installed" ]; then
if [ "$have_pip" = "Not Installed" ]; then
echo "python has no venv, and no pip to install virtualenv, cannot continue"
exit 1
fi
echo "python venv module not found, installing virtualenv"
python3 -m pip install --user virtualenv
python3 -m virtualenv $venv/hbd --system-site-packages $arg
else
python3 -m venv $venv/hbd --system-site-packages $arg
fi
. $venv/hbd/bin/activate
if [ -n "$arg" ]; then
curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py && python3 get-pip.py
fi
deactivate
fi
if [ ! -z "$venv" ]; then
. $venv/hbd/bin/activate
fi
if [ "$what" = "mini" ]; then
curl -s -o $where/hbc_mini https://git.wrede.ca/andreas/heartbeat/raw/branch/master/scripts/hbc_mini.py
chmod +x $where/hbc_mini
else
python3 -mpip install --upgrade --index-url https://git.wrede.ca/api/packages/andreas/pypi/simple/ --extra-index-url https://pypi.org/simple hbd[$what]
fi
if [ ! -z "$venv" ]; then
echo "linking executables to $where"
if [ "$what" = "server" ]; then
rm -f $where/hbd
ln -sf $(which hbd) $where/hbd
elif [ "$what" = "client" ]; then
rm -f $where/hbc
ln -sf $(which hbc) $where/hbc
fi
rm -f $where/hb_install.sh
ln -sf $(which hb_install.sh) $where/hb_install.sh
fi
echo "Installation complete. To upgrade, run the following:"
echo " $where/hb_install.sh $what"
echo "To install on another machine, run the following obtain the install script and run it:"
echo "from https://git.wrede.ca/andreas/heartbeat/raw/branch/master/scripts/hb_install.sh"
echo "and then run sh hb_install.sh [mini|client]"
+1154
View File
File diff suppressed because it is too large Load Diff
-25
View File
@@ -1,25 +0,0 @@
#!/bin/sh
# install the heartbeat tools. By default, this will install the hbc
# client only. The server is installed when the arg 'server' is passed
# to the script. The script will install the heartbeat tools in a python
# virtual environment in ~/venvs/hbd. The hbd and hbc commands will be
# installed from the wheel and symlinked to ~/bin/hbd and ~/bin/hbc,
# respectively. If the virtual environment already exists, it will be
# reused. The script will also remove any existing symlinks for hbd and hbc
# in ~/bin before creating new ones.
# hbd/hbc from wheel and create symlinks for hbd and hbc in ~/bin
set -e
if [ ! -d ~/venvs/hbd ]; then
mkdir -p ~/venvs
python3 -m venv ~/venvs/hbd --system-site-packages
fi
. ~/venvs/hbd/bin/activate
pip install 'git+ssh://git@git.wrede.ca/andreas/heartbeat.git'
rm -f ~/bin/hbd
rm -f ~/bin/hbc
ln -sf $(which hbd) ~/bin/hbd
ln -sf $(which hbc) ~/bin/hbc
+99
View File
@@ -0,0 +1,99 @@
import asyncio
import logging
import os
import stat
from hbd.client.plugins.nagios_runner import (
NagiosRunnerPlugin,
NAGIOS_OK,
NAGIOS_WARNING,
NAGIOS_CRITICAL,
NAGIOS_UNKNOWN,
)
def test_no_commands_sets_skip_reason():
plugin = NagiosRunnerPlugin(config={"commands": []})
result = asyncio.run(plugin.initialize())
assert result is False
assert plugin.skip_reason is not None
assert "nagios_runner.commands" in plugin.skip_reason
def test_stderr_used_when_stdout_empty(tmp_path):
script = tmp_path / "check_err.sh"
script.write_text("#!/bin/sh\necho 'error from stderr' >&2\nexit 2\n")
script.chmod(script.stat().st_mode | stat.S_IEXEC)
config = {"commands": [{"name": "t", "command": str(script)}], "timeout": 5}
plugin = NagiosRunnerPlugin(config=config)
asyncio.run(plugin.initialize())
data = asyncio.run(plugin._collect_metrics())
assert "error from stderr" in data["t_output"]
assert data["t_status_code"] == NAGIOS_CRITICAL
def test_stderr_appended_when_both_present(tmp_path):
script = tmp_path / "check_both.sh"
script.write_text("#!/bin/sh\necho 'OK - all good'\necho 'extra detail' >&2\nexit 0\n")
script.chmod(script.stat().st_mode | stat.S_IEXEC)
config = {"commands": [{"name": "t", "command": str(script)}], "timeout": 5}
plugin = NagiosRunnerPlugin(config=config)
asyncio.run(plugin.initialize())
data = asyncio.run(plugin._collect_metrics())
assert "OK - all good" in data["t_output"]
assert "extra detail" in data["t_output"]
assert data["t_status_code"] == NAGIOS_OK
def test_negative_returncode_maps_to_unknown():
# kill -9 $$ kills the shell itself; asyncio sees returncode -9
config = {"commands": [{"name": "t", "command": "kill -9 $$"}], "timeout": 5}
plugin = NagiosRunnerPlugin(config=config)
asyncio.run(plugin.initialize())
data = asyncio.run(plugin._collect_metrics())
assert data["t_status_code"] == NAGIOS_UNKNOWN
assert "signal" in data["t_output"].lower()
def test_absolute_path_not_found_warns(caplog):
fake_cmd = "/nonexistent_hbc_test_path/check_something"
config = {"commands": [{"name": "t", "command": fake_cmd}]}
plugin = NagiosRunnerPlugin(config=config)
with caplog.at_level(logging.WARNING, logger="plugin.nagios_runner"):
asyncio.run(plugin.initialize())
assert any("not found" in r.message for r in caplog.records)
def test_absolute_path_not_executable_warns(caplog, tmp_path):
non_exec = tmp_path / "check_test"
non_exec.write_text("#!/bin/sh\necho OK\n")
non_exec.chmod(0o644) # readable but not executable
config = {"commands": [{"name": "t", "command": str(non_exec)}]}
plugin = NagiosRunnerPlugin(config=config)
with caplog.at_level(logging.WARNING, logger="plugin.nagios_runner"):
asyncio.run(plugin.initialize())
assert any("not executable" in r.message for r in caplog.records)
def test_relative_path_not_checked(caplog):
# Relative paths (resolved via PATH) must not generate warnings
config = {"commands": [{"name": "t", "command": "echo OK"}]}
plugin = NagiosRunnerPlugin(config=config)
with caplog.at_level(logging.WARNING, logger="plugin.nagios_runner"):
asyncio.run(plugin.initialize())
assert not any(
"not found" in r.message or "not executable" in r.message
for r in caplog.records
)
+83
View File
@@ -0,0 +1,83 @@
import asyncio
import logging
import textwrap
from hbd.client.plugin import PluginLoader, PluginRegistry
def test_plugin_skip_reason_defaults_none(tmp_path):
plugin_code = textwrap.dedent("""
from hbd.client.plugin import MonitorPlugin
class MinimalPlugin(MonitorPlugin):
name = "minimal"
version = "1.0.0"
interval = 60
async def initialize(self):
return True
async def _collect_metrics(self):
return {}
""")
(tmp_path / "minimal.py").write_text(plugin_code)
registry = PluginRegistry()
loader = PluginLoader(registry)
asyncio.run(loader.load_from_directory(tmp_path))
plugin = registry.get("minimal")
assert plugin is not None
assert plugin.skip_reason is None
def test_loader_logs_info_when_skip_reason_set(tmp_path, caplog):
plugin_code = textwrap.dedent("""
from hbd.client.plugin import MonitorPlugin
class SkippablePlugin(MonitorPlugin):
name = "skippable"
version = "1.0.0"
interval = 60
async def initialize(self):
self.skip_reason = "not configured in yaml"
return False
async def _collect_metrics(self):
return {}
""")
(tmp_path / "skippable.py").write_text(plugin_code)
registry = PluginRegistry()
loader = PluginLoader(registry)
with caplog.at_level(logging.INFO, logger="plugin.loader"):
count = asyncio.run(loader.load_from_directory(tmp_path))
assert count == 0
assert any("skipped: not configured in yaml" in r.message for r in caplog.records)
assert not any("failed initialization" in r.message for r in caplog.records)
def test_loader_logs_warning_when_no_skip_reason(tmp_path, caplog):
plugin_code = textwrap.dedent("""
from hbd.client.plugin import MonitorPlugin
class FailPlugin(MonitorPlugin):
name = "fail"
version = "1.0.0"
interval = 60
async def initialize(self):
return False
async def _collect_metrics(self):
return {}
""")
(tmp_path / "fail_plugin.py").write_text(plugin_code)
registry = PluginRegistry()
loader = PluginLoader(registry)
with caplog.at_level(logging.WARNING, logger="plugin.loader"):
count = asyncio.run(loader.load_from_directory(tmp_path))
assert count == 0
assert any("failed initialization" in r.message for r in caplog.records)