Compare commits

..

48 Commits

Author SHA1 Message Date
andreas c70a4807dc version 5.1.2
Release / release (push) Successful in 6s
2026-04-25 07:25:06 +02:00
andreas 1a470e7cfa Fix plugin config lookup shadowed by CLIENT_DEFAULTS plugins key
CLIENT_DEFAULTS seeds "plugins": {} so raw_config.get("plugins", raw_config)
always returned the empty subdict instead of falling back to the full config.
Plugins configured at top-level (e.g. nagios_runner: ...) were therefore
never found, resulting in "No Nagios commands configured".

Now checks the plugins subdict first, then top-level keys, so both
config layouts work correctly.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-24 12:58:42 +02:00
andreas 990c658e65 Apply grace period to all threshold alerts before logging/notifying
Threshold alerts (plugin metrics, RTT) were firing immediately on the
first breach. Now every state transition to WARNING/CRITICAL starts a
grace-period timer (grace_seconds from the 'grace' config key). The
notification is deferred until the next heartbeat after grace_seconds
have elapsed. If the metric recovers within the grace window, both the
alert and the recovery are suppressed — no spurious pages for transient
spikes.

Two helper methods added to ThresholdChecker:
- _apply_grace: handles the state-change path (defer or suppress)
- _check_pending_or_renotify: handles the stable-alert path (fire
  deferred notification once grace expires, or fall through to reminders)

The overdue case is unchanged — on_overdue already fires only after
interval+grace seconds of silence, which is equivalent behaviour.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-24 12:00:40 +02:00
andreas b78d6ac0fe Fix RECOVER routing: use consistent level name and route via alerted channel
threshold.py was emitting level="RECOVERED" for metric recoveries, which
failed the is_recover check in send_notification (which only matched "RECOVER"),
bypassing _alerted_channels routing and the min_level bypass added in the
previous commit. Changed to "RECOVER" so all recovery paths are consistent.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-24 11:29:04 +02:00
andreas afd5060f59 Fix early reminder notifications and lost recovery notifications
- AlertState.update() now resets last_notification when the alert level
  changes, so a WARNING→CRITICAL escalation restarts the reminder interval
  rather than inheriting a nearly-expired timer.
- _dispatch_to_channel() bypasses min_level for RECOVER, so recovery
  notifications are delivered even after a server restart when
  _alerted_channels is empty and the fallback dispatch path is used.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-22 18:11:22 +02:00
andreas f61f7aebc2 Use python3 consistently 2026-04-19 09:49:30 +02:00
Andreas Wrede 5c382d2b8d One more nit 2026-04-13 09:31:35 -04:00
Andreas Wrede 35bba451f5 Various formating nits 2026-04-13 09:27:51 -04:00
Andreas Wrede 80edfba0c0 fix inconsistencies in page layout, add swiss clock 2026-04-13 08:45:50 -04:00
Andreas Wrede 6bc8de192e fix non-alerting of overdue hosts 2026-04-12 18:44:36 -04:00
Andreas Wrede 2d8166d04a unse python3 -mpip instead of plain pip 2026-04-12 18:44:11 -04:00
Andreas Wrede ab33d81b30 catch syntax wanring when parsing version string 2026-04-12 16:39:51 -04:00
Andreas Wrede 2c0328f36d update install.sh to handle missing venv module 2026-04-12 16:39:14 -04:00
Andreas Wrede fb8e27825d make install.sh work on systems withou pip 2026-04-12 14:16:44 -04:00
Andreas Wrede 1366c69cdc version 5.1.1
Release / release (push) Successful in 5s
2026-04-12 13:06:30 -04:00
Andreas Wrede d0c8c186f4 Fix typo 2026-04-12 13:04:17 -04:00
Andreas Wrede 19f7c8312e Mkae columns sortabel agian, check hbc version, provide modile html pages 2026-04-12 12:53:00 -04:00
Andreas Wrede 24b0e362fb provide cli function stop, restart and reload for hbd
Thought for 1s
2026-04-12 12:06:07 -04:00
Andreas Wrede 3a030548c0 Fix profile not updating 2026-04-12 11:57:12 -04:00
Andreas Wrede 094cb7ed9d Merge branch 'master' of git.wrede.ca:andreas/heartbeat 2026-04-12 11:23:28 -04:00
Andreas Wrede 0199ca4693 re-factor notifications, add sms and matrix as channels 2026-04-12 11:21:21 -04:00
Andreas Wrede 75344ebbbd re-factor notifications, add sms and matrix as channels 2026-04-12 11:04:00 -04:00
Andreas Wrede 7f049a4e26 accept websocket connection on http:.../ws 2026-04-12 06:44:32 -04:00
Andreas Wrede 6559f5462c Merge branch 'master' of git.wrede.ca:andreas/heartbeat 2026-04-12 06:34:28 -04:00
Andreas Wrede 6556d35f97 Merge branch 'master' of git.wrede.ca:andreas/heartbeat 2026-04-12 06:32:52 -04:00
Andreas Wrede dec96a0da6 Merge branch 'master' of git.wrede.ca:andreas/heartbeat 2026-04-11 16:40:02 -04:00
Andreas Wrede 8d3de01117 Update install script 2026-04-11 16:36:20 -04:00
Andreas Wrede 5bedf026b1 Update install script 2026-04-11 16:19:41 -04:00
Andreas Wrede daf5277507 version 5.1.0
Release / release (push) Successful in 5s
2026-04-11 15:26:37 -04:00
Andreas Wrede ee3b72878f Add a ping monitor 2026-04-11 15:25:23 -04:00
Andreas Wrede 6217f7a124 fix bogus notification on new clients 2026-04-10 13:39:18 -04:00
Andreas Wrede 2468386f24 adjust default log, pick and config locations. renotify on critical only, make user sessions persistem 2026-04-10 13:24:57 -04:00
Andreas Wrede 2015195112 Grace interval on restart of hbd, fix SIGHUP processing 2026-04-10 12:58:38 -04:00
Andreas Wrede 3426185383 Set SO_TIMESTAMP correctly for the various platforms 2026-04-10 11:19:47 -04:00
Andreas Wrede 9eedbafe97 Show overdue in alerts instead of null 2026-04-10 09:20:28 -04:00
Andreas Wrede a5f31c5cb5 update picked data strucures 2026-04-10 09:18:38 -04:00
Andreas Wrede 2f72cf0118 typo 2026-04-10 09:17:57 -04:00
Andreas Wrede c56e77c2c1 Merge branch 'master' of git.wrede.ca:andreas/heartbeat 2026-04-10 08:20:40 -04:00
Andreas Wrede e9aa7a6f8b info only if no nagios command is defined 2026-04-10 08:19:59 -04:00
Andreas Wrede a75a8a4087 warn only if no nagios command is defined 2026-04-10 08:14:31 -04:00
Andreas Wrede ba27d2e300 Add count to rtt threshold 2026-04-10 08:07:50 -04:00
Andreas Wrede 381e37efce fix log-section height 2026-04-10 08:01:22 -04:00
Andreas Wrede 97dfc08f4d fix log level settiung 2026-04-10 08:00:51 -04:00
Andreas Wrede d281ac5a70 provide defaults for threshold_configs 2026-04-10 07:47:39 -04:00
Andreas Wrede 812bbf8555 Merge branch 'master' of git.wrede.ca:andreas/heartbeat 2026-04-09 13:02:17 -04:00
Andreas Wrede e6b7a1aa27 drop config file 2026-04-09 13:02:10 -04:00
Andreas Wrede 90f47ad018 drop config file 2026-04-09 13:00:07 -04:00
Andreas Wrede cc458e8972 update README 2026-04-09 08:33:25 -04:00
40 changed files with 2273 additions and 1574 deletions
+4 -4
View File
@@ -24,11 +24,11 @@ jobs:
- name: Install build tools - name: Install build tools
run: | run: |
python -m pip install --upgrade pip python3 -m pip install --upgrade pip
pip install build twine python3 -m pip install build twine
- name: Build package - name: Build package
run: python -m build run: python3 -m build
- name: Extract version from tag - name: Extract version from tag
id: get_version id: get_version
@@ -39,7 +39,7 @@ jobs:
TWINE_USERNAME: ${{ secrets.PYPI_USERNAME }} TWINE_USERNAME: ${{ secrets.PYPI_USERNAME }}
TWINE_PASSWORD: ${{ secrets.PYPI_TOKEN }} TWINE_PASSWORD: ${{ secrets.PYPI_TOKEN }}
run: | run: |
python -m twine upload --repository-url https://git.wrede.ca/api/packages/andreas/pypi dist/* python3 -m twine upload --repository-url https://git.wrede.ca/api/packages/andreas/pypi dist/*
- name: Create release - name: Create release
uses: actions/gitea-release-action@v1 uses: actions/gitea-release-action@v1
+1
View File
@@ -11,3 +11,4 @@ dist/
*.egg-info/ *.egg-info/
ssl/ ssl/
uv.lock uv.lock
.hb.yaml
-279
View File
@@ -1,279 +0,0 @@
#name: "w02"
hb_port: 50003
hbd_host: ''
#logfile: "/home/andreas/public_html/messages/andreas"
logfile: "/home/andreas/logs/heartbeat/heartbeat.log"
#logfile: "/Users/andreas/public_html/messages/andreas"
logfmt: "msg"
grace: 40
interval: 10
autosave_interval: 300 # Autosave interval in seconds (default: 5 minutes)
users:
andreas:
full_name: Andreas Wrede
password: pbkdf2:sha256:260000:eece9cdaebc22247566f78983bf5b2a3:f8c74cc057c5590943c115a60bac62f9458e9ba0d2e7e7421b6f0fe5d860e18f # hbd passwd andreas
avatar: /home/andreas/.avatar/Andreas-avatar3-small.png
admin: true
ops:
full_name: Operations Team
password: pbkdf2:sha256:260000:... # hbd passwd ops
admin: false
readonly:
full_name: Read-Only User
password: pbkdf2:sha256:260000:... # hbd
default_owner: andreas
hosts:
weekend:
owner: andreas
managers: [ops]
monitors: [readonly]
# Notification Channels - Define notification providers centrally
# Each channel has a type (pushover, email, signal, mattermost) and type-specific configuration
notification_channels:
pushover_standard:
type: pushover
token: ac7NLX2rPjXFareeDgLpXNoDf4iFmf
user: uDhH33UjQQDYtNzJb1ThRiWb9ingGK
signal_andreas:
type: signal
cli_path: /usr/local/bin/signal-cli
user: +14168226179
recipient: +14168226179
email_andreas:
type: email
recipients: [aew.hbd.notify@wrede.ca]
sender: aew.hbd@wrede.ca
smtp_server: smtp.fastmail.com
smtp_port: 587
smtp_user: andreas@wrede.ca
smtp_password: pvtvefyp5gbhnch2
# Example additional channels (commented out)
# pushover_urgent:
# type: pushover
# token: your-app-token
# user: your-user-key
#
mattermost_devops:
type: mattermost
host: mattermost.example.com
token: webhook-token
channel: devops-alerts
username: heartbeat-bot
icon: https://example.com/heartbeat-icon.png
# Default notification channels (used if host doesn't specify channels)
default_notification_channels: [pushover_standard]
# Host definitions - combines threshold mapping, watch status, DNS updates, and notifications
hosts:
wentworth:
threshold_config: default
watch: true
notification_channels: [pushover_standard]
dyndns: false
y:
threshold_config: default
watch: true
notification_channels: [pushover_standard]
dyndns: false
winter:
threshold_config: default
watch: true
notification_channels: [pushover_standard]
dyndns: false
wally:
threshold_config: freebsd_server
watch: false
notification_channels: [pushover_standard]
dyndns: false
eris:
threshold_config: truenas_server
watch: false
notification_channels: [pushover_standard]
dyndns: false
haschloss:
threshold_config: default
watch: false
dyndns: true
wayback:
threshold_config: default
watch: false
notification_channels: [pushover_standard]
dyndns: true
wertvoll:
threshold_config: default
watch: false
notification_channels: [pushover_standard]
dyndns: true
weekend:
threshold_config: freebsd_server
watch: false
notification_channels: [pushover_standard]
dyndns: true
cotgate:
threshold_config: default
watch: false
dyndns: true
rvgate:
threshold_config: default
watch: false
dyndns: true
draper:
threshold_config: default
watch: false
notification_channels: [pushover_standard]
dyndns: true
# Hosts to drop/ignore
drophosts: {"unknown", "wookie15", "wort"}
nsupdate_bin: "/usr/local/bin/nsupdate"
dyndomains: {"wrede.org"}
ws_port: 50005
# wss_port: 50006 # Commented out - use plain WebSocket instead of secure WSS
# cert_path: "/usr/local/etc/letsencrypt/live/hbd.wrede.ca/"
# cert_path: "test/"
# CERT_PATH = "./test/"
# wss_pem: "fullchain.pem"
# wss_key: "privkey.pem"
journal_enabled: true # Enable/disable journaling
journal_dir: /home/andreas/logs/heartbeat # Journal directory
journal_file: messages.journal # Base filename
journal_max_size: 104857600 # Max size (100MB default)
journal_max_backups: 10 # Number of backups to keep
threshold_configs:
default:
thresholds:
cpu_monitor:
cpu_percent:
warning: 80.0
critical: 90.0
memory_monitor:
percent:
warning: 85.0
critical: 95.0
disk_monitor:
partitions:
/:
percent:
warning: 85.0
critical: 90.0
rtt:
warning: 200
critical: 250.0
freebsd_server:
thresholds:
cpu_monitor:
cpu_percent:
warning: 80.0
critical: 90.0
memory_monitor:
memory_percent:
warning: 97.0
critical: 100.0
disk_monitor:
partitions:
/:
percent:
warning: 85.0
critical: 90.0
nagios_runner:
# overall_status_code:
# warning: 1
# critical: 2
# operator: ">="
load_status:
warning: WARNING
critical: CRITICAL
operator: "=="
ups_load:
display: "load to high: {ups_output}"
warning: 70
critical: 80
operator: ">="
ups_status_code:
display: "{ups_output}"
warning: 1
critical: 2
operator: ">="
nextcloud_apps_status_code:
display: "{nextcloud_apps_output}"
warning: 1
critical: 2
operator: ">="
rtt:
warning: 200
critical: 250.0
truenas_server:
thresholds:
cpu_monitor:
cpu_percent:
warning: 80.0
critical: 90.0
memory_monitor:
percent:
warning: 3.0
critical: 95.0
disk_monitor:
partitions:
/:
percent:
warning: 85.0
critical: 90.0
nagios_runner:
# overall_status_code:
# warning: 1
# critical: 2
# operator: ">="
load_status:
warning: WARNING
critical: CRITICAL
operator: "=="
ups_load:
display: "load to high: {ups_output}"
WARNING: 70
CRITICAL: 80
OPERATOR: ">="
ups_status_code:
DISPLAY: "{ups_output}"
warning: 1
critical: 2
operator: ">="
nextcloud_apps_status_code:
display: "{nextcloud_apps_output}"
warning: 1
critical: 2
operator: ">="
rtt:
warning: 120
critical: 250.0
+6 -5
View File
@@ -4,12 +4,13 @@
// For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387 // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
"version": "0.2.0", "version": "0.2.0",
"configurations": [ "configurations": [
{ {
"name": "Python: Run hbd (module)", "name": "Python: Run hbd (module)",
"type": "debugpy", "type": "debugpy",
"request": "launch", "request": "launch",
"module": "hbd.server.cli", "module": "hbd.server.cli",
"args": ["-c", "/home/andreas/git/heartbeat/.hb.yaml", "-f", "-v", "-x"], "args": ["-c", "~/.hb.yaml", "-f", "-v"],
"cwd": "${workspaceFolder}", "cwd": "${workspaceFolder}",
"env": { "env": {
"PYTHONPATH": "${workspaceFolder}" "PYTHONPATH": "${workspaceFolder}"
@@ -28,14 +29,14 @@
] ]
}, },
{ {
"name": "Python: Run hbd with debugpy (listen)", "name": "Python: Run hbc (module)",
"type": "debugpy", "type": "debugpy",
"request": "launch", "request": "launch",
"module": "debugpy", "module": "hbd.client.main",
"args": ["--listen", "5678", "--wait-for-client", "-m", "hbd.server.cli", "-c", ".hb.yaml", "-f", "-v"], "args": ["-c", "~/.hbc.yaml", "-v", "winter"],
"cwd": "${workspaceFolder}",
"env": { "PYTHONPATH": "${workspaceFolder}" }, "env": { "PYTHONPATH": "${workspaceFolder}" },
"console": "integratedTerminal", "console": "integratedTerminal",
"justMyCode": false
} }
] ]
} }
+54 -36
View File
@@ -76,7 +76,7 @@ See [docs/NAGIOS_INTEGRATION.md](docs/NAGIOS_INTEGRATION.md) for complete integr
### Creating Custom Plugins ### Creating Custom Plugins
```python ```python
from hbd.plugin import MonitorPlugin from hbd.client.plugin import MonitorPlugin
class DiskMonitorPlugin(MonitorPlugin): class DiskMonitorPlugin(MonitorPlugin):
name = "disk_monitor" name = "disk_monitor"
@@ -89,7 +89,7 @@ class DiskMonitorPlugin(MonitorPlugin):
} }
``` ```
Place plugins in `hbd/plugins/` and they'll be automatically discovered and loaded by the client. Place plugins in `hbd/client/plugins/` and they'll be automatically discovered and loaded by the client.
--- ---
@@ -368,7 +368,7 @@ See [docs/HTTP_API.md](docs/HTTP_API.md) for complete API documentation includin
Prerequisites: Prerequisites:
- Python 3.10+ (project uses language features from recent Python) - Python 3.11+ (project uses language features from recent Python)
- `nsupdate` (for DNS updates) if using dynamic DNS - `nsupdate` (for DNS updates) if using dynamic DNS
Install dependencies (recommended into a venv): Install dependencies (recommended into a venv):
@@ -389,7 +389,7 @@ hbd -c .hb.yaml -f -v
You can also run it directly via the package entrypoint after installation: You can also run it directly via the package entrypoint after installation:
```bash ```bash
python -m hbd.cli -c /path/to/config.yaml python -m hbd.server.cli -c /path/to/config.yaml
``` ```
### Running the Client ### Running the Client
@@ -397,14 +397,23 @@ python -m hbd.cli -c /path/to/config.yaml
The heartbeat client (`hbc`) sends periodic heartbeats and plugin data to the server: The heartbeat client (`hbc`) sends periodic heartbeats and plugin data to the server:
```bash ```bash
# Basic usage pointing to server # Basic usage pointing to server (host is a positional argument)
python -m hbd.hbc --server your-server.example.com hbc your-server.example.com
# With custom configuration # Run as daemon with a config file
python -m hbd.hbc --server 192.168.1.100 --port 50003 --interval 30 hbc -d -c /etc/hbc.yaml your-server.example.com
# Run with specific plugins enabled/disabled # Send a one-off boot message
python -m hbd.hbc --server hbd.local --disable-plugin os_info hbc --boot your-server.example.com
# Verbose output
hbc -v your-server.example.com
```
You can also run it via the module entrypoint:
```bash
python -m hbd.client.main your-server.example.com
``` ```
Client configuration can also be specified in YAML: Client configuration can also be specified in YAML:
@@ -438,30 +447,29 @@ This repository includes a ready-to-use `.vscode/launch.json` with configuration
- Ensure the **Python** extension is installed and select the project `.venv` as the interpreter (bottom-left of VS Code). - Ensure the **Python** extension is installed and select the project `.venv` as the interpreter (bottom-left of VS Code).
- Use **F5** and pick one of these configurations from the Run view: - Use **F5** and pick one of these configurations from the Run view:
- **Python: Run hbd (module)** — runs `hbd.cli` as a module and sets `PYTHONPATH` to the workspace root (recommended). - **Python: Run hbd (module)** — runs `hbd.server.cli` as a module and sets `PYTHONPATH` to the workspace root (recommended).
- **Python: Run hbd with debugpy (listen)** — launches `debugpy` and `hbd` together; useful when you want the process to listen for a debugger. - **Python: Run hbd with debugpy (listen)** — launches `debugpy` and `hbd` together; useful when you want the process to listen for a debugger.
- **Python: Attach (localhost:5678)** — attach the debugger to a running process started with `debugpy`. - **Python: Attach (localhost:5678)** — attach the debugger to a running process started with `debugpy`.
To start `hbd` manually and wait for the debugger to attach, run: To start `hbd` manually and wait for the debugger to attach, run:
```bash ```bash
PYTHONPATH=. python -m debugpy --listen 5678 --wait-for-client -m hbd.cli -c .hb.yaml -f -v PYTHONPATH=. python -m debugpy --listen 5678 --wait-for-client -m hbd.server.cli -c .hb.yaml -f -v
``` ```
Set breakpoints in modules such as `hbd/udp.py`, `hbd/dns.py`, or `hbd/server.py`, and use the **Attach** configuration to connect. Use `justMyCode: false` if you need to step into third-party code. Set breakpoints in modules such as `hbd/server/udp.py`, `hbd/server/dns.py`, or `hbd/server/main.py`, and use the **Attach** configuration to connect. Use `justMyCode: false` if you need to step into third-party code.
--- ---
## 🛠 Configuration ## 🛠 Configuration
`hbd` reads YAML configuration (optional). If `PyYAML` is not installed, built-in defaults are used. Example configuration keys (see `hbd/config.py`): `hbd` reads YAML configuration (optional). If `PyYAML` is not installed, built-in defaults are used. Example configuration keys (see `hbd/server/config.py`):
- `hb_port`: UDP port to listen for heartbeats (default: 50003) - `hb_port`: UDP port to listen for heartbeats (default: 50003)
- `hbd_port`: internal control port (default: 50004) - `hbd_port`: internal control port (default: 50004)
- `hbd_host`: bind address for HTTP/WSS - `hbd_host`: bind address for HTTP/WSS
- `pickfile`: path for persisted state - `pickfile`: path for persisted state
- `logfile`: path to log file - `logfile`: path to log file
- `logfmt`: `text` or `msg`
- `pushsrv`: push service (`pushover`|`mattermost`|`all`) - `pushsrv`: push service (`pushover`|`mattermost`|`all`)
- `interval` / `grace`: heartbeat timing configuration - `interval` / `grace`: heartbeat timing configuration
- `dyndomains`: list of dyndomains to update via `nsupdate` - `dyndomains`: list of dyndomains to update via `nsupdate`
@@ -487,29 +495,39 @@ nsupdate_bin: /usr/bin/nsupdate
pushsrv: pushover pushsrv: pushover
``` ```
> Tip: `config.DEFAULTS` in `hbd/config.py` contains the canonical defaults and accepted configuration keys. > Tip: `SERVER_DEFAULTS` in `hbd/server/config.py` contains the canonical defaults and accepted configuration keys.
--- ---
## 🔧 Architecture & Modules ## 🔧 Architecture & Modules
- `hbd.proto` — serialization/deserialization of heartbeat messages (supports compressed payloads and plugin data) The package is organized into three subpackages:
- `hbd.udp` — UDP parsing and `handle_datagram` implementation (main state machine)
- `hbd.dns` — `create_nsupdate_payload`, `nsupdate`, and an asyncio DNS worker (`start_dns_worker`). **`hbd.common`** — shared code used by both client and server:
The DNS worker now runs as an `asyncio` task and the package exposes a - `hbd.common.proto` — serialization/deserialization of heartbeat messages (supports compressed payloads and plugin data)
small thread-safe bridge so legacy synchronous code can `put()` updates - `hbd.common.utils` — small utility helpers (`shortname`, `dur`, `initlog`)
into the queue; there is no longer a permanently-blocking background
`threading.Thread`. **`hbd.server`** — the heartbeat daemon (`hbd`):
- `hbd.notify` — email and push notification helpers - `hbd.server.cli` — CLI entrypoint and argument parsing
- `hbd.ws` — WebSocket server and thread-safe broadcast helpers - `hbd.server.main` — async orchestration to run UDP/HTTP/WSS components
- `hbd.http` — HTTP handler factory for the status UI/API - `hbd.server.udp` — UDP parsing and `handle_datagram` implementation (main state machine)
- `hbd.journal` — message journal with size-based log rotation and backup management - `hbd.server.dns` — `create_nsupdate_payload`, `nsupdate`, and an asyncio DNS worker (`start_dns_worker`).
- `hbd.plugin` — plugin framework with base classes, registry, and dynamic loader The DNS worker runs as an `asyncio` task and the package exposes a small thread-safe bridge
- `hbd.plugins/` — built-in plugins (os_info, cpu_monitor, memory_monitor, disk_monitor, network_monitor, filesystem_info, nagios_runner) so legacy synchronous code can `put()` updates into the queue.
- `hbd.hbc` — heartbeat client that sends heartbeats and plugin data to server - `hbd.server.notify` — email and push notification helpers
- `hbd.utils` — small utility helpers (`shortname`, `dur`, `initlog`) - `hbd.server.ws` — WebSocket server and thread-safe broadcast helpers
- `hbd.cli` — CLI entrypoint and argument parsing - `hbd.server.http` — HTTP handler factory for the status UI/API
- `hbd.server` — async orchestration to run UDP/HTTP/WSS components - `hbd.server.journal` — message journal with size-based log rotation and backup management
- `hbd.server.threshold` — threshold alerting engine
- `hbd.server.monitor` — host state monitoring
- `hbd.server.hbdclass` — `Host` class and shared server state
- `hbd.server.config` — configuration loader and defaults
**`hbd.client`** — the heartbeat client (`hbc`):
- `hbd.client.main` — client entrypoint; sends heartbeats and plugin data to the server
- `hbd.client.plugin` — plugin framework with base classes, registry, and dynamic loader
- `hbd.client.plugins/` — built-in plugins (os_info, cpu_monitor, memory_monitor, disk_monitor, network_monitor, filesystem_info, nagios_runner)
- `hbd.client.config` — client configuration loader
This modular layout makes the code easier to test and maintain. This modular layout makes the code easier to test and maintain.
@@ -517,12 +535,12 @@ This modular layout makes the code easier to test and maintain.
- The main runtime is asyncio-based. Services (UDP listener, HTTP server, WebSocket server, monitor, and DNS worker) run as asyncio tasks. - The main runtime is asyncio-based. Services (UDP listener, HTTP server, WebSocket server, monitor, and DNS worker) run as asyncio tasks.
- On SIGINT/SIGTERM the server triggers a graceful shutdown: it cancels active tasks, signals the DNS worker via a sentinel, and cleans up resources before exit. - On SIGINT/SIGTERM the server triggers a graceful shutdown: it cancels active tasks, signals the DNS worker via a sentinel, and cleans up resources before exit.
- The DNS update worker is implemented as an `asyncio` task; synchronous producers can still enqueue DNS updates via a small thread-safe bridge available at `hbd.hbdclass.Host.dnsQ`. - The DNS update worker is implemented as an `asyncio` task; synchronous producers can still enqueue DNS updates via a small thread-safe bridge available at `hbd.server.hbdclass.Host.dnsQ`.
**Templates & Static Files** **Templates & Static Files**
- Template files are located under `hbd/templates` by default. The HTTP server resolves templates relative to the `hbd` package but the path can be overridden with the `templates_dir` config key. - Template files are located under `hbd/server/templates`. The HTTP server resolves templates relative to the `hbd.server` package but the path can be overridden with the `templates_dir` config key.
- Static assets (CSS/JS/images) are served from `hbd/static` via the `/static/<path>` HTTP route. Place your static files in that directory or configure the HTTP server as needed. - Static assets (CSS/JS/images) are served from `hbd/server/static` via the `/static/<path>` HTTP route.
--- ---
+1 -1
View File
@@ -59,7 +59,7 @@ Server-specific defaults:
- `hb_port`: Port to listen for heartbeats (default: 50003) - `hb_port`: Port to listen for heartbeats (default: 50003)
- `hbd_port`: HTTP API port (default: 50004) - `hbd_port`: HTTP API port (default: 50004)
- `ws_port`: WebSocket port (default: 50005) - `ws_port`: WebSocket port (default: 50005)
- `logfile`, `logfmt`: Logging configuration - `logfile`: Log file path
- `pushsrv`, `pushover_token`, etc.: Notification settings - `pushsrv`, `pushover_token`, etc.: Notification settings
- `watchhosts`, `dyndnshosts`: Host monitoring - `watchhosts`, `dyndnshosts`: Host monitoring
- `smtpserver`, etc.: Email settings - `smtpserver`, etc.: Email settings
+40
View File
@@ -0,0 +1,40 @@
async def send_sms(hass, user, password, sender_did, call):
"""Send SMS message using multipart form-data like MMS."""
_LOGGER = logging.getLogger(__name__)
recipient = call.data.get("recipient")
message = call.data.get("message")
if not recipient or not message:
_LOGGER.error("Recipient or message missing.")
return
# Build form data dictionary
form_data = {
'api_username': str(user),
'api_password': str(password),
'did': str(sender_did),
'dst': str(recipient),
'message': str(message),
'method': 'sendSMS'
}
async with aiohttp.ClientSession() as session:
with aiohttp.MultipartWriter("form-data") as mp:
for key, value in form_data.items():
part = mp.append(value)
part.set_content_disposition('form-data', name=key)
_LOGGER.error("voipms_sms: sending SMS: %s", mp)
async with session.post(REST_ENDPOINT, data=mp) as response:
response_text = await response.text()
if response.status == 200:
response_json = json.loads(response_text)
if response_json['status'] == "success":
_LOGGER.info("voipms_sms: SMS sent successfully: %s", response_text)
else:
_LOGGER.error("voipms_sms: SMS not sent: %s", response_text)
else:
_LOGGER.error("voipms_sms: Failed to send SMS. Status: %s, Response: %s", response.status, response_text)
-1
View File
@@ -81,7 +81,6 @@ The following settings **cannot** be reloaded and require a service restart:
- **Logging** - **Logging**
- `logfile` - Log file path - `logfile` - Log file path
- `logfmt` - Log format
- **Journal Settings** - **Journal Settings**
- `journal_enabled` - Enable/disable journaling - `journal_enabled` - Enable/disable journaling
+235 -473
View File
@@ -2,532 +2,294 @@
## Overview ## Overview
The Heartbeat Monitoring System includes a flexible notification system that can send alerts through multiple channels including Email, Pushover, Signal, and Mattermost. The system supports centralized channel definitions with per-host routing, allowing fine-grained control over notification delivery. Notifications are dispatched to the **owner and managers** of a host, each via their own configured notification channels. Channel definitions are global; users reference them by name. No users configured → no notifications sent.
## Architecture ## Architecture
### Components ```
Alert event (udp.py / threshold.py)
└─ notify.send_notification(host_name, Notification)
├─ look up host.owner + host.managers
├─ for each user → user.notification_channels
└─ for each channel → _dispatch_to_channel (filtered by min_level)
```
1. **Notification Channels** (`notification_channels` in config) Every notification carries:
- Centralized definitions of notification providers - **title** — `[LEVEL] hostname` (e.g. `[CRITICAL] webserver01`)
- Each channel has a type and type-specific credentials - **body** — detail message (metric value, threshold, duration)
- Reusable across multiple hosts - **url** — link to the plugin metrics page (`{base_url}/plugins#{hostname}`)
- **level** — `RECOVER | WARNING | CRITICAL | INFO`
2. **Channel Dispatcher** (`hbd/server/notify.py`)
- `pushmsg_for_host(hostname, message)`: Main entry point for host-specific notifications
- `_dispatch_to_channel(channel_name, channel_config, message)`: Routes to specific provider
- Provider functions: `pushover()`, `pushsignal()`, `pushmattermost()`, `send_email()`
3. **Configuration Utilities** (`hbd/server/config.py`)
- `get_notification_channels_for_host(config, hostname)`: Retrieves channel names for a host
- `get_notification_channels_config(config, hostname)`: Retrieves full channel configurations
- `get_channel_config(config, channel_name)`: Gets configuration for a specific channel
4. **Integration Points**
- **Threshold alerts**: `threshold.py` calls `notify_mod.pushmsg_for_host()`
- **Heartbeat events**: `udp.py` calls `notify_mod.pushmsg_for_host()` for boot/shutdown/overdue
- **Custom alerts**: Any code can call `notify_mod.pushmsg_for_host(hostname, message)`
## Configuration ## Configuration
### Centralized Channel Definitions ### Base URL
Define notification channels once in your configuration file: Set `base_url` so notification links point to your hbd instance:
```yaml
base_url: https://hbd.example.com
```
### Global channel definitions
Define channels once; reference them by name from user configs:
```yaml ```yaml
notification_channels: notification_channels:
# Signal notifications
signal_ops:
type: signal
cli_path: /usr/local/bin/signal-cli
user: +1234567890 # Your Signal number
recipient: +1234567890 # Recipient number
signal_oncall: pushover_ops:
type: signal type: pushover
cli_path: /usr/local/bin/signal-cli token: your-app-token
user: +1234567890 user: your-user-key
recipient: +0987654321 # Different recipient min_level: WARNING # optional, default: WARNING
# Email notifications
email_ops: email_ops:
type: email type: email
recipients: recipients: [ops@example.com]
- ops@example.com sender: hbd@example.com
- alerts@example.com
sender: heartbeat@example.com
smtp_server: smtp.example.com smtp_server: smtp.example.com
smtp_port: 587 smtp_port: 587
smtp_user: heartbeat@example.com smtp_user: hbd@example.com
smtp_password: your-smtp-password smtp_password: secret
min_level: WARNING
email_devteam: matrix_oncall:
type: email type: matrix
recipients: [dev-alerts@example.com] homeserver: https://matrix.example.org
sender: heartbeat-dev@example.com access_token: syt_xxx
smtp_server: smtp.example.com room_id: "!abc:matrix.example.org"
smtp_port: 587 min_level: CRITICAL # only send critical alerts to this room
smtp_user: heartbeat-dev@example.com
smtp_password: your-smtp-password
# Pushover notifications sms_oncall:
pushover_urgent: type: sms_voipms
type: pushover api_user: me@example.com
token: your-pushover-app-token api_password: secret
user: your-pushover-user-key did: "5551234567" # your voip.ms DID number
dst: "5559876543" # destination number
min_level: CRITICAL
pushover_normal: signal_ops:
type: pushover
token: your-pushover-app-token
user: another-user-key
# Mattermost notifications
mattermost_devops:
type: mattermost
host: mattermost.example.com
token: your-webhook-token
channel: devops-alerts
username: heartbeat-bot
icon: https://example.com/heartbeat-icon.png
```
### Default Notification Channels
Specify default channels for hosts that don't have specific channel assignments:
```yaml
default_notification_channels:
- email_ops
- mattermost_devops
```
Hosts without `notification_channels` defined will use these defaults.
### Per-Host Channel Assignment
Assign specific channels to each host in the `hosts` section:
```yaml
hosts:
# Critical production web server - multiple channels for redundancy
prod-web-01:
threshold_config: high_sensitivity
watch: true
notification_channels:
- signal_oncall # Immediate mobile notification
- pushover_urgent # Secondary mobile notification
- email_ops # Email for record keeping
dyndns: false
# Database server - ops team notifications only
prod-db-01:
threshold_config: database
watch: true
notification_channels:
- signal_ops
- email_ops
dyndns: false
# Development server - email only, no urgent notifications
dev-server-01:
threshold_config: low_sensitivity
watch: false
notification_channels:
- email_devteam
dyndns: false
# Test server - uses default_notification_channels
test-server-01:
threshold_config: default
watch: false
dyndns: false
# No notification_channels specified = uses default_notification_channels
```
## Channel Types
### Email
Sends notifications via SMTP.
**Configuration fields:**
```yaml
type: email
recipients: [email1@example.com, email2@example.com] # Required: List of recipients
sender: heartbeat@example.com # Required: From address
smtp_server: smtp.example.com # Required: SMTP server hostname
smtp_port: 587 # Optional: Default 587
smtp_user: heartbeat@example.com # Optional: For authenticated SMTP
smtp_password: your-password # Optional: For authenticated SMTP
```
**Features:**
- Supports multiple recipients
- TLS/STARTTLS support on port 587
- Authenticated and unauthenticated SMTP
**Example:**
```yaml
notification_channels:
email_critical:
type: email
recipients: [admin@example.com, oncall@example.com]
sender: alerts@example.com
smtp_server: smtp.fastmail.com
smtp_port: 587
smtp_user: alerts@example.com
smtp_password: app-specific-password
```
### Pushover
Sends push notifications to mobile devices via Pushover API.
**Configuration fields:**
```yaml
type: pushover
token: your-application-token # Required: Your Pushover app token
user: your-user-key # Required: Recipient's user key
```
**Features:**
- Instant mobile push notifications
- Works on iOS and Android
- Supports delivery confirmations
**Setup:**
1. Create a Pushover account at https://pushover.net
2. Create an application to get your app token
3. Note your user key from your account dashboard
**Example:**
```yaml
notification_channels:
pushover_admin:
type: pushover
token: azGDORePK8gMaC0QOYAMyEEuzJnyUi
user: uQiRzpo4DXghDmr9QzzfQu27cmVRsG
```
### Signal
Sends notifications via Signal messenger using signal-cli.
**Configuration fields:**
```yaml
type: signal
cli_path: /usr/local/bin/signal-cli # Optional: Path to signal-cli binary
user: +1234567890 # Required: Your Signal phone number
recipient: +0987654321 # Required: Recipient phone number
```
**Prerequisites:**
1. Install signal-cli: https://github.com/AsamK/signal-cli
2. Register signal-cli with your phone number:
```bash
signal-cli -u +1234567890 register
signal-cli -u +1234567890 verify CODE
```
3. Ensure signal-cli is in PATH or specify full path in config
**Features:**
- End-to-end encrypted messaging
- Works without phone being online
- No API fees or rate limits
**Example:**
```yaml
notification_channels:
signal_admin:
type: signal type: signal
cli_path: /usr/local/bin/signal-cli cli_path: /usr/local/bin/signal-cli
user: +12025551234 user: +12025551234
recipient: +12025559999 recipient: +12025559999
```
### Mattermost mattermost_devops:
Sends notifications to Mattermost team chat via incoming webhooks.
**Configuration fields:**
```yaml
type: mattermost
host: mattermost.example.com # Required: Mattermost server hostname
token: your-webhook-token # Required: Incoming webhook token
channel: channel-name # Required: Target channel name
username: heartbeat-bot # Optional: Bot display name
icon: https://example.com/icon.png # Optional: Bot icon URL
```
**Prerequisites:**
1. Enable incoming webhooks in Mattermost
2. Create an incoming webhook for your team
3. Note the webhook token from the webhook URL
**Features:**
- Team-wide visibility
- Rich formatting support
- Message threading
**Example:**
```yaml
notification_channels:
mattermost_ops:
type: mattermost type: mattermost
host: chat.example.com host: mattermost.example.com
token: abc123def456ghi789 token: webhook-token
channel: infrastructure-alerts channel: devops-alerts
username: heartbeat-monitor username: heartbeat-bot
icon: https://example.com/heartbeat-icon.png
``` ```
## Notification Events ### Users with notification channels
The system sends notifications for various events: Each user lists which global channels they receive notifications on:
### Threshold Alerts ```yaml
users:
alice:
full_name: Alice Smith
password: pbkdf2:sha256:...
admin: true
notification_channels: [pushover_ops, email_ops]
When monitored metrics exceed configured thresholds: bob:
full_name: Bob Jones
- **State changes**: OK → WARNING, WARNING → CRITICAL, CRITICAL → OK password: pbkdf2:sha256:...
- **Format**: `{LEVEL}: {hostname} - {metric_path} = {value} {threshold_info}` notification_channels: [sms_oncall, matrix_oncall]
- **Example**: `CRITICAL: prod-web-01 - cpu_monitor.cpu_percent = 95.2 (threshold: > 90.0)`
- **Re-notifications**: Periodic reminders for ongoing alerts (default: hourly)
### Heartbeat Events
Host lifecycle events:
- **Host boot**: `{hostname} booted`
- **Host shutdown**: `{hostname} {connection_type} shutdown`
- **Host recovery**: `{hostname} {connection_type} is back`
- **Connection issues**: `{hostname} {message}`
- **Host overdue**: `{hostname} {connection_type} overdue`
Only hosts with `watch: true` send heartbeat event notifications.
### Custom Alerts
Application code can send custom notifications:
```python
from hbd.server import notify as notify_mod
# Send to host-specific channels
notify_mod.pushmsg_for_host("prod-web-01", "Custom alert message")
# Send using global config
notify_mod.pushmsg_from_config("Global notification")
# Send to specific config
notify_mod.pushmsg(custom_config_dict, "Targeted notification")
``` ```
## Design Principles ### Host access — owner and managers
The notification system follows these core principles: Notifications for a host go to its owner and all managers:
- **Centralization**: Define notification providers once, reference them by name
- **Flexibility**: Each host can use different channels for different notification needs
- **Redundancy**: Critical hosts can specify multiple channels for failover
- **Clarity**: Clean separation between channel definition and channel assignment
- **Type Safety**: Provider-specific validation at configuration time
## Best Practices
### Channel Organization
- **Create purpose-specific channels**: `email_ops`, `signal_oncall`, `pushover_urgent`
- **Separate by team/role**: `email_devteam`, `signal_dbateam`, `mattermost_security`
- **Use descriptive names**: Channel names appear in logs and debugging
### Redundancy
For critical hosts, use multiple notification channels:
```yaml ```yaml
hosts: hosts:
critical-db: webserver01:
notification_channels: owner: alice # receives all notifications for this host
- signal_oncall # Primary: Mobile alert managers: [bob] # also receives notifications
- pushover_urgent # Backup: Different mobile platform threshold_config: default
- email_ops # Tertiary: Email for record-keeping watch: true # bold in dashboard (cosmetic only)
dyndns: false
dbserver01:
owner: alice
managers: [bob]
threshold_config: database
dyndns: false
``` ```
### Notification Fatigue Prevention `watch: true` only affects display (bold name in the live dashboard). Notifications are now controlled entirely by owner/managers.
- **Use `watch: false`** for non-critical hosts ## Channel Types
- **Configure appropriate thresholds** to avoid false positives
- **Set different channels for different severities**
- **Use `default_notification_channels`** for baseline, add more for critical systems
### Security ### `min_level` filtering
- **Protect credentials**: Use file permissions to protect config files with passwords/tokens Every channel accepts an optional `min_level` field:
- **Rotate tokens**: Periodically rotate API tokens and passwords
- **Use app-specific passwords**: For email, use app-specific passwords instead of main account password
- **Separate accounts**: Consider separate notification accounts for different environments (prod vs dev)
### Testing | Value | Channels receive |
|---|---|
| `WARNING` (default) | WARNING, CRITICAL, RECOVER |
| `CRITICAL` | CRITICAL only (and RECOVER) |
Test notification channels before relying on them: `RECOVER` is always passed through — you don't want to miss a recovery.
### pushover
Sends push notifications via [Pushover](https://pushover.net). Includes title, body, and a clickable URL.
```yaml
type: pushover
token: your-app-token # Required: Pushover application token
user: your-user-key # Required: Recipient's user key
min_level: WARNING
```
### email
Sends via SMTP. Subject = title, body = message + URL on final line.
```yaml
type: email
recipients: [ops@example.com, oncall@example.com]
sender: hbd@example.com
smtp_server: smtp.example.com
smtp_port: 587 # 587 = STARTTLS (default), 465 = SSL
smtp_user: hbd@example.com
smtp_password: secret
min_level: WARNING
```
### matrix
Sends a formatted HTML message to a Matrix room via [matrix-nio](https://github.com/poljar/matrix-nio).
```yaml
type: matrix
homeserver: https://matrix.example.org
access_token: syt_xxx # Bot account access token
room_id: "!abc:matrix.example.org"
min_level: WARNING
```
**Setup:**
1. Create a bot Matrix account
2. Obtain its access token (Element → Settings → Help & About → Access Token)
3. Invite the bot to the target room and note the room ID
### sms_voipms
Sends SMS via the [voip.ms REST API](https://voip.ms/api/v1/rest.php). Message is truncated to 160 characters.
```yaml
type: sms_voipms
api_user: me@example.com # voip.ms account email
api_password: secret # voip.ms API password
did: "5551234567" # Your voip.ms DID (sending number)
dst: "5559876543" # Destination number
min_level: CRITICAL
```
### signal
Sends via [signal-cli](https://github.com/AsamK/signal-cli).
```yaml
type: signal
cli_path: /usr/local/bin/signal-cli
user: +12025551234 # Your registered Signal number
recipient: +12025559999 # Recipient number
min_level: WARNING
```
**Setup:**
```bash ```bash
# Test signal-cli directly signal-cli -u +12025551234 register
signal-cli -u +1234567890 send -m "Test message" +0987654321 signal-cli -u +12025551234 verify CODE
# Test SMTP
echo "Test" | mail -s "Test Subject" admin@example.com
# Test through heartbeat system (Python REPL)
from hbd.server import notify as notify_mod, config as config_mod
cfg = config_mod.load_config(".hb.yaml")
notify_mod.setup(cfg)
notify_mod.pushmsg_for_host("test-host", "Test notification")
``` ```
### mattermost
Sends via Mattermost incoming webhook. Message is formatted as Markdown.
```yaml
type: mattermost
host: mattermost.example.com
token: your-webhook-token
channel: devops-alerts
username: heartbeat-bot # Optional: display name
icon: https://…/icon.png # Optional: bot icon URL
min_level: WARNING
```
## Notification events
| Source | Level | Title example | Body example |
|---|---|---|---|
| Host overdue | CRITICAL | `[CRITICAL] webserver01` | `IPv4 overdue` |
| Host recover | RECOVER | `[RECOVER] webserver01` | `IPv4 back after being overdue for 5:23` |
| Host boot | INFO | `[INFO] webserver01` | `webserver01 booted` |
| Host shutdown | INFO | `[INFO] webserver01` | `IPv4 shutdown` |
| Threshold breach | WARNING/CRITICAL | `[CRITICAL] webserver01` | `cpu_percent = 95.2 (threshold: > 90.0)` |
| Threshold reminder | CRITICAL | `[REMINDER/CRITICAL] webserver01` | `REMINDER (CRITICAL): … ongoing for 3600s` |
| Connection issue | WARNING | `[WARNING] webserver01` | `new address detected …` |
Reminder notifications (re-notify) are sent only for CRITICAL level alerts.
## API reference
### `send_notification(host_name, notif) -> dict`
Main entry point. Dispatches to owner + managers.
```python
from hbd.server.notify import send_notification, Notification
send_notification(
"webserver01",
Notification(
title="[CRITICAL] webserver01",
body="cpu_percent = 95.2 (threshold: > 90.0)",
level="CRITICAL",
url="https://hbd.example.com/plugins#webserver01",
),
)
```
Returns `{channel_name: bool}` for each channel dispatched.
### `setup(cfg, loop=None)`
Called once at startup from `main.py`. Pass the running asyncio event loop so Matrix sends work correctly.
## Troubleshooting ## Troubleshooting
### Notifications Not Sending **No notifications sent:**
- Check that users are configured (`users:` section in yaml)
- Check that the host has an `owner` or `managers` set
- Check that users have `notification_channels` listed
- Check that the channel names in user config match keys under `notification_channels:`
1. **Check logs**: Look for "Failed to send notification" errors **min_level filtering too aggressive:**
2. **Verify host is watched**: Ensure `watch: true` in host definition - Default is `WARNING` — both WARNING and CRITICAL are sent
3. **Check channel configuration**: Verify credentials and settings - Set `min_level: WARNING` explicitly if you were expecting warnings but set CRITICAL
4. **Test channel directly**: Use command-line tools to test provider
5. **Check network**: Ensure server can reach notification endpoints
### Signal Issues **Matrix sends time out:**
- Verify the access token is valid and the bot is in the room
- `matrix-nio` must be installed: `pip install matrix-nio`
- **signal-cli not found**: Specify full path in `cli_path` **voip.ms SMS fails:**
- **Not registered**: Run `signal-cli -u +NUMBER register` and verify - Enable the API in your voip.ms account (Account → API)
- **Trust issues**: Run `signal-cli -u +NUMBER receive` to sync trust store - Verify the DID is SMS-capable in your voip.ms account
- **Recipient not found**: Ensure recipient is in your Signal contacts
### Email Issues **Signal not found:**
- Specify full `cli_path`
- Run `signal-cli -u +NUMBER receive` to sync trust store
- **Authentication failed**: Check SMTP username/password **Email authentication failed:**
- **TLS errors**: Verify SMTP port (587 for STARTTLS, 465 for SSL) - Use app-specific passwords for Gmail/Fastmail
- **Relay denied**: Ensure SMTP server allows relay from your IP - Verify port: 587 for STARTTLS, 465 for SSL
- **Timeout**: Check firewall rules for SMTP ports
### Pushover Issues **Pushover `400` errors:**
- Double-check `token` (app) and `user` (user key) — they are different values
- **Invalid token/user**: Verify token and user key from Pushover dashboard
- **API rate limits**: Pushover has monthly message limits on free tier
- **HTTP errors**: Check Pushover API status page
### Mattermost Issues
- **Webhook not found**: Verify webhook token and ensure webhook is enabled
- **Channel not found**: Check channel name spelling and permissions
- **Driver import error**: Install mattermostdriver: `pip install mattermostdriver`
## API Reference
### Main Functions
#### `pushmsg_for_host(hostname: str, msg: str, debug: int = 0) -> dict`
Send notification to host-specific channels.
**Parameters:**
- `hostname`: Name of the host (used to look up notification channels)
- `msg`: Message to send
- `debug`: Debug level (0=no debug, 1+=debug output)
**Returns:** Dictionary of results per channel: `{"signal_ops": True, "email_ops": False}`
**Example:**
```python
from hbd.server import notify as notify_mod
notify_mod.pushmsg_for_host("prod-web-01", "Server CPU at 95%")
```
**Behavior:**
1. Looks up notification channels configured for the host
2. If no host-specific channels, uses `default_notification_channels`
3. Dispatches to each channel in parallel
4. Returns dict of results keyed by channel name
5. Logs success/failure for each channel
## Examples
### Complete Configuration Example
```yaml
# Notification channel definitions
notification_channels:
signal_oncall:
type: signal
cli_path: /usr/local/bin/signal-cli
user: +12025551234
recipient: +12025555678
email_ops:
type: email
recipients: [ops@example.com, alerts@example.com]
sender: heartbeat@example.com
smtp_server: smtp.fastmail.com
smtp_port: 587
smtp_user: heartbeat@example.com
smtp_password: app-password-here
# Default channels
default_notification_channels: [email_ops]
# Host definitions with channel assignments
hosts:
prod-web-01:
threshold_config: high_sensitivity
watch: true
notification_channels: [signal_oncall, email_ops]
dyndns: false
dev-server-01:
threshold_config: low_sensitivity
watch: false
notification_channels: [email_ops]
dyndns: false
```
### Multiple Environments Example
```yaml
notification_channels:
# Production channels
signal_prod_oncall:
type: signal
user: +12025551234
recipient: +12025551111 # On-call phone
email_prod_ops:
type: email
recipients: [prod-ops@example.com]
sender: prod-heartbeat@example.com
smtp_server: smtp.example.com
# Staging channels
email_staging:
type: email
recipients: [staging-alerts@example.com]
sender: staging-heartbeat@example.com
smtp_server: smtp.example.com
# Development channels
mattermost_dev:
type: mattermost
host: chat.example.com
token: dev-webhook-token
channel: dev-alerts
hosts:
prod-api-01:
notification_channels: [signal_prod_oncall, email_prod_ops]
staging-api-01:
notification_channels: [email_staging]
dev-api-01:
notification_channels: [mattermost_dev]
```
+17 -5
View File
@@ -1,9 +1,21 @@
Plan Plan the following changes, ask questions to clarify before implementing
Heartbeat is a client/server based network monitor and host observer. hbd, the server portion receives heartbeat and state messages from clients and maintaines state and hisgtory of the informations it receives. Re-factor the notification system:
- use available libraries for pushover, matrix, email and sms notifications.
- notifications have a title/subject: alert_type (recover/warning/critical), a body (info from threshold check) and a link to the host plugin metrix page
- define a list of notification channels for each user
- notifications are dispatched to users that are listed as managers for the host
hbc, the client portion gathers information on various aspects of the
system it is running on, and sends it to hbd. Initially this info is basic, like OS make and version, hardware info (CPU type, memory and disks), fileystem info and some resource info. hbc/hbd support a plugin system to extend the info gathered and stored.
hbd also can send notification based on missed hbc updates, and on violation of pre-set limits for various state paramaters.
1 - correct
2 - for now channels are defined globaly
3 - matrix-nio)sounds good, homeserver URL, access token, room ID per channel?
4 - use the REST api provided by https://voip.ms/api/v1/rest.php
5 - The page does not exist yet, point at the host tab in the /plugins
6 - per-channel minimum severity is a good idea, go fo it
7 - yes
1 - use base_url, there might not have been any incoming requests yet
2 - use same asyncio loop for matrix-nio
3 - for now, just silently do nothing
+1 -1
View File
@@ -14,4 +14,4 @@ Install options:
""" """
__all__ = ["__version__"] __all__ = ["__version__"]
__version__ = "5.0.12" __version__ = "5.1.2"
+8 -4
View File
@@ -2,6 +2,9 @@
import logging import logging
import os import os
import logging
logger = logging.getLogger(__name__)
try: try:
import yaml import yaml
@@ -30,18 +33,19 @@ def load_config(path=None):
If YAML is not available or the file does not exist, defaults are returned. If YAML is not available or the file does not exist, defaults are returned.
Args: Args:
path: Path to YAML config file (default: ~/.hb.yaml) path: Path to YAML config file (default: ~/.hbc.yaml)
Returns: Returns:
Dictionary with configuration Dictionary with configuration
""" """
cfg = CLIENT_DEFAULTS.copy() cfg = CLIENT_DEFAULTS.copy()
if not path: if not path:
# default path (~/.hb.yaml) # default path (~/.hbc.yaml)
path = os.path.join(os.path.expanduser("~"), ".hb.yaml") path = os.path.join(os.path.expanduser("~"), ".hbc.yaml")
if os.path.exists(path): if os.path.exists(path):
if yaml: if yaml:
logger.info("Loading configuration from %s", path)
with open(path) as fh: with open(path) as fh:
data = yaml.safe_load(fh) data = yaml.safe_load(fh)
# Merge YAML data with defaults # Merge YAML data with defaults
@@ -50,5 +54,5 @@ def load_config(path=None):
cfg[k] = v cfg[k] = v
else: else:
# yaml not installed: do not attempt to parse; user must ensure defaults # yaml not installed: do not attempt to parse; user must ensure defaults
pass logger.warning("PyYAML not available - cannot load config from %s, using defaults", path)
return cfg return cfg
+5 -5
View File
@@ -644,13 +644,10 @@ def main(argv=None):
parser = build_parser() parser = build_parser()
args = parser.parse_args(argv) args = parser.parse_args(argv)
# Load config
config = load_config(args.configfile)
# Setup logging # Setup logging
log_level = logging.INFO log_level = logging.WARNING
if args.verbose: if args.verbose:
log_level = logging.DEBUG log_level = logging.INFO
if args.debug: if args.debug:
log_level = logging.DEBUG log_level = logging.DEBUG
@@ -660,6 +657,9 @@ def main(argv=None):
datefmt="%Y-%m-%d %H:%M:%S" datefmt="%Y-%m-%d %H:%M:%S"
) )
# Load config
config = load_config(args.configfile)
# Daemonize if requested # Daemonize if requested
if args.daemon: if args.daemon:
print("Daemonizing...") print("Daemonizing...")
+8 -3
View File
@@ -311,7 +311,11 @@ class PluginLoader:
return 0 return 0
loaded_count = 0 loaded_count = 0
plugin_config = config or {} raw_config = config or {}
# Per-plugin config lives under the 'plugins' key or at top-level.
# CLIENT_DEFAULTS seeds "plugins": {} so the key always exists; check
# both the subdict and top-level so that either layout in .hbc.yaml works.
plugins_subconfig = raw_config.get("plugins", {})
# Scan for Python files # Scan for Python files
for plugin_file in directory.glob("*.py"): for plugin_file in directory.glob("*.py"):
@@ -356,8 +360,9 @@ class PluginLoader:
self.logger.debug(f"Found plugin class: {name}") self.logger.debug(f"Found plugin class: {name}")
# Instantiate plugin with config # Instantiate plugin with config — check plugins subdict first,
plugin_instance_config = plugin_config.get(obj.name, {}) # then top-level keys (e.g. nagios_runner: ... at root of config).
plugin_instance_config = plugins_subconfig.get(obj.name) or raw_config.get(obj.name, {})
plugin = obj(config=plugin_instance_config) plugin = obj(config=plugin_instance_config)
# Initialize plugin # Initialize plugin
+2 -2
View File
@@ -81,7 +81,7 @@ class NagiosRunnerPlugin(MonitorPlugin):
# Validate commands # Validate commands
if not self.commands: if not self.commands:
self.logger.warning( self.logger.info(
"No Nagios commands configured. Add 'nagios_runner.commands' to config." "No Nagios commands configured. Add 'nagios_runner.commands' to config."
) )
@@ -94,7 +94,7 @@ class NagiosRunnerPlugin(MonitorPlugin):
self.logger.info(f"Initializing {self.name} plugin") self.logger.info(f"Initializing {self.name} plugin")
if not self.commands: if not self.commands:
self.logger.error("No Nagios commands configured") self.logger.info("No Nagios commands configured")
return False return False
self.logger.info(f"Configured to run {len(self.commands)} Nagios plugin(s)") self.logger.info(f"Configured to run {len(self.commands)} Nagios plugin(s)")
+2
View File
@@ -48,6 +48,7 @@ class OSInfoPlugin(InfoPlugin):
Dictionary with OS details Dictionary with OS details
""" """
try: try:
from hbd import __version__ as hbc_version
data = { data = {
"system": platform.system(), # e.g., "Linux", "Darwin", "Windows" "system": platform.system(), # e.g., "Linux", "Darwin", "Windows"
"node": platform.node(), # hostname "node": platform.node(), # hostname
@@ -58,6 +59,7 @@ class OSInfoPlugin(InfoPlugin):
"architecture": platform.architecture()[0], # e.g., "64bit" "architecture": platform.architecture()[0], # e.g., "64bit"
"python_version": platform.python_version(), "python_version": platform.python_version(),
"python_implementation": platform.python_implementation(), "python_implementation": platform.python_implementation(),
"hbc_version": hbc_version,
} }
# Add Linux-specific distribution info # Add Linux-specific distribution info
+151
View File
@@ -0,0 +1,151 @@
"""Ping Monitor Plugin for Heartbeat.
Pings one or more hosts and reports round-trip time. Results are sent as
plugin metrics so the server-side threshold system can raise WARNING/CRITICAL
alerts using the same RTT threshold configuration format used for heartbeat RTT.
Example configuration in ~/.hbc.yaml (or the plugins section of ~/.hb.yaml):
```yaml
plugins:
ping_monitor:
interval: 60 # ping every 60 seconds (default)
count: 3 # ICMP packets per ping run (default 3)
timeout: 5 # seconds before a host is considered unreachable (default 5)
hosts:
8.8.8.8:
warning: 20.0 # ms
critical: 100.0 # ms
192.168.1.1:
warning: 5.0
critical: 20.0
```
Reported metrics per host (metric key uses the hostname with dots/colons replaced
by underscores so it is a valid identifier):
ping.<hostname>.rtt_avg average RTT in ms (float, or inf if unreachable)
ping.<hostname>.rtt_min minimum RTT in ms
ping.<hostname>.rtt_max maximum RTT in ms
ping.<hostname>.loss packet loss percentage (0100)
Server-side threshold config example:
```yaml
threshold_configs:
default:
thresholds:
ping_monitor:
8_8_8_8_rtt_avg:
warning: 20.0
critical: 100.0
```
"""
import asyncio
import re
import sys
from typing import Any, Dict, Optional
from hbd.client.plugin import MonitorPlugin
def _host_key(host: str) -> str:
"""Convert a hostname/IP to a safe metric key (replace . and : with _)."""
return re.sub(r"[^a-zA-Z0-9_]", "_", host)
class PingMonitorPlugin(MonitorPlugin):
"""Ping one or more configured hosts and report RTT metrics."""
name = "ping_monitor"
version = "1.0.0"
description = "ICMP ping latency monitoring"
interval = 60
def __init__(self, config: Optional[Dict[str, Any]] = None):
super().__init__(config)
cfg = config or {}
self.interval = cfg.get("interval", 60)
self.count = int(cfg.get("count", 3))
self.timeout = int(cfg.get("timeout", 5))
# hosts: dict of {hostname: {warning: x, critical: y}} or list of hostnames
raw_hosts = cfg.get("hosts", {})
if isinstance(raw_hosts, list):
self.hosts = {h: {} for h in raw_hosts}
else:
self.hosts = dict(raw_hosts)
async def initialize(self) -> bool:
if not self.hosts:
self.logger.warning("ping_monitor: no hosts configured, plugin disabled")
return False
self.logger.info(
"ping_monitor initialized: %d host(s), interval=%ds, count=%d, timeout=%ds",
len(self.hosts), self.interval, self.count, self.timeout,
)
return True
async def _ping(self, host: str) -> Dict[str, float]:
"""Run a system ping command and return rtt_min/avg/max/loss."""
if sys.platform == "win32":
cmd = ["ping", "-n", str(self.count), "-w", str(self.timeout * 1000), host]
else:
cmd = ["ping", "-c", str(self.count), "-W", str(self.timeout), host]
try:
proc = await asyncio.create_subprocess_exec(
*cmd,
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.PIPE,
)
stdout, _ = await asyncio.wait_for(
proc.communicate(),
timeout=self.timeout * self.count + 2,
)
output = stdout.decode(errors="replace")
except (asyncio.TimeoutError, FileNotFoundError, OSError) as e:
self.logger.warning("ping_monitor: ping failed for %s: %s", host, e)
return {"rtt_min": float("inf"), "rtt_avg": float("inf"),
"rtt_max": float("inf"), "loss": 100.0}
# Parse packet loss
loss = 100.0
loss_match = re.search(r"(\d+(?:\.\d+)?)\s*%\s*packet\s*loss", output)
if loss_match:
loss = float(loss_match.group(1))
# Parse rtt min/avg/max — Linux: "rtt min/avg/max/mdev = x/x/x/x ms"
# macOS: "round-trip min/avg/max/stddev = x/x/x/x ms"
rtt_match = re.search(
r"(?:rtt|round-trip)\s+min/avg/max/\S+\s*=\s*([\d.]+)/([\d.]+)/([\d.]+)",
output,
)
if rtt_match:
return {
"rtt_min": float(rtt_match.group(1)),
"rtt_avg": float(rtt_match.group(2)),
"rtt_max": float(rtt_match.group(3)),
"loss": loss,
}
# Host unreachable or all packets lost
return {"rtt_min": float("inf"), "rtt_avg": float("inf"),
"rtt_max": float("inf"), "loss": loss}
async def _collect_metrics(self) -> Dict[str, Any]:
data: Dict[str, Any] = {}
tasks = {host: asyncio.create_task(self._ping(host)) for host in self.hosts}
for host, task in tasks.items():
try:
result = await task
except Exception as e:
self.logger.error("ping_monitor: error pinging %s: %s", host, e)
result = {"rtt_min": float("inf"), "rtt_avg": float("inf"),
"rtt_max": float("inf"), "loss": 100.0}
key = _host_key(host)
for metric, value in result.items():
data[f"{key}_{metric}"] = value
status = "unreachable" if result["loss"] == 100.0 else f"{result['rtt_avg']:.1f}ms"
self.logger.debug("ping_monitor: %s -> %s", host, status)
return data
+9 -4
View File
@@ -52,12 +52,17 @@ def decode_value(val: str) -> Any:
except Exception: except Exception:
return val[1:] # Return as string without @ return val[1:] # Return as string without @
# Try numeric evaluation (original behavior) # Try numeric conversion (avoid eval to prevent SyntaxWarnings on version strings)
if val[0].isdigit() or (val[0] == '-' and len(val) > 1 and val[1].isdigit()): if val[0].isdigit() or (val[0] == '-' and len(val) > 1 and val[1].isdigit()):
try: try:
return eval(val) return int(val)
except Exception: except ValueError:
return val pass
try:
return float(val)
except ValueError:
pass
return val
return val return val
+199
View File
@@ -47,6 +47,48 @@ def build_parser():
help="Username (informational only, for display)", help="Username (informational only, for display)",
) )
# --- notify ---
notify_p = subparsers.add_parser(
"notify",
help="Send a test message via a configured notification channel",
)
notify_p.add_argument("-c", "--config", dest="configfile", help="Config file path (YAML)")
notify_p.add_argument(
"channel",
help="Channel name as defined in notification_channels",
)
notify_p.add_argument(
"message",
nargs="?",
default="Test notification from hbd",
help="Message body (default: 'Test notification from hbd')",
)
notify_p.add_argument(
"--level",
default="WARNING",
choices=["INFO", "WARNING", "CRITICAL", "RECOVER"],
help="Notification level (default: WARNING)",
)
notify_p.add_argument(
"--title",
default=None,
help="Notification title (default: '[LEVEL] test')",
)
# --- stop ---
stop_p = subparsers.add_parser("stop", help="Stop the running hbd instance")
stop_p.add_argument("-c", "--config", dest="configfile", help="Config file path (YAML)")
# --- reload ---
reload_p = subparsers.add_parser("reload", help="Reload configuration (SIGHUP)")
reload_p.add_argument("-c", "--config", dest="configfile", help="Config file path (YAML)")
# --- restart ---
restart_p = subparsers.add_parser("restart", help="Restart the running hbd instance")
restart_p.add_argument("-c", "--config", dest="configfile", help="Config file path (YAML)")
restart_p.add_argument("-f", "--foreground", action="store_true", help="Run in foreground after restart")
restart_p.add_argument("-v", "--verbose", action="store_true", help="Verbose output after restart")
return parser return parser
@@ -75,6 +117,147 @@ def cmd_passwd(args):
print(f" password: {hashed}") print(f" password: {hashed}")
def cmd_notify(args):
"""Send a test message via a single notification channel."""
from .config import load_config
from .notify import Notification, _dispatch_to_channel, setup
config = load_config(args.configfile)
setup(config)
channels = config.get("notification_channels", {})
if args.channel not in channels:
available = ", ".join(channels.keys()) if channels else "(none)"
print(f"Error: channel '{args.channel}' not found in notification_channels.", file=sys.stderr)
print(f"Available channels: {available}", file=sys.stderr)
sys.exit(1)
channel_cfg = channels[args.channel]
level = args.level.upper()
title = args.title or f"[{level}] test"
base_url = config.get("base_url", "").rstrip("/")
notif = Notification(
title=title,
body=args.message,
level=level,
url=f"{base_url}/plugins" if base_url else "",
)
# Bypass min_level for explicit test sends; run async channels directly
import asyncio
ch_type = channel_cfg.get("type", "")
print(f"Sending via {args.channel} ({ch_type}): {title}{args.message}")
if ch_type in ("matrix", "sms_voipms"):
from .notify import _send_matrix_async, _send_sms_voipms_async
driver_async = _send_matrix_async if ch_type == "matrix" else _send_sms_voipms_async
ok = asyncio.run(driver_async(channel_cfg, notif))
else:
from .notify import _DRIVERS
driver = _DRIVERS.get(ch_type)
if driver is None:
print(f"Error: unknown channel type '{ch_type}'", file=sys.stderr)
sys.exit(1)
ok = driver(channel_cfg, notif)
if ok:
print("OK")
else:
print("FAILED — check logs for details", file=sys.stderr)
sys.exit(1)
def _read_pid(configfile) -> int | None:
"""Return the PID from the pidfile, or None if not found / not running."""
import os
config = load_config(configfile)
pidfile = config.get("pidfile", "")
if not pidfile:
print("Error: no pidfile configured.", file=sys.stderr)
return None
try:
with open(pidfile) as f:
pid = int(f.read().strip())
# Verify process is actually running
os.kill(pid, 0)
return pid
except FileNotFoundError:
print(f"PID file not found ({pidfile}). Is hbd running?", file=sys.stderr)
return None
except ProcessLookupError:
print(f"PID file exists but process {pid} is not running.", file=sys.stderr)
return None
except Exception as e:
print(f"Error reading pidfile: {e}", file=sys.stderr)
return None
def cmd_stop(args):
import os, signal as _signal, time
pid = _read_pid(args.configfile)
if pid is None:
sys.exit(1)
print(f"Stopping hbd (pid {pid})...")
os.kill(pid, _signal.SIGTERM)
# Wait up to 10 s for the process to exit
for _ in range(20):
time.sleep(0.5)
try:
os.kill(pid, 0)
except ProcessLookupError:
print("hbd stopped.")
return
print("Warning: hbd did not stop within 10 seconds.", file=sys.stderr)
sys.exit(1)
def cmd_reload(args):
import os, signal as _signal
pid = _read_pid(args.configfile)
if pid is None:
sys.exit(1)
print(f"Sending SIGHUP to hbd (pid {pid})...")
os.kill(pid, _signal.SIGHUP)
print("Reload signal sent.")
def cmd_restart(args):
import os, signal as _signal, time, subprocess
pid = _read_pid(args.configfile)
if pid is not None:
print(f"Stopping hbd (pid {pid})...")
os.kill(pid, _signal.SIGTERM)
for _ in range(20):
time.sleep(0.5)
try:
os.kill(pid, 0)
except ProcessLookupError:
print("hbd stopped.")
break
else:
print("Warning: hbd did not stop within 10 seconds.", file=sys.stderr)
sys.exit(1)
else:
print("hbd does not appear to be running — starting fresh.")
# Re-launch hbd with the same config
cmd = [sys.executable, "-m", "hbd.server.cli", "serve"]
if args.configfile:
cmd += ["-c", args.configfile]
if getattr(args, "foreground", False):
cmd += ["-f"]
if getattr(args, "verbose", False):
cmd += ["-v"]
if getattr(args, "foreground", False):
# Run in foreground — replace current process
os.execv(sys.executable, cmd)
else:
subprocess.Popen(cmd, start_new_session=True)
print("hbd restarted.")
def main(argv=None): def main(argv=None):
parser = build_parser() parser = build_parser()
args = parser.parse_args(argv) args = parser.parse_args(argv)
@@ -83,6 +266,22 @@ def main(argv=None):
cmd_passwd(args) cmd_passwd(args)
return return
if args.command == "notify":
cmd_notify(args)
return
if args.command == "stop":
cmd_stop(args)
return
if args.command == "reload":
cmd_reload(args)
return
if args.command == "restart":
cmd_restart(args)
return
# Default: run the server (supports both `hbd serve ...` and `hbd ...`) # Default: run the server (supports both `hbd serve ...` and `hbd ...`)
config = load_config(args.configfile) config = load_config(args.configfile)
+49 -122
View File
@@ -16,15 +16,14 @@ SERVER_DEFAULTS = {
"hbd_host": "", # Bind address (empty = all interfaces) "hbd_host": "", # Bind address (empty = all interfaces)
# Persistence # Persistence
"pickfile": "/tmp/hb.pick", "pickfile": os.path.join(os.path.expanduser("~"), ".hb.pick"), # File to store host state between restarts
"pidfile": os.path.join(os.path.expanduser("~"), ".hb.pid"), # PID file for stop/restart/reload
# Logging # Logging
"logfile": "/var/log/heartbeat.log", "logfile": os.path.join(os.path.expanduser("~"), ".hb.log"),
"logfmt": "text", # text or msg or json
# Notification channels # Notification channels
"notification_channels": {}, # Named channels with type and credentials "notification_channels": {}, # Named channels with type and credentials
"default_notification_channels": [], # Default channels if host doesn't specify "base_url": "", # Base URL for notification links (e.g. https://hbd.example.com)
# Monitoring settings # Monitoring settings
"interval": 20, # Expected heartbeat interval (for server checks) "interval": 20, # Expected heartbeat interval (for server checks)
@@ -36,8 +35,7 @@ SERVER_DEFAULTS = {
"default_owner": None, # Username that owns hosts with no explicit owner "default_owner": None, # Username that owns hosts with no explicit owner
# Host management # Host management
"hosts": {}, # New unified host definitions (optional) "hosts": {}, # Unified host definitions
"watchhosts": [], # Hosts to monitor and notify about (legacy)
"dyndnshosts": [], # Hosts with dynamic DNS (legacy) "dyndnshosts": [], # Hosts with dynamic DNS (legacy)
"drophosts": [], # Hosts to ignore "drophosts": [], # Hosts to ignore
"dyndomains": ["wrede.org"], "dyndomains": ["wrede.org"],
@@ -69,6 +67,38 @@ SERVER_DEFAULTS = {
"thresholds": {}, "thresholds": {},
} }
THRESHOLD_DEFAULTS = {
'thresholds': {
'cpu_monitor': {
'cpu_percent': {
'warning': 80.0,
'critical': 90.0
}
},
'memory_monitor': {
'percent': {
'warning': 85.0,
'critical': 95.0
}
},
'disk_monitor': {
'partitions': {
'/': {
'percent': {
'warning': 85.0,
'critical': 90.0
}
}
}
},
'rtt': {
'warning': 200,
'critical': 250.0,
'count': 3 # Optional: number of consecutive breaches before alerting
}
}
}
def load_config(path=None): def load_config(path=None):
"""Load configuration from a YAML file and merge with server defaults. """Load configuration from a YAML file and merge with server defaults.
@@ -186,34 +216,18 @@ class ReloadableConfig:
def get_watchhosts(config): def get_watchhosts(config):
"""Extract watchhosts from config, supporting both new and legacy formats. """Extract watched hostnames from config (hosts with watch: true).
Args:
config: Configuration dictionary
Returns: Returns:
List of hostnames to watch List of hostnames to watch
""" """
watchhosts = [] watchhosts = []
hosts_config = config.get("hosts", {})
# New format: hosts section with watch attribute if isinstance(hosts_config, dict):
if "hosts" in config: for host_name, host_attrs in hosts_config.items():
hosts_config = config["hosts"] if isinstance(host_attrs, dict) and host_attrs.get("watch", False):
if isinstance(hosts_config, dict): watchhosts.append(host_name)
for host_name, host_attrs in hosts_config.items(): return watchhosts
if isinstance(host_attrs, dict) and host_attrs.get("watch", False):
watchhosts.append(host_name)
# Legacy format: watchhosts list
if "watchhosts" in config:
legacy_watchhosts = config.get("watchhosts", [])
if isinstance(legacy_watchhosts, (list, set)):
watchhosts.extend(legacy_watchhosts)
elif isinstance(legacy_watchhosts, dict):
# Old dict format: {"host1": {attrs}, "host2": {attrs}}
watchhosts.extend(legacy_watchhosts.keys())
return list(set(watchhosts)) # Remove duplicates
def get_dyndnshosts(config): def get_dyndnshosts(config):
@@ -245,105 +259,18 @@ def get_dyndnshosts(config):
def get_host_config(config, hostname): def get_host_config(config, hostname):
"""Get configuration for a specific host. """Get configuration for a specific host from the hosts section.
Args:
config: Configuration dictionary
hostname: Host name
Returns: Returns:
Dictionary with host attributes or empty dict Dictionary with host attributes or empty dict
""" """
if "hosts" in config: hosts_config = config.get("hosts", {})
hosts_config = config.get("hosts", {}) if isinstance(hosts_config, dict) and hostname in hosts_config:
if isinstance(hosts_config, dict) and hostname in hosts_config: val = hosts_config[hostname]
return hosts_config[hostname] if isinstance(hosts_config[hostname], dict) else {} return val if isinstance(val, dict) else {}
# Check legacy watchhosts for notification settings
if "watchhosts" in config:
watchhosts = config.get("watchhosts", {})
if isinstance(watchhosts, dict) and hostname in watchhosts:
legacy_attrs = watchhosts[hostname]
if isinstance(legacy_attrs, dict):
# Convert legacy format to new format
return {
"watch": True,
"notify": legacy_attrs.get("notify"),
"notify_src": legacy_attrs.get("src"),
}
return {} return {}
def get_notification_channels_for_host(config, hostname):
"""Get notification channels configured for a specific host.
Args:
config: Configuration dictionary
hostname: Host name
Returns:
List of channel names to use for this host
"""
host_config = get_host_config(config, hostname)
# Check if host specifies notification channels
channels = host_config.get("notification_channels", [])
if channels:
if isinstance(channels, str):
return [channels]
elif isinstance(channels, list):
return channels
# Fall back to default channels
default_channels = config.get("default_notification_channels", [])
if default_channels:
if isinstance(default_channels, str):
return [default_channels]
elif isinstance(default_channels, list):
return default_channels
# No channels configured, return empty list (will use legacy global config)
return []
def get_channel_config(config, channel_name):
"""Get configuration for a specific notification channel.
Args:
config: Configuration dictionary
channel_name: Name of the notification channel
Returns:
Dictionary with channel configuration or None if not found
"""
channels = config.get("notification_channels", {})
if isinstance(channels, dict) and channel_name in channels:
return channels[channel_name]
return None
def get_notification_channels_config(config, hostname):
"""Get list of notification channel configurations for a host.
Args:
config: Configuration dictionary
hostname: Host name
Returns:
List of (channel_name, channel_config) tuples
"""
channel_names = get_notification_channels_for_host(config, hostname)
channels = []
for channel_name in channel_names:
channel_config = get_channel_config(config, channel_name)
if channel_config and channel_config.get("type"):
channels.append((channel_name, channel_config))
return channels
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# User / host-access helpers # User / host-access helpers
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
+8
View File
@@ -422,6 +422,14 @@ class Host:
ddict["managers"] = list(getattr(self, "managers", [])) ddict["managers"] = list(getattr(self, "managers", []))
ddict["monitors"] = list(getattr(self, "monitors", [])) ddict["monitors"] = list(getattr(self, "monitors", []))
# hbc version from latest os_info plugin data
hbc_version = None
latest_os = self.get_latest_plugin_data("os_info")
if latest_os:
_, os_data = latest_os
hbc_version = os_data.get("hbc_version")
ddict["hbc_version"] = hbc_version
return ddict return ddict
def jsons(self): def jsons(self):
+37 -12
View File
@@ -12,6 +12,7 @@ from . import data
from . import notify as notify_mod from . import notify as notify_mod
from . import settings as settings_mod from . import settings as settings_mod
from . import users as users_mod from . import users as users_mod
from . import ws as ws_mod
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
@@ -242,11 +243,12 @@ async def start(
env = jinja2.Environment(loader=jinja2.FileSystemLoader(templates_dir)) env = jinja2.Environment(loader=jinja2.FileSystemLoader(templates_dir))
host = config.get("hb_host", "localhost") host = config.get("hb_host", "localhost")
extra_scripts = config.get("http_extra_scripts", "") extra_scripts = config.get("http_extra_scripts", "")
host = request.host.split(":")[0] host = request.host # includes port if non-standard
if config.get("wss_port"): forwarded_proto = request.headers.get("X-Forwarded-Proto", "")
heartbeat_ws_url = f"wss://{host}:{config['wss_port']}/hbd" is_secure = request.secure or forwarded_proto.lower() == "https"
else: scheme = "wss" if is_secure else "ws"
heartbeat_ws_url = f"ws://{host}:{config.get('ws_port', 50005)}/hbd" heartbeat_ws_url = f"{scheme}://{host}/ws"
from hbd import __version__ as hbd_version
tmpl = env.get_template("live.html") tmpl = env.get_template("live.html")
body = tmpl.render( body = tmpl.render(
title="Heartbeat", title="Heartbeat",
@@ -254,6 +256,7 @@ async def start(
request=request, request=request,
heartbeat_ws_url=heartbeat_ws_url, heartbeat_ws_url=heartbeat_ws_url,
extra_scripts=extra_scripts, extra_scripts=extra_scripts,
hbd_version=hbd_version,
hosts=[ hosts=[
hbdclass.Host.hosts[h].stateinfo() for h in sorted(hbdclass.Host.hosts) hbdclass.Host.hosts[h].stateinfo() for h in sorted(hbdclass.Host.hosts)
], ],
@@ -755,17 +758,39 @@ async def start(
templates_dir = config.get("templates_dir", os.path.join(pkg_dir, "templates")) templates_dir = config.get("templates_dir", os.path.join(pkg_dir, "templates"))
env = jinja2.Environment(loader=jinja2.FileSystemLoader(templates_dir)) env = jinja2.Environment(loader=jinja2.FileSystemLoader(templates_dir))
# Build host access summary for this user # Build host access summary for this user.
# Merge live hosts with config-only hosts (not yet seen) so the profile
# reflects the config file immediately after a reload.
from . import config as config_mod
owned, managed, monitored = [], [], [] owned, managed, monitored = [], [], []
if current_user: if current_user:
for hostname, host in sorted(hbdclass.Host.hosts.items()): # Collect all known hostnames: live + configured
if host.is_owner(current_user.username): cfg_hostnames = set(config.get("hosts", {}).keys())
live_hostnames = set(hbdclass.Host.hosts.keys())
all_hostnames = sorted(cfg_hostnames | live_hostnames)
for hostname in all_hostnames:
live_host = hbdclass.Host.hosts.get(hostname)
if live_host is not None:
# Use live object — it has apply_access already called
is_own = live_host.is_owner(current_user.username)
is_mgr = not is_own and live_host.is_manager(current_user.username)
is_mon = not is_own and not is_mgr and live_host.is_monitor(current_user.username)
else:
# Config-only host — read access directly from config
access = config_mod.get_host_access(config, hostname)
is_own = access["owner"] == current_user.username
is_mgr = current_user.username in access["managers"]
is_mon = current_user.username in access["monitors"]
if is_own:
owned.append(hostname) owned.append(hostname)
elif host.is_manager(current_user.username): elif is_mgr:
managed.append(hostname) managed.append(hostname)
elif host.is_monitor(current_user.username): elif is_mon:
monitored.append(hostname) monitored.append(hostname)
# Resolve notification channel configs for display # Resolve notification channel configs for display
notif_channels = [] notif_channels = []
if current_user: if current_user:
@@ -843,6 +868,7 @@ async def start(
web.get("/settings", settings_page), web.get("/settings", settings_page),
web.get("/static/{path:.*}", static), web.get("/static/{path:.*}", static),
web.get("/favicon.ico", favicon), web.get("/favicon.ico", favicon),
web.get("/ws", ws_mod.handler),
] ]
) )
@@ -851,8 +877,7 @@ async def start(
site = web.TCPSite(runner, host, port) site = web.TCPSite(runner, host, port)
await site.start() await site.start()
if verbose: logger.info(f"HTTP server started on {host}:{port}")
print(f"HTTP server started on {host}:{port}")
try: try:
await asyncio.Future() await asyncio.Future()
+61 -48
View File
@@ -27,6 +27,7 @@ def save_state(config, hbdclass):
"""Save current state to pickle file. Safe to call at any time.""" """Save current state to pickle file. Safe to call at any time."""
import pickle import pickle
import os import os
from . import users as users_mod
# Clear timer references before pickling (they can't be serialized) # Clear timer references before pickling (they can't be serialized)
for hostname, host in list(hbdclass.Host.hosts.items()): for hostname, host in list(hbdclass.Host.hosts.items()):
@@ -48,6 +49,7 @@ def save_state(config, hbdclass):
pick = pickle.Pickler(pickf) pick = pickle.Pickler(pickf)
pick.dump(hbdclass.Host.hosts) pick.dump(hbdclass.Host.hosts)
pick.dump(data.msgs) pick.dump(data.msgs)
pick.dump(users_mod.save_sessions())
os.replace(tmpfile, pickfile) os.replace(tmpfile, pickfile)
except Exception as e: except Exception as e:
logger.error("Failed to save state: %s", e) logger.error("Failed to save state: %s", e)
@@ -89,9 +91,13 @@ async def reload_configuration(config_obj, config_path, components):
# Reload users # Reload users
users_mod.load_users(new_config) users_mod.load_users(new_config)
# Re-apply host access from updated config to all known hosts # Re-apply host attributes from updated config to all known hosts
from . import config as config_mod from . import config as config_mod
dyndnshosts = config_mod.get_dyndnshosts(new_config)
watchhosts = config_mod.get_watchhosts(new_config)
for hostname, host in hbdclass.Host.hosts.items(): for hostname, host in hbdclass.Host.hosts.items():
host.dyn = hostname in dyndnshosts
host.watched = hostname in watchhosts
access = config_mod.get_host_access(new_config, hostname) access = config_mod.get_host_access(new_config, hostname)
host.apply_access(access["owner"], access["managers"], access["monitors"]) host.apply_access(access["owner"], access["managers"], access["monitors"])
@@ -126,6 +132,10 @@ async def reload_configuration(config_obj, config_path, components):
async def _run_async(config, config_path=None): async def _run_async(config, config_path=None):
from .config import ReloadableConfig
if not isinstance(config, ReloadableConfig):
config = ReloadableConfig(config, config_path)
loop = asyncio.get_running_loop() loop = asyncio.get_running_loop()
shutdown_event = asyncio.Event() shutdown_event = asyncio.Event()
reload_event = asyncio.Event() reload_event = asyncio.Event()
@@ -152,7 +162,7 @@ async def _run_async(config, config_path=None):
from . import journal as journal_mod from . import journal as journal_mod
from . import threshold as threshold_mod from . import threshold as threshold_mod
notify_mod.setup(config) notify_mod.setup(config, loop=loop)
# Initialize message journal # Initialize message journal
msg_journal = journal_mod.get_journal(config) msg_journal = journal_mod.get_journal(config)
@@ -187,14 +197,14 @@ async def _run_async(config, config_path=None):
sock.bind(bind_addr) sock.bind(bind_addr)
logger.info("Starting UDP server on %s:%s", *bind_addr) logger.info("Starting UDP server on %s:%s", *bind_addr)
# Try to enable kernel receive timestamps (Linux SO_TIMESTAMPNS). # Try to enable kernel receive timestamps (Linux SO_TIMESTAMP).
# If supported, read datagrams via recvmsg() so RTT uses the kernel # If supported, read datagrams via recvmsg() so RTT uses the kernel
# timestamp rather than the time.time() call after asyncio scheduling. # timestamp rather than the time.time() call after asyncio scheduling.
use_kernel_ts = udp.enable_kernel_timestamps(sock) use_kernel_ts = udp.enable_kernel_timestamps(sock)
if use_kernel_ts: if use_kernel_ts:
logger.info("SO_TIMESTAMPNS enabled: using kernel receive timestamps for RTT") logger.info("SO_TIMESTAMP enabled: using kernel receive timestamps for RTT")
else: else:
logger.info("SO_TIMESTAMPNS not available: using time.time() for RTT") logger.info("SO_TIMESTAMP not available: using time.time() for RTT")
def udp_handler(msg, addr, transport, recv_ts=None): def udp_handler(msg, addr, transport, recv_ts=None):
ctx = dict( ctx = dict(
@@ -265,45 +275,17 @@ async def _run_async(config, config_path=None):
except Exception as e: except Exception as e:
logger.exception("dns worker failed to start: %s", e) logger.exception("dns worker failed to start: %s", e)
# Start the websocket servers as a background task # Register WebSocket state — connections are now served through /ws on the HTTP port
if config.get("wss_port", None): ws_task = None
ssl_context = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER) ws_mod.setup(
ssl_path = config.get("cert_path", "") loop=loop,
wss_pem = ssl_path + config.get("wss_pem", "") get_hosts=lambda: [
wss_key = ssl_path + config.get("wss_key", "") hbdclass.Host.hosts[h].stateinfo()
try: for h in sorted(hbdclass.Host.hosts)
ssl_context.load_cert_chain(wss_pem, keyfile=wss_key) ],
except FileNotFoundError: verbose=config.get("verbose", False),
logger.error("error: missing SSL keys %s or %s", wss_pem, wss_key) )
sys.exit(1) logger.info("WebSocket handler registered on /ws (HTTP port %s)", config.get("hbd_port", 50004))
logger.info(
"Starting secure WebSocket server on port %s with cert %s",
config.get("wss_port", None),
wss_pem,
)
else:
ssl_context = None
try:
ws_port = config.get("ws_port", 50005)
logger.info("Starting WebSocket server on port %s", ws_port)
ws_task = asyncio.create_task(
ws_mod.start(
host=config.get("hbd_host", ""),
ws_port=ws_port,
wss_port=config.get("wss_port", None),
ssl_context=ssl_context,
get_hosts=lambda: [
hbdclass.Host.hosts[h].stateinfo()
for h in sorted(hbdclass.Host.hosts)
],
# get_msgs=lambda: msgs,
config=config,
)
)
logger.info("WebSocket task started")
except Exception as e:
logger.exception("websocket server failed to start: %s", e)
# Periodic autosave task # Periodic autosave task
autosave_interval = config.get("autosave_interval", 300) # default: 5 minutes autosave_interval = config.get("autosave_interval", 300) # default: 5 minutes
@@ -365,7 +347,7 @@ async def _run_async(config, config_path=None):
except Exception as e: except Exception as e:
logger.warning("Error closing UDP transport: %s", e) logger.warning("Error closing UDP transport: %s", e)
tasks_to_cancel = [http_task, ws_task, autosave] tasks_to_cancel = [http_task, autosave]
for task in tasks_to_cancel: for task in tasks_to_cancel:
if task: if task:
try: try:
@@ -416,6 +398,13 @@ async def _run_async(config, config_path=None):
except Exception as e: except Exception as e:
logger.warning("Error stopping DNS worker: %s", e) logger.warning("Error stopping DNS worker: %s", e)
# Save state (hosts + sessions) on clean shutdown
try:
save_state(config, hbdclass)
logger.info("State saved on shutdown")
except Exception as e:
logger.warning("Error saving state on shutdown: %s", e)
logger.info("All tasks cancelled") logger.info("All tasks cancelled")
@@ -424,6 +413,7 @@ def load_pickled_hosts(config, hbdclass):
import os import os
import pickle import pickle
from . import config as config_mod from . import config as config_mod
from . import users as users_mod
pickfile = config.get("pickfile", "hbd.pickle") pickfile = config.get("pickfile", "hbd.pickle")
dyndnshosts = config_mod.get_dyndnshosts(config) dyndnshosts = config_mod.get_dyndnshosts(config)
@@ -437,6 +427,10 @@ def load_pickled_hosts(config, hbdclass):
try: try:
hbdclass.Host.hosts = pick.load() hbdclass.Host.hosts = pick.load()
data.msgs = pick.load() data.msgs = pick.load()
try:
users_mod.load_sessions(pick.load())
except Exception:
pass # older pickle without sessions — fine
pickf.close() pickf.close()
except Exception as e: except Exception as e:
logger.exception("load pickled failed: %s", e) logger.exception("load pickled failed: %s", e)
@@ -471,13 +465,26 @@ def run(config, config_path=None):
""" """
import os import os
logging.basicConfig( log_level = logging.WARNING
level=logging.DEBUG if config.get("debug", 0) > 0 else logging.INFO if config.get("verbose", False):
) log_level = logging.INFO
if config.get("debug", 0) > 0:
log_level = logging.DEBUG
logging.basicConfig(level=log_level)
load_pickled_hosts(config, hbdclass) load_pickled_hosts(config, hbdclass)
notify_mod.initlog(logfile=config.get("logfile", "messages.log")) notify_mod.initlog(logfile=config.get("logfile", "messages.log"))
users_mod.load_users(config) users_mod.load_users(config)
# Write pidfile
pidfile = config.get("pidfile", "")
if pidfile:
try:
with open(pidfile, "w") as f:
f.write(str(os.getpid()))
except Exception as e:
logger.warning("Failed to write pidfile %s: %s", pidfile, e)
eventlog(None, "INFO", f"hbd version {__version__} starting up") eventlog(None, "INFO", f"hbd version {__version__} starting up")
if config_path: if config_path:
@@ -500,6 +507,12 @@ def run(config, config_path=None):
logger.info("hbd shutdown complete") logger.info("hbd shutdown complete")
eventlog(None, "INFO", f"hbd version {__version__} shutdown") eventlog(None, "INFO", f"hbd version {__version__} shutdown")
notify_mod.closelog() notify_mod.closelog()
# Remove pidfile
if pidfile:
try:
os.unlink(pidfile)
except Exception:
pass
# Explicitly close the loop # Explicitly close the loop
try: try:
# Cancel all remaining tasks # Cancel all remaining tasks
+406 -221
View File
@@ -1,37 +1,106 @@
"""Notification helpers: email, pushover, mattermost, signal and dispatcher.""" """Notification helpers: email, pushover, matrix, mattermost, signal, sms and dispatcher.
Channel types supported:
pushover - Pushover app notifications
email - SMTP email
matrix - Matrix (via matrix-nio)
mattermost - Mattermost webhook
signal - Signal via signal-cli subprocess
sms_voipms - SMS via voip.ms REST API
Each channel can specify ``min_level: WARNING|CRITICAL`` (default: WARNING).
Notifications are dispatched to the owner + managers of the host, each via
their own ``notification_channels`` list. When no users are configured the
server runs silently (no notifications sent).
"""
import asyncio
import asyncio
import logging import logging
from typing import Optional
import http.client
import urllib.parse
import subprocess
import smtplib import smtplib
import subprocess
import time import time
import sys import sys
from dataclasses import dataclass, field
from typing import Optional
from . import data from . import data
from . import ws as ws_mod from . import ws as ws_mod
from . import main as main_mod
DEFAULT_PUSHPROVIDERS = ["all", "pushover", "mattermost", "signal"]
msg_to_websockets = ws_mod.broadcast
# module-level configuration set via setup()
_config = {}
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
logger = logging.getLogger(__name__)
msg_to_websockets = ws_mod.broadcast
# Module-level state set via setup()
_config: dict = {}
_loop: Optional[asyncio.AbstractEventLoop] = None
# Tracks which channels fired a WARNING/CRITICAL per host.
# {host_name: set of channel_names} — used to route RECOVER to the same channels.
_alerted_channels: dict = {}
logf = None logf = None
# ---------------------------------------------------------------------------
# Level ordering
# ---------------------------------------------------------------------------
_LEVEL_ORDER = {"RECOVER": 0, "INFO": 0, "WARNING": 1, "CRITICAL": 2}
def _level_value(level: str) -> int:
return _LEVEL_ORDER.get(level.upper(), 0)
# ---------------------------------------------------------------------------
# Notification dataclass
# ---------------------------------------------------------------------------
@dataclass
class Notification:
"""Structured notification payload."""
title: str # e.g. "[CRITICAL] webserver01"
body: str # detail message
level: str # RECOVER | WARNING | CRITICAL | INFO
url: str = "" # link to plugin metrics page
# ---------------------------------------------------------------------------
# Module setup
# ---------------------------------------------------------------------------
def setup(cfg: dict, loop: Optional[asyncio.AbstractEventLoop] = None):
"""Initialize notifier from configuration dict and event loop."""
global _config, _loop
_config = dict(cfg)
if loop is not None:
_loop = loop
def reload_config(cfg: dict):
"""Reload notification configuration on SIGHUP."""
global _config
_config = dict(cfg)
logger.info("Notification configuration reloaded")
# ---------------------------------------------------------------------------
# Event log (websocket + file + in-memory)
# ---------------------------------------------------------------------------
def initlog(logfile): def initlog(logfile):
global logf global logf
try: try:
logf = open(logfile, "a+") logf = open(logfile, "a+")
except Exception as e: except Exception as e:
import sys
print("cannot open logfile %s, using STDERR: %s" % (logfile, e)) print("cannot open logfile %s, using STDERR: %s" % (logfile, e))
logf = sys.stderr logf = sys.stderr
return logf return logf
def closelog(): def closelog():
global logf global logf
if logf and logf != sys.stderr: if logf and logf != sys.stderr:
@@ -40,6 +109,7 @@ def closelog():
except Exception: except Exception:
pass pass
def eventlog(host, lvl, m, service=None): def eventlog(host, lvl, m, service=None):
ts = time.time() ts = time.time()
s = f"{time.strftime('%Y-%m-%d %H:%M:%S', time.localtime(ts))} {lvl} " s = f"{time.strftime('%Y-%m-%d %H:%M:%S', time.localtime(ts))} {lvl} "
@@ -56,91 +126,29 @@ def eventlog(host, lvl, m, service=None):
logger.warning("failed to write to logfile: %s", e) logger.warning("failed to write to logfile: %s", e)
msg_to_websockets("message", s) msg_to_websockets("message", s)
def setup(cfg: dict):
"""Initialize notifier defaults from a configuration dict."""
global _config
_config = dict(cfg)
# ---------------------------------------------------------------------------
# Low-level channel drivers
# ---------------------------------------------------------------------------
def reload_config(cfg: dict): def _send_pushover(channel_cfg: dict, notif: Notification) -> bool:
"""Reload notification configuration. import http.client
import urllib.parse
This function updates the module-level notification configuration token = channel_cfg.get("token", "")
during runtime config reloads. user = channel_cfg.get("user", "")
if not token or not user:
Args: logger.warning("pushover: missing token or user")
cfg: New configuration dictionary
"""
global _config
_config = dict(cfg)
logger.info("Notification configuration reloaded")
def send_email(toaddrs, smtpserver, sender, subject, body, debug=0):
"""Send a plain email via SMTP. Returns True on success."""
try:
smtpport = _config.get("smtpport", 587)
server = smtplib.SMTP(smtpserver, smtpport)
if debug > 0:
server.set_debuglevel(1)
if smtpport == 587:
server.starttls()
server.ehlo()
smtpuser = _config.get("smtpuser", None)
smtppassword = _config.get("smtppassword", None)
if smtpuser and smtppassword:
server.login(smtpuser, smtppassword)
server.sendmail(sender, toaddrs, body)
except Exception as e:
logger.warning("email send failed: %s", e)
try:
server.quit()
except Exception:
pass
return False return False
try: params: dict = {"token": token, "user": user, "title": notif.title, "message": notif.body}
server.quit() if notif.url:
except Exception: params["url"] = notif.url
pass params["url_title"] = "Plugin metrics"
return True
def email(subject: str, msg: str, debug: int = 0) -> bool:
"""Convenience wrapper exposed to the rest of the application.
Uses module-level configuration to supply recipient list, smtp server
and sender address.
"""
toaddrs = _config.get("toemail")
fromemail = _config.get("fromemail")
smtpserver = _config.get("smtpserver")
if not toaddrs or not fromemail or not smtpserver:
logger.warning(
"email config incomplete: toemail=%s, fromemail=%s, smtpserver=%s",
toaddrs,
fromemail,
smtpserver,
)
return False
date = time.strftime("%a, %d %b %Y %H:%M:%S %z", time.localtime())
body = "To: %s\nFrom: %s\nSubject: %s\nDate: %s\n\n%s" % (
toaddrs[0] if toaddrs else "",
fromemail,
subject,
date,
msg,
)
return send_email(toaddrs, smtpserver, fromemail, subject, body, debug=debug)
def pushover(token: str, user: str, msg: str, debug: int = 0) -> bool:
"""Send message via Pushover API."""
conn = http.client.HTTPSConnection("api.pushover.net:443") conn = http.client.HTTPSConnection("api.pushover.net:443")
try: try:
conn.request( conn.request(
"POST", "POST",
"/1/messages.json", "/1/messages.json",
urllib.parse.urlencode({"token": token, "user": user, "message": msg}), urllib.parse.urlencode(params),
{"Content-type": "application/x-www-form-urlencoded"}, {"Content-type": "application/x-www-form-urlencoded"},
) )
r = conn.getresponse() r = conn.getresponse()
@@ -151,176 +159,353 @@ def pushover(token: str, user: str, msg: str, debug: int = 0) -> bool:
return False return False
def pushmattermost( def _send_email(channel_cfg: dict, notif: Notification) -> bool:
host: str, recipients = channel_cfg.get("recipients", [])
token: str, sender = channel_cfg.get("sender", "")
channel: str, smtp_server = channel_cfg.get("smtp_server", "")
msg: str, smtp_port = channel_cfg.get("smtp_port", 587)
username: str = "hbd", smtp_user = channel_cfg.get("smtp_user")
icon: Optional[str] = None, smtp_password = channel_cfg.get("smtp_password")
debug: int = 0,
) -> bool:
"""Send a message to Mattermost via simple webhook driver if available.
This helper tries to import mattermostdriver.Driver and uses webhooks if present. if not recipients or not sender or not smtp_server:
If the import fails it returns False. logger.warning("email: missing recipients, sender, or smtp_server")
""" return False
date = time.strftime("%a, %d %b %Y %H:%M:%S %z", time.localtime())
body_text = notif.body
if notif.url:
body_text += f"\n\n{notif.url}"
raw = "To: %s\nFrom: %s\nSubject: %s\nDate: %s\n\n%s" % (
recipients[0] if isinstance(recipients, list) else recipients,
sender,
notif.title,
date,
body_text,
)
try:
server = smtplib.SMTP(smtp_server, smtp_port)
if smtp_port == 587:
server.starttls()
server.ehlo()
if smtp_user and smtp_password:
server.login(smtp_user, smtp_password)
server.sendmail(sender, recipients, raw)
server.quit()
return True
except Exception as e:
logger.warning("email send failed: %s", e)
try:
server.quit()
except Exception:
pass
return False
def _send_mattermost(channel_cfg: dict, notif: Notification) -> bool:
try: try:
from mattermostdriver import Driver from mattermostdriver import Driver
except Exception: except ImportError:
logger.error("mattermostdriver not installed")
return False return False
host = channel_cfg.get("host", "")
token = channel_cfg.get("token", "")
channel = channel_cfg.get("channel", "")
if not host or not token or not channel:
logger.warning("mattermost: missing host, token, or channel")
return False
text = f"**{notif.title}**\n{notif.body}"
if notif.url:
text += f"\n[Plugin metrics]({notif.url})"
ses = {"url": host, "scheme": "http", "basepath": "/api/v4", "port": 8065} ses = {"url": host, "scheme": "http", "basepath": "/api/v4", "port": 8065}
mm = Driver(ses) mm = Driver(ses)
payload = {"text": msg, "channel": channel, "username": username} payload: dict = {"text": text, "channel": channel, "username": channel_cfg.get("username", "hbd")}
icon = channel_cfg.get("icon")
if icon: if icon:
payload["icon_url"] = icon payload["icon_url"] = icon
try: try:
rc = mm.webhooks.call_webhook(token, payload) rc = mm.webhooks.call_webhook(token, payload)
logger.debug("mattermost rc: %s", rc)
return bool(rc is None or rc == "") return bool(rc is None or rc == "")
except Exception as e: except Exception as e:
logger.error("mattermost error: %s", e) logger.error("mattermost error: %s", e)
return False return False
def pushsignal( def _send_signal(channel_cfg: dict, notif: Notification) -> bool:
signal_cli_bin: str, user: str, recipient: str, msg: str, debug: int = 0 cli = channel_cfg.get("cli_path", "/usr/local/bin/signal-cli")
) -> bool: user = channel_cfg.get("user", "")
"""Send a message via signal-cli (requires local installation). recipient = channel_cfg.get("recipient", "")
if not user or not recipient:
Uses subprocess to call signal-cli. Returns True if the command succeeded. logger.warning("signal: missing user or recipient")
""" return False
CLI = [signal_cli_bin, "-u", user, "send", "-m", msg, recipient] msg = f"{notif.title}\n{notif.body}"
logger.debug("signal cli: %s", CLI) if notif.url:
msg += f"\n{notif.url}"
try: try:
res = subprocess.run(CLI, capture_output=True) res = subprocess.run([cli, "-u", user, "send", "-m", msg, recipient], capture_output=True)
if res.returncode != 0: if res.returncode != 0:
logger.error("signal failed: %s".res.stderr.decode()) logger.error("signal failed: %s", res.stderr.decode())
return False return False
logger.debug("signal sent: %s", res.stdout.decode())
return True return True
except Exception as e: except Exception as e:
logger.exception("signal exception: %s", e) logger.exception("signal exception: %s", e)
return False return False
def _dispatch_to_channel(channel_name: str, channel_config: dict, msg: str, debug: int = 0) -> bool: async def _send_sms_voipms_async(channel_cfg: dict, notif: Notification) -> bool:
"""Dispatch a message to a specific notification channel. """Send SMS via voip.ms REST API using multipart form-data POST."""
import json
import aiohttp
Args: api_user = channel_cfg.get("api_user", "")
channel_name: Name of the channel (for logging) api_password = channel_cfg.get("api_password", "")
channel_config: Channel configuration dictionary with 'type' and type-specific fields did = channel_cfg.get("did", "")
msg: Message to send dst = channel_cfg.get("dst", "")
debug: Debug level if not api_user or not api_password or not did or not dst:
logger.warning("sms_voipms: missing api_user, api_password, did, or dst")
return False
Returns: # SMS body: title + body, truncated to 160 chars
True if notification sent successfully, False otherwise text = f"{notif.title}: {notif.body}"
""" if len(text) > 160:
channel_type = channel_config.get("type") text = text[:157] + "..."
if channel_type == "pushover": form_data = {
return pushover( "api_username": api_user,
channel_config.get("token", ""), "api_password": api_password,
channel_config.get("user", ""), "method": "sendSMS",
msg, "did": did,
debug=debug "dst": dst,
) "message": text,
}
elif channel_type == "email": try:
# Build email from channel config async with aiohttp.ClientSession() as session:
recipients = channel_config.get("recipients", []) with aiohttp.MultipartWriter("form-data") as mp:
sender = channel_config.get("sender", "") for key, value in form_data.items():
smtp_server = channel_config.get("smtp_server", "") part = mp.append(value)
smtp_port = channel_config.get("smtp_port", 587) part.set_content_disposition("form-data", name=key)
smtp_user = channel_config.get("smtp_user") async with session.post("https://voip.ms/api/v1/rest.php", data=mp) as resp:
smtp_password = channel_config.get("smtp_password") body = await resp.text()
if resp.status != 200:
if not recipients or not sender or not smtp_server: logger.error("sms_voipms HTTP %s: %s", resp.status, body)
logger.warning( return False
"Email channel '%s' missing required fields: recipients=%s, sender=%s, smtp_server=%s", result = json.loads(body)
channel_name, recipients, sender, smtp_server if result.get("status") == "success":
) return True
return False logger.error("sms_voipms error: %s", result.get("status"))
return False
# Temporarily update _config for email() function except Exception as e:
old_config = dict(_config) logger.error("sms_voipms exception: %s", e)
_config["toemail"] = recipients
_config["fromemail"] = sender
_config["smtpserver"] = smtp_server
_config["smtpport"] = smtp_port
if smtp_user:
_config["smtpuser"] = smtp_user
if smtp_password:
_config["smtppassword"] = smtp_password
result = email("Heartbeat notification", msg, debug=debug)
# Restore config
_config.clear()
_config.update(old_config)
return result
elif channel_type == "signal":
return pushsignal(
channel_config.get("cli_path", "/usr/local/bin/signal-cli"),
channel_config.get("user", ""),
channel_config.get("recipient", ""),
msg,
debug=debug
)
elif channel_type == "mattermost":
return pushmattermost(
channel_config.get("host", ""),
channel_config.get("token", ""),
channel_config.get("channel", ""),
msg,
username=channel_config.get("username", "hbd"),
icon=channel_config.get("icon"),
debug=debug
)
else:
logger.warning("Unknown channel type '%s' for channel '%s'", channel_type, channel_name)
return False return False
def pushmsg_for_host(hostname: str, msg: str, debug: int = 0) -> dict: def _send_sms_voipms(channel_cfg: dict, notif: Notification) -> bool:
"""Send notification for a specific host using its configured channels. """Dispatch voip.ms SMS send onto the shared event loop."""
if _loop is None:
logger.warning("sms_voipms: event loop not available")
return False
future = asyncio.run_coroutine_threadsafe(_send_sms_voipms_async(channel_cfg, notif), _loop)
try:
return future.result(timeout=15)
except Exception as e:
logger.error("sms_voipms send timed out or failed: %s", e)
return False
This function looks up the host's notification channels from the config
and sends the message to those channels.
Args: async def _send_matrix_async(channel_cfg: dict, notif: Notification) -> bool:
hostname: Name of the host to send notification for """Send a Matrix message using matrix-nio."""
msg: Message to send try:
debug: Debug level from nio import AsyncClient, RoomMessageText # noqa: F401
except ImportError:
logger.error("matrix-nio not installed; pip install matrix-nio")
return False
Returns: from nio import AsyncClient
Dictionary of results per channel: {"channel_name": True/False} homeserver = channel_cfg.get("homeserver", "")
access_token = channel_cfg.get("access_token", "")
room_id = channel_cfg.get("room_id", "")
if not homeserver or not access_token or not room_id:
logger.warning("matrix: missing homeserver, access_token, or room_id")
return False
text = f"{notif.title}\n{notif.body}"
if notif.url:
text += f"\n{notif.url}"
html = f"<strong>{notif.title}</strong><br>{notif.body}"
if notif.url:
html += f'<br><a href="{notif.url}">Plugin metrics</a>'
client = AsyncClient(homeserver)
client.access_token = access_token
try:
from nio import RoomSendResponse
content = {
"msgtype": "m.text",
"body": text,
"format": "org.matrix.custom.html",
"formatted_body": html,
}
resp = await client.room_send(room_id, "m.room.message", content)
if hasattr(resp, "event_id"):
return True
logger.error("matrix send failed: %s", resp)
return False
except Exception as e:
logger.error("matrix exception: %s", e)
return False
finally:
await client.close()
def _send_matrix(channel_cfg: dict, notif: Notification) -> bool:
"""Dispatch matrix send onto the shared event loop."""
if _loop is None:
logger.warning("matrix: event loop not available")
return False
future = asyncio.run_coroutine_threadsafe(_send_matrix_async(channel_cfg, notif), _loop)
try:
return future.result(timeout=15)
except Exception as e:
logger.error("matrix send timed out or failed: %s", e)
return False
# ---------------------------------------------------------------------------
# Channel dispatcher
# ---------------------------------------------------------------------------
_DRIVERS = {
"pushover": _send_pushover,
"email": _send_email,
"mattermost": _send_mattermost,
"signal": _send_signal,
"sms_voipms": _send_sms_voipms,
"matrix": _send_matrix,
}
def _dispatch_to_channel(channel_name: str, channel_cfg: dict, notif: Notification) -> bool:
"""Send *notif* to a single named channel, honouring min_level.
RECOVER always bypasses min_level — a recovery is always relevant if the
channel was configured for any alerting (handles the restart-then-recover case
where _alerted_channels is empty and we fall through to the normal loop).
""" """
from . import config as config_mod level = notif.level.upper()
if level != "RECOVER":
min_level = channel_cfg.get("min_level", "WARNING").upper()
if _level_value(level) < _level_value(min_level):
logger.debug(
"channel '%s': skipping level %s (min_level=%s)", channel_name, level, min_level
)
return True # not an error — filtered intentionally
# Get notification channels for this host ch_type = channel_cfg.get("type", "")
channels = config_mod.get_notification_channels_config(_config, hostname) driver = _DRIVERS.get(ch_type)
if driver is None:
logger.warning("unknown channel type '%s' for channel '%s'", ch_type, channel_name)
return False
return driver(channel_cfg, notif)
if not channels:
logger.warning("No notification channels configured for host '%s'", hostname) # ---------------------------------------------------------------------------
# Central dispatch function
# ---------------------------------------------------------------------------
def _build_url(host_name: str) -> str:
base_url = _config.get("base_url", "").rstrip("/")
if not base_url:
return ""
return f"{base_url}/plugins#{host_name}"
def send_notification(host_name: str, notif: Notification) -> dict:
"""Dispatch *notif* to all managers/owner of *host_name*.
Looks up the host's owner + managers, resolves each user's
notification_channels, and dispatches. Silently does nothing if
no users are configured.
Returns a dict of {channel_name: bool} results.
"""
from . import users as users_mod
from . import hbdclass
if not users_mod.users_enabled():
return {} return {}
# Dispatch to each channel # Collect recipient usernames: owner + managers
results = {} host = hbdclass.Host.hosts.get(host_name)
for channel_name, channel_config in channels: if host is None:
try: logger.debug("send_notification: host '%s' not found", host_name)
success = _dispatch_to_channel(channel_name, channel_config, msg, debug=debug) return {}
results[channel_name] = success
if success: recipients: set[str] = set()
logger.info("Notification sent to channel '%s': %s", channel_name, msg) owner = getattr(host, "owner", None)
else: if owner:
logger.warning("Failed to send notification to channel '%s'", channel_name) recipients.add(owner)
except Exception as e: for m in getattr(host, "managers", []):
logger.error("Error sending to channel '%s': %s", channel_name, e) recipients.add(m)
results[channel_name] = False
if not recipients:
logger.debug("send_notification: no owner/managers for '%s'", host_name)
return {}
# Fill url if not already set
if not notif.url:
notif.url = _build_url(host_name)
global_channels: dict = _config.get("notification_channels", {})
results: dict = {}
level = notif.level.upper()
is_alert = level in ("WARNING", "CRITICAL")
is_recover = level in ("RECOVER",)
# For RECOVER: send to every channel that previously fired an alert for this host,
# regardless of that channel's min_level.
if is_recover and host_name in _alerted_channels:
for channel_name in list(_alerted_channels[host_name]):
channel_cfg = global_channels.get(channel_name)
if not channel_cfg:
continue
try:
ch_type = channel_cfg.get("type", "")
driver = _DRIVERS.get(ch_type)
if driver:
ok = driver(channel_cfg, notif)
results[channel_name] = ok
if ok:
logger.info("recover sent to channel '%s': %s", channel_name, notif.title)
except Exception as e:
logger.error("error sending recover to channel '%s': %s", channel_name, e)
# Clear the alerted set once recovery is delivered
del _alerted_channels[host_name]
return results
for username in recipients:
user = users_mod.get_user(username)
if user is None:
logger.debug("send_notification: user '%s' not found", username)
continue
for channel_name in user.notification_channels:
if channel_name in results:
continue # already dispatched to this channel this notification
channel_cfg = global_channels.get(channel_name)
if not channel_cfg:
logger.warning("channel '%s' not defined in notification_channels", channel_name)
results[channel_name] = False
continue
try:
ok = _dispatch_to_channel(channel_name, channel_cfg, notif)
results[channel_name] = ok
if ok:
logger.info("notification sent to channel '%s': %s", channel_name, notif.title)
if is_alert:
_alerted_channels.setdefault(host_name, set()).add(channel_name)
else:
logger.warning("failed to send notification to channel '%s'", channel_name)
except Exception as e:
logger.error("error sending to channel '%s': %s", channel_name, e)
results[channel_name] = False
return results return results
-2
View File
@@ -252,8 +252,6 @@ def get_settings_sections(config: dict) -> list:
"Path to the pickle file used to persist host state across restarts."), "Path to the pickle file used to persist host state across restarts."),
field("logfile", "Event log", "path", field("logfile", "Event log", "path",
"Path to the event log file."), "Path to the event log file."),
field("logfmt", "Log format", "select",
"Format for event log entries: text, msg, or json."),
], ],
}, },
{ {
+64
View File
@@ -140,4 +140,68 @@
float: left; float: left;
} }
/* ── Responsive / mobile ── */
/* Suppress the global transition on mobile to avoid sluggish feel */
@media (max-width: 640px) {
* { transition: none !important; }
html, body {
overflow: auto;
height: auto;
font-size: 16px; /* prevent iOS auto-zoom on inputs */
}
/* Pages that use flex-column full-viewport layout need to relax on mobile */
body[style*="height: 100vh"],
body {
height: auto !important;
min-height: 100vh;
}
/* Containers: full width, no fixed heights */
.container {
max-width: 100% !important;
max-height: none !important;
overflow: visible !important;
padding: 8px !important;
}
/* Log section: fixed reasonable height instead of flex-grow */
.log-section {
flex: none !important;
max-height: 40vh !important;
overflow-y: auto !important;
}
/* Table section: allow vertical scroll, cap height */
.table-section {
max-height: 55vh !important;
overflow-y: auto !important;
overflow-x: auto !important;
padding: 8px !important;
}
/* Slightly larger tap targets in tables */
#ntable td, #ntable th {
padding: 4px 6px !important;
font-size: 0.82em !important;
}
/* Cards on plugin/alerts pages */
.host-card, .alert-card, .card {
padding: 10px !important;
margin-bottom: 8px !important;
}
/* Settings page tables */
table { width: 100%; }
h1 { font-size: 1.2em !important; }
h2 { font-size: 1em !important; }
}
/* Suppress nav-username text on very narrow screens — avatar/initials is enough */
@media (max-width: 400px) {
.nav-username { display: none; }
}
+22 -44
View File
@@ -3,20 +3,13 @@
{% include 'head.html' %} {% include 'head.html' %}
<style> <style>
body {
margin: 20px;
background: #f5f5f5;
}
.container { .container {
max-width: 1400px; max-width: 1400px;
margin: 0 auto; margin: 0 auto;
} }
h1 { h1 { color: #333; margin-bottom: 10px; font-size: 1.5em; }
color: #333;
margin-bottom: 10px;
}
.subtitle { .subtitle {
color: #666; color: #666;
@@ -24,55 +17,40 @@
} }
.summary-cards { .summary-cards {
display: grid; display: flex;
grid-template-columns: repeat(auto-fit, minmax(200px, 1fr)); flex-wrap: wrap;
gap: 20px; gap: 10px;
margin-bottom: 30px; margin-bottom: 16px;
} }
.summary-card { .summary-card {
background: white; background: white;
border-radius: 8px; border-radius: 6px;
padding: 20px; padding: 6px 14px;
box-shadow: 0 2px 8px rgba(0,0,0,0.1); box-shadow: 0 1px 4px rgba(0,0,0,0.1);
text-align: center; display: flex;
align-items: center;
gap: 8px;
border-left: 4px solid #ddd;
} }
.summary-card.critical { .summary-card.critical { border-left-color: #ea1e0f; }
border-left: 5px solid #f44336; .summary-card.warning { border-left-color: #ff9800; }
} .summary-card.ok { border-left-color: #4caf50; }
.summary-card.warning {
border-left: 5px solid #ff9800;
}
.summary-card.ok {
border-left: 5px solid #4caf50;
}
.summary-number { .summary-number {
font-size: 3em; font-size: 1.4em;
font-weight: bold; font-weight: bold;
margin: 10px 0; line-height: 1;
} }
.summary-number.critical { .summary-number.critical { color: #ea1e0f; }
color: #f44336; .summary-number.warning { color: #ff9800; }
} .summary-number.ok { color: #4caf50; }
.summary-number.warning {
color: #ff9800;
}
.summary-number.ok {
color: #4caf50;
}
.summary-label { .summary-label {
color: #666; color: #666;
text-transform: uppercase; font-size: 0.85em;
font-size: 0.9em;
letter-spacing: 1px;
} }
.filters { .filters {
@@ -131,7 +109,7 @@
} }
.alert-item.acknowledged { .alert-item.acknowledged {
opacity: 0.6; opacity: 0.8;
background: #f0f0f0; background: #f0f0f0;
} }
+218 -2
View File
@@ -1,22 +1,40 @@
<head> <head>
<meta http-equiv="content-type" content="text/html; charset=utf-8" /> <meta http-equiv="content-type" content="text/html; charset=utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<link rel="stylesheet" href="/static/style.css" type="text/css" /> <link rel="stylesheet" href="/static/style.css" type="text/css" />
<link rel="icon" href="/static/images/favicon.ico" sizes="32x32" /> <link rel="icon" href="/static/images/favicon.ico" sizes="32x32" />
<title>{{ title }}</title> <title>{{ title }}</title>
{% if extra_scripts %}<script src="{{ extra_scripts }}"></script>{% endif %} {% if extra_scripts %}<script src="{{ extra_scripts }}"></script>{% endif %}
<style> <style>
/* ── Reset / shared baseline ── */
*, *::before, *::after { box-sizing: border-box; }
html {
font-family: 'Segoe UI', system-ui, -apple-system, sans-serif;
font-size: 14px;
}
body {
margin: 0;
padding: 10px;
background: #f5f5f5;
}
h1 { font-size: 1.5em; color: #333; margin: 0 0 5px; }
h2 { font-size: 1.1em; color: #333; margin: 0 0 8px; }
p { margin: 0; }
/* Navigation bar — shared across all pages */ /* Navigation bar — shared across all pages */
.nav { .nav {
background: #fff; background: #fff;
padding: 10px 15px; padding: 6px 12px;
margin-bottom: 10px; margin-bottom: 10px;
box-shadow: 0 2px 4px rgba(0,0,0,.1); box-shadow: 0 2px 4px rgba(0,0,0,.1);
border-radius: 4px; border-radius: 4px;
display: flex; display: flex;
align-items: center; align-items: center;
justify-content: space-between; justify-content: space-between;
flex-wrap: wrap;
gap: 8px;
} }
.nav-links { display: flex; align-items: center; } .nav-links { display: flex; align-items: center; flex-wrap: wrap; gap: 4px; }
.nav a { .nav a {
margin-right: 20px; margin-right: 20px;
text-decoration: none; text-decoration: none;
@@ -39,6 +57,17 @@
transition: background 0.15s; transition: background 0.15s;
} }
.nav-user:hover { background: #f0f4ff; text-decoration: none; } .nav-user:hover { background: #f0f4ff; text-decoration: none; }
.nav-username {
max-width: 0;
overflow: hidden;
white-space: nowrap;
opacity: 0;
transition: max-width 0.2s ease, opacity 0.2s ease;
}
.nav-user:hover .nav-username {
max-width: 160px;
opacity: 1;
}
.nav-avatar { .nav-avatar {
width: 28px; height: 28px; width: 28px; height: 28px;
border-radius: 50%; border-radius: 50%;
@@ -57,5 +86,192 @@
font-weight: 700; font-weight: 700;
flex-shrink: 0; flex-shrink: 0;
} }
/* ── Mobile nav: hamburger toggle ── */
.nav-hamburger {
display: none;
flex-direction: column;
justify-content: space-between;
width: 26px; height: 20px;
cursor: pointer;
flex-shrink: 0;
background: none;
border: none;
padding: 0;
}
.nav-hamburger span {
display: block;
height: 3px;
background: #555;
border-radius: 2px;
}
@media (max-width: 640px) {
.nav-hamburger { display: flex; }
.nav-links {
display: none;
width: 100%;
flex-direction: column;
align-items: flex-start;
padding-top: 8px;
border-top: 1px solid #eee;
order: 3;
}
.nav-links.nav-open { display: flex; }
.nav-links a { margin-right: 0; padding: 6px 0; font-size: 1em; }
}
/* Swiss railway clock — nav */
.nav-clock {
flex-shrink: 0;
line-height: 0;
margin-left: auto;
padding: 4px 4px 4px 0;
cursor: pointer;
}
#swiss-clock { display: block; }
/* Swiss railway clock — full-page overlay */
#clock-overlay {
display: none;
position: fixed;
inset: 0;
z-index: 9999;
background: #1a1a1a;
align-items: center;
justify-content: center;
cursor: pointer;
}
#clock-overlay.visible { display: flex; }
#swiss-clock-overlay { display: block; }
</style> </style>
<script>
/* ── Swiss Federal Railway (SBB) clock ── */
/* Draw one frame of the clock onto any canvas element. */
function drawSwissClock(canvas) {
var SIZE = canvas.width;
var R = SIZE / 2;
var ctx = canvas.getContext('2d');
var now = new Date();
var h = now.getHours() % 12;
var m = now.getMinutes();
var s = now.getSeconds();
var ms = now.getMilliseconds();
/* Seconds hand idles ~1.5 s at 12 before advancing (SBB behaviour) */
var sFrac = s + ms / 1000;
var sAngle = sFrac >= 58.5 ? 0 : (sFrac / 58.5) * Math.PI * 2;
ctx.clearRect(0, 0, SIZE, SIZE);
/* face */
ctx.beginPath();
ctx.arc(R, R, R - 1, 0, Math.PI * 2);
ctx.fillStyle = '#fff';
ctx.fill();
ctx.strokeStyle = '#333';
ctx.lineWidth = SIZE * 0.018;
ctx.stroke();
/* tick marks */
for (var i = 0; i < 60; i++) {
var a = (i / 60) * Math.PI * 2 - Math.PI / 2;
var isHour = (i % 5 === 0);
ctx.beginPath();
ctx.moveTo(R + Math.cos(a) * (isHour ? R * 0.72 : R * 0.88),
R + Math.sin(a) * (isHour ? R * 0.72 : R * 0.88));
ctx.lineTo(R + Math.cos(a) * R * 0.94,
R + Math.sin(a) * R * 0.94);
ctx.strokeStyle = '#222';
ctx.lineWidth = isHour ? SIZE * 0.027 : SIZE * 0.011;
ctx.lineCap = 'butt';
ctx.stroke();
}
/* hands */
function hand(angle, tip, tail, width, color) {
ctx.save();
ctx.translate(R, R);
ctx.rotate(angle);
ctx.beginPath();
ctx.moveTo(tail, 0);
ctx.lineTo(tip, 0);
ctx.strokeStyle = color;
ctx.lineWidth = width;
ctx.lineCap = 'square';
ctx.stroke();
ctx.restore();
}
hand((m + s / 60) / 60 * Math.PI * 2 - Math.PI / 2,
R * 0.88, -R * 0.12, SIZE * 0.027, '#222'); /* minute */
hand((h + m / 60) / 12 * Math.PI * 2 - Math.PI / 2,
R * 0.58, -R * 0.12, SIZE * 0.039, '#222'); /* hour */
hand(sAngle - Math.PI / 2, R * 0.78, -R * 0.22,
SIZE * 0.013, '#e00'); /* second tail+tip */
/* round dot at tip of second hand */
var dotR = SIZE * 0.028;
ctx.save();
ctx.translate(R, R);
ctx.rotate(sAngle - Math.PI / 2);
ctx.beginPath();
ctx.arc(R * 0.78, 0, dotR, 0, Math.PI * 2);
ctx.fillStyle = '#e00';
ctx.fill();
ctx.restore();
/* centre cap */
ctx.beginPath();
ctx.arc(R, R, R * 0.04, 0, Math.PI * 2);
ctx.fillStyle = '#222';
ctx.fill();
}
/* Resize the overlay canvas to fit the viewport, keeping it square. */
function resizeOverlayClock() {
var oc = document.getElementById('swiss-clock-overlay');
if (!oc) return;
var size = Math.min(window.innerWidth, window.innerHeight) * 0.88;
size = Math.floor(size);
oc.width = size;
oc.height = size;
}
/* Main tick — redraws both nav clock and (if visible) overlay clock. */
function clockTick() {
var nav = document.getElementById('swiss-clock');
if (nav) drawSwissClock(nav);
var overlay = document.getElementById('clock-overlay');
if (overlay && overlay.classList.contains('visible')) {
var oc = document.getElementById('swiss-clock-overlay');
if (oc) drawSwissClock(oc);
}
var delay = 100 - (Date.now() % 100);
setTimeout(clockTick, delay);
}
document.addEventListener('DOMContentLoaded', function() {
/* Start the shared tick loop */
clockTick();
/* Overlay toggle — clicking the nav clock opens it */
var navClock = document.querySelector('.nav-clock');
var overlay = document.getElementById('clock-overlay');
if (navClock && overlay) {
navClock.addEventListener('click', function() {
resizeOverlayClock();
overlay.classList.add('visible');
});
overlay.addEventListener('click', function() {
overlay.classList.remove('visible');
});
window.addEventListener('resize', function() {
if (overlay.classList.contains('visible')) resizeOverlayClock();
});
}
});
</script>
<script src="static/sorttable.js"></script>
</head> </head>
+115 -27
View File
@@ -4,17 +4,42 @@
<style> <style>
body { body {
margin: 10px; display: flex;
background: #f5f5f5; flex-direction: column;
height: 100vh;
overflow: hidden; overflow: hidden;
} }
@media (max-width: 640px) {
body {
height: auto;
min-height: 100vh;
overflow: auto;
flex-direction: column;
}
.container {
max-height: none;
overflow: visible;
}
.table-section {
max-height: 55vh;
}
.log-section {
flex: none;
max-height: 40vh;
}
}
.container { .container {
flex: 1;
min-height: 0;
max-width: 1600px; max-width: 1600px;
width: 100%;
margin: 0 auto; margin: 0 auto;
max-height: calc(100vh - 120px); display: flex;
overflow-y: auto; flex-direction: column;
padding-right: 10px; gap: 15px;
overflow: hidden;
} }
h1 { h1 {
@@ -50,14 +75,18 @@
border-radius: 6px; border-radius: 6px;
padding: 15px; padding: 15px;
box-shadow: 0 1px 4px rgba(0,0,0,0.1); box-shadow: 0 1px 4px rgba(0,0,0,0.1);
overflow-x: auto;
overflow-y: auto;
max-height: 60vh;
} }
.log-section { .log-section {
flex: 1;
min-height: 0;
background: white; background: white;
border-radius: 6px; border-radius: 6px;
padding: 15px; padding: 15px;
box-shadow: 0 1px 4px rgba(0,0,0,0.1); box-shadow: 0 1px 4px rgba(0,0,0,0.1);
max-height: 400px;
overflow-y: auto; overflow-y: auto;
} }
@@ -71,7 +100,8 @@
#ntable th { #ntable th {
border: 1px solid #e0e0e0; border: 1px solid #e0e0e0;
text-align: left; text-align: left;
padding: 8px 10px; padding: 2px 4px;
white-space: nowrap;
} }
#ntable tr:nth-child(even) { #ntable tr:nth-child(even) {
@@ -82,8 +112,24 @@
background-color: #e3f2fd; background-color: #e3f2fd;
} }
#ntable tbody tr.row-warning {
background-color: #fff8c5;
}
#ntable tbody tr.row-critical {
background-color: #fde8e8;
}
#ntable tbody tr.row-warning:hover {
background-color: #fff0a0;
}
#ntable tbody tr.row-critical:hover {
background-color: #f9c8c8;
}
#ntable th { #ntable th {
padding: 12px 10px; padding: 6px 8px;
background-color: #2196f3; background-color: #2196f3;
color: white; color: white;
font-weight: 600; font-weight: 600;
@@ -112,24 +158,20 @@
} }
/* Scrollbar styling */ /* Scrollbar styling */
.container::-webkit-scrollbar,
.log-section::-webkit-scrollbar { .log-section::-webkit-scrollbar {
width: 8px; width: 8px;
} }
.container::-webkit-scrollbar-track,
.log-section::-webkit-scrollbar-track { .log-section::-webkit-scrollbar-track {
background: #f1f1f1; background: #f1f1f1;
border-radius: 4px; border-radius: 4px;
} }
.container::-webkit-scrollbar-thumb,
.log-section::-webkit-scrollbar-thumb { .log-section::-webkit-scrollbar-thumb {
background: #888; background: #888;
border-radius: 4px; border-radius: 4px;
} }
.container::-webkit-scrollbar-thumb:hover,
.log-section::-webkit-scrollbar-thumb:hover { .log-section::-webkit-scrollbar-thumb:hover {
background: #555; background: #555;
} }
@@ -137,7 +179,7 @@
/* Message styling */ /* Message styling */
#messages { #messages {
font-size: 0.85em; font-size: 0.85em;
line-height: 1.6; line-height: 1.0;
} }
#messages div { #messages div {
@@ -199,15 +241,37 @@
var nTable = document; var nTable = document;
var name_idx = {}; var name_idx = {};
var c = 0; var c = 0;
var HBD_VERSION = "{{ hbd_version }}";
function hostNameHtml(data) {
var nameHtml = data.name;
if (!data.hbc_version || data.hbc_version !== HBD_VERSION) {
nameHtml += ' 🥀';
}
return data.dyn ? '<b>' + nameHtml + '</b>' : nameHtml;
}
function setup() { function setup() {
name_idx = {}; name_idx = {};
nTable = document.getElementById("ntable"); nTable = document.getElementById("ntable");
for (var i = 0, row; (row = nTable.rows[i]); i++) { for (var i = 0, row; (row = nTable.rows[i]); i++) {
if (i == 0) continue; if (i == 0) continue;
name = nTable.rows[i].cells[0].innerText; var cell = nTable.rows[i].cells[0];
var name = cell.dataset.name || cell.innerText.replace(/\s*🥀\s*$/, '').trim();
name_idx[name] = nTable.rows[i]; name_idx[name] = nTable.rows[i];
/* console.log("name_Id[" + name + "]: " + name_idx[name].innerText); */ }
}
function updateRowAlert(row, data) {
var criticalUnacked = data.alert_critical_unacked || 0;
var criticalAcked = data.alert_critical_acked || 0;
var warningUnacked = data.alert_warning_unacked || 0;
var warningAcked = data.alert_warning_acked || 0;
row.classList.remove('row-warning', 'row-critical');
if (criticalUnacked > 0 || criticalAcked > 0) {
row.classList.add('row-critical');
} else if (warningUnacked > 0 || warningAcked > 0) {
row.classList.add('row-warning');
} }
} }
@@ -245,11 +309,8 @@
row.appendChild(c_ipv6state); row.appendChild(c_ipv6state);
row.appendChild(c_ipv6latency); row.appendChild(c_ipv6latency);
row.appendChild(c_ipv6statets); row.appendChild(c_ipv6statets);
if (data.dyn) { c_name.dataset.name = data.name;
c_name.innerHTML = "<b>" + data.name + "</b>"; c_name.innerHTML = hostNameHtml(data);
} else {
c_name.innerHTML = data.name;
}
// Set alert counts in "x/y" format (unacked/acked) // Set alert counts in "x/y" format (unacked/acked)
var warningUnacked = data.alert_warning_unacked || 0; var warningUnacked = data.alert_warning_unacked || 0;
@@ -278,12 +339,31 @@
var table = document.getElementById("ntablebody"); // find table to append to var table = document.getElementById("ntablebody"); // find table to append to
table.appendChild(row); // append row to table table.appendChild(row); // append row to table
name_idx[c_name] = row; name_idx[c_name] = row;
updateRowAlert(row, data);
} }
function formatTS(ts) { function formatTS(ts) {
const milliseconds = ts * 1000; const now = new Date();
const dateObject = new Date(milliseconds); const d = new Date(ts * 1000);
return dateObject.toLocaleString("de-DE");
const pad = n => String(n).padStart(2, '0');
const timeStr = `${pad(d.getHours())}:${pad(d.getMinutes())}:${pad(d.getSeconds())}`;
// Same calendar day → show time only
if (d.toDateString() === now.toDateString()) {
return timeStr;
}
// Within 8 days → show "-X d hh:mm:ss"
const todayStart = new Date(now.getFullYear(), now.getMonth(), now.getDate());
const dStart = new Date(d.getFullYear(), d.getMonth(), d.getDate());
const diffDays = Math.round((todayStart - dStart) / 86400000);
if (diffDays < 8) {
return `-${diffDays}d ${timeStr}`;
}
// Older → date only
return `${d.getFullYear()}-${pad(d.getMonth() + 1)}-${pad(d.getDate())}`;
} }
function update_table(data) { function update_table(data) {
@@ -292,6 +372,11 @@
setup(); setup();
} }
// Update name cell (version indicator)
var nameCell = name_idx[data.name].cells[0];
nameCell.dataset.name = data.name;
nameCell.innerHTML = hostNameHtml(data);
// Update warning and critical counts in "x/y" format (unacked/acked) // Update warning and critical counts in "x/y" format (unacked/acked)
var warningUnacked = data.alert_warning_unacked || 0; var warningUnacked = data.alert_warning_unacked || 0;
var warningAcked = data.alert_warning_acked || 0; var warningAcked = data.alert_warning_acked || 0;
@@ -339,6 +424,7 @@
name_idx[data.name].cells[4 + i * 4].innerHTML = state; name_idx[data.name].cells[4 + i * 4].innerHTML = state;
name_idx[data.name].cells[5 + i * 4].innerHTML = latency; name_idx[data.name].cells[5 + i * 4].innerHTML = latency;
} }
updateRowAlert(name_idx[data.name], data);
} }
function WS_Connect() { function WS_Connect() {
@@ -399,8 +485,10 @@
{% include 'menu.html' %} {% include 'menu.html' %}
<div class="container"> <div class="container">
<h1>{{ header }}</h1> <div>
<p class="subtitle">Real-time host monitoring and event log</p> <h1>{{ header }}</h1>
<p class="subtitle">Real-time host monitoring and event log</p>
</div>
<div class="table-section"> <div class="table-section">
<table id="ntable" class="sortable"> <table id="ntable" class="sortable">
@@ -421,8 +509,8 @@
</thead> </thead>
<tbody id="ntablebody"> <tbody id="ntablebody">
{% for host in hosts %} {% for host in hosts %}
<tr> <tr class="{% if host.alert_critical_unacked > 0 or host.alert_critical_acked > 0 %}row-critical{% elif host.alert_warning_unacked > 0 or host.alert_warning_acked > 0 %}row-warning{% endif %}">
<td>{{ host.name }}</td> <td data-name="{{ host.name }}">{{ host.name }}{% if not host.hbc_version or host.hbc_version != hbd_version %} 🥀{% endif %}</td>
<td style="text-align: center; color: #ff9800; font-weight: bold;"> <td style="text-align: center; color: #ff9800; font-weight: bold;">
{%- set warning_unacked = host.alert_warning_unacked -%} {%- set warning_unacked = host.alert_warning_unacked -%}
{%- set warning_acked = host.alert_warning_acked -%} {%- set warning_acked = host.alert_warning_acked -%}
+26 -1
View File
@@ -1,5 +1,8 @@
<div class="nav"> <div class="nav">
<div class="nav-links"> <button class="nav-hamburger" id="nav-hamburger-btn" aria-label="Menu" aria-expanded="false">
<span></span><span></span><span></span>
</button>
<div class="nav-links" id="nav-links">
<a href="/live"{% if active_page == "live" %} class="active"{% endif %}>Live Dashboard</a> <a href="/live"{% if active_page == "live" %} class="active"{% endif %}>Live Dashboard</a>
<a href="/plugins"{% if active_page == "plugins" %} class="active"{% endif %}>Plugin Metrics</a> <a href="/plugins"{% if active_page == "plugins" %} class="active"{% endif %}>Plugin Metrics</a>
<a href="/alerts"{% if active_page == "alerts" %} class="active"{% endif %}>Alerts</a> <a href="/alerts"{% if active_page == "alerts" %} class="active"{% endif %}>Alerts</a>
@@ -7,6 +10,9 @@
<a href="/settings"{% if active_page == "settings" %} class="active"{% endif %}>Settings</a> <a href="/settings"{% if active_page == "settings" %} class="active"{% endif %}>Settings</a>
{% endif %} {% endif %}
</div> </div>
<div class="nav-clock" title="Click for full-screen clock">
<canvas id="swiss-clock" width="44" height="44"></canvas>
</div>
{% if current_user %} {% if current_user %}
<a href="/profile" class="nav-user{% if active_page == 'profile' %} active{% endif %}" title="{{ current_user.full_name or current_user.username }}"> <a href="/profile" class="nav-user{% if active_page == 'profile' %} active{% endif %}" title="{{ current_user.full_name or current_user.username }}">
{% if current_user.avatar %} {% if current_user.avatar %}
@@ -14,6 +20,25 @@
{% else %} {% else %}
<span class="nav-initials">{{ (current_user.full_name or current_user.username)[:1] | upper }}</span> <span class="nav-initials">{{ (current_user.full_name or current_user.username)[:1] | upper }}</span>
{% endif %} {% endif %}
<span class="nav-username">{{ current_user.full_name or current_user.username }}</span>
</a> </a>
{% endif %} {% endif %}
</div> </div>
<!-- Full-page clock overlay (click anywhere to dismiss) -->
<div id="clock-overlay">
<canvas id="swiss-clock-overlay" width="400" height="400"></canvas>
</div>
<script>
(function() {
var btn = document.getElementById('nav-hamburger-btn');
var links = document.getElementById('nav-links');
if (btn && links) {
btn.addEventListener('click', function() {
var open = links.classList.toggle('nav-open');
btn.setAttribute('aria-expanded', open ? 'true' : 'false');
});
}
})();
</script>
+1 -5
View File
@@ -3,11 +3,7 @@
{% include 'head.html' %} {% include 'head.html' %}
<style> <style>
body { body { overflow: hidden; }
margin: 10px;
background: #f5f5f5;
overflow: hidden;
}
.container { .container {
max-width: 1400px; max-width: 1400px;
+1 -5
View File
@@ -3,11 +3,7 @@
{% include 'head.html' %} {% include 'head.html' %}
<style> <style>
body { html, body { overflow: visible; }
margin: 20px;
background: #f5f5f5;
font-family: 'Segoe UI', system-ui, sans-serif;
}
.container { .container {
max-width: 900px; max-width: 900px;
+68 -7
View File
@@ -3,15 +3,10 @@
{% include 'head.html' %} {% include 'head.html' %}
<style> <style>
body { html, body { overflow: visible; }
margin: 20px;
background: #f5f5f5;
font-family: 'Segoe UI', system-ui, sans-serif;
}
.container { .container {
max-width: 960px; max-width: 960px;
margin: 0 auto;
} }
h1 { color: #333; margin-bottom: 4px; font-size: 1.5em; } h1 { color: #333; margin-bottom: 4px; font-size: 1.5em; }
@@ -213,6 +208,49 @@
.channel-field-value { color: #333; word-break: break-all; } .channel-field-value { color: #333; word-break: break-all; }
/* ---- Hosts table ---- */ /* ---- Hosts table ---- */
/* ---- Mobile: collapsible sidebar ---- */
.sidebar-toggle {
display: none;
width: 100%;
padding: 8px 12px;
background: #e8eaf6;
border: none;
border-radius: 6px;
font-size: 0.9em;
font-weight: 600;
color: #283593;
cursor: pointer;
text-align: left;
margin-bottom: 16px;
}
.sidebar-toggle::after { content: ' ▾'; float: right; }
.sidebar-toggle.open::after { content: ' ▴'; }
@media (max-width: 640px) {
.sidebar-toggle { display: block; }
.settings-layout { flex-direction: column; gap: 0; }
.settings-sidebar {
width: 100%;
position: static;
margin-bottom: 0;
}
.sidebar-nav {
display: none;
background: white;
border-radius: 6px;
box-shadow: 0 1px 4px rgba(0,0,0,.1);
margin-bottom: 16px;
padding: 4px 0;
}
.sidebar-nav.open { display: block; }
.sidebar-nav a { padding: 10px 16px; font-size: 1em; }
.field-row { flex-direction: column; gap: 4px; }
.field-label { width: 100%; font-size: 0.82em; color: #888; }
}
.host-bool { text-align: center; } .host-bool { text-align: center; }
.dot-yes { color: #2e7d32; font-size: 1.1em; } .dot-yes { color: #2e7d32; font-size: 1.1em; }
.dot-no { color: #ddd; font-size: 1.1em; } .dot-no { color: #ddd; font-size: 1.1em; }
@@ -229,9 +267,10 @@
<!-- Sidebar navigation --> <!-- Sidebar navigation -->
<nav class="settings-sidebar"> <nav class="settings-sidebar">
<button class="sidebar-toggle" id="sidebar-toggle" aria-expanded="false">Sections</button>
<div class="sidebar-nav" id="sidebar-nav"> <div class="sidebar-nav" id="sidebar-nav">
{% for section in sections %} {% for section in sections %}
<a href="#{{ section.id }}">{{ section.title }}</a> <a href="#{{ section.id }}" onclick="closeSidebar()">{{ section.title }}</a>
{% endfor %} {% endfor %}
</div> </div>
</nav> </nav>
@@ -424,6 +463,28 @@
}, { threshold: 0.25 }); }, { threshold: 0.25 });
sections.forEach(s => observer.observe(s)); sections.forEach(s => observer.observe(s));
// Collapsible sidebar on mobile
var sidebarToggle = document.getElementById('sidebar-toggle');
var sidebarNav = document.getElementById('sidebar-nav');
if (sidebarToggle && sidebarNav) {
sidebarToggle.addEventListener('click', function() {
var open = sidebarNav.classList.toggle('open');
sidebarToggle.classList.toggle('open', open);
sidebarToggle.setAttribute('aria-expanded', open ? 'true' : 'false');
});
}
</script>
<script>
function closeSidebar() {
var sidebarNav = document.getElementById('sidebar-nav');
var sidebarToggle = document.getElementById('sidebar-toggle');
if (sidebarNav) { sidebarNav.classList.remove('open'); }
if (sidebarToggle) {
sidebarToggle.classList.remove('open');
sidebarToggle.setAttribute('aria-expanded', 'false');
}
}
</script> </script>
</body> </body>
</html> </html>
+173 -50
View File
@@ -14,6 +14,7 @@ import time
from enum import Enum from enum import Enum
from typing import Dict, Any, Optional, Tuple, Callable from typing import Dict, Any, Optional, Tuple, Callable
from . import notify as notify_mod from . import notify as notify_mod
from .config import THRESHOLD_DEFAULTS
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
eventlog = notify_mod.eventlog eventlog = notify_mod.eventlog
@@ -58,6 +59,8 @@ class AlertState:
self.formatted_message = None # Formatted display message for UI self.formatted_message = None # Formatted display message for UI
self.acknowledged = False # Whether alert has been acknowledged self.acknowledged = False # Whether alert has been acknowledged
self.acknowledged_at = None # Timestamp when acknowledged self.acknowledged_at = None # Timestamp when acknowledged
self.consecutive_count = 0 # Consecutive exceedances while still OK (for count gating)
self.pending_since: Optional[float] = None # non-None while waiting out grace period before notifying
def update( def update(
self, self,
@@ -103,6 +106,7 @@ class AlertState:
self.level = level self.level = level
self.since = now self.since = now
self.notification_count = 0 self.notification_count = 0
self.last_notification = None # restart reminder interval on level change
# Reset acknowledgment on state change # Reset acknowledgment on state change
if level != AlertLevel.OK: if level != AlertLevel.OK:
# Only reset if changing to a different alert level # Only reset if changing to a different alert level
@@ -118,8 +122,11 @@ class AlertState:
# Helper to sanitize numeric values for JSON (handle inf/nan) # Helper to sanitize numeric values for JSON (handle inf/nan)
def sanitize_value(val): def sanitize_value(val):
if isinstance(val, float) and (math.isinf(val) or math.isnan(val)): if isinstance(val, float):
return None if math.isinf(val):
return "overdue"
if math.isnan(val):
return None
return val return val
result = { result = {
@@ -146,6 +153,12 @@ class AlertState:
return result return result
def __setstate__(self, state):
"""Restore from pickle, backfilling fields added after the pickle was written."""
self.__dict__.update(state)
if not hasattr(self, 'consecutive_count'):
self.consecutive_count = 0
def acknowledge(self): def acknowledge(self):
"""Acknowledge this alert to stop reminder notifications.""" """Acknowledge this alert to stop reminder notifications."""
self.acknowledged = True self.acknowledged = True
@@ -167,6 +180,7 @@ class ThresholdConfig:
operator: str = ">", operator: str = ">",
hysteresis: float = 0.0, hysteresis: float = 0.0,
enabled: bool = True, enabled: bool = True,
count: int = 1,
): ):
""" """
Initialize threshold configuration. Initialize threshold configuration.
@@ -178,6 +192,7 @@ class ThresholdConfig:
operator: Comparison operator (>, >=, <, <=, ==, !=) operator: Comparison operator (>, >=, <, <=, ==, !=)
hysteresis: Hysteresis percentage to prevent flapping (0.0-1.0) hysteresis: Hysteresis percentage to prevent flapping (0.0-1.0)
enabled: Whether this threshold is enabled enabled: Whether this threshold is enabled
count: Number of consecutive exceedances required before alerting (default 1)
""" """
self.metric_path = metric_path self.metric_path = metric_path
self.warning = warning self.warning = warning
@@ -185,6 +200,7 @@ class ThresholdConfig:
self.enabled = enabled self.enabled = enabled
self.hysteresis = hysteresis self.hysteresis = hysteresis
self.display = display self.display = display
self.count = max(1, int(count))
# Parse operator # Parse operator
try: try:
@@ -325,6 +341,7 @@ class ThresholdChecker:
self.default_config = "default" self.default_config = "default"
self.renotify_interval = renotify_interval self.renotify_interval = renotify_interval
self.grace_seconds: float = float(config.get("grace", 2))
self.journal = journal self.journal = journal
# Parse configuration # Parse configuration
@@ -357,6 +374,7 @@ class ThresholdChecker:
self.threshold_configs.clear() self.threshold_configs.clear()
self.thresholds.clear() self.thresholds.clear()
self.host_config_mapping.clear() self.host_config_mapping.clear()
self.grace_seconds = float(config.get("grace", 2))
# Parse new configuration # Parse new configuration
self._parse_config(config) self._parse_config(config)
@@ -391,8 +409,28 @@ class ThresholdChecker:
logger.info("No threshold configurations defined") logger.info("No threshold configurations defined")
return return
# Parse each named configuration # Build effective_defaults: THRESHOLD_DEFAULTS merged with the 'default' config (if present).
# All other configs inherit any metric not explicitly defined from effective_defaults.
effective_defaults: Dict[str, ThresholdConfig] = {}
for plugin_name, plugin_thresholds in THRESHOLD_DEFAULTS.get("thresholds", {}).items():
if isinstance(plugin_thresholds, dict):
self._parse_plugin_thresholds(plugin_name, plugin_thresholds, target_dict=effective_defaults)
if "default" in threshold_configs:
default_data = threshold_configs["default"]
if isinstance(default_data, dict) and "thresholds" in default_data:
for plugin_name, plugin_thresholds in default_data["thresholds"].items():
if isinstance(plugin_thresholds, dict):
self._parse_plugin_thresholds(plugin_name, plugin_thresholds, target_dict=effective_defaults)
self.threshold_configs["default"] = dict(effective_defaults)
logger.info("Registered 'default' threshold config with %d metrics", len(effective_defaults))
# Parse each named configuration, seeding it with effective_defaults first
for config_name, config_data in threshold_configs.items(): for config_name, config_data in threshold_configs.items():
if config_name == "default":
continue # already handled above
if not isinstance(config_data, dict): if not isinstance(config_data, dict):
logger.warning("Invalid threshold config '%s', skipping", config_name) logger.warning("Invalid threshold config '%s', skipping", config_name)
continue continue
@@ -402,7 +440,7 @@ class ThresholdChecker:
continue continue
logger.info("Parsing threshold configuration: %s", config_name) logger.info("Parsing threshold configuration: %s", config_name)
self.threshold_configs[config_name] = {} self.threshold_configs[config_name] = dict(effective_defaults)
thresholds_config = config_data["thresholds"] thresholds_config = config_data["thresholds"]
for plugin_name, plugin_thresholds in thresholds_config.items(): for plugin_name, plugin_thresholds in thresholds_config.items():
@@ -600,6 +638,7 @@ class ThresholdChecker:
hysteresis = rtt_thresholds.get("hysteresis", 0.1) # 10% default hysteresis = rtt_thresholds.get("hysteresis", 0.1) # 10% default
enabled = rtt_thresholds.get("enabled", True) enabled = rtt_thresholds.get("enabled", True)
display = rtt_thresholds.get("display") display = rtt_thresholds.get("display")
count = rtt_thresholds.get("count", 1)
if warning is None and critical is None: if warning is None and critical is None:
logger.warning("No RTT thresholds defined, skipping") logger.warning("No RTT thresholds defined, skipping")
@@ -612,14 +651,16 @@ class ThresholdChecker:
operator=operator, operator=operator,
hysteresis=hysteresis, hysteresis=hysteresis,
enabled=enabled, enabled=enabled,
display=display display=display,
count=count,
) )
target_dict[metric_path] = threshold target_dict[metric_path] = threshold
logger.debug( logger.debug(
"Registered RTT threshold: warn=%s ms, crit=%s ms", "Registered RTT threshold: warn=%s ms, crit=%s ms, count=%d",
warning, warning,
critical critical,
count,
) )
def get_thresholds_for_host(self, host_name: str) -> Dict[str, ThresholdConfig]: def get_thresholds_for_host(self, host_name: str) -> Dict[str, ThresholdConfig]:
@@ -692,6 +733,26 @@ class ThresholdChecker:
alert_state.level alert_state.level
) )
# Apply consecutive-count gating: when currently OK, require threshold.count
# consecutive exceedances before escalating to WARNING/CRITICAL.
if new_level == AlertLevel.OK:
# Value is fine (or recovered) — reset the pending counter immediately.
alert_state.consecutive_count = 0
elif alert_state.level == AlertLevel.OK and new_level != AlertLevel.OK:
# First time we exceed while still OK: count up.
alert_state.consecutive_count += 1
if alert_state.consecutive_count < threshold.count:
logger.debug(
"RTT threshold exceeded %d/%d consecutive times for %s on %s",
alert_state.consecutive_count,
threshold.count,
metric_path,
host_name,
)
return None
# Count reached — fire the alert and reset the counter.
alert_state.consecutive_count = 0
# Determine which threshold was exceeded # Determine which threshold was exceeded
threshold_value = None threshold_value = None
if new_level == AlertLevel.CRITICAL and threshold.critical is not None: if new_level == AlertLevel.CRITICAL and threshold.critical is not None:
@@ -702,15 +763,10 @@ class ThresholdChecker:
# Update state and check for changes # Update state and check for changes
old_level = alert_state.level old_level = alert_state.level
if alert_state.update(new_level, value, threshold_value, threshold.operator.value): if alert_state.update(new_level, value, threshold_value, threshold.operator.value):
# For check_value, we don't have full plugin data, pass None self._apply_grace(host_name, alert_state, metric_path, old_level, new_level, value, threshold, None)
lvl, message, formatted_msg = self._trigger_notification(host_name, metric_path, old_level, new_level, value, threshold, None)
# Update alert state with formatted message
alert_state.formatted_message = formatted_msg
self._send_notification(host_name, lvl, message, metric_path, old_level, new_level, value)
return (old_level, new_level) return (old_level, new_level)
elif new_level != AlertLevel.OK: elif new_level != AlertLevel.OK:
# Check if we should re-notify self._check_pending_or_renotify(host_name, alert_state, metric_path, value, threshold, None)
self._check_renotify(host_name, alert_state, metric_path, value, threshold, None)
return None return None
def check_plugin_data( def check_plugin_data(
@@ -769,13 +825,9 @@ class ThresholdChecker:
old_level = alert_state.level old_level = alert_state.level
if alert_state.update(new_level, value, threshold_value, threshold.operator.value): if alert_state.update(new_level, value, threshold_value, threshold.operator.value):
state_changes.append((metric_path, old_level, new_level, value)) state_changes.append((metric_path, old_level, new_level, value))
lvl, message, formatted_msg = self._trigger_notification(host_name, metric_path, old_level, new_level, value, threshold, data) self._apply_grace(host_name, alert_state, metric_path, old_level, new_level, value, threshold, data)
# Update alert state with formatted message
alert_state.formatted_message = formatted_msg
self._send_notification(host_name, lvl, message, metric_path, old_level, new_level, value)
elif new_level != AlertLevel.OK: elif new_level != AlertLevel.OK:
# Check if we should re-notify self._check_pending_or_renotify(host_name, alert_state, metric_path, value, threshold, data)
self._check_renotify(host_name, alert_state, metric_path, value, threshold, data)
# Check nested metrics (e.g., partition data in disk_monitor) # Check nested metrics (e.g., partition data in disk_monitor)
self._check_nested_metrics( self._check_nested_metrics(
@@ -838,20 +890,9 @@ class ThresholdChecker:
old_level = alert_state.level old_level = alert_state.level
if alert_state.update(new_level, value, threshold_value, threshold.operator.value): if alert_state.update(new_level, value, threshold_value, threshold.operator.value):
state_changes.append((metric_path, old_level, new_level, value)) state_changes.append((metric_path, old_level, new_level, value))
lvl, message, formatted_msg = self._trigger_notification( self._apply_grace(host_name, alert_state, metric_path, old_level, new_level, value, threshold, data)
host_name,
metric_path,
old_level,
new_level,
value,
threshold,
data # Pass full plugin data for format string
)
# Update alert state with formatted message
alert_state.formatted_message = formatted_msg
self._send_notification(host_name, lvl, message, metric_path, old_level, new_level, value)
elif new_level != AlertLevel.OK: elif new_level != AlertLevel.OK:
self._check_renotify(host_name, alert_state, metric_path, value, threshold, data) self._check_pending_or_renotify(host_name, alert_state, metric_path, value, threshold, data)
def _trigger_notification( def _trigger_notification(
self, self,
@@ -884,48 +925,50 @@ class ThresholdChecker:
# Format operator symbol # Format operator symbol
op_symbol = threshold.operator.value op_symbol = threshold.operator.value
# Use a display-friendly value (inf is the sentinel for "overdue")
import math
display_value = "overdue" if isinstance(value, float) and math.isinf(value) else value
# Format message # Format message
if new_level == AlertLevel.OK: if new_level == AlertLevel.OK:
lvl = "RECOVERED" lvl = "RECOVER"
message = f"{metric_path} = {value} ({old_level.name} -> OK)" message = f"{metric_path} = {display_value} ({old_level.name} -> OK)"
elif new_level == AlertLevel.WARNING: elif new_level == AlertLevel.WARNING:
lvl = "WARNING" lvl = "WARNING"
if threshold_value is not None: if threshold_value is not None:
# Use display format string
threshold_info = self._format_display( threshold_info = self._format_display(
threshold.display, threshold.display,
value=value, value=display_value,
threshold_value=threshold_value, threshold_value=threshold_value,
op_symbol=op_symbol, op_symbol=op_symbol,
plugin_data=plugin_data plugin_data=plugin_data
) )
message = f"{metric_path} = {value} {threshold_info}" message = f"{metric_path} = {display_value} {threshold_info}"
else: else:
message = f"{metric_path} = {value}" message = f"{metric_path} = {display_value}"
elif new_level == AlertLevel.CRITICAL: elif new_level == AlertLevel.CRITICAL:
lvl = "CRITICAL" lvl = "CRITICAL"
if threshold_value is not None: if threshold_value is not None:
# Use display format string
threshold_info = self._format_display( threshold_info = self._format_display(
threshold.display, threshold.display,
value=value, value=display_value,
threshold_value=threshold_value, threshold_value=threshold_value,
op_symbol=op_symbol, op_symbol=op_symbol,
plugin_data=plugin_data plugin_data=plugin_data
) )
message = f"{metric_path} = {value} {threshold_info}" message = f"{metric_path} = {display_value} {threshold_info}"
else: else:
message = f"{metric_path} = {value}" message = f"{metric_path} = {display_value}"
else: else:
lvl = "UNKNOWN" lvl = "UNKNOWN"
message = f"{metric_path} = {value}" message = f"{metric_path} = {display_value}"
# Return the formatted threshold info for storing in AlertState # Return the formatted threshold info for storing in AlertState
formatted_threshold_msg = None formatted_threshold_msg = None
if threshold_value is not None and new_level != AlertLevel.OK: if threshold_value is not None and new_level != AlertLevel.OK:
formatted_threshold_msg = self._format_display( formatted_threshold_msg = self._format_display(
threshold.display, threshold.display,
value=value, value=display_value,
threshold_value=threshold_value, threshold_value=threshold_value,
op_symbol=op_symbol, op_symbol=op_symbol,
plugin_data=plugin_data plugin_data=plugin_data
@@ -944,9 +987,15 @@ class ThresholdChecker:
value: Any, value: Any,
): ):
"""Send notification and log to journal/eventlog.""" """Send notification and log to journal/eventlog."""
# Send notification using host-specific channels
try: try:
notify_mod.pushmsg_for_host(host_name, f"{lvl}: {host_name} - {message}") notify_mod.send_notification(
host_name,
notify_mod.Notification(
title=f"[{lvl}] {host_name}",
body=message,
level=lvl,
),
)
logger.info("Notification sent: %s", message) logger.info("Notification sent: %s", message)
except Exception as e: except Exception as e:
logger.error("Failed to send notification: %s", e) logger.error("Failed to send notification: %s", e)
@@ -1018,6 +1067,74 @@ class ThresholdChecker:
) )
return f"(threshold: {op_symbol} {threshold_value})" return f"(threshold: {op_symbol} {threshold_value})"
def _apply_grace(
self,
host_name: str,
alert_state: AlertState,
metric_path: str,
old_level: AlertLevel,
new_level: AlertLevel,
value: Any,
threshold: ThresholdConfig,
plugin_data: Optional[Dict[str, Any]],
) -> None:
"""Handle a state-change transition with grace-period logic.
Transitioning INTO alert: defers the notification for grace_seconds.
Transitioning TO OK:
- Still in grace window (pending_since set): suppresses both the alert
and the recovery — the spike never warranted a page.
- Past grace: fires the RECOVER notification normally.
"""
lvl, message, formatted_msg = self._trigger_notification(
host_name, metric_path, old_level, new_level, value, threshold, plugin_data
)
alert_state.formatted_message = formatted_msg
if new_level == AlertLevel.OK:
if alert_state.pending_since is not None:
logger.info(
"Alert suppressed (recovered within %.0fs grace): %s on %s",
self.grace_seconds, metric_path, host_name,
)
alert_state.pending_since = None
else:
self._send_notification(host_name, lvl, message, metric_path, old_level, new_level, value)
else:
alert_state.pending_since = time.time()
logger.debug(
"Alert deferred (%.0fs grace): %s on %s = %s",
self.grace_seconds, metric_path, host_name, value,
)
def _check_pending_or_renotify(
self,
host_name: str,
alert_state: AlertState,
metric_path: str,
value: Any,
threshold: ThresholdConfig,
plugin_data: Optional[Dict[str, Any]],
) -> None:
"""Called when alert level is unchanged and non-OK.
If a deferred notification is pending and grace_seconds have elapsed,
fires it now. Otherwise falls through to normal reminder logic.
"""
if alert_state.pending_since is not None:
if time.time() - alert_state.pending_since >= self.grace_seconds:
lvl, message, formatted_msg = self._trigger_notification(
host_name, metric_path, AlertLevel.OK, alert_state.level, value, threshold, plugin_data
)
alert_state.formatted_message = formatted_msg
self._send_notification(
host_name, lvl, message, metric_path, AlertLevel.OK, alert_state.level, value
)
alert_state.pending_since = None
# else: still within grace window, do nothing
else:
self._check_renotify(host_name, alert_state, metric_path, value, threshold, plugin_data)
def _check_renotify( def _check_renotify(
self, self,
host_name: str, host_name: str,
@@ -1037,7 +1154,7 @@ class ThresholdChecker:
threshold: Threshold configuration threshold: Threshold configuration
plugin_data: Optional dictionary of all plugin data fields plugin_data: Optional dictionary of all plugin data fields
""" """
if alert_state.level == AlertLevel.OK: if alert_state.level != AlertLevel.CRITICAL:
return return
# Skip reminders if alert has been acknowledged # Skip reminders if alert has been acknowledged
@@ -1078,9 +1195,15 @@ class ThresholdChecker:
else: else:
message = f"REMINDER ({alert_state.level.name}): {host_name} - {metric_path} = {value} (ongoing for {int(now - alert_state.since)}s)" message = f"REMINDER ({alert_state.level.name}): {host_name} - {metric_path} = {value} (ongoing for {int(now - alert_state.since)}s)"
# Send re-notification using host-specific channels
try: try:
notify_mod.pushmsg_for_host(host_name, message) notify_mod.send_notification(
host_name,
notify_mod.Notification(
title=f"[REMINDER/{alert_state.level.name}] {host_name}",
body=message,
level=alert_state.level.name,
),
)
alert_state.last_notification = now alert_state.last_notification = now
alert_state.notification_count += 1 alert_state.notification_count += 1
logger.info("Re-notification sent: %s", message) logger.info("Re-notification sent: %s", message)
+79 -34
View File
@@ -7,6 +7,8 @@ import time
import zlib import zlib
import logging import logging
from platform import system as platform_system
from ..common.proto import stodict, oldmtodict from ..common.proto import stodict, oldmtodict
from ..common.utils import dur from ..common.utils import dur
from . import notify as notify_mod from . import notify as notify_mod
@@ -16,9 +18,18 @@ eventlog = notify_mod.eventlog
# SO_TIMESTAMP: kernel attaches a struct timeval to each received datagram. # SO_TIMESTAMP: kernel attaches a struct timeval to each received datagram.
# Supported on Linux, FreeBSD, and macOS. The constant is not exposed by # Supported on Linux, FreeBSD, and macOS. The constant is not exposed by
# Python's socket module on all platforms, so fall back to the Linux value (29) # Python's socket module on all platforms
# when absent. platform = platform_system()
_SO_TIMESTAMP = getattr(socket, 'SO_TIMESTAMP', 29) if platform == "Darwin":
_SO_TIMESTAMP = 1024 # SO_TIMESTAMP on macOS (not in Python's socket module)
elif platform == "Linux":
_SO_TIMESTAMP = 29 # Linux value (not in older Python versions)
elif platform == "FreeBSD":
_SO_TIMESTAMP = 32 # FreeBSD value (not in older Python versions)
else:
logger.warning("SO_TIMESTAMP may not be supported on this platform (%s)", platform)
_SO_TIMESTAMP = None
# struct timeval uses two native C longs: tv_sec and tv_usec # struct timeval uses two native C longs: tv_sec and tv_usec
_TIMEVAL = struct.Struct('@ll') _TIMEVAL = struct.Struct('@ll')
@@ -160,7 +171,25 @@ def dicttos(ID, d):
DROPOVERDUE = 7 * 24 * 3600 # seconds before an overdue host becomes UNKNOWN DROPOVERDUE = 7 * 24 * 3600 # seconds before an overdue host becomes UNKNOWN
def _make_timer_callbacks(uname, host, watchhosts, ctx): def _set_connectivity_alert(host, afam, level_name):
"""Update (or clear) a connectivity alert_state entry for a host/address-family.
level_name is "CRITICAL", "WARNING", or "OK". "OK" removes the entry so
that recovered hosts don't clutter the Alerts Dashboard.
"""
from .threshold import AlertState, AlertLevel
metric_path = f"connectivity.{afam}"
level = getattr(AlertLevel, level_name, AlertLevel.OK)
if level == AlertLevel.OK:
host.alert_states.pop(metric_path, None)
return
if metric_path not in host.alert_states:
host.alert_states[metric_path] = AlertState(metric_path)
state = host.alert_states[metric_path]
state.update(level, level_name)
def _make_timer_callbacks(uname, host, ctx):
"""Return (on_overdue, on_unknown) async callbacks for connection timer logic. """Return (on_overdue, on_unknown) async callbacks for connection timer logic.
Captured values are bound at call time so callbacks are safe to use in loops. Captured values are bound at call time so callbacks are safe to use in loops.
@@ -171,6 +200,7 @@ def _make_timer_callbacks(uname, host, watchhosts, ctx):
async def on_unknown(connection): async def on_unknown(connection):
connection.newstate(connection.__class__.UNKNOWN, connection.lastbeat) connection.newstate(connection.__class__.UNKNOWN, connection.lastbeat)
# Keep connectivity alert active when host transitions to unknown
if msg_to_websockets: if msg_to_websockets:
msg_to_websockets("host", host.stateinfo()) msg_to_websockets("host", host.stateinfo())
@@ -180,9 +210,13 @@ def _make_timer_callbacks(uname, host, watchhosts, ctx):
now = time.time() now = time.time()
connection.newstate(connection.__class__.OVERDUE, now, cfg.get("grace", 2)) connection.newstate(connection.__class__.OVERDUE, now, cfg.get("grace", 2))
msg = f"{connection.afam} overdue" msg = f"{connection.afam} overdue"
eventlog(uname, "CRITICAL" if uname in watchhosts else "WARNING", msg) eventlog(uname, "CRITICAL", msg)
if uname in watchhosts: notify_mod.send_notification(
notify_mod.pushmsg_for_host(uname, f"{uname} {msg}") uname,
notify_mod.Notification(title=f"[CRITICAL] {uname}", body=msg, level="CRITICAL"),
)
# Track in alert_states so the Alerts Dashboard shows this
_set_connectivity_alert(host, connection.afam, "CRITICAL")
if threshold_checker: if threshold_checker:
threshold_checker.check_value( threshold_checker.check_value(
host_name=uname, host_name=uname,
@@ -207,8 +241,6 @@ def restore_connection_timers(hbdclass, ctx):
now = time.time() now = time.time()
cfg = ctx.get("config", {}) cfg = ctx.get("config", {})
grace = cfg.get("grace", 2) grace = cfg.get("grace", 2)
from . import config as config_mod
watchhosts = config_mod.get_watchhosts(cfg)
restored = 0 restored = 0
for uname, host in list(hbdclass.Host.hosts.items()): for uname, host in list(hbdclass.Host.hosts.items()):
@@ -218,15 +250,19 @@ def restore_connection_timers(hbdclass, ctx):
if state == hbdclass.Connection.DOWN: if state == hbdclass.Connection.DOWN:
continue continue
on_overdue, on_unknown = _make_timer_callbacks(uname, host, watchhosts, ctx) on_overdue, on_unknown = _make_timer_callbacks(uname, host, ctx)
if state == hbdclass.Connection.UP and interval > 0: if state == hbdclass.Connection.UP and interval > 0:
elapsed = now - conn.lastbeat elapsed = now - conn.lastbeat
remaining = max(1.0, (interval + grace) - elapsed) # Give hosts one full (interval + grace) of extra time on startup
# so hosts that were silent while hbd was down are not immediately
# flagged as overdue before they have a chance to check in.
startup_grace = interval + grace
remaining = max(startup_grace, 2 * startup_grace - elapsed)
conn.reset_overdue_timer(remaining, on_overdue) conn.reset_overdue_timer(remaining, on_overdue)
logger.debug( logger.debug(
"Restored UP timer %s/%s: %.0fs remaining (elapsed %.0fs)", "Restored UP timer %s/%s: %.0fs remaining (elapsed %.0fs, startup grace %.0fs)",
uname, afam, remaining, elapsed, uname, afam, remaining, elapsed, startup_grace,
) )
restored += 1 restored += 1
@@ -307,9 +343,6 @@ def handle_datagram(msg: dict, addr, transport, ctx: dict):
host = hbdcls.Host.hosts[uname] host = hbdcls.Host.hosts[uname]
newh = False newh = False
# Get watchhosts once for use throughout message handling
watchhosts = config_mod.get_watchhosts(cfg)
cid = msg.get("id", 0) cid = msg.get("id", 0)
try: try:
rtt = float(msg.get("rtt")) rtt = float(msg.get("rtt"))
@@ -375,8 +408,10 @@ def handle_datagram(msg: dict, addr, transport, ctx: dict):
if res: if res:
eventlog(uname, "WARNING", res) eventlog(uname, "WARNING", res)
if uname in watchhosts: notify_mod.send_notification(
notify_mod.pushmsg_for_host(uname, "%s %s" % (host.name, res)) uname,
notify_mod.Notification(title=f"[WARNING] {uname}", body=res, level="WARNING"),
)
interval = int(msg.get("interval", 0) or 0) interval = int(msg.get("interval", 0) or 0)
shutdown = msg.get("shutdown", 0) shutdown = msg.get("shutdown", 0)
@@ -386,24 +421,30 @@ def handle_datagram(msg: dict, addr, transport, ctx: dict):
if boot: if boot:
eventlog(uname, "INFO", "booted") eventlog(uname, "INFO", "booted")
if uname in watchhosts: notify_mod.send_notification(
m = "%s booted" % (host.name) uname,
notify_mod.pushmsg_for_host(uname, m) notify_mod.Notification(title=f"[INFO] {uname}", body=f"{host.name} booted", level="INFO"),
)
if message: if message:
eventlog(uname, "INFO", "msg: %s" % message, service=service) eventlog(uname, "INFO", "msg: %s" % message, service=service)
if uname in watchhosts:
notify_mod.pushmsg_for_host(uname, message)
if conn.getstate() != hbdcls.Connection.UP: if conn.getstate() != hbdcls.Connection.UP:
lasts = conn.state lasts = conn.state
d = conn.newstate(hbdcls.Connection.UP, now) d = conn.newstate(hbdcls.Connection.UP, now)
if d == 0 or lasts == "unknown": # Clear connectivity alert now that the host is back up
m = "%s is up" % (conn.afam) _set_connectivity_alert(host, conn.afam, "OK")
else: # Don't log/notify RECOVER for a brand-new host seen for the first time —
m = "%s back after being %s for %s" % (conn.afam, lasts, dur(d)) # it was never down, it just hasn't been seen before.
eventlog(uname, "RECOVER", m) if not newh:
if uname in watchhosts: if d == 0 or lasts == "unknown":
notify_mod.pushmsg_for_host(uname, "%s %s is back" % (uname, conn.afam)) m = "%s is up" % (conn.afam)
else:
m = "%s back after being %s for %s" % (conn.afam, lasts, dur(d))
eventlog(uname, "RECOVER", m)
notify_mod.send_notification(
uname,
notify_mod.Notification(title=f"[RECOVER] {uname}", body=m, level="RECOVER"),
)
if boot or newh: if boot or newh:
host.upcount = host.doesack host.upcount = host.doesack
@@ -411,10 +452,14 @@ def handle_datagram(msg: dict, addr, transport, ctx: dict):
host.upcount += 1 host.upcount += 1
if shutdown: if shutdown:
eventlog(uname, "INFO", "%s shutdown" % conn.afam) m = "%s shutdown" % conn.afam
if uname in watchhosts: eventlog(uname, "INFO", m)
notify_mod.pushmsg_for_host(uname, "%s %s shutdown" % (uname, conn.afam)) notify_mod.send_notification(
uname,
notify_mod.Notification(title=f"[INFO] {uname}", body=m, level="INFO"),
)
conn.newstate(hbdcls.Connection.DOWN, now) conn.newstate(hbdcls.Connection.DOWN, now)
_set_connectivity_alert(host, conn.afam, "CRITICAL")
if interval > 0: if interval > 0:
host.interval = interval host.interval = interval
@@ -424,7 +469,7 @@ def handle_datagram(msg: dict, addr, transport, ctx: dict):
if interval > 0 and conn.getstate() != hbdcls.Connection.DOWN: if interval > 0 and conn.getstate() != hbdcls.Connection.DOWN:
grace = cfg.get("grace", 2) grace = cfg.get("grace", 2)
timeout_seconds = interval + grace timeout_seconds = interval + grace
on_overdue, _ = _make_timer_callbacks(uname, host, watchhosts, ctx) on_overdue, _ = _make_timer_callbacks(uname, host, ctx)
conn.reset_overdue_timer(timeout_seconds, on_overdue) conn.reset_overdue_timer(timeout_seconds, on_overdue)
# Check RTT thresholds using the threshold checker # Check RTT thresholds using the threshold checker
+14
View File
@@ -226,3 +226,17 @@ def _purge_expired_sessions() -> None:
expired = [t for t, s in list(_sessions.items()) if s["expires"] < now] expired = [t for t, s in list(_sessions.items()) if s["expires"] < now]
for t in expired: for t in expired:
del _sessions[t] del _sessions[t]
def save_sessions() -> dict:
"""Return a snapshot of non-expired sessions suitable for pickling."""
_purge_expired_sessions()
return dict(_sessions)
def load_sessions(snapshot: dict) -> None:
"""Restore sessions from a pickled snapshot, dropping any that have expired."""
global _sessions
now = time.time()
_sessions = {t: s for t, s in snapshot.items() if s.get("expires", 0) > now}
logger.debug("Restored %d session(s) from pickle", len(_sessions))
+76 -123
View File
@@ -1,7 +1,8 @@
"""WebSocket server and broadcast helpers for hbd. """WebSocket handler and broadcast helpers for hbd.
Provides an asyncio-based WebSocket server and a thread-safe broadcast WebSocket connections are served through the regular HTTP port via the
function that other threads or synchronous code can call. /ws route registered in http.py (aiohttp WebSocketResponse upgrade).
The separate standalone WebSocket server on ws_port is no longer used.
""" """
import asyncio import asyncio
@@ -10,147 +11,99 @@ import logging
from typing import Callable, Iterable, Optional from typing import Callable, Iterable, Optional
from . import data from . import data
import websockets
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)
_connections = set() _connections: set = set()
_loop: Optional[asyncio.AbstractEventLoop] = None _loop: Optional[asyncio.AbstractEventLoop] = None
_get_hosts: Optional[Callable[[], Iterable]] = None _get_hosts: Optional[Callable[[], Iterable]] = None
#_get_msgs: Optional[Callable[[], Iterable]] = None _verbose: bool = False
_verbose = False
async def _handler(websocket, path=None): def setup(
_connections.add(websocket) loop: asyncio.AbstractEventLoop,
remote_address = websocket.remote_address get_hosts: Optional[Callable[[], Iterable]] = None,
if path is None: verbose: bool = False,
path = getattr(websocket, "path", None)
logger.info("WebSocket connection from %s: %s", remote_address, path)
try:
# send initial hosts
if _get_hosts:
try:
hosts = list(_get_hosts())
logger.debug("Sending %d hosts to new WebSocket client", len(hosts))
for h in hosts:
jmsg = json.dumps({"type": "host", "data": h})
await websocket.send(jmsg)
except Exception as e:
logger.error("Error sending initial hosts: %s", e, exc_info=True)
# send recent messages
if data.msgs:
try:
# msgs = list(_get_msgs())[-100:]
logger.debug("Sending %d recent messages to new WebSocket client", len(data.msgs))
for m in data.msgs:
jmsg = json.dumps({"type": "message", "data": m})
await websocket.send(jmsg)
except Exception as e:
logger.error("Error sending initial messages: %s", e, exc_info=True)
# keep connection open until client disconnects
async for _ in websocket:
# we don't expect meaningful incoming messages besides the initial
# client 'hello' that some clients send; ignore for now
if _verbose:
logger.debug("received ws data: %s", _)
except (
websockets.exceptions.ConnectionClosedOK,
websockets.exceptions.ConnectionClosedError,
) as e:
logger.info("WebSocket closed from %s: %r", remote_address, e)
except Exception as e:
logger.exception("WebSocket handler exception from %s: %s", remote_address, e)
finally:
logger.debug("Removing WebSocket connection from %s", remote_address)
_connections.discard(websocket)
async def start(
host: str,
ws_port: int,
wss_port: Optional[int] = None,
ssl_context=None,
get_hosts: Optional[Callable] = None,
# get_msgs: Optional[Callable] = None,
config: dict = {},
): ):
"""Start WebSocket servers and block until cancelled. """Register the running loop and initial-state callback.
This is intended to be awaited inside the main asyncio event loop. Call this once from _run_async before starting the HTTP server.
If `wss_port` and `ssl_context` are provided, a WSS server will also be
started.
""" """
global _loop, _get_hosts, _verbose global _loop, _get_hosts, _verbose
_loop = asyncio.get_running_loop() _loop = loop
_get_hosts = get_hosts _get_hosts = get_hosts
_verbose = config.get("verbose", False), _verbose = verbose
_debug = config.get("debug", 0),
# Start servers and keep the server objects for clean shutdown
running_servers = []
ws_server = await websockets.serve(_handler, host, ws_port)
running_servers.append(ws_server)
if wss_port and ssl_context:
wss_server = await websockets.serve(_handler, host, wss_port, ssl=ssl_context)
running_servers.append(wss_server)
logger.info( async def handler(request):
"WebSocket server(s) started on port %s (wss %s)", ws_port, wss_port """aiohttp WebSocket upgrade handler — register as GET /ws."""
) from aiohttp import web
ws = web.WebSocketResponse()
await ws.prepare(request)
_connections.add(ws)
remote = request.remote
logger.info("WebSocket connected from %s", remote)
try: try:
# Block until cancelled # Send current host state to the new client
await asyncio.Future() if _get_hosts:
except asyncio.CancelledError: try:
pass for h in list(_get_hosts()):
await ws.send_str(json.dumps({"type": "host", "data": h}))
except Exception as e:
logger.error("Error sending initial hosts: %s", e)
# Send recent messages
if data.msgs:
try:
for m in data.msgs:
await ws.send_str(json.dumps({"type": "message", "data": m}))
except Exception as e:
logger.error("Error sending initial messages: %s", e)
# Keep connection open, ignore incoming frames
async for msg in ws:
from aiohttp import WSMsgType
if msg.type == WSMsgType.TEXT:
if _verbose:
logger.debug("ws recv from %s: %s", remote, msg.data)
elif msg.type in (WSMsgType.ERROR, WSMsgType.CLOSE):
break
except Exception as e:
logger.exception("WebSocket handler error from %s: %s", remote, e)
finally: finally:
# Close all active browser connections so their handler coroutines exit _connections.discard(ws)
active = list(_connections) logger.info("WebSocket disconnected from %s", remote)
if active:
logger.info("Closing %d active WebSocket connection(s)...", len(active)) return ws
await asyncio.gather(
*[ws.close() for ws in active],
return_exceptions=True,
)
# Stop the listening servers and wait for all handlers to finish
for srv in running_servers:
srv.close()
await asyncio.gather(
*[srv.wait_closed() for srv in running_servers],
return_exceptions=True,
)
logger.info("WebSocket server(s) stopped")
def broadcast(typ: str, data) -> bool: def broadcast(typ: str, payload) -> bool:
"""Thread-safe broadcast helper. """Thread-safe broadcast to all connected WebSocket clients.
Schedules coroutine(s) on the running loop to send message to all Can be called from any thread; schedules sends on the event loop.
connected websockets. Returns False if server was not running. Returns False if the loop is not running yet.
""" """
if not _loop: if not _loop:
return False return False
jmsg = json.dumps({"type": typ, "data": data}) jmsg = json.dumps({"type": typ, "data": payload})
to_close = []
for ws in list(_connections): async def _send_all():
if ws.state != websockets.protocol.State.OPEN: dead = set()
to_close.append(ws) for ws in list(_connections):
continue try:
try: if not ws.closed:
asyncio.run_coroutine_threadsafe(ws.send(jmsg), _loop) await ws.send_str(jmsg)
except Exception: else:
to_close.append(ws) dead.add(ws)
logger.debug("ws.send exception: closed") except Exception:
for ws in to_close: dead.add(ws)
try: for ws in dead:
asyncio.run_coroutine_threadsafe(ws.wait_closed(), _loop) _connections.discard(ws)
except Exception:
pass asyncio.run_coroutine_threadsafe(_send_all(), _loop)
if ws in _connections:
_connections.remove(ws)
return True return True
+2 -1
View File
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
[project] [project]
name = "hbd" name = "hbd"
version = "5.0.12" version = "5.1.2"
description = "Heartbeat monitoring system — client (hbc) and server (hbd)" description = "Heartbeat monitoring system — client (hbc) and server (hbd)"
readme = "README.md" readme = "README.md"
requires-python = ">=3.11" requires-python = ">=3.11"
@@ -31,6 +31,7 @@ server = [
"mattermostdriver>=7.3.0", "mattermostdriver>=7.3.0",
"aiohttp>=3.11", "aiohttp>=3.11",
"Jinja2>=3.1.6", "Jinja2>=3.1.6",
"matrix-nio>=0.24",
] ]
# Install both client and server # Install both client and server
+74 -11
View File
@@ -1,7 +1,7 @@
#!/bin/sh #!/bin/sh
# install the heartbeat tools. By default, this will install the hbc # install the heartbeat client, hbc. The server is installed when the arg 'server' is passed
# client only. The server is installed when the arg 'server' is passed # install the heartbeat client, hbc. The server is installed when the arg 'server' is passed
# to the script. The script will install the heartbeat tools in a python # to the script. The script will install the heartbeat tools in a python
# virtual environment in ~/venvs/hbd. The hbd and hbc commands will be # virtual environment in ~/venvs/hbd. The hbd and hbc commands will be
# installed from the wheel and symlinked to ~/bin/hbd and ~/bin/hbc, # installed from the wheel and symlinked to ~/bin/hbd and ~/bin/hbc,
@@ -13,13 +13,76 @@
# hbd/hbc from wheel and create symlinks for hbd and hbc in ~/bin # hbd/hbc from wheel and create symlinks for hbd and hbc in ~/bin
set -e set -e
if [ ! -d ~/venvs/hbd ]; then what=$1
mkdir -p ~/venvs on_ha=0
python3 -m venv ~/venvs/hbd --system-site-packages [ -z "$what" ] && what="client"
if [ -d /homeassistant ]; then
echo "cannot install in HA, run \"docker exec -it homeassistant $0 $@\""
exit 1
fi
if [ -d /config ]; then
echo "Installing on HA"
where="/config/bin"
venv="/config/venvs"
on_ha=1
else
if [ ! -d $HOME/.local/bin ] && [ ! -d $HOME/bin ]; then
echo "No suitable bin directory found in PATH, please add either $HOME/.local/bin or $HOME/bin to your PATH"
exit 1
fi
for where in $HOME/bin $HOME/.local/bin notset ; do
if echo ":$PATH:" | grep -q ":$where:" ; then
break
fi
done
if [ "$where" = "notset" ]; then
echo "No suitable bin directory found in PATH, please add either $HOME/.local/bin or $HOME/bin to your PATH"
exit 1
fi
venv="$HOME/venvs"
fi
echo "Installing heartbeat $what"
if [ ! -d $venv/hbd ]; then
python3 -m pip --version > /dev/null 2>&1
if [ $? -ne 0 ]; then
# truenas does not have pip installed by default, so we need to fetch get-pip.py and install pip
echo "pip is not installed, fetching get-pip.py and installing pip"
arg="--without-pip"
fi
mkdir -p $venv
have_venv=$(python3 -c "import venv" &> /dev/null && echo "Installed" || echo "Not Installed")
if [ "$have_venv" = "Not Installed" ]; then
echo "python venv module not found, installing virtualenv"
python3 -m pip install --user virtualenv
python3 -m virtualenv $venv/hbd --system-site-packages $arg
else
python3 -m venv $venv/hbd --system-site-packages $arg
fi
. $venv/hbd/bin/activate
if [ -n "$arg" ]; then
curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py && python3 get-pip.py
fi
deactivate
fi
. $venv/hbd/bin/activate
python3 -mpip install --upgrade --index-url https://git.wrede.ca/api/packages/andreas/pypi/simple/ --extra-index-url https://pypi.org/simple hbd[$what]
if [ "$what" = "server" ]; then
rm -f $where/hbd
ln -sf $(which hbd) $where/hbd
echo "hbd installed, you can run it with \"$where/hbd\" or \"hbd\" if $where is in your PATH"
else
rm -f $where/hbc
ln -sf $(which hbc) $where/hbc
if [ $on_ha -eq 1 ]; then
echo "restarting hbc "
job=$(grep run_hbc configuration.yaml | sed 's/run_hbc://')
$job
else
echo "hbc installed, you can run it with \"$where/hbc\" or \"hbc\" if $where is in your PATH"
fi
fi fi
. ~/venvs/hbd/bin/activate
pip install 'git+ssh://git@git.wrede.ca/andreas/heartbeat.git'
rm -f ~/bin/hbd
rm -f ~/bin/hbc
ln -sf $(which hbd) ~/bin/hbd
ln -sf $(which hbc) ~/bin/hbc