andreas/heartbeat
Public Access
1
0
Fork 1
Code Issues Pull Requests Actions Packages Projects Releases 44 Wiki Activity

Major refactoring of the codebase, including restructuring of files and directories, renaming of modules and classes, and improvements to the overall organization and readability of the code. This refactoring aims to enhance maintainability, scalability, and clarity of the codebase while preserving existing functionality. The changes include:

Browse Source 
- Restructuring of the project directory into client and server components
- Renaming of modules and classes to better reflect their purpose and functionality
- Moving common utilities and configurations to a shared location
- Updating import statements to reflect the new structure
- Adding new documentation files for better clarity on various aspects of the project
- Removing deprecated or unused code to streamline the codebase
- Ensuring that all existing functionality is preserved and that the codebase remains functional after the refactoring.
This commit is contained in:
Andreas Wrede
2026-03-29 11:13:40 -04:00
parent 7e2038ecac
commit 0543266c92
65 changed files with 11371 additions and 140 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.hb.yaml
+29 -6
View File
@@ -2,7 +2,8 @@
hb_port: 50003
hbd_host: ''
#logfile: "/home/andreas/public_html/messages/andreas"
logfile: "/Users/andreas/public_html/messages/andreas"
logfile: "/home/andreas/logs/heartbeat/andreas"
#logfile: "/Users/andreas/public_html/messages/andreas"
logfmt: "msg"
grace: 40
interval: 10
@@ -35,10 +36,32 @@ smtppassword: "r8psra6wj6gcakkp"
smtpport: 587
ws_port: 50005
wss_port: 50006
cert_path: "/usr/local/etc/letsencrypt/live/hbd.wrede.ca/"
cert_path: "ssl/"
# wss_port: 50006 # Commented out - use plain WebSocket instead of secure WSS
# cert_path: "/usr/local/etc/letsencrypt/live/hbd.wrede.ca/"
# cert_path: "test/"
# CERT_PATH = "./test/"
wss_pem: "fullchain.pem"
wss_key: "privkey.pem"
# wss_pem: "fullchain.pem"
# wss_key: "privkey.pem"
journal_enabled: true # Enable/disable journaling
journal_dir: /home/andreas/logs/heartbeat # Journal directory
journal_file: messages.journal # Base filename
journal_max_size: 104857600 # Max size (100MB default)
journal_max_backups: 10 # Number of backups to keep
thresholds:
cpu_monitor:
cpu_percent:
warning: 1.0
critical: 90.0
memory_monitor:
percent:
warning: 3.0
critical: 95.0
disk_monitor:
partitions:
/:
percent:
warning: 8.0
critical: 90.0
.hb.yaml.swp
BIN
View File
Binary file not shown.
.vscode/launch.json
Vendored
+3 -3
View File
@@ -8,8 +8,8 @@
"name": "Python: Run hbd (module)",
"type": "debugpy",
"request": "launch",
"module": "hbd.cli",
"args": ["-c", ".hb.yaml", "-f", "-v", "-x", "-x", "-x"],
"module": "hbd.server.cli",
"args": ["-c", "/home/andreas/git/heartbeat/.hb.yaml", "-f", "-v", "-x", "-x", "-x"],
"cwd": "${workspaceFolder}",
"env": {
"PYTHONPATH": "${workspaceFolder}"
@@ -32,7 +32,7 @@
"type": "debugpy",
"request": "launch",
"module": "debugpy",
"args": ["--listen", "5678", "--wait-for-client", "-m", "hbd.cli", "-c", ".hb.yaml", "-f", "-v"],
"args": ["--listen", "5678", "--wait-for-client", "-m", "hbd.server.cli", "-c", ".hb.yaml", "-f", "-v"],
"env": { "PYTHONPATH": "${workspaceFolder}" },
"console": "integratedTerminal",
"justMyCode": false
.vscode/settings.json
Vendored
+4 -1
View File
@@ -2,5 +2,8 @@
"python.pythonPath": "/usr/bin/python3",
"python.linting.enabled": true,
"python.formatting.provider": "black",
"python.linting.flake8Enabled": true
"python.linting.flake8Enabled": true,
"chat.tools.terminal.autoApprove": {
"mv": true
}
}
README.md
+329 -1
View File
@@ -11,10 +11,294 @@ A lightweight daemon that listens for UDP heartbeat messages and acts on them: k
- Queue DNS updates via `nsupdate` and run them in a background thread ✅
- WebSocket API for live updates (hosts & messages) ✅
- Notification pipeline (email, Pushover, Mattermost, Signal) ✅
- **HTTP API & Web UI** ✅
- REST API for plugin data, alerts, and host information
- Live dashboard with WebSocket updates
- Interactive plugin metrics visualization
- Alerts dashboard with filtering and summaries
- **Message journal with automatic log rotation** ✅
- Logs all received messages in JSON format
- Size-based automatic rotation
- Configurable retention and backup management
- **Plugin system for extensible monitoring** ✅
- Collect system metrics (CPU, memory, disk, network)
- Execute existing Nagios monitoring plugins
- Create custom plugins with simple Python classes
- **Threshold alerting system** ✅
- Monitor metrics against configurable WARNING/CRITICAL thresholds
- Hysteresis to prevent alert flapping
- Automatic notifications on state changes
- Re-notification for ongoing alerts
- Modular codebase suitable for unit testing and CI ✅
---
## 🔌 Plugin System
Heartbeat includes a comprehensive plugin architecture that extends monitoring beyond simple heartbeats. The plugin system allows you to:
- **Collect system information**: OS details, hardware info, system configuration
- **Monitor resources**: CPU usage, memory, disk space, network statistics
- **Run Nagios plugins**: Execute thousands of existing Nagios monitoring plugins without modification
- **Create custom plugins**: Build your own monitoring logic with simple Python classes
### Plugin Types
- **InfoPlugin**: Collects static information once (e.g., OS version, hardware specs)
- **MonitorPlugin**: Collects metrics periodically (e.g., CPU usage every 30 seconds)
### Built-in Plugins
- `os_info`: Collects OS, kernel, distribution, and architecture information
- `cpu_monitor`: Monitors CPU usage, load average, frequency, and process counts
- `memory_monitor`: Monitors RAM and swap usage, available memory
- `disk_monitor`: Monitors disk usage, I/O statistics, and filesystem metrics
- `network_monitor`: Monitors network interface statistics, bandwidth, and connections
- `filesystem_info`: Collects mounted filesystem information (physical filesystems only by default)
- `nagios_runner`: Executes Nagios monitoring plugins (check_disk, check_load, check_http, etc.)
### Nagios Integration
The `nagios_runner` plugin provides seamless integration with the vast Nagios plugin ecosystem. You can run any Nagios-compatible plugin and have the results automatically parsed and stored:
- Executes plugins via subprocess with timeout protection
- Parses exit codes (OK/WARNING/CRITICAL/UNKNOWN)
- Extracts performance data with thresholds
- Reports aggregated status across all configured checks
See [docs/NAGIOS_INTEGRATION.md](docs/NAGIOS_INTEGRATION.md) for complete integration guide including configuration examples and custom plugin development.
### Creating Custom Plugins
```python
from hbd.plugin import MonitorPlugin
class DiskMonitorPlugin(MonitorPlugin):
name = "disk_monitor"
interval = 60 # Run every 60 seconds
async def collect(self):
return {
"disk_usage": get_disk_usage(),
"timestamp": time.time()
}
```
Place plugins in `hbd/plugins/` and they'll be automatically discovered and loaded by the client.
---
## 📝 Message Journal
Heartbeat includes a message journal that logs all received messages with automatic rotation.
### Features
- **JSON Format**: All messages logged in JSONL (JSON Lines) format for easy parsing
- **Automatic Rotation**: Size-based rotation with configurable thresholds
- **Backup Management**: Keeps configurable number of rotated log files
- **Non-blocking**: Async logging with minimal performance impact
### Configuration
```yaml
# Message journal settings
journal_enabled: true # Enable/disable journaling
journal_dir: /var/log/heartbeat # Journal directory
journal_file: messages.journal # Base filename
journal_max_size: 104857600 # Max size (100MB default)
journal_max_backups: 10 # Number of backups to keep
```
### Example Journal Entry
```json
{"timestamp":1711234567.123,"datetime":"2026-03-28T12:34:56","source_ip":"192.168.1.100","source_port":50003,"message":{"ID":"HTB","name":"webserver1","interval":30}}
```
### Analyzing Journal Files
```bash
# View recent messages
tail -100 /var/log/heartbeat/messages.journal | jq .
# Count messages by type
cat /var/log/heartbeat/messages.journal | jq -r '.message.ID' | sort | uniq -c
# Filter by hostname
cat /var/log/heartbeat/messages.journal | jq 'select(.message.name == "webserver1")'
```
See [docs/MESSAGE_JOURNAL.md](docs/MESSAGE_JOURNAL.md) for complete documentation including rotation behavior, integration with log management systems, and analysis examples.
---
## 🚨 Threshold Alerting
Heartbeat includes a sophisticated threshold alerting system that monitors plugin metrics and triggers notifications when values exceed configured limits.
### Features
- **Multi-level alerts**: WARNING and CRITICAL severity levels
- **Flexible operators**: Support for >, >=, <, <=, ==, != comparisons
- **Hysteresis**: Prevents alert flapping with configurable recovery thresholds
- **Smart notifications**: Alerts only on state changes, not every check
- **Re-notifications**: Periodic reminders for ongoing alerts
- **Journal integration**: All threshold events logged for audit trail
### Configuration
```yaml
thresholds:
cpu_monitor:
cpu_percent:
warning: 80.0 # Warn when CPU > 80%
critical: 90.0 # Critical when CPU > 90%
operator: ">"
hysteresis: 0.1 # 10% hysteresis to prevent flapping
memory_monitor:
percent:
warning: 85.0
critical: 95.0
disk_monitor:
partitions:
/:
percent:
warning: 80.0
critical: 90.0
free_gb:
warning: 10.0 # Alert when < 10GB free
critical: 5.0
operator: "<" # Inverse threshold
# Global settings
threshold_renotify_interval: 3600 # Re-notify every hour for ongoing alerts
```
### Alert Behavior
1. **State Changes**: Notifications sent when crossing thresholds
- OK → WARNING: Early notification
- WARNING → CRITICAL: Escalation
- CRITICAL → OK: Recovery
2. **Hysteresis**: Prevents rapid state transitions
```
Critical threshold: 90%
Hysteresis: 10%
Recovery threshold: 81% (90 - 10% of 90)
Value 91% → CRITICAL (threshold crossed)
Value 85% → CRITICAL (still above 81%)
Value 79% → OK (below recovery threshold)
```
3. **Re-notifications**: Periodic reminders for ongoing alerts
- Default: Every 60 minutes
- Configurable via `threshold_renotify_interval`
### Example Notifications
```
WARNING: webserver01 - cpu_monitor.cpu_percent = 85.0
CRITICAL: webserver01 - memory_monitor.percent = 96.0
RECOVERED: database01 - disk_monitor./.percent = 75.0 (WARNING -> OK)
REMINDER (CRITICAL): mailserver - cpu_monitor.load_1min = 12.5 (ongoing for 3600s)
```
### Supported Metrics
All plugin metrics can be thresholded:
- **CPU**: cpu_percent, load_1min, load_5min, load_15min
- **Memory**: percent, available_mb, swap_percent
- **Disk**: Per-partition percent, free_gb, free_mb
- **Network**: errors_total, dropped packets, connection counts
- **Nagios**: exit_code mapping (0=OK, 1=WARNING, 2=CRITICAL)
See [docs/THRESHOLD_ALERTING.md](docs/THRESHOLD_ALERTING.md) for comprehensive documentation including best practices, troubleshooting, and advanced configuration.
---
## 🌐 HTTP API & Web UI
Heartbeat includes a built-in HTTP/WebSocket server that provides both a REST API and web-based dashboards for monitoring and visualization.
### Features
- **REST API**: JSON endpoints for accessing plugin data, alerts, and host information
- **Live Dashboard**: Real-time WebSocket-powered host status view
- **Plugin Metrics**: Interactive visualization of all plugin data with auto-refresh
- **Alerts Dashboard**: Comprehensive alert monitoring with filtering and summaries
- **CORS Support**: Configurable for integration with external applications
### Web Dashboards
- **Live View** (`/live`): Real-time host connectivity, latency, and messages
- **Plugin Metrics** (`/plugins`): Browse and visualize metrics from all plugins
- **Alerts Dashboard** (`/alerts`): Monitor active alerts with severity filtering
### API Endpoints
```bash
# List all monitored hosts
curl http://localhost:50004/api/0/hosts
# Get all plugin data for a host
curl http://localhost:50004/api/0/hosts/webserver01/plugins
# Get detailed plugin history (last 50 samples)
curl http://localhost:50004/api/0/hosts/webserver01/plugins/cpu_monitor?limit=50
# Get alert states for a specific host
curl http://localhost:50004/api/0/hosts/webserver01/alerts
# Get all active alerts across all hosts
curl http://localhost:50004/api/0/alerts
```
### Integration Examples
**Python Client:**
```python
import requests
# Monitor for critical alerts
response = requests.get('http://localhost:50004/api/0/alerts')
alerts = response.json()
if alerts['summary']['critical'] > 0:
print(f"⚠️ {alerts['summary']['critical']} CRITICAL alerts!")
for alert in alerts['alerts']:
if alert['level'] == 'CRITICAL':
print(f" {alert['hostname']}: {alert['metric_path']} = {alert['last_value']}")
```
**Bash Monitoring Script:**
```bash
#!/bin/bash
# Check for critical alerts
CRITICAL=$(curl -s http://localhost:50004/api/0/alerts | jq '.summary.critical')
if [ "$CRITICAL" -gt 0 ]; then
echo "CRITICAL: $CRITICAL critical alerts detected!"
# Send notification
fi
```
### Demo & Testing
Run the API demo script to test all endpoints:
```bash
python3 scripts/demo_http_api.py
```
See [docs/HTTP_API.md](docs/HTTP_API.md) for complete API documentation including response formats, error handling, and integration examples.
---
## ⚙️ Quickstart
Prerequisites:
@@ -43,6 +327,46 @@ You can also run it directly via the package entrypoint after installation:
python -m hbd.cli -c /path/to/config.yaml
```
### Running the Client
The heartbeat client (`hbc`) sends periodic heartbeats and plugin data to the server:
```bash
# Basic usage pointing to server
python -m hbd.hbc --server your-server.example.com
# With custom configuration
python -m hbd.hbc --server 192.168.1.100 --port 50003 --interval 30
# Run with specific plugins enabled/disabled
python -m hbd.hbc --server hbd.local --disable-plugin os_info
```
Client configuration can also be specified in YAML:
```yaml
server: hbd.example.com
port: 50003
interval: 30
plugins:
cpu_monitor:
interval: 300 # Check every 5 minutes (default)
per_core: true
memory_monitor:
interval: 300 # Check every 5 minutes (default)
disk_monitor:
interval: 300 # Check every 5 minutes (default)
network_monitor:
interval: 300 # Check every 5 minutes (default)
nagios_runner:
interval: 300 # Check every 5 minutes (default)
commands:
- /usr/lib/nagios/plugins/check_load -w 5,4,3 -c 10,8,6
- /usr/lib/nagios/plugins/check_disk -w 20% -c 10% -p /
```
All monitoring plugins default to 5-minute (300 second) intervals, but can be customized as needed.
## 🐞 Debugging in VS Code
This repository includes a ready-to-use `.vscode/launch.json` with configurations to run or attach the VS Code debugger to `hbd`.
@@ -102,7 +426,7 @@ pushsrv: pushover
## 🔧 Architecture & Modules
- `hbd.proto` — serialization/deserialization of heartbeat messages (supports compressed payloads)
- `hbd.proto` — serialization/deserialization of heartbeat messages (supports compressed payloads and plugin data)
- `hbd.udp` — UDP parsing and `handle_datagram` implementation (main state machine)
- `hbd.dns` — `create_nsupdate_payload`, `nsupdate`, and an asyncio DNS worker (`start_dns_worker`).
The DNS worker now runs as an `asyncio` task and the package exposes a
@@ -112,6 +436,10 @@ pushsrv: pushover
- `hbd.notify` — email and push notification helpers
- `hbd.ws` — WebSocket server and thread-safe broadcast helpers
- `hbd.http` — HTTP handler factory for the status UI/API
- `hbd.journal` — message journal with size-based log rotation and backup management
- `hbd.plugin` — plugin framework with base classes, registry, and dynamic loader
- `hbd.plugins/` — built-in plugins (os_info, cpu_monitor, memory_monitor, disk_monitor, network_monitor, filesystem_info, nagios_runner)
- `hbd.hbc` — heartbeat client that sends heartbeats and plugin data to server
- `hbd.utils` — small utility helpers (`shortname`, `dur`, `initlog`)
- `hbd.cli` — CLI entrypoint and argument parsing
- `hbd.server` — async orchestration to run UDP/HTTP/WSS components
REFACTORING.md
+234
View File
@@ -0,0 +1,234 @@
# HBD/HBC Separation Refactoring
## Overview
The heartbeat monitoring system has been refactored into a modular package structure with separate client and server components. This allows users to install only what they need and provides clear separation of concerns.
## New Package Structure
```
hbd/
├── __init__.py # Main package (minimal)
├── client/ # HBC - System monitoring client
│ ├── __init__.py
│ ├── main.py # Entry point (was hbc.py)
│ ├── config.py # Client-specific configuration
│ ├── plugin.py # Plugin framework
│ ├── threshold.py # Threshold checking
│ └── plugins/ # Monitoring plugins
│ ├── cpu_monitor.py
│ ├── disk_monitor.py
│ ├── memory_monitor.py
│ ├── network_monitor.py
│ ├── filesystem_info.py
│ ├── os_info.py
│ └── nagios_runner.py
├── server/ # HBD - Heartbeat daemon/server
│ ├── __init__.py
│ ├── main.py # Server runtime (was server.py)
│ ├── cli.py # Command-line interface
│ ├── config.py # Server-specific configuration
│ ├── http.py # HTTP/REST API
│ ├── ws.py # WebSocket server
│ ├── udp.py # UDP heartbeat listener
│ ├── dns.py # DNS update functionality
│ ├── notify.py # Notification handlers
│ ├── monitor.py # Host monitoring
│ ├── hbdclass.py # Host class definitions
│ ├── journal.py # Message journaling
│ ├── templates/ # Jinja2 web templates
│ └── static/ # Web UI assets
└── common/ # Shared utilities
├── __init__.py
├── proto.py # Protocol encoding/decoding
└── utils.py # Common utilities
## Configuration Files
### Client Configuration (hbd/client/config.py)
Client-specific defaults:
- `hb_port`: Port where hbd servers listen (default: 50003)
- `interval`: Heartbeat interval in seconds (default: 10)
- `plugins`: Per-plugin configuration
- `thresholds`: Threshold configuration for monitoring
### Server Configuration (hbd/server/config.py)
Server-specific defaults:
- `hb_port`: Port to listen for heartbeats (default: 50003)
- `hbd_port`: HTTP API port (default: 50004)
- `ws_port`: WebSocket port (default: 50005)
- `logfile`, `logfmt`: Logging configuration
- `pushsrv`, `pushover_token`, etc.: Notification settings
- `watchhosts`, `dyndnshosts`: Host monitoring
- `smtpserver`, etc.: Email settings
- `journal_*`: Message journaling settings
## Installation Options
### Install Core Only (minimal, PyYAML only)
```bash
pip install hbd
```
### Install Client Only (for monitoring)
```bash
pip install hbd[client]
# Installs: PyYAML, psutil
```
### Install Server Only (for daemon)
```bash
pip install hbd[server]
# Installs: PyYAML, websockets, mattermostdriver, aiohttp, Jinja2
```
### Install Everything
```bash
pip install hbd[all]
# Installs all dependencies for both client and server
```
### Development Installation
```bash
pip install -e ".[dev]"
# Includes all dependencies plus testing/linting tools
```
## Command-Line Interfaces
### HBC (Client)
```bash
hbc [options] host1 [host2 ...]
# Entry point: hbd.client.main:main
# Location: hbd/client/main.py
```
### HBD (Server)
```bash
hbd [options]
# Entry point: hbd.server.cli:main
# Location: hbd/server/cli.py → hbd/server/main.py
```
## Import Changes
### Client Code
```python
# Old imports
from .config import load_config
from .proto import dicttos, stodict
from .plugin import PluginRegistry
# New imports
from .config import load_config # Still in client/
from ..common.proto import dicttos # Moved to common/
from .plugin import PluginRegistry # Still in client/
```
### Server Code
```python
# Old imports
from .config import load_config
from .proto import stodict
from .threshold import AlertLevel
# New imports
from .config import load_config # Server-specific config
from ..common.proto import stodict # Moved to common/
from ..client.threshold import AlertLevel # Client module
```
### Plugin Code
```python
# Old import
from hbd.plugin import MonitorPlugin
# New import
from hbd.client.plugin import MonitorPlugin
```
## Benefits
1. **Modular Installation**: Install only what you need
- Client-only systems don't need web server dependencies
- Server-only systems don't need psutil
2. **Clearer Architecture**: Explicit separation of concerns
- Client: System monitoring and data collection
- Server: Heartbeat reception, web UI, notifications
- Common: Shared protocol and utilities
3. **Independent Evolution**: Client and server can evolve separately
- Different release cycles possible
- Clear API boundaries via common/
4. **Smaller Footprint**: Reduced dependency installation
- Client: ~1 dependency (psutil)
- Server: ~4 dependencies (websockets, aiohttp, Jinja2, mattermostdriver)
## Migration Guide
### For Existing Installations
1. **Reinstall the package**:
```bash
pip install -e ".[all]" # For development
# or
pip install hbd[all] # For production
```
2. **Configuration files remain unchanged**:
- Both client and server read from `~/.hb.yaml`
- All existing config keys are supported in both configs
- Server has additional keys (journal, websocket, email, etc.)
- Client has minimal keys (interval, plugins, thresholds)
3. **Commands remain the same**:
- `hbc` command works identically
- `hbd` command works identically
### For New Deployments
1. **Client-only system** (monitoring host):
```bash
pip install hbd[client]
hbc server1.example.com server2.example.com
```
2. **Server-only system** (monitoring daemon):
```bash
pip install hbd[server]
hbd -c /etc/hbd.yaml -f
```
3. **Combined system** (dev/test):
```bash
pip install hbd[all]
```
## Testing
All imports and entry points have been tested and validated:
- ✅ Package imports work correctly
- ✅ `hbc` command entry point functional
- ✅ `hbd` command entry point functional
- ✅ Optional dependencies properly configured
- ✅ All internal imports updated
## Files Archived
The following files were renamed to avoid conflicts:
- `hbd/config.py` → `hbd/config.py.old` (split into client/server configs)
- `hbd/hbc_old.py` → `hbd/hbc_old.py.bak` (backup file)
## Next Steps
1. Test client functionality with a monitoring host
2. Test server functionality with web UI and notifications
3. Update documentation (README.md) with new structure
4. Consider publishing to PyPI with new structure
5. Update any deployment scripts/Dockerfiles to use optional dependencies
demo_threshold.py
+320
View File
@@ -0,0 +1,320 @@
#!/usr/bin/env python3
"""
Demonstration of the threshold alerting system.
This script shows how thresholds work by simulating plugin data
with values that cross various threshold boundaries.
"""
import sys
import time
from pathlib import Path
sys.path.insert(0, str(Path(__file__).parent.parent))
from hbd.threshold import ThresholdChecker, AlertLevel
def demo_basic_thresholds():
"""Demonstrate basic threshold checking."""
print("=" * 70)
print("DEMO 1: Basic Threshold Checking")
print("=" * 70)
config = {
"thresholds": {
"cpu_monitor": {
"cpu_percent": {
"warning": 80.0,
"critical": 90.0,
"operator": ">",
"hysteresis": 0.1,
}
}
}
}
notifications = []
def notifier(msg):
notifications.append(msg)
print(f" 📧 NOTIFICATION: {msg}")
checker = ThresholdChecker(config, notification_callback=notifier)
alert_states = {}
# Simulate CPU values over time
test_values = [
(50.0, "Normal operation"),
(85.0, "Crosses WARNING threshold"),
(87.0, "Still in WARNING"),
(95.0, "Escalates to CRITICAL"),
(92.0, "Still CRITICAL (in hysteresis)"),
(85.0, "Still CRITICAL (above recovery threshold of 81)"),
(79.0, "Recovers to OK"),
(50.0, "Back to normal"),
]
print("\nSimulating CPU usage over time:")
print("-" * 70)
for value, description in test_values:
print(f"\n📊 CPU: {value}% - {description}")
plugin_data = {"cpu_percent": value}
state_changes = checker.check_plugin_data(
host_name="testhost",
plugin_name="cpu_monitor",
data=plugin_data,
alert_states=alert_states,
)
current_state = alert_states.get("cpu_monitor.cpu_percent")
if current_state:
print(f" Current state: {current_state.level.name}")
if state_changes:
for metric, old_level, new_level, val in state_changes:
print(f" ⚠️ State change: {old_level.name} → {new_level.name}")
print(f"\n📈 Summary: {len(notifications)} notifications sent")
print("=" * 70)
def demo_multiple_metrics():
"""Demonstrate monitoring multiple metrics."""
print("\n\n" + "=" * 70)
print("DEMO 2: Multiple Metrics and Alert Summary")
print("=" * 70)
config = {
"thresholds": {
"cpu_monitor": {
"cpu_percent": {"warning": 80.0, "critical": 90.0},
"load_1min": {"warning": 4.0, "critical": 8.0},
},
"memory_monitor": {
"percent": {"warning": 85.0, "critical": 95.0},
"available_mb": {
"warning": 1000,
"critical": 500,
"operator": "<",
},
},
}
}
notifications = []
checker = ThresholdChecker(config, notification_callback=lambda m: notifications.append(m))
alert_states = {}
# Simulate problematic system state
print("\nSimulating a system under load:")
print("-" * 70)
scenarios = [
{
"name": "Initial state - all OK",
"cpu_monitor": {"cpu_percent": 50.0, "load_1min": 2.0},
"memory_monitor": {"percent": 60.0, "available_mb": 2000},
},
{
"name": "CPU spikes to WARNING",
"cpu_monitor": {"cpu_percent": 85.0, "load_1min": 2.0},
"memory_monitor": {"percent": 60.0, "available_mb": 2000},
},
{
"name": "Memory also reaches WARNING",
"cpu_monitor": {"cpu_percent": 85.0, "load_1min": 2.0},
"memory_monitor": {"percent": 88.0, "available_mb": 800},
},
{
"name": "CPU escalates to CRITICAL",
"cpu_monitor": {"cpu_percent": 95.0, "load_1min": 5.0},
"memory_monitor": {"percent": 88.0, "available_mb": 800},
},
{
"name": "System recovering",
"cpu_monitor": {"cpu_percent": 70.0, "load_1min": 2.0},
"memory_monitor": {"percent": 65.0, "available_mb": 1500},
},
]
for scenario in scenarios:
print(f"\n📍 {scenario['name']}")
# Check CPU metrics
checker.check_plugin_data(
"testhost",
"cpu_monitor",
scenario["cpu_monitor"],
alert_states
)
# Check memory metrics
checker.check_plugin_data(
"testhost",
"memory_monitor",
scenario["memory_monitor"],
alert_states
)
# Show alert summary
summary = checker.get_alert_summary(alert_states)
print(f" Alerts: OK={summary['ok']}, WARNING={summary['warning']}, CRITICAL={summary['critical']}")
# Show active alerts
active = checker.get_active_alerts(alert_states)
if active:
print(f" Active alerts:")
for alert in active:
print(f" - {alert.metric_path}: {alert.level.name} (value={alert.last_value})")
print(f"\n📈 Total notifications sent: {len(notifications)}")
print("=" * 70)
def demo_hysteresis():
"""Demonstrate hysteresis effect."""
print("\n\n" + "=" * 70)
print("DEMO 3: Hysteresis Prevents Flapping")
print("=" * 70)
config = {
"thresholds": {
"cpu_monitor": {
"cpu_percent": {
"warning": 80.0,
"critical": 90.0,
"hysteresis": 0.1, # 10% hysteresis
}
}
}
}
notifications = []
checker = ThresholdChecker(config, notification_callback=lambda m: notifications.append(m))
alert_states = {}
print("\nCritical threshold: 90%")
print("Hysteresis: 10%")
print("Recovery threshold: 81% (90 - 10% of 90)")
print("\nSimulating CPU fluctuating near CRITICAL threshold:")
print("-" * 70)
# Simulate fluctuating values
test_values = [
(75.0, "Normal"),
(92.0, "Crosses CRITICAL"),
(88.0, "Drops but still above 81% (stays CRITICAL)"),
(86.0, "Still above 81% (stays CRITICAL)"),
(83.0, "Still above 81% (stays CRITICAL)"),
(80.0, "Below 81% - recovers to OK"),
(88.0, "Rises again but below 90% (stays OK)"),
(91.0, "Crosses CRITICAL again"),
]
for value, description in test_values:
print(f"\n📊 CPU: {value:5.1f}% - {description}")
plugin_data = {"cpu_percent": value}
state_changes = checker.check_plugin_data(
"testhost",
"cpu_monitor",
plugin_data,
alert_states,
)
current_state = alert_states.get("cpu_monitor.cpu_percent")
print(f" State: {current_state.level.name}")
if state_changes:
print(f" 📧 Notification sent (state changed)")
else:
print(f" ✓ No notification (state unchanged - hysteresis working)")
print(f"\n📈 Notifications sent: {len(notifications)} (without hysteresis would be ≥6)")
print("=" * 70)
def demo_inverse_threshold():
"""Demonstrate inverse thresholds (less than)."""
print("\n\n" + "=" * 70)
print("DEMO 4: Inverse Thresholds (Alert When Low)")
print("=" * 70)
config = {
"thresholds": {
"memory_monitor": {
"available_mb": {
"warning": 1000, # Warn when < 1000 MB
"critical": 500, # Critical when < 500 MB
"operator": "<",
"hysteresis": 0.1,
}
}
}
}
notifications = []
checker = ThresholdChecker(config, notification_callback=lambda m: notifications.append(m))
alert_states = {}
print("\nMonitoring available memory (alert when LOW):")
print("WARNING when < 1000 MB, CRITICAL when < 500 MB")
print("-" * 70)
test_values = [
(2000, "Plenty of memory"),
(800, "Drops below 1000 MB - WARNING"),
(450, "Drops below 500 MB - CRITICAL"),
(520, "Rises but still in hysteresis zone - stays CRITICAL"),
(600, "Enough recovery - back to WARNING"),
(1200, "Fully recovered - OK"),
]
for value, description in test_values:
print(f"\n💾 Available: {value} MB - {description}")
plugin_data = {"available_mb": value}
state_changes = checker.check_plugin_data(
"testhost",
"memory_monitor",
plugin_data,
alert_states,
)
current_state = alert_states.get("memory_monitor.available_mb")
print(f" State: {current_state.level.name}")
if state_changes:
for metric, old_level, new_level, val in state_changes:
print(f" 📧 {old_level.name} → {new_level.name}")
print(f"\n📈 Notifications sent: {len(notifications)}")
print("=" * 70)
if __name__ == "__main__":
print("\n")
print("╔" + "═" * 68 + "╗")
print("║" + " " * 15 + "THRESHOLD ALERTING DEMONSTRATION" + " " * 21 + "║")
print("╚" + "═" * 68 + "╝")
demo_basic_thresholds()
demo_multiple_metrics()
demo_hysteresis()
demo_inverse_threshold()
print("\n\n" + "=" * 70)
print("DEMONSTRATION COMPLETE")
print("=" * 70)
print("\nKey takeaways:")
print(" • Thresholds detect when metrics exceed configured limits")
print(" • Notifications sent only on state changes, not every check")
print(" • Hysteresis prevents alert flapping")
print(" • Supports both 'greater than' and 'less than' thresholds")
print(" • Multiple metrics can be monitored simultaneously")
print("\nFor full documentation, see docs/THRESHOLD_ALERTING.md")
print("=" * 70)
print()
docs/HTTP_API.md
+532
View File
@@ -0,0 +1,532 @@
# HTTP API and Web UI Documentation
## Overview
The Heartbeat Daemon provides a comprehensive HTTP API and web-based UI for monitoring plugin data and alert states. The API follows RESTful conventions and returns JSON responses.
## Base URL
All API endpoints are relative to the server base URL:
```
http://your-server:50004
```
Default port is `50004` (configurable via `hbd_port` in configuration).
---
## API Endpoints
### Host Management
#### GET /api/0/hosts
Get list of all monitored hosts with their state information.
**Response:**
```json
[
{
"name": "webserver01",
"dyn": false,
"ver": 6,
"connections": [...]
}
]
```
#### GET /api/0/messages
Get recent heartbeat messages (last 30).
**Response:**
```json
[
{
"time": 1711234567.123,
"host": "webserver01",
"msg": "heartbeat received"
}
]
```
---
### Plugin Data Endpoints
#### GET /api/0/hosts/{hostname}/plugins
Get all plugin data for a specific host.
**Parameters:**
- `hostname` (path): Name of the host
**Response:**
```json
{
"hostname": "webserver01",
"plugins": {
"cpu_monitor": {
"timestamp": 1711234567.123,
"data": {
"cpu_percent": 45.2,
"load_1min": 2.5,
"load_5min": 2.1,
"load_15min": 1.8
},
"sample_count": 100
},
"memory_monitor": {
"timestamp": 1711234568.456,
"data": {
"percent": 65.4,
"available_mb": 4096,
"total_mb": 16384
},
"sample_count": 100
}
}
}
```
**Example:**
```bash
curl http://localhost:50004/api/0/hosts/webserver01/plugins
```
#### GET /api/0/hosts/{hostname}/plugins/{plugin_name}
Get detailed historical data for a specific plugin.
**Parameters:**
- `hostname` (path): Name of the host
- `plugin_name` (path): Name of the plugin
- `limit` (query, optional): Number of recent samples to return (default: 10)
**Response:**
```json
{
"hostname": "webserver01",
"plugin": "cpu_monitor",
"samples": [
{
"timestamp": 1711234567.123,
"data": {
"cpu_percent": 45.2,
"load_1min": 2.5
}
},
{
"timestamp": 1711234267.123,
"data": {
"cpu_percent": 42.1,
"load_1min": 2.3
}
}
],
"sample_count": 2
}
```
**Examples:**
```bash
# Get last 1 sample (most recent)
curl http://localhost:50004/api/0/hosts/webserver01/plugins/cpu_monitor?limit=1
# Get last 50 samples
curl http://localhost:50004/api/0/hosts/webserver01/plugins/memory_monitor?limit=50
# Get disk monitor data
curl http://localhost:50004/api/0/hosts/database01/plugins/disk_monitor
```
---
### Alert Endpoints
#### GET /api/0/hosts/{hostname}/alerts
Get alert states for a specific host.
**Parameters:**
- `hostname` (path): Name of the host
**Response:**
```json
{
"hostname": "webserver01",
"alerts": [
{
"metric_path": "cpu_monitor.cpu_percent",
"level": "WARNING",
"since": 1711234000.0,
"last_value": 85.5,
"last_check": 1711234567.123,
"notification_count": 2
},
{
"metric_path": "disk_monitor./.percent",
"level": "OK",
"since": 1711230000.0,
"last_value": 65.0,
"last_check": 1711234567.123,
"notification_count": 0
}
],
"summary": {
"ok": 15,
"warning": 1,
"critical": 0,
"unknown": 0
}
}
```
**Example:**
```bash
curl http://localhost:50004/api/0/hosts/webserver01/alerts
```
#### GET /api/0/alerts
Get all active alerts across all monitored hosts.
**Response:**
```json
{
"alerts": [
{
"hostname": "webserver01",
"metric_path": "cpu_monitor.cpu_percent",
"level": "CRITICAL",
"since": 1711234000.0,
"last_value": 95.5,
"last_check": 1711234567.123,
"notification_count": 3
},
{
"hostname": "database01",
"metric_path": "memory_monitor.percent",
"level": "WARNING",
"since": 1711233000.0,
"last_value": 88.2,
"last_check": 1711234567.123,
"notification_count": 1
}
],
"summary": {
"critical": 1,
"warning": 1,
"unknown": 0,
"total": 2
},
"host_count": 5
}
```
**Example:**
```bash
curl http://localhost:50004/api/0/alerts | jq .
```
---
## Web UI Pages
### Live Dashboard
**URL:** `/live`
Real-time dashboard showing:
- Host connection states
- IPv4/IPv6 connectivity
- Latency metrics
- Recent messages
**Features:**
- WebSocket-powered live updates
- Sortable columns
- Color-coded status indicators
### Plugin Metrics
**URL:** `/plugins`
Interactive visualization of plugin metrics:
- Select host and plugin from dropdown
- View current metric values
- Automatic refresh every 30 seconds
- Support for nested metrics (e.g., per-partition disk stats)
**Features:**
- Card-based metric display
- Unit formatting (%, MB, GB)
- Nested object visualization
- Auto-refresh
**Screenshots of available data:**
- CPU usage, load average, frequency
- Memory usage, available memory, swap
- Disk usage per partition, I/O statistics
- Network interface statistics, connection counts
- Custom plugin data
### Alerts Dashboard
**URL:** `/alerts`
Comprehensive alert monitoring:
- Summary cards (Critical, Warning, Total Hosts)
- Filter by severity (All, Critical, Warning)
- Alert details with duration
- Auto-refresh every 15 seconds
**Features:**
- Color-coded alert levels
- Duration tracking
- Filterable list
- Real-time updates
- Summary statistics
---
## Integration Examples
### Monitoring Script
```bash
#!/bin/bash
# Check for critical alerts and send notification
RESPONSE=$(curl -s http://localhost:50004/api/0/alerts)
CRITICAL_COUNT=$(echo "$RESPONSE" | jq '.summary.critical')
if [ "$CRITICAL_COUNT" -gt 0 ]; then
echo "CRITICAL: $CRITICAL_COUNT critical alerts detected!"
echo "$RESPONSE" | jq '.alerts[] | select(.level=="CRITICAL")'
# Send notification
# mail -s "Critical Alerts" admin@example.com < alert_details.txt
fi
```
### Python Client
```python
import requests
import json
# Get all plugin data for a host
response = requests.get('http://localhost:50004/api/0/hosts/webserver01/plugins')
data = response.json()
print(f"Host: {data['hostname']}")
print(f"Plugins: {', '.join(data['plugins'].keys())}")
for plugin, info in data['plugins'].items():
print(f"\n{plugin}:")
for metric, value in info['data'].items():
print(f" {metric}: {value}")
# Check for alerts
response = requests.get('http://localhost:50004/api/0/alerts')
alerts = response.json()
if alerts['summary']['critical'] > 0:
print(f"\n⚠️ {alerts['summary']['critical']} CRITICAL ALERTS!")
for alert in alerts['alerts']:
if alert['level'] == 'CRITICAL':
print(f" - {alert['hostname']}: {alert['metric_path']} = {alert['last_value']}")
```
### Grafana Integration
The API endpoints can be used with Grafana's JSON datasource plugin:
1. Install the SimpleJSON datasource plugin
2. Configure datasource URL: `http://your-server:50004`
3. Create queries:
- Metrics: `/api/0/hosts/webserver01/plugins/cpu_monitor?limit=100`
- Alerts: `/api/0/alerts`
### Prometheus Integration
Export metrics in Prometheus format (future enhancement):
```python
# Example prometheus exporter
from prometheus_client import Gauge, generate_latest
import requests
cpu_usage = Gauge('heartbeat_cpu_percent', 'CPU usage percentage', ['hostname'])
memory_usage = Gauge('heartbeat_memory_percent', 'Memory usage percentage', ['hostname'])
def collect_metrics():
hosts = requests.get('http://localhost:50004/api/0/hosts').json()
for host in hosts:
hostname = host['name']
plugins = requests.get(f'http://localhost:50004/api/0/hosts/{hostname}/plugins').json()
if 'cpu_monitor' in plugins['plugins']:
cpu_data = plugins['plugins']['cpu_monitor']['data']
cpu_usage.labels(hostname=hostname).set(cpu_data.get('cpu_percent', 0))
if 'memory_monitor' in plugins['plugins']:
mem_data = plugins['plugins']['memory_monitor']['data']
memory_usage.labels(hostname=hostname).set(mem_data.get('percent', 0))
```
---
## Response Formats
### Success Response
All successful API calls return HTTP 200 with JSON body:
```json
{
"field": "value",
...
}
```
### Error Response
API errors return appropriate HTTP status codes with JSON:
```json
{
"error": "Host 'unknown-host' not found"
}
```
**Common Status Codes:**
- `200 OK` - Success
- `400 Bad Request` - Invalid parameters
- `404 Not Found` - Resource not found
- `500 Internal Server Error` - Server error
---
## WebSocket API
For real-time updates, connect to the WebSocket endpoint:
**URL:** `ws://your-server:50005/hbd` (or `wss://` for secure)
**Messages:**
```json
{
"type": "host",
"data": {
"name": "webserver01",
"state": "UP"
}
}
```
```json
{
"type": "plugin",
"data": {
"host": "webserver01",
"plugin": "cpu_monitor",
"data": {...},
"timestamp": 1711234567.123
}
}
```
---
## Configuration
### Enable HTTP Server
```yaml
# In your hbd configuration file
hbd_host: "" # Listen on all interfaces
hbd_port: 50004 # HTTP port
ws_port: 50005 # WebSocket port (optional)
# wss_port: 50006 # Secure WebSocket (requires SSL)
```
### SSL/TLS Configuration
For secure WebSocket connections:
```yaml
wss_port: 50006
cert_path: /etc/heartbeat/certs/
wss_pem: server.pem
wss_key: server.key
```
---
## Rate Limiting
The API currently does not implement rate limiting. For production use, consider:
- Placing behind a reverse proxy (nginx, Apache)
- Using API gateway for rate limiting
- Implementing caching for frequently accessed endpoints
---
## CORS Support
By default, CORS is not enabled. To enable for web applications:
```python
# In http.py, add CORS middleware
from aiohttp_cors import setup as cors_setup
app = web.Application()
cors = cors_setup(app)
# Configure CORS for all routes
for route in list(app.router.routes()):
cors.add(route, {
"*": aiohttp_cors.ResourceOptions(
allow_credentials=True,
expose_headers="*",
allow_headers="*",
)
})
```
---
## Performance Considerations
### Caching
- Plugin data is cached in memory (last 100 samples per plugin)
- No database queries required
- Responses are fast (<10ms typical)
### Scalability
- Each host stores its own data independently
- Memory usage: ~1KB per host + ~1KB per plugin sample
- For 100 hosts with 5 plugins: ~50MB memory
### Best Practices
1. Use `limit` parameter to control response size
2. Cache responses on client side when appropriate
3. Use WebSocket for real-time updates instead of polling
4. Consider pagination for large deployments (future enhancement)
---
## Troubleshooting
### API Returns 404
- Verify hostname in URL matches actual host name
- Check host is sending heartbeats: `curl http://localhost:50004/api/0/hosts`
### No Plugin Data
- Verify client is configured with plugins
- Check client logs for plugin errors
- Ensure plugins are sending data (check journal logs)
### Empty Alerts
- Verify thresholds are configured
- Check host is in `watchhosts` list
- Ensure plugins are collecting metrics
- Review server logs for threshold checker errors
---
## See Also
- [Plugin Development Guide](PLUGIN_DEVELOPMENT.md)
- [Threshold Alerting Documentation](THRESHOLD_ALERTING.md)
- [Message Journal Documentation](MESSAGE_JOURNAL.md)
- Configuration examples: `hbd/config_example.yaml`
docs/MESSAGE_JOURNAL.md
+413
View File
@@ -0,0 +1,413 @@
# Message Journal
The message journal provides persistent logging of all received heartbeat messages with automatic size-based log rotation.
## Overview
The journal logs every message received by the heartbeat daemon (hbd) in JSON format, making it easy to:
- Audit message history
- Debug connection issues
- Analyze traffic patterns
- Replay messages for testing
- Create historical reports
## Features
- **JSON Format**: Each message is logged as a single JSON line for easy parsing
- **Size-Based Rotation**: Automatically rotates logs when size threshold is reached
- **Automatic Cleanup**: Keeps only a configurable number of backup files
- **Thread-Safe**: Safe for concurrent access from multiple async tasks
- **Configurable**: All settings controllable via configuration file
- **Performance**: Non-blocking async operation with minimal overhead
## Configuration
Add these settings to your hbd configuration file (e.g., `.hb.yaml`):
```yaml
# Message journal configuration
journal_enabled: true # Enable/disable journaling
journal_dir: /var/log/heartbeat # Directory for journal files
journal_file: messages.journal # Base filename
journal_max_size: 104857600 # Max size in bytes (100MB default)
journal_max_backups: 10 # Number of backup files to keep
```
### Configuration Options
| Option | Default | Description |
|--------|---------|-------------|
| `journal_enabled` | `true` | Enable or disable message journaling |
| `journal_dir` | `/var/log/heartbeat` | Directory where journal files are stored |
| `journal_file` | `messages.journal` | Base filename for the journal |
| `journal_max_size` | `104857600` (100MB) | Maximum file size before rotation |
| `journal_max_backups` | `10` | Number of rotated backup files to keep |
## File Format
Messages are logged in JSONL (JSON Lines) format - one JSON object per line:
```json
{"timestamp":1711234567.123,"datetime":"2026-03-28T12:34:56","source_ip":"192.168.1.100","source_port":50003,"message":{"ID":"HTB","name":"webserver1","interval":30,"ver":1}}
{"timestamp":1711234597.456,"datetime":"2026-03-28T12:35:37","source_ip":"192.168.1.101","source_port":50003,"message":{"ID":"PLG","plugin":"cpu_monitor","cpu_percent":45.2,"load_1min":1.5}}
```
### Entry Structure
Each journal entry contains:
| Field | Type | Description |
|-------|------|-------------|
| `timestamp` | float | Unix timestamp (seconds since epoch) |
| `datetime` | string | ISO 8601 formatted datetime |
| `source_ip` | string | Source IP address |
| `source_port` | integer | Source UDP port |
| `message` | object | Complete parsed message dictionary |
## Log Rotation
### How Rotation Works
1. Journal writes messages to the current file
2. When file size exceeds `journal_max_size`, rotation is triggered
3. Current file is renamed with timestamp: `messages.journal.YYYYMMDD-HHMMSS`
4. New empty file is created as the current journal
5. Old backup files exceeding `journal_max_backups` are deleted
### Example File Structure
```
/var/log/heartbeat/
├── messages.journal # Current active journal
├── messages.journal.20260328-120000 # Rotated backup
├── messages.journal.20260328-140000 # Rotated backup
└── messages.journal.20260328-160000 # Rotated backup (oldest)
```
### Rotation Behavior
- Rotation is triggered when the next message would exceed the size limit
- Rotation is automatic and requires no manual intervention
- Old backups are deleted in FIFO order (oldest first)
- Rotation is thread-safe and won't lose messages
## Usage Examples
### Reading Journal Files
#### Using Python
```python
import json
# Read all entries from current journal
with open('/var/log/heartbeat/messages.journal', 'r') as f:
for line in f:
entry = json.loads(line)
print(f"{entry['datetime']} - {entry['source_ip']} - {entry['message']['ID']}")
```
#### Using jq (command line)
```bash
# View all messages
cat /var/log/heartbeat/messages.journal | jq .
# Filter by message type
cat /var/log/heartbeat/messages.journal | jq 'select(.message.ID == "HTB")'
# Filter by hostname
cat /var/log/heartbeat/messages.journal | jq 'select(.message.name == "webserver1")'
# Count messages by type
cat /var/log/heartbeat/messages.journal | jq -r '.message.ID' | sort | uniq -c
# Extract timestamps and source IPs
cat /var/log/heartbeat/messages.journal | jq -r '[.datetime, .source_ip, .message.ID] | @tsv'
```
#### Using shell tools
```bash
# Count total messages
wc -l /var/log/heartbeat/messages.journal
# View recent messages
tail -n 100 /var/log/heartbeat/messages.journal | jq .
# Search for specific host
grep -F '"name":"webserver1"' /var/log/heartbeat/messages.journal
# Check journal file size
du -h /var/log/heartbeat/messages.journal
```
### Analyzing Historical Data
```bash
# Combine all journal files (current + backups)
cat /var/log/heartbeat/messages.journal* | jq . > all_messages.json
# Count messages per host
cat /var/log/heartbeat/messages.journal* | jq -r '.message.name // "unknown"' | sort | uniq -c
# Find all plugin messages
cat /var/log/heartbeat/messages.journal* | jq 'select(.message.ID == "PLG")'
# Extract CPU metrics from plugin messages
cat /var/log/heartbeat/messages.journal* | \
jq 'select(.message.plugin == "cpu_monitor") | {time: .datetime, host: .message.name, cpu: .message.cpu_percent}'
```
## Integration with Log Management
### Logrotate
While the journal has built-in rotation, you can also use logrotate for additional management:
```
/var/log/heartbeat/messages.journal.* {
daily
rotate 30
compress
delaycompress
missingok
notifempty
}
```
### Elasticsearch/OpenSearch
Import journal data into Elasticsearch for advanced analysis:
```python
from elasticsearch import Elasticsearch
import json
es = Elasticsearch(['localhost:9200'])
with open('/var/log/heartbeat/messages.journal', 'r') as f:
for line in f:
entry = json.loads(line)
es.index(index='heartbeat-messages', body=entry)
```
### Splunk
Create a Splunk input for the journal:
```ini
[monitor:///var/log/heartbeat/messages.journal*]
sourcetype = heartbeat_json
index = heartbeat
```
## Performance Considerations
### Overhead
- Journal writing is async and non-blocking
- Typical overhead: < 1ms per message
- Minimal impact on heartbeat processing
### Disk Usage
Calculate expected disk usage:
```
Messages per day = (86400 seconds / interval) * number_of_hosts
Average message size ≈ 200-500 bytes
Daily disk usage = Messages per day * Average message size
Example:
- 100 hosts
- 30 second interval
- 2880 messages/day per host
- 288,000 messages/day total
- ~60-140 MB/day
```
### Recommendations
- **Small deployments** (< 50 hosts): Default settings work well
- **Medium deployments** (50-500 hosts): Increase `journal_max_size` to 500MB, `journal_max_backups` to 20
- **Large deployments** (> 500 hosts): Consider 1GB+ journal files, 30+ backups, or external log aggregation
## Monitoring
### Check Journal Status
The journal exposes statistics that can be queried:
```python
from hbd.journal import get_journal
journal = get_journal()
stats = journal.get_stats()
print(f"Current size: {stats['current_size']:,} bytes")
print(f"Rotation threshold: {stats['rotation_threshold']}")
```
### Log Messages
Journal operations are logged at appropriate levels:
- `INFO`: Initialization, rotation events, cleanup
- `DEBUG`: Individual message logging
- `WARNING`: Non-critical issues
- `ERROR`: Critical failures
Check hbd logs for journal-related messages:
```bash
grep journal /var/log/heartbeat.log
```
## Troubleshooting
### Journal Files Not Created
**Problem**: No journal files appear in the configured directory.
**Solutions**:
- Check `journal_enabled: true` in configuration
- Verify directory exists and hbd has write permissions
- Check hbd logs for initialization errors
- Verify disk space is available
### Rotation Not Working
**Problem**: Journal file grows beyond `journal_max_size`.
**Solutions**:
- Check that `journal_max_size` is properly configured
- Verify hbd has permission to rename/create files
- Check for filesystem issues
- Review hbd logs for rotation errors
### Missing Messages
**Problem**: Some messages don't appear in journal.
**Solutions**:
- Verify `journal_enabled: true`
- Check for write errors in hbd logs
- Verify sufficient disk space
- Check if filesystem is read-only
### Performance Issues
**Problem**: Journal causing slow message processing.
**Solutions**:
- Use faster storage (SSD) for journal directory
- Increase `journal_max_size` to reduce rotation frequency
- Disable journal if not needed: `journal_enabled: false`
- Consider async syslog forwarding instead
## Security Considerations
### File Permissions
Ensure proper permissions on journal files:
```bash
# Journal directory
chmod 750 /var/log/heartbeat
chown hbd:hbd /var/log/heartbeat
# Journal files
chmod 640 /var/log/heartbeat/messages.journal*
```
### Sensitive Data
Journal files may contain:
- Hostnames and IP addresses
- System metrics
- Custom message content
**Recommendations**:
- Restrict read access to authorized users only
- Consider encryption for archived journals
- Implement log retention policies
- Sanitize data if sharing for debugging
## API Reference
### MessageJournal Class
```python
class MessageJournal:
def __init__(self, config: Dict[str, Any])
async def initialize(self) -> bool
async def log_message(self, msg: Dict, addr: tuple, timestamp: float)
async def close(self)
def get_stats(self) -> Dict[str, Any]
```
### Module Functions
```python
def get_journal(config: Dict = None) -> MessageJournal
async def log_message(msg: Dict, addr: tuple, timestamp: float = None)
```
## Example: Custom Message Processing
Process journal messages in real-time:
```python
import asyncio
import json
from pathlib import Path
async def tail_journal(journal_path):
"""Follow journal file and process new messages."""
path = Path(journal_path)
with open(path, 'r') as f:
# Jump to end
f.seek(0, 2)
while True:
line = f.readline()
if line:
entry = json.loads(line)
await process_message(entry)
else:
await asyncio.sleep(0.1)
async def process_message(entry):
"""Process a journal entry."""
msg = entry['message']
# Alert on boot messages
if msg.get('boot'):
print(f"ALERT: {msg['name']} rebooted at {entry['datetime']}")
# Track CPU usage
if msg.get('ID') == 'PLG' and msg.get('plugin') == 'cpu_monitor':
cpu = msg.get('cpu_percent', 0)
if cpu > 90:
print(f"WARNING: {entry['source_ip']} CPU usage: {cpu}%")
```
## Future Enhancements
Potential improvements for future versions:
- Compression of rotated logs (gzip)
- Time-based rotation in addition to size-based
- Filtering to exclude certain message types
- Structured logging output formats (CEF, GELF)
- Remote syslog forwarding
- Message deduplication
- Journal file encryption
- Signed journal entries
## See Also
- [Configuration Guide](../hbd/config.py) - Full configuration options
- [UDP Protocol](../hbd/udp.py) - Message handling
- [Server Architecture](../hbd/server.py) - Server initialization
docs/NAGIOS_INTEGRATION.md
+331
View File
@@ -0,0 +1,331 @@
# Nagios Plugin Integration Guide
The Heartbeat monitoring system now supports running existing Nagios-compatible monitoring plugins through the `nagios_runner` plugin. This allows you to leverage the thousands of existing Nagios plugins without modification.
## Quick Start
### 1. Install Nagios Plugins
**Debian/Ubuntu:**
```bash
sudo apt-get install nagios-plugins
```
**RHEL/CentOS/Fedora:**
```bash
sudo yum install nagios-plugins-all
# or
sudo dnf install nagios-plugins-all
```
**Arch Linux:**
```bash
sudo pacman -S monitoring-plugins
```
### 2. Configure Heartbeat
Add the `nagios_runner` section to your `~/.hb.yaml` config:
```yaml
nagios_runner:
interval: 60 # Run plugins every 60 seconds
timeout: 30 # Command timeout in seconds
commands:
- name: check_disk_root
command: /usr/lib/nagios/plugins/check_disk -w 20% -c 10% -p /
- name: check_load
command: /usr/lib/nagios/plugins/check_load -w 5,4,3 -c 10,8,6
- name: check_procs
command: /usr/lib/nagios/plugins/check_procs -w 250 -c 400
```
### 3. Start Heartbeat Client
```bash
hbc -v localhost
```
The client will now execute the configured Nagios plugins and send their results to the server.
## How It Works
### Nagios Plugin Standard
Nagios plugins follow a simple interface:
1. **Exit Codes:**
- `0` = OK
- `1` = WARNING
- `2` = CRITICAL
- `3` = UNKNOWN
2. **Output Format:**
```
STATUS - Message | performance_data
```
3. **Performance Data Format:**
```
'label'=value[UOM];[warn];[crit];[min];[max]
```
### Example Plugin Output
```bash
$ /usr/lib/nagios/plugins/check_disk -w 20% -c 10% -p /
DISK OK - free space: / 156 GB (78%); | /=44GB;127;142;0;159
```
This output includes:
- **Status:** `DISK OK`
- **Message:** `free space: / 156 GB (78%)`
- **Performance Data:** `/=44GB;127;142;0;159`
- Current value: 44GB
- Warning threshold: 127GB
- Critical threshold: 142GB
- Min: 0GB
- Max: 159GB
### Data Collected
The `nagios_runner` plugin collects:
**For each configured command:**
- `{name}_status` - Status string (OK, WARNING, CRITICAL, UNKNOWN)
- `{name}_status_code` - Numeric exit code (0-3)
- `{name}_output` - Status message
- `{name}_{metric}` - Each performance metric value
- `{name}_{metric}_uom` - Unit of measurement (if present)
- `{name}_{metric}_warn` - Warning threshold (if present)
- `{name}_{metric}_crit` - Critical threshold (if present)
- `{name}_{metric}_min` - Minimum value (if present)
- `{name}_{metric}_max` - Maximum value (if present)
**Overall:**
- `overall_status` - Worst status from all commands
- `overall_status_code` - Worst status code
- `plugin_count` - Number of Nagios plugins executed
## Configuration Options
```yaml
nagios_runner:
# Collection interval in seconds (default: 60)
interval: 60
# Command execution timeout in seconds (default: 30)
timeout: 30
# Execute commands via shell (default: true)
# Set to false for direct execution (more secure but less flexible)
shell: true
# List of Nagios plugins to run
commands:
- name: unique_name # Required: unique identifier
command: /path/to/plugin [args] # Required: full command to execute
```
## Common Nagios Plugins
### System Resources
**Disk Space:**
```yaml
- name: check_disk_root
command: /usr/lib/nagios/plugins/check_disk -w 20% -c 10% -p /
```
**Load Average:**
```yaml
- name: check_load
command: /usr/lib/nagios/plugins/check_load -w 5,4,3 -c 10,8,6
```
**Swap Usage:**
```yaml
- name: check_swap
command: /usr/lib/nagios/plugins/check_swap -w 20% -c 10%
```
**Process Count:**
```yaml
- name: check_procs
command: /usr/lib/nagios/plugins/check_procs -w 250 -c 400
```
**Users Logged In:**
```yaml
- name: check_users
command: /usr/lib/nagios/plugins/check_users -w 5 -c 10
```
### Network Services
**SSH:**
```yaml
- name: check_ssh
command: /usr/lib/nagios/plugins/check_ssh localhost
```
**HTTP:**
```yaml
- name: check_http_local
command: /usr/lib/nagios/plugins/check_http -H localhost
- name: check_http_ssl
command: /usr/lib/nagios/plugins/check_http -H example.com --ssl
```
**DNS:**
```yaml
- name: check_dns
command: /usr/lib/nagios/plugins/check_dns -H google.com
```
**Ping:**
```yaml
- name: check_ping_gateway
command: /usr/lib/nagios/plugins/check_ping -H 192.168.1.1 -w 100,20% -c 500,60%
```
### Databases
**MySQL:**
```yaml
- name: check_mysql
command: /usr/lib/nagios/plugins/check_mysql -H localhost -u user -p password
```
**PostgreSQL:**
```yaml
- name: check_pgsql
command: /usr/lib/nagios/plugins/check_pgsql -H localhost -d database
```
## Writing Custom Nagios Plugins
You can write your own Nagios-compatible plugins in any language. Here's a simple example:
**Bash:**
```bash
#!/bin/bash
# /usr/local/bin/check_example.sh
# Get the value to check
value=$(some_command)
# Define thresholds
warn=80
crit=90
# Check and output result
if [ $value -ge $crit ]; then
echo "CRITICAL - Value is $value | value=${value};${warn};${crit};0;100"
exit 2
elif [ $value -ge $warn ]; then
echo "WARNING - Value is $value | value=${value};${warn};${crit};0;100"
exit 1
else
echo "OK - Value is $value | value=${value};${warn};${crit};0;100"
exit 0
fi
```
**Python:**
```python
#!/usr/bin/env python3
# /usr/local/bin/check_example.py
import sys
def check_something():
value = get_value() # Your check logic here
warn = 80
crit = 90
perfdata = f"value={value};{warn};{crit};0;100"
if value >= crit:
print(f"CRITICAL - Value is {value} | {perfdata}")
sys.exit(2)
elif value >= warn:
print(f"WARNING - Value is {value} | {perfdata}")
sys.exit(1)
else:
print(f"OK - Value is {value} | {perfdata}")
sys.exit(0)
if __name__ == "__main__":
check_something()
```
Then configure in Heartbeat:
```yaml
nagios_runner:
commands:
- name: my_custom_check
command: /usr/local/bin/check_example.sh
```
## Troubleshooting
### Plugin not found
```
Error: Command not found
```
**Solution:** Use the full path to the plugin. Common locations:
- `/usr/lib/nagios/plugins/`
- `/usr/lib64/nagios/plugins/`
- `/usr/local/nagios/libexec/`
### Permission denied
```
Error: Permission denied
```
**Solution:** Ensure the plugin is executable:
```bash
chmod +x /path/to/plugin
```
### Timeout errors
```
Command timed out after 30s
```
**Solution:** Increase the timeout in config:
```yaml
nagios_runner:
timeout: 60 # Increase timeout
```
### No performance data
If performance data is not being parsed:
1. Check plugin output includes `|` separator
2. Verify performance data format: `'label'=value[UOM];...`
3. Enable debug logging: `hbc -v -x localhost`
## Benefits
1. **Massive Plugin Library:** Thousands of existing Nagios plugins available
2. **No Rewriting:** Use plugins as-is without modification
3. **Community Support:** Well-documented and maintained plugins
4. **Flexibility:** Mix Nagios plugins with native Heartbeat plugins
5. **Standard Interface:** Consistent exit codes and output format
6. **Performance Data:** Automatic extraction of metrics
## Resources
- [Nagios Plugin Development Guidelines](https://nagios-plugins.org/doc/guidelines.html)
- [Monitoring Plugins Project](https://www.monitoring-plugins.org/)
- [Nagios Exchange](https://exchange.nagios.org/) - Plugin repository
- [Check_MK Local Checks](https://docs.checkmk.com/latest/en/localchecks.html) - Compatible format
## Next Steps
- Configure threshold alerts based on Nagios plugin status codes
- View plugin data in the Heartbeat web UI
- Create custom plugins for your specific monitoring needs
- Integrate with existing Nagios/Icinga configurations
docs/PLUGIN_DEVELOPMENT.md
+544
View File
@@ -0,0 +1,544 @@
# Plugin Development Guide
This guide explains how to create custom plugins for the Heartbeat monitoring system.
## Table of Contents
- [Plugin Architecture](#plugin-architecture)
- [Plugin Types](#plugin-types)
- [Creating a Plugin](#creating-a-plugin)
- [Plugin Lifecycle](#plugin-lifecycle)
- [Configuration](#configuration)
- [Best Practices](#best-practices)
- [Examples](#examples)
- [Testing](#testing)
## Plugin Architecture
Heartbeat's plugin system is designed to be simple yet powerful. Plugins are Python classes that inherit from one of the base plugin types and implement a few key methods.
### Key Concepts
- **Plugin Registry**: Central registry that manages all loaded plugins
- **Plugin Loader**: Automatically discovers and loads plugins from the `hbd/plugins/` directory
- **Plugin Types**: InfoPlugin (static data) and MonitorPlugin (periodic metrics)
- **Async/Await**: All plugin methods are async for non-blocking operation
## Plugin Types
### InfoPlugin
InfoPlugins collect static information that doesn't change frequently (OS version, hardware specs, etc.).
- **Runs once** at startup (interval = 0)
- **Cached** - data is collected once and reused
- **Lightweight** - no periodic overhead
**Use InfoPlugin for:**
- Operating system details
- Hardware information
- Software versions
- Configuration data
- Static inventory
### MonitorPlugin
MonitorPlugins collect metrics that change over time (CPU usage, memory, network traffic).
- **Runs periodically** based on configured interval
- **Scheduled** - collected at regular intervals
- **Dynamic** - captures changing system state
**Use MonitorPlugin for:**
- Resource usage (CPU, memory, disk, network)
- Performance metrics
- Counters and gauges
- Time-series data
## Creating a Plugin
### Step 1: Choose Plugin Type
Decide whether your plugin collects static information (InfoPlugin) or dynamic metrics (MonitorPlugin).
### Step 2: Create Plugin File
Create a new Python file in `hbd/plugins/` directory:
```python
"""
My awesome plugin for Heartbeat.
Brief description of what this plugin does.
"""
import logging
from typing import Dict, Any, Optional
# Import psutil or other dependencies if needed
try:
import psutil
except ImportError:
psutil = None
from hbd.plugin import MonitorPlugin # or InfoPlugin
logger = logging.getLogger(__name__)
class MyAwesomePlugin(MonitorPlugin): # or InfoPlugin
"""
One-line description of the plugin.
Collects:
- List of metrics/data collected
- Another metric
Configuration:
interval: Collection interval in seconds (default: 60)
option1: Description of option1 (default: value)
option2: Description of option2 (default: value)
"""
name = "my_awesome_plugin" # Unique plugin name
interval = 60 # For MonitorPlugin, use 0 for InfoPlugin
def __init__(self, config: Optional[Dict[str, Any]] = None):
"""Initialize the plugin with optional configuration."""
super().__init__(config)
# Extract configuration options
self.option1 = self.config.get('option1', 'default_value')
self.option2 = self.config.get('option2', True)
# Check dependencies
if psutil is None:
raise ImportError("psutil is required for my_awesome_plugin")
async def initialize(self):
"""
Initialize the plugin.
This is called once when the plugin is loaded.
Use this to verify dependencies, establish connections, etc.
Returns:
True if initialization successful, False otherwise
"""
logger.info(f"My awesome plugin initialized (option1: {self.option1})")
return True
async def collect(self) -> Dict[str, Any]:
"""
Collect data.
This is called periodically (MonitorPlugin) or once (InfoPlugin).
Returns:
Dictionary of collected data (will be sent to server)
"""
try:
data = await self._collect_metrics()
logger.debug(f"Collected {len(data)} metrics")
return data
except Exception as e:
logger.error(f"Error collecting data: {e}")
return {"error": str(e)}
async def _collect_metrics(self) -> Dict[str, Any]:
"""Internal method to collect actual metrics."""
metrics = {}
# Collect your data here
metrics['metric1'] = self._get_metric1()
metrics['metric2'] = self._get_metric2()
return metrics
def _get_metric1(self):
"""Helper method for metric collection."""
# Implementation here
return 42
def _get_metric2(self):
"""Helper method for metric collection."""
# Implementation here
return "hello"
async def cleanup(self):
"""
Cleanup resources.
This is called when the plugin is unloaded or the client shuts down.
Use this to close connections, release resources, etc.
"""
logger.info("My awesome plugin cleanup")
# Plugin instance for automatic discovery
plugin = MyAwesomePlugin
```
### Step 3: Test Your Plugin
Create a test script to verify your plugin works:
```python
#!/usr/bin/env python3
import asyncio
import sys
from pathlib import Path
# Add parent directory to path
sys.path.insert(0, str(Path(__file__).parent))
from hbd.plugins.my_awesome_plugin import MyAwesomePlugin
async def test():
# Create plugin instance
plugin = MyAwesomePlugin({'option1': 'test_value'})
# Initialize
if not await plugin.initialize():
print("Failed to initialize")
return False
# Collect data
data = await plugin.collect()
print(f"Collected data: {data}")
# Cleanup
await plugin.cleanup()
return True
if __name__ == '__main__':
success = asyncio.run(test())
sys.exit(0 if success else 1)
```
## Plugin Lifecycle
Understanding the plugin lifecycle helps you implement plugins correctly:
```
1. Plugin Discovery
└─> Loader scans hbd/plugins/ directory
└─> Finds Python files (except those starting with _)
└─> Imports modules
2. Plugin Instantiation
└─> Creates instance with configuration
└─> __init__() is called
3. Plugin Initialization
└─> initialize() is called
└─> Plugin verifies dependencies, establishes connections
└─> Returns True/False for success/failure
4. Plugin Registration
└─> If initialization succeeds, plugin is registered
└─> Plugin becomes active
5. Data Collection
└─> For InfoPlugin: collect() called once after initialization
└─> For MonitorPlugin: collect() called periodically based on interval
└─> Data is sent to server via PLG message
6. Plugin Shutdown
└─> cleanup() is called
└─> Plugin releases resources, closes connections
```
## Configuration
### Plugin-Specific Configuration
Plugins receive configuration through the `config` parameter in `__init__`:
```python
def __init__(self, config: Optional[Dict[str, Any]] = None):
super().__init__(config)
# Access configuration with defaults
self.interval = self.config.get('interval', 60)
self.threshold = self.config.get('threshold', 80)
self.enabled_features = self.config.get('features', ['feature1', 'feature2'])
```
### Client Configuration File
Users configure plugins in the client configuration YAML:
```yaml
plugins:
my_awesome_plugin:
enabled: true
interval: 120
option1: custom_value
option2: false
```
## Best Practices
### 1. Error Handling
Always handle errors gracefully:
```python
async def collect(self) -> Dict[str, Any]:
try:
return await self._collect_metrics()
except Exception as e:
logger.error(f"Error collecting metrics: {e}")
return {"error": str(e)}
```
### 2. Logging
Use appropriate log levels:
```python
logger.debug("Detailed information for debugging")
logger.info("Normal operation messages")
logger.warning("Warning messages for unusual but handled situations")
logger.error("Error messages for failures")
```
### 3. Dependencies
Check for optional dependencies:
```python
try:
import some_optional_library
except ImportError:
some_optional_library = None
# Later in __init__:
if some_optional_library is None:
raise ImportError("some_optional_library is required")
```
### 4. Performance
- Keep collection methods fast (< 1 second)
- Use async/await for I/O operations
- Cache expensive computations
- Don't block the event loop
### 5. Data Structure
Return clean, structured data:
```python
{
'metric_name': value,
'nested_data': {
'sub_metric': value
},
'list_data': [item1, item2],
'timestamp': time.time() # Optional timestamp
}
```
### 6. Documentation
Document your plugin thoroughly:
- Class docstring with description and configuration
- Method docstrings explaining purpose and return values
- Inline comments for complex logic
## Examples
### Example 1: Simple InfoPlugin
```python
from hbd.plugin import InfoPlugin
import platform
class SimpleInfoPlugin(InfoPlugin):
"""Collect basic system information."""
name = "simple_info"
interval = 0 # InfoPlugin
async def initialize(self):
return True
async def collect(self) -> Dict[str, Any]:
return {
'hostname': platform.node(),
'system': platform.system(),
'python_version': platform.python_version()
}
async def cleanup(self):
pass
plugin = SimpleInfoPlugin
```
### Example 2: MonitorPlugin with State
```python
from hbd.plugin import MonitorPlugin
import time
class CounterPlugin(MonitorPlugin):
"""Track a counter over time."""
name = "counter"
interval = 30
def __init__(self, config=None):
super().__init__(config)
self._counter = 0
self._start_time = time.time()
async def initialize(self):
return True
async def collect(self) -> Dict[str, Any]:
self._counter += 1
uptime = time.time() - self._start_time
return {
'count': self._counter,
'uptime': uptime,
'rate': self._counter / uptime
}
async def cleanup(self):
pass
plugin = CounterPlugin
```
### Example 3: Plugin with External Command
```python
from hbd.plugin import MonitorPlugin
import asyncio
class CommandPlugin(MonitorPlugin):
"""Execute external command and capture output."""
name = "command_executor"
interval = 60
def __init__(self, config=None):
super().__init__(config)
self.command = self.config.get('command', 'echo "no command"')
async def initialize(self):
return True
async def collect(self) -> Dict[str, Any]:
try:
process = await asyncio.create_subprocess_shell(
self.command,
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.PIPE
)
stdout, stderr = await asyncio.wait_for(
process.communicate(),
timeout=30
)
return {
'exit_code': process.returncode,
'stdout': stdout.decode('utf-8'),
'stderr': stderr.decode('utf-8')
}
except Exception as e:
return {'error': str(e)}
async def cleanup(self):
pass
plugin = CommandPlugin
```
## Testing
### Unit Testing
Create unit tests for your plugins:
```python
import unittest
import asyncio
class TestMyPlugin(unittest.TestCase):
def setUp(self):
self.plugin = MyAwesomePlugin({'option1': 'test'})
def test_initialization(self):
result = asyncio.run(self.plugin.initialize())
self.assertTrue(result)
def test_collection(self):
asyncio.run(self.plugin.initialize())
data = asyncio.run(self.plugin.collect())
self.assertIsInstance(data, dict)
self.assertIn('metric1', data)
self.assertGreater(data['metric1'], 0)
def tearDown(self):
asyncio.run(self.plugin.cleanup())
if __name__ == '__main__':
unittest.main()
```
### Integration Testing
Test your plugin with the actual client:
```bash
# Create test configuration
cat > test_config.yaml <<EOF
server: localhost
plugins:
my_awesome_plugin:
enabled: true
interval: 10
option1: test_value
EOF
# Run client in test mode
python -m hbd.hbc -c test_config.yaml --verbose
```
## Troubleshooting
### My plugin isn't loading
1. Check filename doesn't start with underscore
2. Verify plugin class inherits from InfoPlugin or MonitorPlugin
3. Check `initialize()` returns True
4. Look for import errors in logs
### Plugin loads but doesn't collect data
1. Check `interval` is set correctly (0 for InfoPlugin, > 0 for MonitorPlugin)
2. Verify `collect()` returns a dictionary
3. Check for exceptions in `collect()` method
4. Enable DEBUG logging to see detailed errors
### Data isn't appearing on server
1. Verify client is connected to server
2. Check server logs for PLG message handling
3. Verify returned data is JSON-serializable
4. Check for large data sizes (may exceed UDP packet size)
## Further Reading
- [Plugin Framework Source](../hbd/plugin.py) - Core plugin implementation
- [Built-in Plugins](../hbd/plugins/) - Examples of working plugins
- [Nagios Integration](NAGIOS_INTEGRATION.md) - Running external plugins
- [Configuration Guide](../hbd/config_example.yaml) - Full configuration reference
docs/THRESHOLD_ALERTING.md
+742
View File
@@ -0,0 +1,742 @@
# Threshold Alerting System
## Overview
The Heartbeat Monitoring System includes a comprehensive threshold alerting system that monitors plugin metrics and triggers notifications when values exceed configured thresholds. This system is designed to:
- **Detect anomalies**: Automatically identify when system metrics exceed safe operating ranges
- **Prevent alert fatigue**: Use hysteresis to prevent notification flapping
- **Escalate appropriately**: Support WARNING and CRITICAL severity levels
- **Track state**: Maintain alert history and state transitions per host
- **Integrate seamlessly**: Work with existing notification infrastructure (email, pushover, etc.)
## Architecture
### Components
1. **ThresholdChecker** (`hbd/threshold.py`)
- Main threshold checking engine
- Parses configuration
- Evaluates metrics against thresholds
- Triggers notifications on state changes
2. **ThresholdConfig**
- Individual threshold configuration
- Supports multiple comparison operators
- Implements hysteresis logic
3. **AlertState**
- Tracks current alert state per metric
- Records state transitions
- Manages notification timing
4. **Integration Points**
- UDP handler: Checks thresholds when plugin data arrives
- Host objects: Store alert states per host
- Notification system: Sends alerts via configured channels
### Alert Levels
- **OK**: Metric is within normal range
- **WARNING**: Metric has exceeded warning threshold (first-level concern)
- **CRITICAL**: Metric has exceeded critical threshold (requires immediate attention)
- **UNKNOWN**: Metric value cannot be evaluated (e.g., non-numeric data)
## Configuration
### Basic Structure
Thresholds are configured in the YAML configuration file under the `thresholds` section:
```yaml
thresholds:
plugin_name:
metric_name:
warning: 80.0
critical: 90.0
operator: ">"
hysteresis: 0.1
enabled: true
```
### Configuration Parameters
#### Required Parameters
- **warning**: Warning threshold value (numeric)
- **critical**: Critical threshold value (numeric)
Note: At least one of `warning` or `critical` must be specified.
#### Optional Parameters
- **operator**: Comparison operator (default: `">"`)
- `">"` - Greater than
- `">="` - Greater than or equal
- `"<"` - Less than
- `"<="` - Less than or equal
- `"=="` - Equal to
- `"!="` - Not equal to
- **hysteresis**: Hysteresis percentage to prevent flapping (default: `0.1` = 10%)
- Range: 0.0 to 1.0
- Prevents rapid state transitions when value hovers near threshold
- **enabled**: Whether this threshold is active (default: `true`)
### Comparison Operators
#### Greater Than (`>`, `>=`)
Used for metrics where **higher values are problematic**:
```yaml
cpu_monitor:
cpu_percent:
warning: 80.0 # Alert when CPU > 80%
critical: 90.0 # Alert when CPU > 90%
operator: ">"
```
Examples:
- CPU usage percentage
- Memory usage percentage
- Disk usage percentage
- Load average
- Error counters
#### Less Than (`<`, `<=`)
Used for metrics where **lower values are problematic**:
```yaml
memory_monitor:
available_mb:
warning: 1000 # Alert when available memory < 1GB
critical: 500 # Alert when available memory < 500MB
operator: "<"
```
Examples:
- Available memory
- Free disk space
- Connection pool availability
- Battery level
## Hysteresis
Hysteresis prevents alert flapping by requiring values to improve by a certain amount before recovering from an alert state.
### How It Works
When a metric crosses a threshold (e.g., CPU goes from 85% to 91%, triggering CRITICAL), hysteresis is applied when the value improves:
```
Threshold: 90
Hysteresis: 0.1 (10%)
Recovery threshold: 90 - (90 * 0.1) = 81
Value 91 -> CRITICAL (threshold crossed)
Value 89 -> CRITICAL (still above recovery threshold of 81)
Value 85 -> CRITICAL (still above recovery threshold)
Value 80 -> WARNING or OK (below recovery threshold, re-evaluated normally)
```
### Configuration Recommendations
- **Stable metrics** (CPU, memory): 10-15% hysteresis
```yaml
hysteresis: 0.1
```
- **Very stable metrics** (disk usage): 5% hysteresis
```yaml
hysteresis: 0.05
```
- **Counter metrics** (errors, packets): 20% hysteresis
```yaml
hysteresis: 0.2
```
- **Binary states** (exit codes): No hysteresis
```yaml
hysteresis: 0.0
```
## Plugin-Specific Configuration
### CPU Monitor
```yaml
cpu_monitor:
cpu_percent:
warning: 80.0
critical: 90.0
operator: ">"
hysteresis: 0.1
load_1min:
warning: 4.0
critical: 8.0
operator: ">"
hysteresis: 0.15
load_5min:
warning: 3.0
critical: 6.0
operator: ">"
load_15min:
warning: 2.0
critical: 4.0
operator: ">"
```
### Memory Monitor
```yaml
memory_monitor:
# Percentage-based threshold
percent:
warning: 85.0
critical: 95.0
operator: ">"
# Absolute value threshold (inverse - alert when LOW)
available_mb:
warning: 1000
critical: 500
operator: "<"
# Swap usage
swap_percent:
warning: 50.0
critical: 80.0
operator: ">"
```
### Disk Monitor
Disk thresholds support **partition-specific configuration**:
```yaml
disk_monitor:
partitions:
/:
percent:
warning: 80.0
critical: 90.0
operator: ">"
hysteresis: 0.05
free_gb:
warning: 10.0
critical: 5.0
operator: "<"
/home:
percent:
warning: 85.0
critical: 95.0
operator: ">"
/var:
percent:
warning: 80.0
critical: 90.0
operator: ">"
free_gb:
warning: 5.0
critical: 2.0
operator: "<"
```
### Network Monitor
```yaml
network_monitor:
# Error counters
errors_total:
warning: 100
critical: 1000
operator: ">"
hysteresis: 0.2
# Dropped packets
dropin_total:
warning: 50
critical: 200
operator: ">"
dropout_total:
warning: 50
critical: 200
operator: ">"
# Connection states
connections_TIME_WAIT:
warning: 1000
critical: 5000
operator: ">"
connections_ESTABLISHED:
warning: 500
critical: 1000
operator: ">"
```
### Nagios Runner
The Nagios plugin runner reports exit codes that can be thresholded:
```yaml
nagios_runner:
exit_code:
warning: 1 # Map Nagios WARNING to our WARNING
critical: 2 # Map Nagios CRITICAL to our CRITICAL
operator: ">="
hysteresis: 0.0 # No hysteresis for exit codes
```
## Notification Behavior
### When Notifications Are Sent
Notifications are triggered on **state changes**:
1. **Escalation**: OK → WARNING, OK → CRITICAL, WARNING → CRITICAL
```
WARNING: webserver01 - cpu_monitor.cpu_percent = 85.0
```
2. **Recovery**: CRITICAL → WARNING, CRITICAL → OK, WARNING → OK
```
RECOVERED: webserver01 - cpu_monitor.cpu_percent = 70.0 (CRITICAL -> OK)
```
3. **Re-notifications**: Periodic reminders for ongoing alerts
```
REMINDER (CRITICAL): webserver01 - cpu_monitor.cpu_percent = 95.0 (ongoing for 3600s)
```
### Notification Frequency
- **State changes**: Immediate notification
- **Re-notifications**: Controlled by `threshold_renotify_interval` (default: 3600 seconds = 1 hour)
```yaml
threshold_renotify_interval: 3600 # Re-notify every hour for ongoing alerts
```
### Notification Channels
Thresholds use the same notification infrastructure as heartbeat monitoring:
- **Email** (via SMTP)
- **Pushover** (mobile notifications)
- **Mattermost** (team chat)
- **Custom webhooks**
Configuration:
```yaml
# Email
toemail:
- admin@example.com
- oncall@example.com
fromemail: heartbeat@example.com
smtpserver: smtp.example.com
smtpport: 587
smtpuser: heartbeat@example.com
smtppassword: your-password
# Pushover
pushover_token: your-app-token
pushover_user: your-user-key
```
### Watched Hosts
Only hosts in the `watchhosts` list will trigger notifications:
```yaml
watchhosts:
- webserver01
- database01
- mailserver
```
Hosts not in this list will still have thresholds checked and alert states tracked, but won't send notifications.
## Alert State Tracking
Each host maintains alert states for all monitored metrics:
```python
host.alert_states = {
"cpu_monitor.cpu_percent": AlertState(level=WARNING, since=1234567890),
"memory_monitor.percent": AlertState(level=CRITICAL, since=1234567800),
"disk_monitor./.percent": AlertState(level=OK, since=1234567700),
}
```
Alert states persist in memory and are saved with host data (pickle).
### Alert State Information
Each `AlertState` tracks:
- **level**: Current alert level (OK, WARNING, CRITICAL, UNKNOWN)
- **since**: Timestamp when current state started
- **last_value**: Most recent metric value
- **last_check**: Timestamp of last threshold check
- **notification_count**: Number of notifications sent for this alert
- **last_notification**: Timestamp of last notification
### Querying Alert States
Via HTTP API (future enhancement):
```bash
GET /api/hosts/webserver01/alerts
```
Response:
```json
{
"active_alerts": [
{
"metric": "cpu_monitor.cpu_percent",
"level": "WARNING",
"since": 1234567890,
"value": 85.0,
"duration": 300
}
],
"summary": {
"ok": 15,
"warning": 1,
"critical": 0
}
}
```
## Testing
A comprehensive test suite is provided in `test_threshold.py`:
```bash
python test_threshold.py
```
Tests cover:
- Threshold configuration and parsing
- All comparison operators
- Hysteresis functionality
- Alert state tracking
- State change detection
- Notification triggering
- Nested metrics (partitions)
- Alert summaries
## Best Practices
### 1. Start Conservative
Begin with higher thresholds to avoid alert fatigue:
```yaml
cpu_monitor:
cpu_percent:
warning: 85.0 # Start higher
critical: 95.0 # Very high for critical
```
Adjust downward based on observed behavior.
### 2. Consider Workload Patterns
Different systems have different normal ranges:
**Web servers** (bursty traffic):
```yaml
cpu_percent:
warning: 80.0
critical: 90.0
hysteresis: 0.15 # Higher hysteresis for burstiness
```
**Database servers** (steady load):
```yaml
cpu_percent:
warning: 70.0
critical: 85.0
hysteresis: 0.1 # Lower hysteresis for steady metrics
```
### 3. Use Appropriate Operators
Match the operator to the metric:
| Metric Type | Example | Operator | Reason |
|-------------|---------|----------|--------|
| Resource usage | CPU%, Memory% | `>` | Alert when high |
| Available resources | Free memory, Free disk | `<` | Alert when low |
| Error counters | Network errors | `>` | Alert when increasing |
| Health checks | Nagios exit code | `>=` | Map to standard codes |
### 4. Align with Monitoring Intervals
Ensure threshold checks align with plugin collection intervals:
```yaml
plugins:
cpu_monitor:
interval: 300 # Check every 5 minutes
thresholds:
cpu_monitor:
cpu_percent:
warning: 80.0
# Will be checked every 5 minutes
```
### 5. Test Before Production
1. **Start with disabled thresholds**:
```yaml
enabled: false
```
2. **Observe metric ranges** over a week
3. **Set thresholds** based on observed data
4. **Enable gradually**:
```yaml
enabled: true
```
5. **Monitor for false positives**
### 6. Document Baseline Values
Keep a record of normal operating ranges:
```yaml
# Production web server baseline (observed over 30 days):
# CPU: 20-40% normal, 60% peak
# Memory: 60-70% normal, 80% peak
# Disk /: 40-50% usage, growing 2%/month
cpu_monitor:
cpu_percent:
warning: 75.0 # Above peak + margin
critical: 90.0 # Danger zone
```
### 7. Layer Alerts
Use WARNING for early notification, CRITICAL for immediate action:
```yaml
disk_monitor:
partitions:
/:
percent:
warning: 75.0 # Early warning: "check in next few days"
critical: 90.0 # Critical: "act now before outage"
```
## Troubleshooting
### No Notifications Being Sent
1. **Check if host is watched**:
```yaml
watchhosts:
- your-host-name
```
2. **Verify notification configuration**:
```yaml
toemail:
- admin@example.com
smtpserver: smtp.example.com
```
3. **Check threshold configuration**:
```bash
# Look for parsing errors in server logs
grep "threshold" /var/log/heartbeat/hbd.log
```
4. **Verify metric names**:
- Metric names must match exactly (case-sensitive)
- Check journal or logs for actual metric names
### Too Many Alerts (Flapping)
1. **Increase hysteresis**:
```yaml
hysteresis: 0.2 # Increase from 0.1 to 0.2 (20%)
```
2. **Adjust thresholds**:
```yaml
warning: 85.0 # Increase from 80.0
```
3. **Increase renotification interval**:
```yaml
threshold_renotify_interval: 7200 # 2 hours instead of 1
```
### Alerts Not Triggering
1. **Check threshold operator**:
```yaml
# For available memory (alert when LOW):
operator: "<" # NOT ">"
```
2. **Verify numeric values**:
- Ensure metric values are numeric
- Check for unit mismatches (MB vs GB)
3. **Check if threshold is enabled**:
```yaml
enabled: true # NOT false
```
4. **Review hysteresis settings**:
- Very high hysteresis may prevent state changes
- Try reducing or disabling temporarily
### Alert State Not Recovering
1. **Check recovery threshold calculation**:
```
Threshold: 90
Hysteresis: 0.1
Recovery: 90 - (90 * 0.1) = 81
Value must drop below 81 to recover
```
2. **Temporarily disable hysteresis**:
```yaml
hysteresis: 0.0
```
3. **Monitor actual metric values**:
```bash
# Check journal for actual values
grep "cpu_percent" /var/log/heartbeat/messages.journal | tail -20
```
## Advanced Topics
### Custom Notification Callbacks
The ThresholdChecker supports custom notification functions:
```python
def custom_notifier(message):
# Send to incident management system
pagerduty.trigger(message)
# Log to custom system
logger.critical(message)
# Update dashboard
metrics.alert_count.inc()
checker = ThresholdChecker(
config=config,
notification_callback=custom_notifier
)
```
### Programmatic Access
Query alert states programmatically:
```python
# Get all active alerts for a host
active = threshold_checker.get_active_alerts(host.alert_states)
for alert in active:
print(f"{alert.metric_path}: {alert.level.name} for {time.time() - alert.since}s")
# Get alert summary
summary = threshold_checker.get_alert_summary(host.alert_states)
print(f"WARNING: {summary['warning']}, CRITICAL: {summary['critical']}")
```
### Integration with External Systems
Threshold violations can be integrated with:
- **PagerDuty**: Incident creation and escalation
- **OpsGenie**: On-call scheduling and routing
- **ServiceNow**: Ticket creation
- **Grafana**: Dashboard annotations
- **Elasticsearch**: Alert indexing and analysis
## Future Enhancements
Planned features:
1. **Composite thresholds**: Alert based on multiple metrics
```yaml
composite:
high_load_with_low_memory:
conditions:
- cpu_monitor.load_1min > 8.0
- memory_monitor.available_mb < 500
```
2. **Time-based thresholds**: Different thresholds by time of day
```yaml
schedule:
business_hours:
warning: 70.0
off_hours:
warning: 85.0
```
3. **Rate-of-change thresholds**: Alert on rapid changes
```yaml
rate_of_change:
metric: cpu_percent
period: 300
threshold: 30.0 # Alert if changes >30% in 5 minutes
```
4. **Alert grouping**: Combine related alerts
```yaml
groups:
disk_critical:
metrics:
- disk_monitor./.percent
- disk_monitor./var.percent
action: single_notification
```
5. **Maintenance windows**: Suppress alerts during planned maintenance
```yaml
maintenance:
- host: webserver01
start: 2024-01-15T02:00:00Z
end: 2024-01-15T04:00:00Z
```
## See Also
- [Plugin Development Guide](PLUGIN_DEVELOPMENT.md)
- [Message Journal Documentation](MESSAGE_JOURNAL.md)
- Configuration examples: `hbd/config_thresholds_example.yaml`
- Test suite: `test_threshold.py`
hbd/Plan.md
+9
View File
@@ -0,0 +1,9 @@
Plan
Heartbeat is a client/server based network monitor and host observer. hbd, the server portion receives heartbeat and state messages from clients and maintaines state and hisgtory of the informations it receives.
hbc, the client portion gathers information on various aspects of the
system it is running on, and sends it to hbd. Initially this info is basic, like OS make and version, hardware info (CPU type, memory and disks), fileystem info and some resource info. hbc/hbd support a plugin system to extend the info gathered and stored.
hbd also can send notification based on missed hbc updates, and on violation of pre-set limits for various state paramaters.
hbd/__init__.py
+13 -7
View File
@@ -1,11 +1,17 @@
"""hbd package - scaffolding for heartbeat daemon
"""hbd package - heartbeat monitoring system
This package contains the refactored modules for the original monolithic
`hbd` script. The initial implementation contains small scaffolds so you can
start moving functionality into the package.
This package contains both the heartbeat client (hbc) and server (hbd) components,
organized into separate subpackages:
- hbd.client: Client component with system monitoring plugins
- hbd.server: Server/daemon component with web UI and notifications
- hbd.common: Shared utilities and protocol definitions
Install options:
- pip install hbd[client] # Client only
- pip install hbd[server] # Server only
- pip install hbd[all] # Both client and server
"""
__all__ = ["main", "__version__"]
__all__ = ["__version__"]
__version__ = "5.0.5"
from .cli import main
hbd/client/__init__.py
+3
View File
@@ -0,0 +1,3 @@
"""HeartBeat Client (hbc) - System monitoring client."""
__version__ = "5.0.5"
hbd/client/config.py
+54
View File
@@ -0,0 +1,54 @@
"""Configuration loader and defaults for hbc (HeartBeat Client)."""
import logging
import os
try:
import yaml
except Exception:
yaml = None
CLIENT_DEFAULTS = {
# Network settings
"hb_port": 50003, # Port where hbd servers listen
"interval": 10, # Heartbeat interval in seconds
# Runtime flags
"foreground": False,
"verbose": False,
"debug": 0,
# Plugin configuration
"plugins": {}, # Per-plugin configuration
"thresholds": {}, # Threshold configuration for monitoring
}
def load_config(path=None):
"""Load configuration from a YAML file and merge with client defaults.
If YAML is not available or the file does not exist, defaults are returned.
Args:
path: Path to YAML config file (default: ~/.hb.yaml)
Returns:
Dictionary with configuration
"""
cfg = CLIENT_DEFAULTS.copy()
if not path:
# default path (~/.hb.yaml)
path = os.path.join(os.path.expanduser("~"), ".hb.yaml")
if os.path.exists(path):
if yaml:
with open(path) as fh:
data = yaml.safe_load(fh)
# Merge YAML data with defaults
# Keep all keys from YAML to support plugin configs and future extensions
for k, v in data.items():
cfg[k] = v
else:
# yaml not installed: do not attempt to parse; user must ensure defaults
pass
return cfg
hbd/client/main.py
+643
View File
@@ -0,0 +1,643 @@
#!/usr/bin/env python3
"""
HeartBeat Client (hbc) - Async version with plugin support.
Sends heartbeat messages to HeartBeat Daemon (hbd) servers and collects
system information via plugins.
"""
import argparse
import asyncio
import logging
import os
import signal
import socket
import sys
import time
from hashlib import md5
from pathlib import Path
from typing import Dict, List, Optional
# Import protocol and config
from .config import load_config
from ..common.proto import dicttos, stodict
# Import plugin system
from .plugin import PluginRegistry, PluginLoader, InfoPlugin, MonitorPlugin
# Constants
PORT = 50003
INTERVAL = 10
VER = 6
MAXRECV = 32767
# Global state
running = True
dorestart = False
class AsyncConnection:
"""Async UDP connection to a heartbeat server."""
def __init__(self, conn_id: int, addr: str, port: int, af: int, name: str):
self.conn_id = conn_id
self.addr = addr
self.port = port
self.af = af
self.name = name
self.ackcount = 0
self.lastack = 0.0
self.send_count = 0
self.lastsend = 0.0
self.rtts = [0.0]
self.transport: Optional[asyncio.DatagramTransport] = None
self.protocol: Optional[asyncio.DatagramProtocol] = None
self.logger = logging.getLogger(f"hbc.conn.{addr}")
async def open(self) -> bool:
"""Open the UDP connection.
Returns:
True if successful, False otherwise
"""
try:
loop = asyncio.get_event_loop()
# Create datagram endpoint
self.transport, self.protocol = await loop.create_datagram_endpoint(
lambda: HeartbeatProtocol(self),
family=self.af
)
self.logger.debug(f"Opened connection to {self.addr}:{self.port}")
return True
except Exception as e:
self.logger.error(f"Failed to open connection: {e}")
return False
def close(self):
"""Close the connection."""
if self.transport:
self.transport.close()
self.transport = None
self.protocol = None
async def sendto(self, msg: dict, msg_id: str = "HTB"):
"""Send a message to the server.
Args:
msg: Message dictionary
msg_id: Message ID (HTB, PLG, etc.)
"""
if not self.transport:
await self.open()
if not self.transport:
self.logger.error("Cannot send - no transport")
return
# Add standard fields
msg["name"] = shortname(self.name)
msg["id"] = self.conn_id
msg["ver"] = VER
msg["time"] = time.time()
# Encode message
data = dicttos(msg_id, msg, compress=True)
# Send
self.transport.sendto(data, (self.addr, self.port))
self.send_count += 1
self.lastsend = time.time()
self.logger.debug(f"Sent {msg_id} message ({len(data)} bytes)")
def handle_ack(self, msg: dict, now: float):
"""Handle ACK message from server."""
try:
self.lastack = msg.get("time", now)
rtt = (self.lastack - self.lastsend) * 2000.0 # Convert to ms
except Exception:
self.lastack = now
rtt = (self.lastack - self.lastsend) * 1000.0
self.rtts.append(rtt)
if len(self.rtts) > 10:
self.rtts.pop(0)
self.ackcount += 1
self.logger.debug(f"ACK received, RTT: {rtt:.1f}ms")
class HeartbeatProtocol(asyncio.DatagramProtocol):
"""Protocol handler for incoming UDP messages."""
def __init__(self, connection: AsyncConnection):
self.connection = connection
self.logger = logging.getLogger("hbc.protocol")
def datagram_received(self, data: bytes, addr):
"""Handle incoming datagram."""
try:
msg = stodict(data)
if not msg:
self.logger.warning(f"Failed to parse message from {addr}")
return
now = time.time()
msg_id = msg.get("ID")
if msg_id == "ACK":
self.connection.handle_ack(msg, now)
elif msg_id == "CMD":
# Command from server
asyncio.create_task(handle_command(self.connection, msg))
elif msg_id == "UPD":
# Update from server
asyncio.create_task(handle_update(self.connection, msg))
else:
self.logger.warning(f"Unknown message type: {msg_id}")
except Exception as e:
self.logger.error(f"Error processing datagram: {e}", exc_info=True)
def error_received(self, exc):
"""Handle protocol errors."""
self.logger.error(f"Protocol error: {exc}")
async def handle_command(conn: AsyncConnection, msg: dict):
"""Execute a command received from server."""
import subprocess
cmd = msg.get("cmd", "")
if not cmd:
return
logger = logging.getLogger("hbc.command")
logger.info(f"Executing command: {cmd}")
try:
result = subprocess.check_output(
cmd, shell=True, stderr=subprocess.STDOUT, timeout=30
).decode()
status = "OK"
except subprocess.CalledProcessError as e:
result = str(e)
status = "CalledProcessError"
except subprocess.TimeoutExpired:
result = "Command timed out"
status = "Timeout"
except Exception as e:
result = str(e)
status = "Error"
# Send response
response = {
"service": "command",
"msg": f"{status} {result}"
}
await conn.sendto(response)
async def handle_update(conn: AsyncConnection, msg: dict):
"""Handle self-update from server."""
import codecs
import shutil
logger = logging.getLogger("hbc.update")
try:
code = codecs.decode(msg["code"], "base64").decode()
csum = msg["csum"]
except Exception as e:
error = f"Missing code/csum: {e}"
logger.error(error)
await conn.sendto({"service": "update", "msg": error})
return
# Verify checksum
m = md5()
m.update(code.encode())
if m.hexdigest() != csum:
error = "Checksum mismatch"
logger.error(error)
await conn.sendto({"service": "update", "msg": error})
return
# Backup current file
fn = sys.argv[0]
ofn = f"{fn}.sav"
try:
shutil.copy2(fn, ofn)
except Exception as e:
error = f"Backup failed: {e}"
logger.error(error)
await conn.sendto({"service": "update", "msg": error})
return
# Write new code
try:
with open(fn, "w") as fh:
fh.write(code)
except Exception as e:
error = f"Write failed: {e}"
logger.error(error)
await conn.sendto({"service": "update", "msg": error})
return
logger.info("Update successful, restart required")
await conn.sendto({"service": "update", "msg": "OK"})
# Trigger restart
global dorestart
dorestart = True
stop()
async def heartbeat_sender(conn: AsyncConnection, interval: int):
"""Send periodic heartbeats.
Args:
conn: Connection to send on
interval: Heartbeat interval in seconds
"""
logger = logging.getLogger("hbc.heartbeat")
while running:
try:
msg = {
"acks": conn.ackcount,
"rtt": conn.rtts[-1],
"interval": interval
}
await conn.sendto(msg, "HTB")
except Exception as e:
logger.error(f"Error sending heartbeat: {e}", exc_info=True)
# Wait for next interval
await asyncio.sleep(interval)
async def plugin_collector(conn: AsyncConnection, registry: PluginRegistry):
"""Collect and send plugin data.
Args:
conn: Connection to send on
registry: Plugin registry
"""
logger = logging.getLogger("hbc.plugins")
# Collect InfoPlugins once at startup
info_plugins = registry.get_by_type(InfoPlugin)
for plugin in info_plugins:
try:
data = await plugin.collect()
if data:
# Create PLG message with plugin name
plugin_msg = {"plugin": plugin.name, **data}
await conn.sendto(plugin_msg, "PLG")
logger.info(f"Sent {plugin.name} data")
except Exception as e:
logger.error(f"Error collecting {plugin.name}: {e}", exc_info=True)
# Schedule MonitorPlugins
# Group plugins by interval
from collections import defaultdict
by_interval = defaultdict(list)
monitor_plugins = registry.get_by_type(MonitorPlugin)
for plugin in monitor_plugins:
by_interval[plugin.interval].append(plugin)
# Create tasks for each interval
tasks = []
for interval, plugins in by_interval.items():
task = asyncio.create_task(
plugin_collector_interval(conn, plugins, interval)
)
tasks.append(task)
# Wait for all tasks
if tasks:
await asyncio.gather(*tasks)
async def plugin_collector_interval(
conn: AsyncConnection,
plugins: List,
interval: int
):
"""Collect plugins on a specific interval.
Args:
conn: Connection to send on
plugins: List of plugins to collect
interval: Collection interval in seconds
"""
logger = logging.getLogger(f"hbc.plugins.{interval}s")
while running:
for plugin in plugins:
try:
data = await plugin.collect()
if data:
# Don't use encode_plugin_data - create dict directly
plugin_msg = {"plugin": plugin.name, **data}
await conn.sendto(plugin_msg, "PLG")
logger.debug(f"Sent {plugin.name} data")
except Exception as e:
logger.error(
f"Error collecting {plugin.name}: {e}",
exc_info=True
)
await asyncio.sleep(interval)
def shortname(name: str) -> str:
"""Extract short hostname."""
return name.split(".")[0]
def stop():
"""Stop the event loop."""
global running
running = False
async def cleanup(connections: List[AsyncConnection]):
"""Cleanup connections on shutdown."""
logger = logging.getLogger("hbc.cleanup")
logger.info("Cleaning up connections")
for conn in connections:
try:
msg = {
"shutdown": 1,
"acks": conn.ackcount
}
await conn.sendto(msg)
except Exception as e:
logger.error(f"Error sending shutdown: {e}")
conn.close()
# Give messages time to send
await asyncio.sleep(0.5)
async def async_main(args, config):
"""Async main function."""
global running
logger = logging.getLogger("hbc.main")
# Setup
iam = socket.gethostname()
if args.name:
iam = args.name
hb_hosts = args.hosts
hb_port = config.get("hb_port", PORT)
interval = config.get("interval", INTERVAL)
logger.info(f"Starting hbc for {iam} -> {hb_hosts}")
logger.info(f"Port: {hb_port}, Interval: {interval}s")
# Create connections
connections = []
conn_id = 1
for host in hb_hosts:
try:
addrs = socket.getaddrinfo(host, hb_port, 0, 0, socket.SOL_UDP)
except socket.gaierror as e:
logger.error(f"Cannot resolve {host}: {e}")
continue
for addr_info in addrs:
af = addr_info[0]
addr = addr_info[4][0]
conn = AsyncConnection(conn_id, addr, hb_port, af, iam)
if await conn.open():
connections.append(conn)
conn_id += 1
if not connections:
logger.error("No connections established")
return 1
logger.info(f"Created {len(connections)} connections")
# Send boot/message if requested
if args.boot or args.message:
boot_msg = {}
if args.boot:
boot_msg["boot"] = 1
if args.message:
boot_msg["service"] = "service"
boot_msg["msg"] = args.message
boot_msg["acks"] = 0
for conn in connections:
await conn.sendto(boot_msg)
if args.message and not args.daemon:
# Message-only mode
await cleanup(connections)
return 0
# Load plugins
registry = PluginRegistry()
loader = PluginLoader(registry)
plugin_dir = Path(__file__).parent / "plugins"
if plugin_dir.exists():
count = await loader.load_from_directory(plugin_dir, config)
logger.info(f"Loaded {count} plugins")
else:
logger.warning(f"Plugin directory not found: {plugin_dir}")
# Start async tasks
tasks = []
# Heartbeat senders (one per connection)
for conn in connections:
task = asyncio.create_task(heartbeat_sender(conn, interval))
tasks.append(task)
# Plugin collector (uses all connections, but we'll use first one)
if connections and registry.get_enabled():
task = asyncio.create_task(plugin_collector(connections[0], registry))
tasks.append(task)
# Setup signal handlers
loop = asyncio.get_event_loop()
for sig in (signal.SIGTERM, signal.SIGINT):
loop.add_signal_handler(sig, stop)
# Wait for stop or tasks to complete
try:
await asyncio.gather(*tasks)
except asyncio.CancelledError:
pass
# Cleanup
await cleanup(connections)
await loader.unload_all()
return 0
def daemonize(
working_dir="/",
stdin="/dev/zero",
stdout="/dev/null",
stderr="/dev/null"
):
"""UNIX double-fork daemonization."""
try:
pid = os.fork()
if pid > 0:
os._exit(0)
except OSError as e:
sys.stderr.write(f"fork #1 failed: {e}\n")
os._exit(1)
os.chdir(working_dir)
os.setsid()
os.umask(0)
try:
pid = os.fork()
if pid > 0:
os._exit(0)
except OSError as e:
sys.stderr.write(f"fork #2 failed: {e}\n")
sys.exit(1)
sys.stdout.flush()
sys.stderr.flush()
si = open(stdin, "r")
so = open(stdout, "a+")
se = open(stderr, "a+")
os.dup2(si.fileno(), sys.stdin.fileno())
os.dup2(so.fileno(), sys.stdout.fileno())
os.dup2(se.fileno(), sys.stderr.fileno())
def build_parser():
"""Build argument parser."""
parser = argparse.ArgumentParser(
prog="hbc",
description="HeartBeatClient - send heartbeat messages to HeartBeatDaemon",
formatter_class=argparse.RawDescriptionHelpFormatter,
)
parser.add_argument(
"-b", "--boot",
action="store_true",
help="Send a boot message"
)
parser.add_argument(
"-c", "--config",
dest="configfile",
help="Config file path (YAML)"
)
parser.add_argument(
"-m", "--message",
dest="message",
help="Send a message"
)
parser.add_argument(
"-n", "--name",
dest="name",
help="Name to use in heartbeat message"
)
parser.add_argument(
"-d", "--daemon",
action="store_true",
help="Run in daemon mode"
)
parser.add_argument(
"-v", "--verbose",
action="store_true",
help="Verbose output"
)
parser.add_argument(
"-x", "--debug",
action="count",
default=0,
help="Increase debug level"
)
parser.add_argument(
"hosts",
nargs="+",
help="Heartbeat daemon hosts to send to"
)
return parser
def main(argv=None):
"""Main entry point."""
global running, dorestart
parser = build_parser()
args = parser.parse_args(argv)
# Load config
config = load_config(args.configfile)
# Setup logging
log_level = logging.INFO
if args.verbose:
log_level = logging.DEBUG
if args.debug:
log_level = logging.DEBUG
logging.basicConfig(
level=log_level,
format="%(asctime)s %(name)s %(levelname)s: %(message)s",
datefmt="%Y-%m-%d %H:%M:%S"
)
# Daemonize if requested
if args.daemon:
print("Daemonizing...")
import syslog
syslog.openlog("hbc", syslog.LOG_PID, syslog.LOG_DAEMON)
syslog.syslog(syslog.LOG_INFO, f"Starting heartbeat to {', '.join(args.hosts)}")
daemonize()
# Reconfigure logging for syslog
logging.basicConfig(
level=log_level,
format="hbc[%(process)d]: %(name)s %(levelname)s: %(message)s"
)
# Run async main
try:
exit_code = asyncio.run(async_main(args, config))
except KeyboardInterrupt:
logging.info("Interrupted by user")
exit_code = 0
except Exception as e:
logging.error(f"Fatal error: {e}", exc_info=True)
exit_code = 1
# Handle restart
if dorestart:
logging.info("Restarting...")
os.execv(sys.argv[0], sys.argv)
sys.exit(exit_code)
if __name__ == "__main__":
main()
hbd/client/plugin.py
+410
View File
@@ -0,0 +1,410 @@
"""Plugin system for extending Heartbeat data collection and monitoring.
This module provides the base classes and infrastructure for the plugin system
that enables extending hbc (client) data collection and hbd (server) processing.
Plugin Types:
- InfoPlugin: Collects static or rarely-changing information (OS, hardware)
- MonitorPlugin: Collects periodic monitoring data (CPU, memory, disk usage)
Plugins run on the client (hbc) to gather data, which is then sent to the server
(hbd) for storage, threshold checking, and display.
"""
import importlib.util
import inspect
import logging
import sys
from abc import ABC, abstractmethod
from pathlib import Path
from typing import Any, Dict, List, Optional, Type
class Plugin(ABC):
"""Base class for all plugins.
Attributes:
name: Unique plugin identifier (e.g., "os_info", "cpu_monitor")
version: Plugin version string
description: Human-readable description
interval: Collection interval in seconds (0 for InfoPlugin = collect once)
enabled: Whether plugin is active (can be disabled via config)
"""
name: str = ""
version: str = "1.0.0"
description: str = ""
interval: int = 0
enabled: bool = True
def __init__(self, config: Optional[Dict[str, Any]] = None):
"""Initialize plugin with optional configuration.
Args:
config: Plugin-specific configuration from YAML (e.g., thresholds, paths)
"""
self.config = config or {}
self.logger = logging.getLogger(f"plugin.{self.name}")
self._initialized = False
@abstractmethod
async def initialize(self) -> bool:
"""Initialize plugin (load resources, check dependencies).
Called once when plugin is loaded. Plugins should validate dependencies
(e.g., check if psutil is available) and prepare any resources.
Returns:
True if initialization succeeded, False otherwise
"""
pass
@abstractmethod
async def collect(self) -> Dict[str, Any]:
"""Collect data from the system.
This is the main method called on each collection interval. Should return
a dictionary of key-value pairs representing the collected data.
Keys should be strings (metric names). Values can be:
- Scalars: int, float, str, bool
- Lists/dicts (will be serialized appropriately)
Returns:
Dictionary of collected metrics, or empty dict on error
"""
pass
async def cleanup(self) -> None:
"""Cleanup plugin resources before shutdown.
Called when plugin is being unloaded or on system shutdown.
Override to release resources, close connections, etc.
"""
pass
def validate_data(self, data: Dict[str, Any]) -> bool:
"""Validate collected data before sending to server.
Override to implement custom validation logic.
Args:
data: Data returned from collect()
Returns:
True if data is valid, False otherwise
"""
return isinstance(data, dict)
class InfoPlugin(Plugin):
"""Plugin for collecting static or rarely-changing information.
InfoPlugins collect data that doesn't change frequently:
- OS name and version
- Hardware specifications (CPU model, RAM size)
- Network interface MAC addresses
Characteristics:
- interval = 0 (collected once at startup by default)
- Can specify interval > 0 for periodic refresh (e.g., check for hardware changes)
- Data is cached and reused until next collection
"""
interval: int = 0 # Collect once at startup
def __init__(self, config: Optional[Dict[str, Any]] = None):
super().__init__(config)
self._cached_data: Optional[Dict[str, Any]] = None
async def get_cached_data(self) -> Optional[Dict[str, Any]]:
"""Get cached data if available (avoids re-collection).
Returns:
Cached data dict, or None if not yet collected
"""
return self._cached_data
async def collect(self) -> Dict[str, Any]:
"""Collect and cache static information."""
if self._cached_data is None:
self._cached_data = await self._collect_info()
return self._cached_data
@abstractmethod
async def _collect_info(self) -> Dict[str, Any]:
"""Internal method to perform actual data collection.
Override this method instead of collect() for InfoPlugins.
"""
pass
def invalidate_cache(self) -> None:
"""Force re-collection on next collect() call."""
self._cached_data = None
class MonitorPlugin(Plugin):
"""Plugin for collecting periodic monitoring data.
MonitorPlugins collect time-series metrics that change frequently:
- CPU usage percentage
- Memory consumption
- Disk I/O statistics
- Network traffic
Characteristics:
- interval > 0 (e.g., 30 seconds for CPU, 60 for disk)
- Collected continuously on schedule
- Data includes timestamps for time-series tracking
"""
interval: int = 30 # Default: collect every 30 seconds
def __init__(self, config: Optional[Dict[str, Any]] = None):
super().__init__(config)
self._last_reading: Optional[Dict[str, Any]] = None
def get_last_reading(self) -> Optional[Dict[str, Any]]:
"""Get the last collected reading.
Returns:
Last reading dict with timestamp, or None if not yet collected
"""
return self._last_reading
async def collect(self) -> Dict[str, Any]:
"""Collect monitoring data and store as last reading."""
data = await self._collect_metrics()
if data:
# Add collection timestamp
import time
data['_timestamp'] = time.time()
self._last_reading = data
return data
@abstractmethod
async def _collect_metrics(self) -> Dict[str, Any]:
"""Internal method to perform actual metric collection.
Override this method instead of collect() for MonitorPlugins.
"""
pass
class PluginRegistry:
"""Registry for managing loaded plugins.
Maintains a collection of loaded plugins and provides methods to
query plugins by name, type, or interval.
"""
def __init__(self):
self._plugins: Dict[str, Plugin] = {}
self.logger = logging.getLogger("plugin.registry")
def register(self, plugin: Plugin) -> bool:
"""Register a plugin instance.
Args:
plugin: Plugin instance to register
Returns:
True if registered successfully, False if name conflict
"""
if plugin.name in self._plugins:
self.logger.error(f"Plugin '{plugin.name}' already registered")
return False
self._plugins[plugin.name] = plugin
self.logger.info(f"Registered plugin: {plugin.name} v{plugin.version}")
return True
def unregister(self, name: str) -> bool:
"""Unregister a plugin by name.
Args:
name: Plugin name to unregister
Returns:
True if unregistered, False if not found
"""
if name in self._plugins:
del self._plugins[name]
self.logger.info(f"Unregistered plugin: {name}")
return True
return False
def get(self, name: str) -> Optional[Plugin]:
"""Get plugin by name.
Args:
name: Plugin name
Returns:
Plugin instance or None if not found
"""
return self._plugins.get(name)
def get_all(self) -> List[Plugin]:
"""Get all registered plugins."""
return list(self._plugins.values())
def get_enabled(self) -> List[Plugin]:
"""Get all enabled plugins."""
return [p for p in self._plugins.values() if p.enabled]
def get_by_type(self, plugin_type: Type[Plugin]) -> List[Plugin]:
"""Get all plugins of a specific type.
Args:
plugin_type: Plugin class (InfoPlugin or MonitorPlugin)
Returns:
List of plugins matching the type
"""
return [p for p in self._plugins.values() if isinstance(p, plugin_type)]
def get_by_interval(self, interval: int) -> List[Plugin]:
"""Get all plugins with a specific collection interval.
Args:
interval: Interval in seconds (0 for one-time collection)
Returns:
List of plugins with matching interval
"""
return [p for p in self._plugins.values() if p.interval == interval]
class PluginLoader:
"""Load plugins from filesystem and instantiate them.
Scans plugin directories for Python modules containing Plugin subclasses,
loads them dynamically, and registers them with the PluginRegistry.
"""
def __init__(self, registry: PluginRegistry):
self.registry = registry
self.logger = logging.getLogger("plugin.loader")
self._loaded_modules: Dict[str, Any] = {}
async def load_from_directory(
self,
directory: Path,
config: Optional[Dict[str, Any]] = None
) -> int:
"""Load all plugins from a directory.
Scans for .py files, imports them, finds Plugin subclasses,
instantiates them with config, initializes, and registers.
Args:
directory: Path to plugin directory
config: Configuration dict (may contain per-plugin config)
Returns:
Number of plugins successfully loaded
"""
if not directory.exists() or not directory.is_dir():
self.logger.warning(f"Plugin directory not found: {directory}")
return 0
loaded_count = 0
plugin_config = config or {}
# Scan for Python files
for plugin_file in directory.glob("*.py"):
if plugin_file.name.startswith("_"):
continue # Skip __init__.py and private modules
self.logger.debug(f"Processing plugin file: {plugin_file.name}")
try:
# Load module dynamically
module_name = f"plugins.{plugin_file.stem}"
spec = importlib.util.spec_from_file_location(module_name, plugin_file)
if not spec or not spec.loader:
self.logger.warning(f"Could not create spec for {plugin_file}")
continue
module = importlib.util.module_from_spec(spec)
sys.modules[module_name] = module
spec.loader.exec_module(module)
self._loaded_modules[module_name] = module
self.logger.debug(f"Loaded module: {module_name}")
# Track which plugin classes we've already processed to avoid duplicates
processed_classes = set()
# Find Plugin subclasses in module
for name, obj in inspect.getmembers(module, inspect.isclass):
# Skip base classes and non-Plugin classes
if obj in (Plugin, InfoPlugin, MonitorPlugin):
self.logger.debug(f"Skipping base class: {name}")
continue
if not issubclass(obj, Plugin):
self.logger.debug(f"Skipping non-Plugin class: {name}")
continue
# Skip if we've already processed this class (handles module-level aliases)
if id(obj) in processed_classes:
self.logger.debug(f"Skipping duplicate reference to: {obj.__name__}")
continue
processed_classes.add(id(obj))
self.logger.debug(f"Found plugin class: {name}")
# Instantiate plugin with config
plugin_instance_config = plugin_config.get(obj.name, {})
plugin = obj(config=plugin_instance_config)
# Initialize plugin
try:
initialized = await plugin.initialize()
if not initialized:
self.logger.warning(
f"Plugin {plugin.name} failed initialization, skipping"
)
continue
except Exception as e:
self.logger.error(
f"Error initializing plugin {plugin.name}: {e}",
exc_info=True
)
continue
# Register with registry
if self.registry.register(plugin):
loaded_count += 1
self.logger.info(
f"Loaded plugin: {plugin.name} v{plugin.version} "
f"(interval: {plugin.interval}s)"
)
except Exception as e:
self.logger.error(
f"Error loading plugin from {plugin_file}: {e}",
exc_info=True
)
return loaded_count
async def unload_all(self) -> None:
"""Unload all plugins and cleanup resources."""
for plugin in self.registry.get_all():
try:
await plugin.cleanup()
except Exception as e:
self.logger.error(
f"Error cleaning up plugin {plugin.name}: {e}",
exc_info=True
)
self.registry.unregister(plugin.name)
# Remove loaded modules
for module_name in self._loaded_modules:
if module_name in sys.modules:
del sys.modules[module_name]
self._loaded_modules.clear()
hbd/client/plugins/cpu_monitor.py
+129
View File
@@ -0,0 +1,129 @@
"""CPU Monitoring Plugin for Heartbeat.
Collects CPU usage statistics including overall CPU percentage, per-core usage,
load average, and process counts.
"""
from typing import Any, Dict, Optional
import sys
from pathlib import Path
# Import from parent package
from hbd.client.plugin import MonitorPlugin
class CPUMonitorPlugin(MonitorPlugin):
"""Monitor CPU usage and load.
Collects:
- Overall CPU usage percentage
- Per-core CPU usage (if enabled in config)
- Load average (1min, 5min, 15min)
- Process count
- CPU frequency (if available)
"""
name = "cpu_monitor"
version = "1.0.0"
description = "CPU usage and load monitoring"
interval = 300 # MonitorPlugin: collect every 5 minutes by default
def __init__(self, config: Optional[Dict[str, Any]] = None):
super().__init__(config)
self.psutil = None
self.per_core = config.get("per_core", False) if config else False
self.interval = config.get("interval", 300) if config else 300
async def initialize(self) -> bool:
"""Initialize the CPU monitor plugin.
Checks if psutil is available.
Returns:
True if psutil is available, False otherwise
"""
self.logger.info(f"Initializing {self.name} plugin")
try:
import psutil
self.psutil = psutil
self.logger.info(f"{self.name} initialized successfully")
return True
except ImportError:
self.logger.error(
"psutil module not available. Install with: pip install psutil"
)
return False
async def _collect_metrics(self) -> Dict[str, Any]:
"""Collect CPU metrics.
Returns:
Dictionary with CPU metrics
"""
if not self.psutil:
return {}
try:
data = {}
# Overall CPU usage percentage (non-blocking, interval=1 for accuracy)
# Note: first call to cpu_percent() returns 0.0, subsequent calls work correctly
data["cpu_percent"] = self.psutil.cpu_percent(interval=1)
# Per-core CPU usage (if enabled)
if self.per_core:
per_core_percents = self.psutil.cpu_percent(interval=0, percpu=True)
data["cpu_per_core"] = per_core_percents
data["cpu_core_count"] = len(per_core_percents)
else:
# Just report core count
data["cpu_core_count"] = self.psutil.cpu_count()
# Load average (Unix-like systems only)
try:
load_avg = self.psutil.getloadavg()
data["load_1min"] = round(load_avg[0], 2)
data["load_5min"] = round(load_avg[1], 2)
data["load_15min"] = round(load_avg[2], 2)
except (AttributeError, OSError):
# Not available on Windows
pass
# Process count
try:
data["process_count"] = len(self.psutil.pids())
except Exception as e:
self.logger.warning(f"Could not get process count: {e}")
# CPU frequency (if available)
try:
freq = self.psutil.cpu_freq()
if freq:
data["cpu_freq_current"] = round(freq.current, 2)
data["cpu_freq_min"] = round(freq.min, 2)
data["cpu_freq_max"] = round(freq.max, 2)
except (AttributeError, OSError, RuntimeError, SystemError) as e:
# Not available on all systems, or may fail on FreeBSD with sysctl issues
self.logger.debug(f"CPU frequency not available: {e}")
pass
# CPU times (user, system, idle, etc.)
try:
cpu_times = self.psutil.cpu_times_percent(interval=0)
data["cpu_user"] = round(cpu_times.user, 1)
data["cpu_system"] = round(cpu_times.system, 1)
data["cpu_idle"] = round(cpu_times.idle, 1)
if hasattr(cpu_times, "iowait"):
data["cpu_iowait"] = round(cpu_times.iowait, 1)
except Exception as e:
self.logger.debug(f"Could not get CPU times: {e}")
self.logger.debug(
f"Collected CPU metrics: {data.get('cpu_percent', 'N/A')}% usage"
)
return data
except Exception as e:
self.logger.error(f"Error collecting CPU metrics: {e}", exc_info=True)
return {}
hbd/client/plugins/disk_monitor.py
+199
View File
@@ -0,0 +1,199 @@
"""
Disk monitoring plugin for Heartbeat.
Collects disk usage and I/O statistics using psutil.
"""
import logging
from typing import Dict, Any, Optional, List
try:
import psutil
except ImportError:
psutil = None
from hbd.client.plugin import MonitorPlugin
logger = logging.getLogger(__name__)
class DiskMonitorPlugin(MonitorPlugin):
"""
Monitor disk usage and I/O statistics.
Collects:
- Disk partition information
- Disk usage per partition (total, used, free, percent)
- Disk I/O counters (read/write bytes, read/write count)
- Disk I/O time statistics
Configuration:
interval: Collection interval in seconds (default: 300)
partitions: List of mount points to monitor (default: all)
include_io: Include disk I/O statistics (default: True)
exclude_types: List of filesystem types to exclude (default: tmpfs, devtmpfs, squashfs)
"""
name = "disk_monitor"
interval = 300 # Collect every 5 minutes by default
def __init__(self, config: Optional[Dict[str, Any]] = None):
"""
Initialize the disk monitor plugin.
Args:
config: Optional configuration dict with keys:
- interval: Collection interval in seconds (default: 300)
- partitions: List of specific mount points to monitor
- include_io: Include I/O statistics (default: True)
- exclude_types: List of filesystem types to exclude
"""
super().__init__(config)
self.partitions = self.config.get('partitions', None) # None = all partitions
self.include_io = self.config.get('include_io', True)
self.exclude_types = set(self.config.get('exclude_types', ['tmpfs', 'devtmpfs', 'squashfs']))
self.interval = self.config.get('interval', 300)
if psutil is None:
raise ImportError("psutil library is required for disk_monitor plugin")
# Store previous I/O counters for delta calculation
self._prev_io = {}
async def initialize(self):
"""Initialize the plugin (check psutil availability)."""
if psutil is None:
logger.error("psutil not available - disk_monitor cannot run")
return False
logger.info(f"Disk monitor initialized (interval: {self.interval}s, io: {self.include_io})")
# Initialize I/O counters if available
if self.include_io:
try:
self._prev_io = psutil.disk_io_counters(perdisk=True)
except Exception as e:
logger.warning(f"Could not initialize disk I/O counters: {e}")
return True
async def collect(self) -> Dict[str, Any]:
"""
Collect current disk statistics.
Returns:
Dictionary with disk metrics organized by partition:
- partitions: Dict of partition data, keyed by mount point
- device: Device name (e.g., /dev/sda1)
- fstype: Filesystem type (e.g., ext4)
- total: Total space in bytes
- used: Used space in bytes
- free: Free space in bytes
- percent: Usage percentage
- io_counters: Dict of I/O statistics, keyed by disk name (if include_io)
- read_count: Number of reads
- write_count: Number of writes
- read_bytes: Bytes read
- write_bytes: Bytes written
- read_time: Time spent reading in ms
- write_time: Time spent writing in ms
- read_bytes_delta: Bytes read since last collection
- write_bytes_delta: Bytes written since last collection
"""
if psutil is None:
logger.error("psutil not available")
return {}
try:
data = await self._collect_metrics()
logger.debug(f"Collected disk metrics: {len(data.get('partitions', {}))} partitions")
return data
except Exception as e:
logger.error(f"Error collecting disk metrics: {e}")
return {"error": str(e)}
async def _collect_metrics(self) -> Dict[str, Any]:
"""Collect disk metrics from psutil."""
metrics = {}
# Collect partition usage
partitions_data = {}
partitions = psutil.disk_partitions(all=False)
for partition in partitions:
# Skip unwanted filesystem types
if partition.fstype in self.exclude_types:
continue
# Skip if we're only monitoring specific partitions
if self.partitions and partition.mountpoint not in self.partitions:
continue
try:
usage = psutil.disk_usage(partition.mountpoint)
partitions_data[partition.mountpoint] = {
'device': partition.device,
'fstype': partition.fstype,
'total': usage.total,
'used': usage.used,
'free': usage.free,
'percent': usage.percent
}
except PermissionError:
logger.debug(f"Permission denied accessing {partition.mountpoint}")
continue
except Exception as e:
logger.warning(f"Error reading {partition.mountpoint}: {e}")
continue
metrics['partitions'] = partitions_data
# Collect I/O statistics
if self.include_io:
try:
io_counters = psutil.disk_io_counters(perdisk=True)
io_data = {}
for disk_name, counters in io_counters.items():
disk_stats = {
'read_count': counters.read_count,
'write_count': counters.write_count,
'read_bytes': counters.read_bytes,
'write_bytes': counters.write_bytes,
}
# Add time statistics if available
if hasattr(counters, 'read_time'):
disk_stats['read_time'] = counters.read_time
if hasattr(counters, 'write_time'):
disk_stats['write_time'] = counters.write_time
if hasattr(counters, 'busy_time'):
disk_stats['busy_time'] = counters.busy_time
# Calculate deltas from previous collection
if disk_name in self._prev_io:
prev = self._prev_io[disk_name]
disk_stats['read_bytes_delta'] = counters.read_bytes - prev.read_bytes
disk_stats['write_bytes_delta'] = counters.write_bytes - prev.write_bytes
disk_stats['read_count_delta'] = counters.read_count - prev.read_count
disk_stats['write_count_delta'] = counters.write_count - prev.write_count
io_data[disk_name] = disk_stats
metrics['io_counters'] = io_data
# Store current counters for next delta calculation
self._prev_io = io_counters
except Exception as e:
logger.warning(f"Could not collect disk I/O statistics: {e}")
return metrics
async def cleanup(self):
"""Cleanup (nothing to do for this plugin)."""
logger.info("Disk monitor cleanup")
# Plugin instance for automatic discovery
plugin = DiskMonitorPlugin
hbd/client/plugins/filesystem_info.py
+168
View File
@@ -0,0 +1,168 @@
"""
Filesystem information plugin for Heartbeat.
Collects static filesystem and partition information using psutil.
"""
import logging
from typing import Dict, Any, Optional
try:
import psutil
except ImportError:
psutil = None
from hbd.client.plugin import InfoPlugin
logger = logging.getLogger(__name__)
class FilesystemInfoPlugin(InfoPlugin):
"""
Collect filesystem and partition information.
This is an InfoPlugin that collects static information once during startup.
By default, only reports physical mounted filesystems (e.g., ext4, xfs, btrfs).
Set include_pseudo=True to also include pseudo filesystems (proc, sysfs, tmpfs, etc.).
Collects:
- List of mounted filesystems
- Partition details (device, mount point, filesystem type, options)
- Filesystem capabilities and features
Configuration:
include_pseudo: Include pseudo/virtual filesystems (default: False)
exclude_types: List of additional filesystem types to exclude (default: [])
"""
name = "filesystem_info"
interval = 0 # InfoPlugin - collect once
def __init__(self, config: Optional[Dict[str, Any]] = None):
"""
Initialize the filesystem info plugin.
Args:
config: Optional configuration dict with keys:
- include_pseudo: Include pseudo/virtual filesystems (default: False)
- exclude_types: List of filesystem types to exclude (default: [])
"""
super().__init__(config)
self.include_pseudo = self.config.get('include_pseudo', False)
# By default, no exclusions since all=False filters most pseudo filesystems
# Users can add specific types to exclude if needed
self.exclude_types = set(self.config.get('exclude_types', []))
if psutil is None:
raise ImportError("psutil library is required for filesystem_info plugin")
async def initialize(self):
"""Initialize the plugin (check psutil availability)."""
if psutil is None:
logger.error("psutil not available - filesystem_info cannot run")
return False
logger.info(f"Filesystem info initialized (pseudo: {self.include_pseudo})")
return True
async def collect(self) -> Dict[str, Any]:
"""
Collect filesystem information.
Returns only physical mounted filesystems by default.
Returns:
Dictionary with filesystem data:
- filesystems: List of filesystem dictionaries:
- device: Device name (e.g., /dev/sda1)
- mountpoint: Mount point path
- fstype: Filesystem type (e.g., ext4, xfs, btrfs)
- opts: Mount options (comma-separated string)
- maxfile: Maximum filename length
- maxpath: Maximum path length
- filesystem_types: List of unique filesystem types found
- mount_count: Total number of mounted filesystems
"""
if psutil is None:
logger.error("psutil not available")
return {}
try:
data = await self._collect_info()
logger.info(f"Collected filesystem info: {len(data.get('filesystems', []))} filesystems")
return data
except Exception as e:
logger.error(f"Error collecting filesystem info: {e}")
return {"error": str(e)}
async def _collect_info(self) -> Dict[str, Any]:
"""Collect filesystem information from psutil."""
info = {}
filesystems = []
filesystem_types = set()
# Get mounted disk partitions
# all=False returns only physical devices (real mounted filesystems)
# all=True would include pseudo filesystems (proc, sysfs, etc.)
partitions = psutil.disk_partitions(all=self.include_pseudo)
for partition in partitions:
# Additional filtering if exclude_types is specified
if partition.fstype in self.exclude_types:
continue
fs_info = {
'device': partition.device,
'mountpoint': partition.mountpoint,
'fstype': partition.fstype,
'opts': partition.opts,
}
# Try to get filesystem capabilities
try:
# Get path configuration for this mount point
import os
if hasattr(os, 'pathconf'):
try:
# Maximum filename length
max_name = os.pathconf(partition.mountpoint, 'PC_NAME_MAX')
if max_name:
fs_info['maxfile'] = max_name
except (OSError, ValueError):
pass
try:
# Maximum path length
max_path = os.pathconf(partition.mountpoint, 'PC_PATH_MAX')
if max_path:
fs_info['maxpath'] = max_path
except (OSError, ValueError):
pass
except Exception as e:
logger.debug(f"Could not get pathconf for {partition.mountpoint}: {e}")
filesystems.append(fs_info)
filesystem_types.add(partition.fstype)
info['filesystems'] = filesystems
info['filesystem_types'] = sorted(list(filesystem_types))
info['mount_count'] = len(filesystems)
# Add some additional filesystem statistics
try:
# Get boot time (useful for determining filesystem mount times)
boot_time = psutil.boot_time()
info['boot_time'] = boot_time
except Exception as e:
logger.debug(f"Could not get boot time: {e}")
return info
async def cleanup(self):
"""Cleanup (nothing to do for this plugin)."""
logger.info("Filesystem info cleanup")
# Plugin instance for automatic discovery
plugin = FilesystemInfoPlugin
hbd/client/plugins/memory_monitor.py
+147
View File
@@ -0,0 +1,147 @@
"""
Memory monitoring plugin for Heartbeat.
Collects memory and swap usage statistics using psutil.
"""
import logging
from typing import Dict, Any, Optional
try:
import psutil
except ImportError:
psutil = None
from hbd.client.plugin import MonitorPlugin
logger = logging.getLogger(__name__)
class MemoryMonitorPlugin(MonitorPlugin):
"""
Monitor memory and swap usage.
Collects:
- Physical memory (RAM) usage and statistics
- Virtual memory details
- Swap memory usage and statistics
- Memory available for applications
Configuration:
interval: Collection interval in seconds (default: 300)
include_swap: Include swap statistics (default: True)
"""
name = "memory_monitor"
interval = 300 # Collect every 5 minutes by default
def __init__(self, config: Optional[Dict[str, Any]] = None):
"""
Initialize the memory monitor plugin.
Args:
config: Optional configuration dict with keys:
- interval: Collection interval in seconds (default: 300)
- include_swap: Include swap statistics (default: True)
"""
super().__init__(config)
self.include_swap = self.config.get('include_swap', True)
self.interval = self.config.get('interval', 300)
if psutil is None:
raise ImportError("psutil library is required for memory_monitor plugin")
async def initialize(self):
"""Initialize the plugin (check psutil availability)."""
if psutil is None:
logger.error("psutil not available - memory_monitor cannot run")
return False
logger.info(f"Memory monitor initialized (interval: {self.interval}s, swap: {self.include_swap})")
return True
async def collect(self) -> Dict[str, Any]:
"""
Collect current memory statistics.
Returns:
Dictionary with memory metrics:
- memory_total: Total physical RAM in bytes
- memory_available: Available memory in bytes
- memory_used: Used memory in bytes
- memory_free: Free memory in bytes
- memory_percent: Memory usage percentage
- memory_active: Active memory (Unix)
- memory_inactive: Inactive memory (Unix)
- memory_buffers: Buffers (Linux)
- memory_cached: Cached (Linux)
- memory_shared: Shared (Linux)
- swap_total: Total swap in bytes (if include_swap)
- swap_used: Used swap in bytes (if include_swap)
- swap_free: Free swap in bytes (if include_swap)
- swap_percent: Swap usage percentage (if include_swap)
- swap_sin: Bytes swapped in from disk (if include_swap)
- swap_sout: Bytes swapped out to disk (if include_swap)
"""
if psutil is None:
logger.error("psutil not available")
return {}
try:
data = await self._collect_metrics()
logger.debug(f"Collected memory metrics: {len(data)} fields")
return data
except Exception as e:
logger.error(f"Error collecting memory metrics: {e}")
return {"error": str(e)}
async def _collect_metrics(self) -> Dict[str, Any]:
"""Collect memory metrics from psutil."""
metrics = {}
# Virtual (physical) memory statistics
vmem = psutil.virtual_memory()
metrics['memory_total'] = vmem.total
metrics['memory_available'] = vmem.available
metrics['memory_used'] = vmem.used
metrics['memory_free'] = vmem.free
metrics['memory_percent'] = vmem.percent
# Platform-specific memory details
if hasattr(vmem, 'active'):
metrics['memory_active'] = vmem.active
if hasattr(vmem, 'inactive'):
metrics['memory_inactive'] = vmem.inactive
if hasattr(vmem, 'buffers'):
metrics['memory_buffers'] = vmem.buffers
if hasattr(vmem, 'cached'):
metrics['memory_cached'] = vmem.cached
if hasattr(vmem, 'shared'):
metrics['memory_shared'] = vmem.shared
# Swap memory statistics
if self.include_swap:
try:
swap = psutil.swap_memory()
metrics['swap_total'] = swap.total
metrics['swap_used'] = swap.used
metrics['swap_free'] = swap.free
metrics['swap_percent'] = swap.percent
# Swap in/out counters (may not be available on all platforms)
if hasattr(swap, 'sin'):
metrics['swap_sin'] = swap.sin
if hasattr(swap, 'sout'):
metrics['swap_sout'] = swap.sout
except Exception as e:
logger.warning(f"Could not collect swap statistics: {e}")
return metrics
async def cleanup(self):
"""Cleanup (nothing to do for this plugin)."""
logger.info("Memory monitor cleanup")
# Plugin instance for automatic discovery
plugin = MemoryMonitorPlugin
hbd/client/plugins/nagios_runner.py
+283
View File
@@ -0,0 +1,283 @@
"""Nagios Plugin Runner for Heartbeat.
Executes Nagios-compatible monitoring plugins and parses their output.
Nagios Plugin Standard:
- Exit codes: 0=OK, 1=WARNING, 2=CRITICAL, 3=UNKNOWN
- Output format: Single line status message, optional performance data
- Performance data format: 'label'=value[UOM];[warn];[crit];[min];[max]
Example configuration in ~/.hb.yaml:
```yaml
nagios_runner:
interval: 60
commands:
- name: check_disk_root
command: /usr/lib/nagios/plugins/check_disk -w 20% -c 10% -p /
- name: check_procs
command: /usr/lib/nagios/plugins/check_procs -w 250 -c 400
- name: check_load
command: /usr/lib/nagios/plugins/check_load -w 5,4,3 -c 10,8,6
```
"""
import re
import subprocess
from typing import Any, Dict, List, Optional, Tuple
from hbd.client.plugin import MonitorPlugin
# Nagios exit codes
NAGIOS_OK = 0
NAGIOS_WARNING = 1
NAGIOS_CRITICAL = 2
NAGIOS_UNKNOWN = 3
STATUS_NAMES = {
NAGIOS_OK: "OK",
NAGIOS_WARNING: "WARNING",
NAGIOS_CRITICAL: "CRITICAL",
NAGIOS_UNKNOWN: "UNKNOWN"
}
class NagiosRunnerPlugin(MonitorPlugin):
"""Run Nagios-compatible monitoring plugins.
This plugin executes external Nagios plugins and collects their output,
including status codes, messages, and performance data.
Configuration:
interval: Collection interval in seconds (default: 300)
commands: List of command definitions with 'name' and 'command' keys
timeout: Command execution timeout in seconds (default: 30)
shell: Whether to execute commands via shell (default: True)
Example:
nagios_runner:
interval: 300 # Check every 5 minutes
timeout: 30
commands:
- name: check_disk
command: /usr/lib/nagios/plugins/check_disk -w 20% -c 10%
- name: check_load
command: /usr/lib/nagios/plugins/check_load -w 5,4,3 -c 10,8,6
"""
name = "nagios_runner"
version = "1.0.0"
description = "Execute Nagios-compatible monitoring plugins"
interval = 300 # MonitorPlugin: collect every 5 minutes by default
def __init__(self, config: Optional[Dict[str, Any]] = None):
super().__init__(config)
# Extract configuration
self.commands: List[Dict[str, str]] = config.get("commands", []) if config else []
self.timeout: int = config.get("timeout", 30) if config else 30
self.shell: bool = config.get("shell", True) if config else True
self.interval = config.get("interval", 300) if config else 300
# Validate commands
if not self.commands:
self.logger.warning(
"No Nagios commands configured. Add 'nagios_runner.commands' to config."
)
async def initialize(self) -> bool:
"""Initialize the Nagios runner plugin.
Returns:
True if at least one command is configured, False otherwise
"""
self.logger.info(f"Initializing {self.name} plugin")
if not self.commands:
self.logger.error("No Nagios commands configured")
return False
self.logger.info(f"Configured to run {len(self.commands)} Nagios plugin(s)")
for cmd_config in self.commands:
name = cmd_config.get("name", "unnamed")
self.logger.info(f" - {name}: {cmd_config.get('command', 'N/A')}")
return True
async def _collect_metrics(self) -> Dict[str, Any]:
"""Collect metrics from all configured Nagios plugins.
Returns:
Dictionary with results from all plugins
"""
results = {}
# Track overall status (worst status wins)
worst_status = NAGIOS_OK
for cmd_config in self.commands:
name = cmd_config.get("name")
command = cmd_config.get("command")
if not name or not command:
self.logger.warning("Skipping command with missing name or command")
continue
# Execute plugin
try:
status_code, output, perfdata = await self._run_nagios_plugin(command)
# Store results
results[f"{name}_status"] = STATUS_NAMES.get(status_code, "UNKNOWN")
results[f"{name}_status_code"] = status_code
results[f"{name}_output"] = output
# Track worst status
if status_code > worst_status:
worst_status = status_code
# Parse and add performance data
if perfdata:
for metric_name, metric_value in perfdata.items():
results[f"{name}_{metric_name}"] = metric_value
self.logger.debug(
f"Executed {name}: {STATUS_NAMES.get(status_code, 'UNKNOWN')} - {output[:50]}"
)
except Exception as e:
self.logger.error(f"Error running {name}: {e}", exc_info=True)
results[f"{name}_status"] = "ERROR"
results[f"{name}_status_code"] = NAGIOS_UNKNOWN
results[f"{name}_output"] = str(e)
worst_status = NAGIOS_UNKNOWN
# Add overall status
results["overall_status"] = STATUS_NAMES.get(worst_status, "UNKNOWN")
results["overall_status_code"] = worst_status
results["plugin_count"] = len(self.commands)
return results
async def _run_nagios_plugin(
self,
command: str
) -> Tuple[int, str, Dict[str, Any]]:
"""Execute a Nagios plugin and parse its output.
Args:
command: Command string to execute
Returns:
Tuple of (status_code, output_message, performance_data_dict)
"""
try:
# Run command
result = subprocess.run(
command,
shell=self.shell,
capture_output=True,
timeout=self.timeout,
text=True
)
status_code = result.returncode
output = result.stdout.strip()
# Nagios plugins can return codes > 3, treat as UNKNOWN
if status_code > 3:
status_code = NAGIOS_UNKNOWN
# Parse performance data
perfdata = self._parse_perfdata(output)
# Extract just the status message (before the pipe if present)
if '|' in output:
output_msg = output.split('|')[0].strip()
else:
output_msg = output
return status_code, output_msg, perfdata
except subprocess.TimeoutExpired:
self.logger.error(f"Command timed out: {command}")
return NAGIOS_UNKNOWN, f"Command timed out after {self.timeout}s", {}
except Exception as e:
self.logger.error(f"Error executing command: {e}")
return NAGIOS_UNKNOWN, f"Execution error: {str(e)}", {}
def _parse_perfdata(self, output: str) -> Dict[str, Any]:
"""Parse Nagios performance data from plugin output.
Nagios performance data format:
'label'=value[UOM];[warn];[crit];[min];[max]
Multiple metrics separated by spaces.
Args:
output: Plugin output string
Returns:
Dictionary of metric_name: value
"""
perfdata = {}
# Performance data comes after the pipe character
if '|' not in output:
return perfdata
perf_section = output.split('|', 1)[1].strip()
# Regex to match performance data format
# Matches: 'label'=value or label=value
perf_regex = r"'?([^'=]+)'?=([\d.]+)([a-zA-Z%]*);?([\d.]*);?([\d.]*);?([\d.]*);?([\d.]*)"
for match in re.finditer(perf_regex, perf_section):
label = match.group(1).strip()
value_str = match.group(2)
uom = match.group(3) or ""
warn = match.group(4)
crit = match.group(5)
min_val = match.group(6)
max_val = match.group(7)
# Convert value to float
try:
value = float(value_str)
except ValueError:
continue
# Store the value
perfdata[label] = value
# Optionally store UOM as separate field
if uom:
perfdata[f"{label}_uom"] = uom
# Store thresholds if present
if warn:
try:
perfdata[f"{label}_warn"] = float(warn)
except ValueError:
pass
if crit:
try:
perfdata[f"{label}_crit"] = float(crit)
except ValueError:
pass
if min_val:
try:
perfdata[f"{label}_min"] = float(min_val)
except ValueError:
pass
if max_val:
try:
perfdata[f"{label}_max"] = float(max_val)
except ValueError:
pass
return perfdata
hbd/client/plugins/network_monitor.py
+240
View File
@@ -0,0 +1,240 @@
"""
Network monitoring plugin for Heartbeat.
Collects network interface statistics and connection information using psutil.
"""
import logging
from typing import Dict, Any, Optional, List
try:
import psutil
except ImportError:
psutil = None
from hbd.client.plugin import MonitorPlugin
logger = logging.getLogger(__name__)
class NetworkMonitorPlugin(MonitorPlugin):
"""
Monitor network interface statistics and connections.
Collects:
- Network interface I/O counters (bytes sent/received, packets, errors, drops)
- Per-interface statistics
- Network connection counts by state
- Interface addresses and configuration
Configuration:
interval: Collection interval in seconds (default: 300)
interfaces: List of interfaces to monitor (default: all)
include_connections: Include connection statistics (default: True)
include_addresses: Include interface addresses (default: False)
"""
name = "network_monitor"
interval = 300 # Collect every 5 minutes by default
def __init__(self, config: Optional[Dict[str, Any]] = None):
"""
Initialize the network monitor plugin.
Args:
config: Optional configuration dict with keys:
- interval: Collection interval in seconds (default: 300)
- interfaces: List of specific interfaces to monitor
- include_connections: Include connection stats (default: True)
- include_addresses: Include interface addresses (default: False)
"""
super().__init__(config)
self.interfaces = self.config.get('interfaces', None) # None = all interfaces
self.include_connections = self.config.get('include_connections', True)
self.include_addresses = self.config.get('include_addresses', False)
self.interval = self.config.get('interval', 300)
if psutil is None:
raise ImportError("psutil library is required for network_monitor plugin")
# Store previous I/O counters for delta calculation
self._prev_io = {}
async def initialize(self):
"""Initialize the plugin (check psutil availability)."""
if psutil is None:
logger.error("psutil not available - network_monitor cannot run")
return False
logger.info(f"Network monitor initialized (interval: {self.interval}s, "
f"connections: {self.include_connections})")
# Initialize I/O counters
try:
self._prev_io = psutil.net_io_counters(pernic=True)
except Exception as e:
logger.warning(f"Could not initialize network I/O counters: {e}")
return True
async def collect(self) -> Dict[str, Any]:
"""
Collect current network statistics.
Returns:
Dictionary with network metrics:
- interfaces: Dict of interface statistics, keyed by interface name
- bytes_sent: Total bytes sent
- bytes_recv: Total bytes received
- packets_sent: Total packets sent
- packets_recv: Total packets received
- errin: Total incoming errors
- errout: Total outgoing errors
- dropin: Total incoming packets dropped
- dropout: Total outgoing packets dropped
- bytes_sent_delta: Bytes sent since last collection
- bytes_recv_delta: Bytes received since last collection
- packets_sent_delta: Packets sent since last collection
- packets_recv_delta: Packets received since last collection
- connections: Connection statistics by state (if include_connections)
- ESTABLISHED: Count of established connections
- LISTEN: Count of listening sockets
- TIME_WAIT: Count of TIME_WAIT connections
- etc.
- addresses: Interface address information (if include_addresses)
- Dict keyed by interface name with address details
"""
if psutil is None:
logger.error("psutil not available")
return {}
try:
data = await self._collect_metrics()
logger.debug(f"Collected network metrics: {len(data.get('interfaces', {}))} interfaces")
return data
except Exception as e:
logger.error(f"Error collecting network metrics: {e}")
return {"error": str(e)}
async def _collect_metrics(self) -> Dict[str, Any]:
"""Collect network metrics from psutil."""
metrics = {}
# Collect per-interface I/O counters
try:
io_counters = psutil.net_io_counters(pernic=True)
interfaces_data = {}
for iface_name, counters in io_counters.items():
# Skip if we're only monitoring specific interfaces
if self.interfaces and iface_name not in self.interfaces:
continue
iface_stats = {
'bytes_sent': counters.bytes_sent,
'bytes_recv': counters.bytes_recv,
'packets_sent': counters.packets_sent,
'packets_recv': counters.packets_recv,
'errin': counters.errin,
'errout': counters.errout,
'dropin': counters.dropin,
'dropout': counters.dropout,
}
# Calculate deltas from previous collection
if iface_name in self._prev_io:
prev = self._prev_io[iface_name]
iface_stats['bytes_sent_delta'] = counters.bytes_sent - prev.bytes_sent
iface_stats['bytes_recv_delta'] = counters.bytes_recv - prev.bytes_recv
iface_stats['packets_sent_delta'] = counters.packets_sent - prev.packets_sent
iface_stats['packets_recv_delta'] = counters.packets_recv - prev.packets_recv
interfaces_data[iface_name] = iface_stats
metrics['interfaces'] = interfaces_data
# Store current counters for next delta calculation
self._prev_io = io_counters
except Exception as e:
logger.warning(f"Could not collect network I/O counters: {e}")
# Collect connection statistics
if self.include_connections:
try:
connections = psutil.net_connections(kind='inet')
conn_stats = {}
# Count connections by state
for conn in connections:
state = conn.status
conn_stats[state] = conn_stats.get(state, 0) + 1
metrics['connections'] = conn_stats
except (PermissionError, psutil.AccessDenied):
logger.debug("Permission denied for net_connections (requires root/admin)")
except Exception as e:
logger.warning(f"Could not collect connection statistics: {e}")
# Collect interface addresses
if self.include_addresses:
try:
addresses = psutil.net_if_addrs()
addr_data = {}
for iface_name, addrs in addresses.items():
# Skip if we're only monitoring specific interfaces
if self.interfaces and iface_name not in self.interfaces:
continue
iface_addrs = []
for addr in addrs:
addr_info = {
'family': str(addr.family),
'address': addr.address,
}
if addr.netmask:
addr_info['netmask'] = addr.netmask
if addr.broadcast:
addr_info['broadcast'] = addr.broadcast
iface_addrs.append(addr_info)
addr_data[iface_name] = iface_addrs
metrics['addresses'] = addr_data
except Exception as e:
logger.warning(f"Could not collect interface addresses: {e}")
# Add interface stats (up/down status, speed, mtu)
try:
if_stats = psutil.net_if_stats()
stats_data = {}
for iface_name, stats in if_stats.items():
# Skip if we're only monitoring specific interfaces
if self.interfaces and iface_name not in self.interfaces:
continue
stats_data[iface_name] = {
'isup': stats.isup,
'duplex': str(stats.duplex) if hasattr(stats, 'duplex') else None,
'speed': stats.speed,
'mtu': stats.mtu,
}
metrics['interface_stats'] = stats_data
except Exception as e:
logger.warning(f"Could not collect interface stats: {e}")
return metrics
async def cleanup(self):
"""Cleanup (nothing to do for this plugin)."""
logger.info("Network monitor cleanup")
# Plugin instance for automatic discovery
plugin = NetworkMonitorPlugin
hbd/client/plugins/os_info.py
+136
View File
@@ -0,0 +1,136 @@
"""OS Information Plugin for Heartbeat.
Collects static operating system information including OS name, version,
kernel, architecture, and distribution details.
"""
import platform
import sys
from pathlib import Path
from typing import Any, Dict, Optional
# Import from parent package
from hbd.client.plugin import InfoPlugin
class OSInfoPlugin(InfoPlugin):
"""Collect operating system information.
This plugin gathers static OS information that rarely changes:
- OS name and version
- Kernel version
- Architecture (x86_64, arm64, etc.)
- Distribution details (for Linux)
- Python version (used by hbc)
"""
name = "os_info"
version = "1.0.0"
description = "Operating system and platform information"
interval = 0 # InfoPlugin: collect once at startup
def __init__(self, config: Optional[Dict[str, Any]] = None):
super().__init__(config)
async def initialize(self) -> bool:
"""Initialize the OS info plugin.
Returns:
True (always succeeds - platform module is stdlib)
"""
self.logger.info(f"Initializing {self.name} plugin")
return True
async def _collect_info(self) -> Dict[str, Any]:
"""Collect OS information.
Returns:
Dictionary with OS details
"""
try:
data = {
"system": platform.system(), # e.g., "Linux", "Darwin", "Windows"
"node": platform.node(), # hostname
"release": platform.release(), # kernel version
"version": platform.version(), # detailed version
"machine": platform.machine(), # e.g., "x86_64", "arm64"
"processor": platform.processor(), # processor name
"architecture": platform.architecture()[0], # e.g., "64bit"
"python_version": platform.python_version(),
"python_implementation": platform.python_implementation(),
}
# Add Linux-specific distribution info
if platform.system() == "Linux":
data.update(self._get_linux_distro())
# Add macOS-specific info
elif platform.system() == "Darwin":
data["macos_version"] = platform.mac_ver()[0]
# Add Windows-specific info
elif platform.system() == "Windows":
win_ver = platform.win32_ver()
data["windows_release"] = win_ver[0]
data["windows_version"] = win_ver[1]
data["windows_sp"] = win_ver[2]
data["windows_type"] = win_ver[3]
self.logger.debug(f"Collected OS info: {data['system']} {data['release']}")
return data
except Exception as e:
self.logger.error(f"Error collecting OS info: {e}", exc_info=True)
return {}
def _get_linux_distro(self) -> Dict[str, str]:
"""Get Linux distribution information.
Returns:
Dictionary with distribution details
"""
distro_info = {}
# Try reading /etc/os-release (standard on modern Linux)
os_release = Path("/etc/os-release")
if os_release.exists():
try:
with open(os_release) as f:
for line in f:
line = line.strip()
if "=" in line and not line.startswith("#"):
key, value = line.split("=", 1)
# Remove quotes from value
value = value.strip('"').strip("'")
# Map common keys
if key == "NAME":
distro_info["distro_name"] = value
elif key == "VERSION":
distro_info["distro_version"] = value
elif key == "ID":
distro_info["distro_id"] = value
elif key == "VERSION_ID":
distro_info["distro_version_id"] = value
elif key == "PRETTY_NAME":
distro_info["distro_pretty_name"] = value
except Exception as e:
self.logger.warning(f"Could not read /etc/os-release: {e}")
# Fallback: try lsb_release (older systems)
elif Path("/etc/lsb-release").exists():
try:
with open("/etc/lsb-release") as f:
for line in f:
line = line.strip()
if "=" in line:
key, value = line.split("=", 1)
if key == "DISTRIB_ID":
distro_info["distro_id"] = value
elif key == "DISTRIB_RELEASE":
distro_info["distro_version"] = value
elif key == "DISTRIB_DESCRIPTION":
distro_info["distro_name"] = value
except Exception as e:
self.logger.warning(f"Could not read /etc/lsb-release: {e}")
return distro_info
hbd/client/threshold.py
+579
View File
@@ -0,0 +1,579 @@
"""
Threshold checking and alerting for plugin metrics.
This module provides a flexible threshold checking system that:
- Evaluates plugin metrics against configured warning/critical thresholds
- Tracks alert states per host and metric
- Prevents alert flapping with hysteresis
- Triggers notifications only on state changes
- Supports multiple comparison operators
"""
import logging
import time
from enum import Enum
from typing import Dict, Any, Optional, Tuple, Callable
logger = logging.getLogger(__name__)
class AlertLevel(Enum):
"""Alert severity levels."""
OK = 0
WARNING = 1
CRITICAL = 2
UNKNOWN = 3
class ComparisonOperator(Enum):
"""Supported comparison operators for threshold checks."""
GT = ">" # Greater than
GTE = ">=" # Greater than or equal
LT = "<" # Less than
LTE = "<=" # Less than or equal
EQ = "==" # Equal to
NEQ = "!=" # Not equal to
class AlertState:
"""Represents the current alert state for a specific metric."""
def __init__(self, metric_path: str):
"""
Initialize alert state.
Args:
metric_path: Full path to metric (e.g., "cpu_monitor.cpu_percent")
"""
self.metric_path = metric_path
self.level = AlertLevel.OK
self.since = time.time()
self.last_value = None
self.last_check = time.time()
self.notification_count = 0
self.last_notification = None
def update(self, level: AlertLevel, value: Any) -> bool:
"""
Update alert state.
Args:
level: New alert level
value: Current metric value
Returns:
True if state changed (notification needed), False otherwise
"""
now = time.time()
self.last_check = now
self.last_value = value
# Check if state changed
if level != self.level:
logger.info(
"Alert state change for %s: %s -> %s (value: %s)",
self.metric_path,
self.level.name,
level.name,
value
)
self.level = level
self.since = now
self.notification_count = 0
return True
return False
def to_dict(self) -> dict:
"""Convert alert state to dictionary for serialization."""
return {
"metric_path": self.metric_path,
"level": self.level.name,
"since": self.since,
"last_value": self.last_value,
"last_check": self.last_check,
"notification_count": self.notification_count,
}
class ThresholdConfig:
"""Configuration for a single threshold check."""
def __init__(
self,
metric_path: str,
warning: Optional[float] = None,
critical: Optional[float] = None,
operator: str = ">",
hysteresis: float = 0.0,
enabled: bool = True,
):
"""
Initialize threshold configuration.
Args:
metric_path: Full path to metric (e.g., "cpu_monitor.cpu_percent")
warning: Warning threshold value
critical: Critical threshold value
operator: Comparison operator (>, >=, <, <=, ==, !=)
hysteresis: Hysteresis percentage to prevent flapping (0.0-1.0)
enabled: Whether this threshold is enabled
"""
self.metric_path = metric_path
self.warning = warning
self.critical = critical
self.enabled = enabled
self.hysteresis = hysteresis
# Parse operator
try:
self.operator = ComparisonOperator(operator)
except ValueError:
logger.warning(
"Invalid operator '%s' for %s, using '>' as default",
operator,
metric_path
)
self.operator = ComparisonOperator.GT
def evaluate(self, value: float) -> AlertLevel:
"""
Evaluate a value against this threshold.
Args:
value: Metric value to check
Returns:
AlertLevel indicating the severity
"""
if not self.enabled:
return AlertLevel.OK
try:
# Convert value to float for comparison
value = float(value)
except (TypeError, ValueError):
logger.warning("Cannot convert value %s to float for %s", value, self.metric_path)
return AlertLevel.UNKNOWN
# Check critical threshold first
if self.critical is not None:
if self._compare(value, self.critical):
return AlertLevel.CRITICAL
# Then check warning threshold
if self.warning is not None:
if self._compare(value, self.warning):
return AlertLevel.WARNING
return AlertLevel.OK
def evaluate_with_hysteresis(
self,
value: float,
current_level: AlertLevel
) -> AlertLevel:
"""
Evaluate with hysteresis to prevent flapping.
Args:
value: Current metric value
current_level: Current alert level
Returns:
New alert level considering hysteresis
"""
new_level = self.evaluate(value)
# If no hysteresis, return new level
if self.hysteresis == 0.0:
return new_level
# If improving (going to a lower severity), apply hysteresis
if new_level.value < current_level.value:
# For recovery, value must be better by hysteresis amount
if current_level == AlertLevel.CRITICAL and self.critical is not None:
threshold = self.critical
elif current_level == AlertLevel.WARNING and self.warning is not None:
threshold = self.warning
else:
return new_level
# Calculate hysteresis threshold
hysteresis_amount = abs(threshold * self.hysteresis)
if self.operator in [ComparisonOperator.GT, ComparisonOperator.GTE]:
# For "greater than" thresholds, value must go below by hysteresis
recovery_threshold = threshold - hysteresis_amount
if value >= recovery_threshold:
# Not enough improvement, keep current level
return current_level
elif self.operator in [ComparisonOperator.LT, ComparisonOperator.LTE]:
# For "less than" thresholds, value must go above by hysteresis
recovery_threshold = threshold + hysteresis_amount
if value <= recovery_threshold:
# Not enough improvement, keep current level
return current_level
return new_level
def _compare(self, value: float, threshold: float) -> bool:
"""Perform comparison based on operator."""
if self.operator == ComparisonOperator.GT:
return value > threshold
elif self.operator == ComparisonOperator.GTE:
return value >= threshold
elif self.operator == ComparisonOperator.LT:
return value < threshold
elif self.operator == ComparisonOperator.LTE:
return value <= threshold
elif self.operator == ComparisonOperator.EQ:
return abs(value - threshold) < 1e-9 # Float comparison
elif self.operator == ComparisonOperator.NEQ:
return abs(value - threshold) >= 1e-9
return False
class ThresholdChecker:
"""Main threshold checking and alerting system."""
def __init__(
self,
config: Dict[str, Any],
notification_callback: Optional[Callable] = None,
renotify_interval: int = 3600,
journal: Optional[Any] = None,
):
"""
Initialize threshold checker.
Args:
config: Threshold configuration dictionary from YAML
notification_callback: Function to call for notifications
renotify_interval: Seconds between repeat notifications (default: 1 hour)
journal: Optional MessageJournal instance for logging threshold events
"""
self.thresholds = {} # {metric_path: ThresholdConfig}
self.notification_callback = notification_callback
self.renotify_interval = renotify_interval
self.journal = journal
# Parse configuration
self._parse_config(config)
logger.info("ThresholdChecker initialized with %d thresholds", len(self.thresholds))
def _parse_config(self, config: Dict[str, Any]):
"""Parse threshold configuration from YAML structure."""
if not config or "thresholds" not in config:
logger.info("No thresholds configured")
return
thresholds_config = config["thresholds"]
for plugin_name, plugin_thresholds in thresholds_config.items():
if not isinstance(plugin_thresholds, dict):
continue
self._parse_plugin_thresholds(plugin_name, plugin_thresholds)
def _parse_plugin_thresholds(self, plugin_name: str, thresholds: Dict[str, Any]):
"""Parse thresholds for a specific plugin."""
for metric_name, threshold_config in thresholds.items():
if not isinstance(threshold_config, dict):
continue
# Handle nested metrics (e.g., partitions./.percent)
if metric_name == "partitions":
self._parse_partition_thresholds(plugin_name, threshold_config)
continue
metric_path = f"{plugin_name}.{metric_name}"
# Extract threshold values
warning = threshold_config.get("warning")
critical = threshold_config.get("critical")
operator = threshold_config.get("operator", ">")
hysteresis = threshold_config.get("hysteresis", 0.1) # 10% default
enabled = threshold_config.get("enabled", True)
if warning is None and critical is None:
logger.warning("No thresholds defined for %s, skipping", metric_path)
continue
threshold = ThresholdConfig(
metric_path=metric_path,
warning=warning,
critical=critical,
operator=operator,
hysteresis=hysteresis,
enabled=enabled,
)
self.thresholds[metric_path] = threshold
logger.debug(
"Registered threshold for %s: warn=%s, crit=%s, op=%s",
metric_path,
warning,
critical,
operator
)
def _parse_partition_thresholds(self, plugin_name: str, partitions: Dict[str, Any]):
"""Parse partition-specific thresholds for disk monitoring."""
for partition, metrics in partitions.items():
if not isinstance(metrics, dict):
continue
for metric_name, threshold_config in metrics.items():
if not isinstance(threshold_config, dict):
continue
# Create metric path like "disk_monitor./dev/sda1.percent"
metric_path = f"{plugin_name}.{partition}.{metric_name}"
warning = threshold_config.get("warning")
critical = threshold_config.get("critical")
operator = threshold_config.get("operator", ">")
hysteresis = threshold_config.get("hysteresis", 0.1)
enabled = threshold_config.get("enabled", True)
if warning is None and critical is None:
continue
threshold = ThresholdConfig(
metric_path=metric_path,
warning=warning,
critical=critical,
operator=operator,
hysteresis=hysteresis,
enabled=enabled,
)
self.thresholds[metric_path] = threshold
def check_plugin_data(
self,
host_name: str,
plugin_name: str,
data: Dict[str, Any],
alert_states: Dict[str, AlertState],
) -> list:
"""
Check plugin data against configured thresholds.
Args:
host_name: Name of the host
plugin_name: Name of the plugin
data: Plugin data dictionary
alert_states: Host's alert_states dictionary
Returns:
List of (metric_path, old_level, new_level, value) tuples for state changes
"""
state_changes = []
# Check flat metrics
for metric_name, value in data.items():
metric_path = f"{plugin_name}.{metric_name}"
if metric_path not in self.thresholds:
continue
threshold = self.thresholds[metric_path]
# Get or create alert state
if metric_path not in alert_states:
alert_states[metric_path] = AlertState(metric_path)
alert_state = alert_states[metric_path]
# Evaluate threshold with hysteresis
new_level = threshold.evaluate_with_hysteresis(
value,
alert_state.level
)
# Update state and check for changes
old_level = alert_state.level
if alert_state.update(new_level, value):
state_changes.append((metric_path, old_level, new_level, value))
self._trigger_notification(host_name, metric_path, old_level, new_level, value)
elif new_level != AlertLevel.OK:
# Check if we should re-notify
self._check_renotify(host_name, alert_state, metric_path, value)
# Check nested metrics (e.g., partition data in disk_monitor)
self._check_nested_metrics(
host_name,
plugin_name,
data,
alert_states,
state_changes
)
return state_changes
def _check_nested_metrics(
self,
host_name: str,
plugin_name: str,
data: Dict[str, Any],
alert_states: Dict[str, AlertState],
state_changes: list,
):
"""Check nested metrics like partition-specific thresholds."""
# Look for partition data in disk_monitor
if plugin_name == "disk_monitor" and "partitions" in data:
partitions = data["partitions"]
if not isinstance(partitions, dict):
return
for partition, metrics in partitions.items():
if not isinstance(metrics, dict):
continue
for metric_name, value in metrics.items():
metric_path = f"{plugin_name}.{partition}.{metric_name}"
if metric_path not in self.thresholds:
continue
threshold = self.thresholds[metric_path]
if metric_path not in alert_states:
alert_states[metric_path] = AlertState(metric_path)
alert_state = alert_states[metric_path]
new_level = threshold.evaluate_with_hysteresis(
value,
alert_state.level
)
old_level = alert_state.level
if alert_state.update(new_level, value):
state_changes.append((metric_path, old_level, new_level, value))
self._trigger_notification(
host_name,
metric_path,
old_level,
new_level,
value
)
elif new_level != AlertLevel.OK:
self._check_renotify(host_name, alert_state, metric_path, value)
def _trigger_notification(
self,
host_name: str,
metric_path: str,
old_level: AlertLevel,
new_level: AlertLevel,
value: Any,
):
"""Trigger a notification for an alert state change."""
# Format message
if new_level == AlertLevel.OK:
message = f"RECOVERED: {host_name} - {metric_path} = {value} ({old_level.name} -> OK)"
elif new_level == AlertLevel.WARNING:
message = f"WARNING: {host_name} - {metric_path} = {value}"
elif new_level == AlertLevel.CRITICAL:
message = f"CRITICAL: {host_name} - {metric_path} = {value}"
else:
message = f"UNKNOWN: {host_name} - {metric_path} = {value}"
# Send notification
if self.notification_callback is not None:
try:
self.notification_callback(message)
logger.info("Notification sent: %s", message)
except Exception as e:
logger.error("Failed to send notification: %s", e)
# Log to journal
if self.journal is not None:
try:
import asyncio
loop = asyncio.get_event_loop()
loop.create_task(self.journal.log_threshold_event(
host_name=host_name,
metric_path=metric_path,
old_level=old_level.name,
new_level=new_level.name,
value=value,
))
except Exception as e:
logger.debug(f"Failed to log threshold event to journal: {e}")
def _check_renotify(
self,
host_name: str,
alert_state: AlertState,
metric_path: str,
value: Any,
):
"""Check if we should send a repeat notification."""
if alert_state.level == AlertLevel.OK:
return
now = time.time()
# Check if we should re-notify
if alert_state.last_notification is None:
# First notification already sent during state change
alert_state.last_notification = now
alert_state.notification_count = 1
return
if (now - alert_state.last_notification) >= self.renotify_interval:
# Time to re-notify
message = f"REMINDER ({alert_state.level.name}): {host_name} - {metric_path} = {value} (ongoing for {int(now - alert_state.since)}s)"
if self.notification_callback:
try:
self.notification_callback(message)
alert_state.last_notification = now
alert_state.notification_count += 1
logger.info("Re-notification sent: %s", message)
except Exception as e:
logger.error("Failed to send re-notification: %s", e)
def get_active_alerts(self, alert_states: Dict[str, AlertState]) -> list:
"""
Get all currently active (non-OK) alerts.
Args:
alert_states: Host's alert_states dictionary
Returns:
List of AlertState objects that are not OK
"""
return [
state for state in alert_states.values()
if state.level != AlertLevel.OK
]
def get_alert_summary(self, alert_states: Dict[str, AlertState]) -> Dict[str, int]:
"""
Get summary counts of alert levels.
Args:
alert_states: Host's alert_states dictionary
Returns:
Dictionary with counts: {"ok": N, "warning": N, "critical": N}
"""
summary = {"ok": 0, "warning": 0, "critical": 0, "unknown": 0}
for state in alert_states.values():
if state.level == AlertLevel.OK:
summary["ok"] += 1
elif state.level == AlertLevel.WARNING:
summary["warning"] += 1
elif state.level == AlertLevel.CRITICAL:
summary["critical"] += 1
elif state.level == AlertLevel.UNKNOWN:
summary["unknown"] += 1
return summary
hbd/common/__init__.py
+3
View File
@@ -0,0 +1,3 @@
"""Common utilities shared between hbc and hbd."""
__version__ = "5.0.5"
hbd/common/proto.py
+160
View File
@@ -0,0 +1,160 @@
"""Message encoding/decoding utilities for hbd protocol.
Message Types:
HTB: Heartbeat message (client -> server)
ACK: Acknowledgment (server -> client)
CMD: Command message (server -> client)
UPD: Update message (server -> client)
PLG: Plugin data message (client -> server)
"""
from typing import Dict, Any, Union
import json
import zlib
def encode_value(v: Any) -> str:
"""Encode a value for protocol transmission.
Args:
v: Value to encode (int, float, str, bool, list, dict, etc.)
Returns:
String representation suitable for protocol
"""
if isinstance(v, float):
return f"{v:0.5f}"
elif isinstance(v, (list, dict)):
# Use JSON encoding for complex types, prefixed with @
return "@" + json.dumps(v)
elif isinstance(v, bool):
return str(int(v)) # True->1, False->0
else:
return str(v)
def decode_value(val: str) -> Any:
"""Decode a value from protocol format.
Args:
val: String value from protocol
Returns:
Decoded Python object
"""
if not val:
return val
# Check for JSON-encoded complex types
if val.startswith("@"):
try:
return json.loads(val[1:])
except Exception:
return val[1:] # Return as string without @
# Try numeric evaluation (original behavior)
if val[0].isdigit() or (val[0] == '-' and len(val) > 1 and val[1].isdigit()):
try:
return eval(val)
except Exception:
return val
return val
def dicttos(ID: str, d: Dict[str, Any], compress: bool = False):
"""Serialize a dict to protocol message bytes.
If compress is True, the payload is zlib-compressed and the message is
prefixed with `!ID:` as the original script did. Otherwise the format is
`ID:key=value;...` (bytes).
"""
s = []
for k in d:
v = d[k]
encoded_val = encode_value(v)
s.append(f"{k}={encoded_val}")
pk = ";".join(s)
if compress:
zpk = zlib.compress(pk.encode(), 6)
hdr = ("!" + ID + ":").encode()
return hdr + zpk
else:
return (ID + ":" + pk).encode()
def stodict(msg: bytes):
"""Deserialize a protocol message into a dict.
Mirrors original behaviour: detects compressed messages starting with
'!' and decodes accordingly. Returns a dict with key 'ID' set to the
message ID and the parsed key/value pairs.
"""
d = {}
if len(msg) > 0 and chr(msg[0]) == "!":
# message is: b'!ID:' + compressed_payload
# original code used msg[1:4].decode() for ID (3 bytes including colon)
try:
pk = zlib.decompress(msg[5:]).decode()
except Exception:
# malformed compressed payload
return {}
d["ID"] = msg[1:4].decode()
else:
try:
r0 = msg.split(b":", 1)
pk = r0[1].decode()
d["ID"] = r0[0].decode()
except Exception:
return {}
if not pk:
return d
parts = pk.split(";")
for v in parts:
if not v:
continue
vr = v.split("=", 1)
k = vr[0].strip()
if len(vr) == 1:
d[k] = None
else:
val = vr[1].strip()
d[k] = decode_value(val)
return d
def oldmtodict(msg: bytes):
"""Compatibility wrapper for old-style messages (no ID prefix).
The original implementation prefixed with 'HTB:' and called stodict.
"""
return stodict(b"HTB:" + msg)
def encode_plugin_data(plugin_name: str, data: Dict[str, Any], compress: bool = False) -> bytes:
"""Encode plugin data into a PLG message.
Args:
plugin_name: Name of the plugin (e.g., "os_info", "cpu_monitor")
data: Plugin data dictionary
compress: Whether to compress the payload
Returns:
Encoded message bytes
"""
# Add plugin name to data
full_data = {"plugin": plugin_name, **data}
return dicttos("PLG", full_data, compress)
def decode_plugin_data(msg: bytes) -> Dict[str, Any]:
"""Decode a PLG message into plugin data.
Args:
msg: Raw message bytes
Returns:
Dictionary with 'ID', 'plugin', and plugin data fields
"""
return stodict(msg)
hbd/utils.py → hbd/common/utils.py
View File
hbd/config.py → hbd/config.py.old
+11 -4
View File
@@ -39,6 +39,15 @@ DEFAULTS = {
"cert_path": "/usr/local/etc/ssl/",
"wss_pem": "fullchain.pem",
"wss_key": "privkey.pem",
# Message journal configuration
"journal_enabled": True,
"journal_dir": "/var/log/heartbeat",
"journal_file": "messages.journal",
"journal_max_size": 100 * 1024 * 1024, # 100MB
"journal_max_backups": 10,
"plugins": {},
"thresholds": {},
"threshold_renotify_interval": 3600,
}
@@ -56,12 +65,10 @@ def load_config(path=None):
if yaml:
with open(path) as fh:
data = yaml.safe_load(fh)
# only keep known keys
# Merge YAML data with defaults
# Keep all keys from YAML to support plugin configs and future extensions
for k, v in data.items():
if k in cfg:
cfg[k] = v
else:
logging.warning("unknown config key %s in %s", k, path)
else:
# yaml not installed: do not attempt to parse; user must ensure defaults
pass
hbd/config_example.yaml
+196
View File
@@ -0,0 +1,196 @@
# Example Heartbeat Client Configuration
# This file demonstrates all available configuration options for the heartbeat client (hbc)
# and its plugin system.
# ==============================================================================
# Server Configuration
# ==============================================================================
server: hbd.example.com # Heartbeat server hostname or IP
port: 50003 # Server UDP port (default: 50003)
interval: 30 # Heartbeat interval in seconds (default: 30)
# ==============================================================================
# Plugin Configuration
# ==============================================================================
# Plugins are configured under the "plugins" section. Each plugin can be enabled/disabled
# and configured with plugin-specific settings.
plugins:
# --------------------------------------------------------------------------
# OS Information Plugin (InfoPlugin - runs once at startup)
# --------------------------------------------------------------------------
os_info:
enabled: true
# No additional configuration needed
# --------------------------------------------------------------------------
# CPU Monitor Plugin (MonitorPlugin - periodic collection)
# --------------------------------------------------------------------------
cpu_monitor:
enabled: true
interval: 300 # Collection interval in seconds (default: 300 = 5 minutes)
per_core: false # Collect per-core CPU statistics (default: false)
# When per_core is true, will report CPU usage for each core separately
# --------------------------------------------------------------------------
# Memory Monitor Plugin (MonitorPlugin)
# --------------------------------------------------------------------------
memory_monitor:
enabled: true
interval: 300 # Collection interval in seconds (default: 300 = 5 minutes)
include_swap: true # Include swap memory statistics (default: true)
# --------------------------------------------------------------------------
# Disk Monitor Plugin (MonitorPlugin)
# --------------------------------------------------------------------------
disk_monitor:
enabled: true
interval: 300 # Collection interval in seconds (default: 300 = 5 minutes)
include_io: true # Include I/O statistics (default: true)
# Optional: Monitor only specific partitions
# partitions:
# - /
# - /home
# - /var
# Optional: Exclude specific filesystem types
exclude_types:
- tmpfs
- devtmpfs
- squashfs
# --------------------------------------------------------------------------
# Network Monitor Plugin (MonitorPlugin)
# --------------------------------------------------------------------------
network_monitor:
enabled: true
interval: 300 # Collection interval in seconds (default: 300 = 5 minutes)
include_connections: true # Include connection statistics (default: true)
include_addresses: false # Include interface addresses (default: false)
# Optional: Monitor only specific interfaces
# interfaces:
# - eth0
# - wlan0
# --------------------------------------------------------------------------
# Filesystem Info Plugin (InfoPlugin - runs once at startup)
# --------------------------------------------------------------------------
filesystem_info:
enabled: true
include_pseudo: false # Include pseudo/virtual filesystems (default: false)
# When false (default), only reports physical mounted filesystems (ext4, zfs, xfs, etc.)
# When true, also includes pseudo filesystems (proc, sysfs, tmpfs, devtmpfs, etc.)
# Optional: Exclude additional specific filesystem types
# exclude_types:
# - squashfs
# - iso9660
# --------------------------------------------------------------------------
# Nagios Runner Plugin (MonitorPlugin)
# --------------------------------------------------------------------------
nagios_runner:
enabled: true
interval: 300 # Collection interval in seconds (default: 300 = 5 minutes)
timeout: 30 # Plugin execution timeout in seconds (default: 30)
# List of Nagios plugins to execute
# Each command is executed as-is, so provide full paths and arguments
commands:
# System load monitoring
- /usr/lib/nagios/plugins/check_load -w 5,4,3 -c 10,8,6
# Disk space monitoring
- /usr/lib/nagios/plugins/check_disk -w 20% -c 10% -p /
- /usr/lib/nagios/plugins/check_disk -w 20% -c 10% -p /home
# Process monitoring
- /usr/lib/nagios/plugins/check_procs -w 250 -c 400 -s RSZDT
# Swap usage
- /usr/lib/nagios/plugins/check_swap -w 20% -c 10%
# Custom script example
# - /usr/local/bin/check_my_app.sh
# ==============================================================================
# Advanced Options
# ==============================================================================
# These options control client behavior
# Compression: Enable zlib compression for heartbeat messages (default: true)
compress: true
# Hostname: Override the system hostname (default: auto-detect)
# hostname: myhost.example.com
# Message: Custom message included in heartbeat (optional)
# message: "Production web server"
# Logging
log_level: INFO # Log level: DEBUG, INFO, WARNING, ERROR (default: INFO)
# logfile: /var/log/hbc.log # Optional log file path
# ==============================================================================
# Example Profiles
# ==============================================================================
# Below are example configuration profiles for different use cases
# Minimal Configuration (default settings):
# -----------------------------------------
# server: hbd.example.com
# interval: 30
# Monitoring Server (comprehensive metrics):
# ------------------------------------------
# server: monitoring.example.com
# interval: 30
# plugins:
# cpu_monitor:
# enabled: true
# interval: 15
# per_core: true
# memory_monitor:
# enabled: true
# interval: 15
# disk_monitor:
# enabled: true
# interval: 60
# network_monitor:
# enabled: true
# interval: 30
# include_connections: true
# Nagios Integration (leverage existing plugins):
# -----------------------------------------------
# server: hbd.example.com
# plugins:
# nagios_runner:
# enabled: true
# interval: 300 # Check every 5 minutes
# commands:
# - /usr/lib/nagios/plugins/check_http -H localhost -p 80
# - /usr/lib/nagios/plugins/check_mysql -H localhost -u monitor -p password
# - /usr/lib/nagios/plugins/check_smtp -H mail.example.com
# ==============================================================================
# Threshold Configuration (for Heartbeat Daemon)
# ==============================================================================
# NOTE: Thresholds are configured on the SERVER side (hbd), not the client (hbc).
# This is just an example - see config_thresholds_example.yaml for comprehensive examples.
#
# Basic threshold example:
# thresholds:
# cpu_monitor:
# cpu_percent:
# warning: 80.0
# critical: 90.0
# memory_monitor:
# percent:
# warning: 85.0
# critical: 95.0
# disk_monitor:
# partitions:
# /:
# percent:
# warning: 80.0
# critical: 90.0
hbd/config_nagios_example.yaml
+111
View File
@@ -0,0 +1,111 @@
# Heartbeat Configuration Example with Nagios Plugin Runner
# This example shows how to configure the Nagios Runner plugin
# to execute existing Nagios-compatible monitoring plugins
# Basic server settings (existing config)
hb_port: 50003
hbd_port: 50004
interval: 20
grace: 2
# Plugin configuration
# Each plugin can have its own configuration section
# CPU Monitor Plugin
cpu_monitor:
interval: 300 # Collect every 5 minutes (default)
per_core: false # Set to true to get per-core CPU usage
# Nagios Runner Plugin
nagios_runner:
interval: 300 # Run Nagios plugins every 5 minutes (default)
timeout: 30 # Command execution timeout in seconds
shell: true # Execute commands via shell
# List of Nagios plugins to run
commands:
# Example 1: Check disk space
- name: check_disk_root
command: /usr/lib/nagios/plugins/check_disk -w 20% -c 10% -p /
# Example 2: Check disk space for /home
- name: check_disk_home
command: /usr/lib/nagios/plugins/check_disk -w 20% -c 10% -p /home
# Example 3: Check system load
- name: check_load
command: /usr/lib/nagios/plugins/check_load -w 5,4,3 -c 10,8,6
# Example 4: Check process count
- name: check_procs
command: /usr/lib/nagios/plugins/check_procs -w 250 -c 400
# Example 5: Check SSH service
- name: check_ssh
command: /usr/lib/nagios/plugins/check_ssh localhost
# Example 6: Check HTTP service
- name: check_http
command: /usr/lib/nagios/plugins/check_http -H localhost
# Example 7: Check swap usage
- name: check_swap
command: /usr/lib/nagios/plugins/check_swap -w 20% -c 10%
# Example 8: Custom script (Nagios plugin format)
- name: check_custom
command: /usr/local/bin/my_custom_check.sh
# Example 9: Check specific log file
- name: check_logs
command: /usr/lib/nagios/plugins/check_log -F /var/log/syslog -O /var/tmp/check_log.old -q "ERROR"
# Notes:
#
# 1. Nagios Plugin Output Format:
# - Single line: STATUS - Message | performance_data
# - Performance data format: 'label'=value[UOM];[warn];[crit];[min];[max]
#
# 2. Exit Codes:
# - 0 = OK
# - 1 = WARNING
# - 2 = CRITICAL
# - 3 = UNKNOWN
#
# 3. Performance Data:
# - Automatically parsed and included in heartbeat data
# - Metrics are stored as: {plugin_name}_{metric_name}
# - Example: check_disk_root_/ will contain the disk usage percentage
#
# 4. Overall Status:
# - The plugin reports the worst status from all commands
# - Useful for quick health checks
#
# 5. Plugin Paths:
# Common Nagios plugin directories:
# - Debian/Ubuntu: /usr/lib/nagios/plugins/
# - RHEL/CentOS: /usr/lib64/nagios/plugins/
# - Custom installs: /usr/local/nagios/libexec/
#
# 6. Installing Nagios Plugins:
# Debian/Ubuntu: sudo apt-get install nagios-plugins
# RHEL/CentOS: sudo yum install nagios-plugins-all
# Arch Linux: sudo pacman -S monitoring-plugins
#
# 7. Writing Custom Nagios Plugins:
# Any script can be a Nagios plugin if it:
# - Returns appropriate exit codes (0-3)
# - Prints status message to stdout
# - Optionally includes performance data after "|"
#
# Example custom plugin (save as /usr/local/bin/check_example.sh):
# #!/bin/bash
# if [ $(uptime | awk '{print $1}') -gt 50 ]; then
# echo "CRITICAL - Too many users | users=52;40;50;0"
# exit 2
# else
# echo "OK - Normal user count | users=25;40;50;0"
# exit 0
# fi
hbd/config_thresholds_example.yaml
+254
View File
@@ -0,0 +1,254 @@
# ==============================================================================
# Heartbeat Daemon Threshold Configuration Example
# ==============================================================================
# This file demonstrates threshold configuration for the Heartbeat monitoring system.
# Thresholds can be defined for any metric collected by monitoring plugins.
#
# Threshold levels:
# - WARNING: First level of concern, typically for early notification
# - CRITICAL: Severe condition requiring immediate attention
#
# Alert notifications are sent when:
# - A metric crosses from OK to WARNING or CRITICAL
# - A metric crosses from WARNING to CRITICAL
# - A metric recovers (returns to a lower severity level)
#
# Re-notifications are sent for ongoing alerts based on threshold_renotify_interval.
# ==============================================================================
# Global threshold settings
threshold_renotify_interval: 3600 # Re-notify every hour for ongoing alerts (seconds)
# Threshold definitions per plugin
thresholds:
# ----------------------------------------------------------------------------
# CPU Monitor Thresholds
# ----------------------------------------------------------------------------
cpu_monitor:
# Overall CPU usage percentage (0-100)
cpu_percent:
warning: 80.0 # Warn when CPU usage exceeds 80%
critical: 90.0 # Critical when CPU usage exceeds 90%
operator: ">" # Alert when value is GREATER than threshold
hysteresis: 0.1 # 10% hysteresis to prevent flapping
enabled: true
# 1-minute load average
load_1min:
warning: 4.0 # Warn when 1-min load exceeds 4.0
critical: 8.0 # Critical when 1-min load exceeds 8.0
operator: ">"
hysteresis: 0.15 # 15% hysteresis
enabled: true
# 5-minute load average
load_5min:
warning: 3.0
critical: 6.0
operator: ">"
hysteresis: 0.15
enabled: true
# 15-minute load average
load_15min:
warning: 2.0
critical: 4.0
operator: ">"
hysteresis: 0.15
enabled: true
# ----------------------------------------------------------------------------
# Memory Monitor Thresholds
# ----------------------------------------------------------------------------
memory_monitor:
# Memory usage percentage
percent:
warning: 85.0 # Warn at 85% memory usage
critical: 95.0 # Critical at 95% memory usage
operator: ">"
hysteresis: 0.1
enabled: true
# Available memory in MB (inverse threshold - alert when LOW)
available_mb:
warning: 1000 # Warn when less than 1GB available
critical: 500 # Critical when less than 500MB available
operator: "<" # Alert when value is LESS than threshold
hysteresis: 0.1
enabled: true
# Swap usage percentage
swap_percent:
warning: 50.0 # Warn at 50% swap usage
critical: 80.0 # Critical at 80% swap usage
operator: ">"
hysteresis: 0.1
enabled: true
# ----------------------------------------------------------------------------
# Disk Monitor Thresholds
# ----------------------------------------------------------------------------
disk_monitor:
# Partition-specific thresholds
# Use the mount point as the key
partitions:
# Root filesystem
/:
percent:
warning: 80.0 # Warn at 80% disk usage
critical: 90.0 # Critical at 90% disk usage
operator: ">"
hysteresis: 0.05 # 5% hysteresis for disk (more stable)
enabled: true
free_gb:
warning: 10.0 # Warn when less than 10GB free
critical: 5.0 # Critical when less than 5GB free
operator: "<"
hysteresis: 0.1
enabled: true
# Home filesystem (if separate partition)
/home:
percent:
warning: 85.0
critical: 95.0
operator: ">"
hysteresis: 0.05
enabled: true
# Var filesystem (logs, etc.)
/var:
percent:
warning: 80.0
critical: 90.0
operator: ">"
hysteresis: 0.05
enabled: true
free_gb:
warning: 5.0 # Var needs space for logs
critical: 2.0
operator: "<"
hysteresis: 0.1
enabled: true
# ----------------------------------------------------------------------------
# Network Monitor Thresholds
# ----------------------------------------------------------------------------
network_monitor:
# Total error count across all interfaces
errors_total:
warning: 100 # Warn at 100 errors
critical: 1000 # Critical at 1000 errors
operator: ">"
hysteresis: 0.2 # 20% hysteresis for counters
enabled: true
# Total dropped packets
dropin_total:
warning: 50
critical: 200
operator: ">"
hysteresis: 0.2
enabled: true
dropout_total:
warning: 50
critical: 200
operator: ">"
hysteresis: 0.2
enabled: true
# TCP connections in TIME_WAIT state
connections_TIME_WAIT:
warning: 1000 # Warn at 1000 TIME_WAIT connections
critical: 5000 # Critical at 5000 TIME_WAIT connections
operator: ">"
hysteresis: 0.2
enabled: true
# Total established connections
connections_ESTABLISHED:
warning: 500
critical: 1000
operator: ">"
hysteresis: 0.1
enabled: true
# ----------------------------------------------------------------------------
# Nagios Plugin Thresholds (if using nagios_runner)
# ----------------------------------------------------------------------------
nagios_runner:
# Nagios plugins report exit codes:
# 0 = OK, 1 = WARNING, 2 = CRITICAL, 3 = UNKNOWN
# We can threshold on the exit_code directly
exit_code:
warning: 1 # Map Nagios WARNING to our WARNING
critical: 2 # Map Nagios CRITICAL to our CRITICAL
operator: ">=" # Alert when exit code >= threshold
hysteresis: 0.0 # No hysteresis for exit codes
enabled: true
# ==============================================================================
# Notification Configuration
# ==============================================================================
# Configure notification methods (email, pushover, etc.)
# These are used when threshold violations occur
# Email notifications
toemail:
- admin@example.com
- oncall@example.com
fromemail: heartbeat@example.com
smtpserver: smtp.example.com
smtpport: 587
smtpuser: heartbeat@example.com
smtppassword: your-password-here
# Pushover notifications (optional)
# pushover_token: your-pushover-app-token
# pushover_user: your-pushover-user-key
# Mattermost webhook (optional)
# mattermost_url: https://mattermost.example.com/hooks/your-webhook-id
# ==============================================================================
# Watched Hosts
# ==============================================================================
# Hosts in this list will trigger notifications for:
# - Heartbeat timeouts/overdue
# - Threshold violations
# - Boot messages
watchhosts:
- webserver01
- database01
- mailserver
- critical-app
# ==============================================================================
# Additional Server Settings
# ==============================================================================
hb_port: 50003 # UDP port for heartbeat messages
hbd_port: 50004 # HTTP port for web interface
grace: 10 # Grace period for overdue detection (seconds)
debug: 0 # Debug level (0-3)
verbose: false # Verbose output
# Journal settings (message logging)
journal_enabled: true
journal_path: /var/log/heartbeat/messages.journal
journal_max_size: 104857600 # 100MB before rotation
journal_max_backups: 10
# ==============================================================================
# Example: Production Configuration with Conservative Thresholds
# ==============================================================================
# For production systems, consider:
# - Higher warning thresholds to reduce alert fatigue
# - Appropriate hysteresis values (5-15% typical)
# - Re-notification intervals matching on-call rotation
# - Multiple escalation contacts
# - Integration with incident management systems
# ==============================================================================
hbd/hbc.py → hbd/hbc_old.py.bak
View File
hbd/proto.py
-82
View File
@@ -1,82 +0,0 @@
"""Message encoding/decoding utilities for hbd protocol."""
from typing import Dict, Any
import zlib
def dicttos(ID: str, d: Dict[str, Any], compress: bool = False):
"""Serialize a dict to protocol message bytes.
If compress is True, the payload is zlib-compressed and the message is
prefixed with `!ID:` as the original script did. Otherwise the format is
`ID:key=value;...` (bytes).
"""
s = []
for k in d:
v = d[k]
if isinstance(v, float):
s.append(f"{k}={v:0.5f}")
else:
s.append(f"{k}={v}")
pk = ";".join(s)
if compress:
zpk = zlib.compress(pk.encode(), 6)
hdr = ("!" + ID + ":").encode()
return hdr + zpk
else:
return (ID + ":" + pk).encode()
def stodict(msg: bytes):
"""Deserialize a protocol message into a dict.
Mirrors original behaviour: detects compressed messages starting with
'!' and decodes accordingly. Returns a dict with key 'ID' set to the
message ID and the parsed key/value pairs.
"""
d = {}
if len(msg) > 0 and chr(msg[0]) == "!":
# message is: b'!ID:' + compressed_payload
# original code used msg[1:4].decode() for ID (3 bytes including colon)
try:
pk = zlib.decompress(msg[5:]).decode()
except Exception:
# malformed compressed payload
return {}
d["ID"] = msg[1:4].decode()
else:
try:
r0 = msg.split(b":", 1)
pk = r0[1].decode()
d["ID"] = r0[0].decode()
except Exception:
return {}
if not pk:
return d
parts = pk.split(";")
for v in parts:
if not v:
continue
vr = v.split("=", 1)
k = vr[0].strip()
if len(vr) == 1:
d[k] = None
else:
val = vr[1].strip()
if val and val[0].isdigit():
try:
val_e = eval(val)
except Exception:
val_e = val
d[k] = val_e
else:
d[k] = val
return d
def oldmtodict(msg: bytes):
"""Compatibility wrapper for old-style messages (no ID prefix).
The original implementation prefixed with 'HTB:' and called stodict.
"""
return stodict(b"HTB:" + msg)
hbd/server/__init__.py
+3
View File
@@ -0,0 +1,3 @@
"""HeartBeat Daemon (hbd) - Server/daemon component."""
__version__ = "5.0.5"
hbd/cli.py → hbd/server/cli.py
+1 -1
View File
@@ -3,7 +3,7 @@
import argparse
from .config import load_config
from .server import run as run_server
from .main import run as run_server
PUSHSRVS = ["all", "pushover", "mattermost"]
hbd/server/config.py
+103
View File
@@ -0,0 +1,103 @@
"""Configuration loader and defaults for hbd (HeartBeat Daemon/Server)."""
import logging
import os
try:
import yaml
except Exception:
yaml = None
SERVER_DEFAULTS = {
# Network settings
"hb_port": 50003, # Port to listen for heartbeats
"hbd_port": 50004, # HTTP API port
"hbd_host": "", # Bind address (empty = all interfaces)
# Persistence
"pickfile": "/tmp/hb.pick",
# Logging
"logfile": "/var/log/heartbeat.log",
"logfmt": "text", # text or msg or json
# Notification settings
"pushsrv": "pushover", # pushover, mattermost, or all
"pushover_token": "",
"pushover_user": "",
# Monitoring settings
"interval": 20, # Expected heartbeat interval (for server checks)
"grace": 2, # Grace multiplier (interval * grace = timeout)
"threshold_renotify_interval": 3600, # Seconds between threshold re-notifications
# Host management
"watchhosts": [], # Hosts to monitor and notify about
"dyndnshosts": [], # Hosts with dynamic DNS
"drophosts": [], # Hosts to ignore
"dyndomains": ["wrede.org"],
# DNS updates
"nsupdate_bin": "/usr/bin/nsupdate",
# Email settings
"smtpserver": "smtp.fastmail.com",
"smtpuser": "andreas@wrede.ca",
"smtppassword": "pvtvefyp5gbhnch2",
"smtpport": 587,
"toemail": ["aew.hbd.notify@wrede.ca"],
"fromemail": "aew.hbd@wrede.ca",
# WebSocket settings
"ws_port": 50005,
"wss_port": None,
"cert_path": "/usr/local/etc/ssl/",
"wss_pem": "fullchain.pem",
"wss_key": "privkey.pem",
# Message journal configuration
"journal_enabled": True,
"journal_dir": "/var/log/heartbeat",
"journal_file": "messages.journal",
"journal_max_size": 100 * 1024 * 1024, # 100MB
"journal_max_backups": 10,
# Runtime flags
"foreground": False,
"verbose": False,
"debug": 0,
# Plugin/threshold configs (for clients reporting to this server)
"plugins": {},
"thresholds": {},
}
def load_config(path=None):
"""Load configuration from a YAML file and merge with server defaults.
If YAML is not available or the file does not exist, defaults are returned.
Args:
path: Path to YAML config file (default: ~/.hb.yaml)
Returns:
Dictionary with configuration
"""
cfg = SERVER_DEFAULTS.copy()
if not path:
# default path (~/.hb.yaml)
path = os.path.join(os.path.expanduser("~"), ".hb.yaml")
if os.path.exists(path):
if yaml:
with open(path) as fh:
data = yaml.safe_load(fh)
# Merge YAML data with defaults
# Keep all keys from YAML to support plugin configs and future extensions
for k, v in data.items():
cfg[k] = v
else:
# yaml not installed: do not attempt to parse; user must ensure defaults
pass
return cfg
hbd/dns.py → hbd/server/dns.py
View File
hbd/hbdclass.py → hbd/server/hbdclass.py
+69
View File
@@ -179,6 +179,11 @@ class Host:
self.cver = 0
self.connections = {}
self.hdwcounts = [[0, 0], [0, 0], [0, 0]]
# Plugin data storage: {plugin_name: [(timestamp, data), ...]}
self.plugin_data = {}
self.plugin_retention = 100 # Keep last N samples per plugin
# Alert state tracking: {metric_path: AlertState}
self.alert_states = {}
def statedict(self):
d = {}
@@ -272,8 +277,72 @@ class Host:
addr = addr[7:]
self.connections[c].addr = addr
# Add plugin_data if missing (for backward compatibility)
if not hasattr(self, "plugin_data"):
self.plugin_data = {}
if not hasattr(self, "plugin_retention"):
self.plugin_retention = 100
if not hasattr(self, "alert_states"):
self.alert_states = {}
pass
def add_plugin_data(self, plugin_name, data, timestamp=None):
"""Store plugin data with timestamp.
Args:
plugin_name: Name of the plugin (e.g., "cpu_monitor")
data: Dict of plugin data
timestamp: Optional timestamp (default: current time)
"""
if timestamp is None:
timestamp = time.time()
if plugin_name not in self.plugin_data:
self.plugin_data[plugin_name] = []
# Add new data
self.plugin_data[plugin_name].append((timestamp, data))
# Enforce retention limit (keep last N samples)
if len(self.plugin_data[plugin_name]) > self.plugin_retention:
self.plugin_data[plugin_name] = self.plugin_data[plugin_name][-self.plugin_retention:]
def get_plugin_data(self, plugin_name, limit=None):
"""Retrieve plugin data for a specific plugin.
Args:
plugin_name: Name of the plugin
limit: Optional limit on number of recent samples to return
Returns:
List of (timestamp, data) tuples, most recent last
"""
data = self.plugin_data.get(plugin_name, [])
if limit and len(data) > limit:
return data[-limit:]
return data
def get_latest_plugin_data(self, plugin_name):
"""Get the most recent plugin data for a plugin.
Args:
plugin_name: Name of the plugin
Returns:
(timestamp, data) tuple or None if no data
"""
data = self.plugin_data.get(plugin_name, [])
return data[-1] if data else None
def get_all_plugin_data(self):
"""Get all plugin data for this host.
Returns:
Dict of {plugin_name: [(timestamp, data), ...]}
"""
return self.plugin_data
# def dispstate(self):
# if self.state in ["down", "overdue"]:
# state = "<b>%s</b>" % self.state
hbd/http.py → hbd/server/http.py
+193
View File
@@ -32,6 +32,7 @@ async def start(
verbose=False,
get_now=None,
VER="",
threshold_checker=None,
):
"""Start an aiohttp web server and block until cancelled.
@@ -183,17 +184,209 @@ async def start(
return web.Response(status=404, text="Not Found")
return web.FileResponse(path=target)
# -------------------------------------------------------------------------
# Plugin Data API Endpoints
# -------------------------------------------------------------------------
async def api_host_plugins(request):
"""Get all plugin data for a specific host."""
hostname = request.match_info.get("hostname")
if hostname not in hbdclass.Host.hosts:
return web.json_response(
{"error": f"Host '{hostname}' not found"},
status=404
)
host = hbdclass.Host.hosts[hostname]
# Get plugin data with most recent sample for each plugin
plugins_summary = {}
for plugin_name, samples in host.plugin_data.items():
if samples:
# Get most recent sample
timestamp, data = samples[-1]
plugins_summary[plugin_name] = {
"timestamp": timestamp,
"data": data,
"sample_count": len(samples),
}
return web.json_response({
"hostname": hostname,
"plugins": plugins_summary,
})
async def api_host_plugin_detail(request):
"""Get detailed data for a specific plugin on a host."""
hostname = request.match_info.get("hostname")
plugin_name = request.match_info.get("plugin_name")
if hostname not in hbdclass.Host.hosts:
return web.json_response(
{"error": f"Host '{hostname}' not found"},
status=404
)
host = hbdclass.Host.hosts[hostname]
# Get limit from query parameter
limit = request.rel_url.query.get("limit", "10")
try:
limit = int(limit)
except ValueError:
limit = 10
# Get plugin data
samples = host.get_plugin_data(plugin_name, limit=limit)
if not samples:
return web.json_response(
{"error": f"No data for plugin '{plugin_name}' on host '{hostname}'"},
status=404
)
# Format samples
formatted_samples = [
{
"timestamp": ts,
"data": data,
}
for ts, data in samples
]
return web.json_response({
"hostname": hostname,
"plugin": plugin_name,
"samples": formatted_samples,
"sample_count": len(formatted_samples),
})
async def api_host_alerts(request):
"""Get alert states for a specific host."""
hostname = request.match_info.get("hostname")
if hostname not in hbdclass.Host.hosts:
return web.json_response(
{"error": f"Host '{hostname}' not found"},
status=404
)
host = hbdclass.Host.hosts[hostname]
# Get alert states
alerts = []
for metric_path, alert_state in host.alert_states.items():
alerts.append(alert_state.to_dict())
# Get summary if threshold_checker available
summary = {"ok": 0, "warning": 0, "critical": 0, "unknown": 0}
if threshold_checker:
summary = threshold_checker.get_alert_summary(host.alert_states)
return web.json_response({
"hostname": hostname,
"alerts": alerts,
"summary": summary,
})
async def api_all_alerts(request):
"""Get all active alerts across all hosts."""
all_alerts = []
for hostname, host in hbdclass.Host.hosts.items():
if threshold_checker:
active_alerts = threshold_checker.get_active_alerts(host.alert_states)
else:
# Fallback if no threshold checker
from hbd.client.threshold import AlertLevel
active_alerts = [
state for state in host.alert_states.values()
if state.level != AlertLevel.OK
]
for alert in active_alerts:
alert_dict = alert.to_dict()
alert_dict["hostname"] = hostname
all_alerts.append(alert_dict)
# Sort by level (critical first) then by hostname
level_order = {"CRITICAL": 0, "WARNING": 1, "UNKNOWN": 2, "OK": 3}
all_alerts.sort(
key=lambda a: (level_order.get(a["level"], 99), a["hostname"], a["metric_path"])
)
# Get summary counts
summary = {"critical": 0, "warning": 0, "unknown": 0, "total": len(all_alerts)}
for alert in all_alerts:
level = alert["level"].lower()
if level in summary:
summary[level] += 1
return web.json_response({
"alerts": all_alerts,
"summary": summary,
"host_count": len(hbdclass.Host.hosts),
})
# -------------------------------------------------------------------------
# UI Pages
# -------------------------------------------------------------------------
async def plugins_page(request):
"""Render the plugin metrics visualization page."""
pkg_dir = os.path.dirname(__file__)
templates_dir = config.get("templates_dir", os.path.join(pkg_dir, "templates"))
env = jinja2.Environment(loader=jinja2.FileSystemLoader(templates_dir))
# Collect all hosts with plugin data
hosts_with_plugins = []
for hostname in sorted(hbdclass.Host.hosts.keys()):
host = hbdclass.Host.hosts[hostname]
if host.plugin_data:
hosts_with_plugins.append({
"name": hostname,
"plugins": list(host.plugin_data.keys()),
})
tmpl = env.get_template("plugins.html")
body = tmpl.render(
title="Plugin Metrics - Heartbeat",
header="Plugin Metrics",
hosts=hosts_with_plugins,
)
return web.Response(text=body, content_type="text/html")
async def alerts_page(request):
"""Render the alerts dashboard page."""
pkg_dir = os.path.dirname(__file__)
templates_dir = config.get("templates_dir", os.path.join(pkg_dir, "templates"))
env = jinja2.Environment(loader=jinja2.FileSystemLoader(templates_dir))
tmpl = env.get_template("alerts.html")
body = tmpl.render(
title="Alerts Dashboard - Heartbeat",
header="Alerts Dashboard",
)
return web.Response(text=body, content_type="text/html")
app = web.Application()
app.add_routes(
[
web.get("/", index),
web.get("/api/0/hosts", api_hosts),
web.get("/api/0/messages", api_messages),
web.get("/api/0/hosts/{hostname}/plugins", api_host_plugins),
web.get("/api/0/hosts/{hostname}/plugins/{plugin_name}", api_host_plugin_detail),
web.get("/api/0/hosts/{hostname}/alerts", api_host_alerts),
web.get("/api/0/alerts", api_all_alerts),
web.get("/c", cmd),
web.get("/d", drop),
web.get("/n", register),
web.get("/u", update),
web.get("/live", live),
web.get("/plugins", plugins_page),
web.get("/alerts", alerts_page),
web.get("/static/{path:.*}", static),
web.get("/favicon.ico", favicon),
]
hbd/server/journal.py
+342
View File
@@ -0,0 +1,342 @@
"""
Journal logging for heartbeat messages.
Provides size-based rotating log files for all received heartbeat messages.
Messages are logged in JSON format for easy parsing and analysis.
"""
import json
import logging
import os
import asyncio
from datetime import datetime
from pathlib import Path
from typing import Dict, Any, Optional
logger = logging.getLogger(__name__)
class MessageJournal:
"""
Journal logger for heartbeat messages with size-based rotation.
Features:
- Logs all received messages in JSON format
- Automatic rotation when file size exceeds threshold
- Keeps configurable number of rotated logs
- Thread-safe and async-safe operation
- Configurable log directory and file naming
Configuration:
journal_dir: Directory for journal files (default: /var/log/heartbeat/)
journal_file: Base filename (default: messages.journal)
max_size: Maximum file size in bytes before rotation (default: 100MB)
max_backups: Number of backup files to keep (default: 10)
enabled: Enable/disable journaling (default: True)
"""
def __init__(self, config: Optional[Dict[str, Any]] = None):
"""
Initialize the message journal.
Args:
config: Configuration dictionary with journal settings
"""
self.config = config or {}
# Configuration options
self.journal_dir = Path(self.config.get('journal_dir', '/var/log/heartbeat'))
self.journal_file = self.config.get('journal_file', 'messages.journal')
self.max_size = self.config.get('journal_max_size', 100 * 1024 * 1024) # 100MB default
self.max_backups = self.config.get('journal_max_backups', 10)
self.enabled = self.config.get('journal_enabled', True)
# Runtime state
self._file_handle = None
self._current_size = 0
self._lock = asyncio.Lock()
self._initialized = False
# Full path to current journal file
self.journal_path = self.journal_dir / self.journal_file
async def initialize(self) -> bool:
"""
Initialize the journal.
Creates journal directory if needed and opens the journal file.
Returns:
True if initialization successful, False otherwise
"""
if not self.enabled:
logger.info("Message journal disabled in configuration")
return True
try:
# Create journal directory if it doesn't exist
self.journal_dir.mkdir(parents=True, exist_ok=True)
# Open journal file in append mode
self._file_handle = open(self.journal_path, 'a', encoding='utf-8')
# Get current file size
try:
self._current_size = os.path.getsize(self.journal_path)
except OSError:
self._current_size = 0
self._initialized = True
logger.info(f"Message journal initialized: {self.journal_path} "
f"(current size: {self._current_size:,} bytes, "
f"max: {self.max_size:,} bytes)")
return True
except Exception as e:
logger.error(f"Failed to initialize message journal: {e}")
self.enabled = False
return False
async def log_message(
self,
msg: Dict[str, Any],
addr: tuple,
timestamp: Optional[float] = None
):
"""
Log a received message to the journal.
Args:
msg: Parsed message dictionary
addr: Source address (ip, port) tuple
timestamp: Message timestamp (defaults to current time)
"""
if not self.enabled or not self._initialized:
return
# Skip HTB (heartbeat) messages - too verbose
msg_id = msg.get('ID', '')
if msg_id == 'HTB':
return
async with self._lock:
try:
# Prepare journal entry
if timestamp is None:
import time
timestamp = time.time()
entry = {
'timestamp': timestamp,
'datetime': datetime.fromtimestamp(timestamp).isoformat(),
'source_ip': addr[0] if isinstance(addr, (tuple, list)) else str(addr),
'source_port': addr[1] if isinstance(addr, (tuple, list)) and len(addr) > 1 else None,
'message': msg
}
# Serialize to JSON (one line per entry)
json_line = json.dumps(entry, separators=(',', ':')) + '\n'
json_bytes = json_line.encode('utf-8')
# Check if rotation is needed
if self._current_size + len(json_bytes) > self.max_size:
await self._rotate()
# Write to journal
if self._file_handle:
self._file_handle.write(json_line)
self._file_handle.flush() # Ensure data is written
self._current_size += len(json_bytes)
logger.debug(f"Logged message from {addr[0]}: {msg.get('ID', 'UNKNOWN')}")
except Exception as e:
logger.error(f"Error writing to journal: {e}")
async def _rotate(self):
"""
Rotate the journal file.
Renames current file with timestamp, opens new file, and removes
old backups exceeding max_backups limit.
"""
try:
# Close current file
if self._file_handle:
self._file_handle.close()
self._file_handle = None
# Generate backup filename with timestamp
timestamp_str = datetime.now().strftime('%Y%m%d-%H%M%S')
backup_name = f"{self.journal_file}.{timestamp_str}"
backup_path = self.journal_dir / backup_name
# Rename current file to backup
if self.journal_path.exists():
self.journal_path.rename(backup_path)
logger.info(f"Rotated journal: {backup_path} "
f"(size: {self._current_size:,} bytes)")
# Open new journal file
self._file_handle = open(self.journal_path, 'a', encoding='utf-8')
self._current_size = 0
# Clean up old backups
await self._cleanup_old_backups()
except Exception as e:
logger.error(f"Error rotating journal: {e}")
# Try to reopen the file even if rotation failed
try:
self._file_handle = open(self.journal_path, 'a', encoding='utf-8')
except Exception as e2:
logger.error(f"Failed to reopen journal after rotation error: {e2}")
self.enabled = False
async def _cleanup_old_backups(self):
"""
Remove old backup files exceeding max_backups limit.
Keeps only the most recent backups based on filename (which includes timestamp).
"""
try:
# Find all backup files
backup_pattern = f"{self.journal_file}.*"
backup_files = sorted(self.journal_dir.glob(backup_pattern))
# Remove oldest backups if we have too many
if len(backup_files) > self.max_backups:
files_to_remove = backup_files[:len(backup_files) - self.max_backups]
for backup_file in files_to_remove:
try:
backup_file.unlink()
logger.info(f"Removed old backup: {backup_file.name}")
except Exception as e:
logger.warning(f"Failed to remove old backup {backup_file}: {e}")
except Exception as e:
logger.error(f"Error cleaning up old backups: {e}")
async def log_threshold_event(
self,
host_name: str,
metric_path: str,
old_level: str,
new_level: str,
value: Any,
timestamp: Optional[float] = None
):
"""
Log a threshold state change event.
Args:
host_name: Name of the host
metric_path: Full metric path (e.g., "cpu_monitor.cpu_percent")
old_level: Previous alert level
new_level: New alert level
value: Current metric value
timestamp: Event timestamp (default: current time)
"""
if not self.enabled or not self._initialized:
return
try:
if timestamp is None:
timestamp = __import__('time').time()
event = {
'timestamp': timestamp,
'iso_time': datetime.fromtimestamp(timestamp).isoformat(),
'event_type': 'threshold',
'host': host_name,
'metric': metric_path,
'old_level': old_level,
'new_level': new_level,
'value': value,
}
async with self._lock:
if not self._file_handle:
return
# Check if rotation is needed
if self._current_size >= self.max_size:
await self._rotate()
# Write event
line = json.dumps(event) + '\n'
self._file_handle.write(line)
self._file_handle.flush()
# Update size
self._current_size += len(line.encode('utf-8'))
except Exception as e:
logger.error(f"Error logging threshold event: {e}")
async def close(self):
"""
Close the journal and release resources.
Should be called during shutdown.
"""
async with self._lock:
if self._file_handle:
try:
self._file_handle.close()
logger.info("Message journal closed")
except Exception as e:
logger.error(f"Error closing journal: {e}")
finally:
self._file_handle = None
self._initialized = False
def get_stats(self) -> Dict[str, Any]:
"""
Get journal statistics.
Returns:
Dictionary with journal stats
"""
return {
'enabled': self.enabled,
'initialized': self._initialized,
'current_file': str(self.journal_path),
'current_size': self._current_size,
'max_size': self.max_size,
'max_backups': self.max_backups,
'rotation_threshold': f"{(self._current_size / self.max_size * 100):.1f}%"
}
# Global journal instance
_journal_instance: Optional[MessageJournal] = None
def get_journal(config: Optional[Dict[str, Any]] = None) -> MessageJournal:
"""
Get or create the global journal instance.
Args:
config: Configuration dictionary (only used on first call)
Returns:
MessageJournal instance
"""
global _journal_instance
if _journal_instance is None:
_journal_instance = MessageJournal(config)
return _journal_instance
async def log_message(msg: Dict[str, Any], addr: tuple, timestamp: Optional[float] = None):
"""
Convenience function to log a message using the global journal.
Args:
msg: Parsed message dictionary
addr: Source address (ip, port) tuple
timestamp: Message timestamp (defaults to current time)
"""
journal = get_journal()
await journal.log_message(msg, addr, timestamp)
hbd/server.py → hbd/server/main.py
+27 -1
View File
@@ -83,9 +83,24 @@ async def _run_async(config):
from . import dns as dns_mod
from . import notify as notify_mod
from . import monitor as monitor_mod
from . import journal as journal_mod
from ..client import threshold as threshold_mod
notify_mod.setup(config)
# Initialize message journal
msg_journal = journal_mod.get_journal(config)
await msg_journal.initialize()
# Initialize threshold checker
threshold_checker = threshold_mod.ThresholdChecker(
config=config,
notification_callback=notify_mod.pushmsg_from_config,
renotify_interval=config.get("threshold_renotify_interval", 3600),
journal=msg_journal,
)
logger.info("Threshold checker initialized")
pushmsg = notify_mod.pushmsg_from_config
sock = socket.socket(socket.AF_INET6, socket.SOCK_DGRAM)
@@ -113,6 +128,8 @@ async def _run_async(config):
log=log,
pushmsg=pushmsg,
msg_to_websockets=msg_to_websockets,
msg_journal=msg_journal,
threshold_checker=threshold_checker,
DEBUG=config.get("debug", 0),
verbose=config.get("verbose", False),
)
@@ -135,6 +152,7 @@ async def _run_async(config):
log=log,
pushmsg=pushmsg,
msg_to_websockets=msg_to_websockets,
threshold_checker=threshold_checker,
tcss=None,
DEBUG=config.get("debug", 0),
verbose=config.get("verbose", False),
@@ -180,10 +198,12 @@ async def _run_async(config):
ssl_context = None
try:
ws_port = config.get("ws_port", 50005)
logger.info("Starting WebSocket server on port %s", ws_port)
ws_task = asyncio.create_task(
ws_mod.start(
host=config.get("hbd_host", ""),
ws_port=config.get("ws_port", None),
ws_port=ws_port,
wss_port=config.get("wss_port", None),
ssl_context=ssl_context,
get_hosts=lambda: [
@@ -249,6 +269,12 @@ async def _run_async(config):
except Exception as e:
logger.debug("Exception during task cancellation: %s", e)
# Close message journal
try:
await msg_journal.close()
except Exception as e:
logger.warning("Error closing message journal: %s", e)
# Signal DNS worker to exit and await it
try:
if "dns_task" in locals() and dns_task:
hbd/monitor.py → hbd/server/monitor.py
View File
hbd/notify.py → hbd/server/notify.py
View File
hbd/static/images/favicon.ico → hbd/server/static/images/favicon.ico
View File
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 5.3 KiB

After

Width:  |  Height:  |  Size: 5.3 KiB

hbd/static/style.css → hbd/server/static/style.css
View File
hbd/server/templates/alerts.html
+466
View File
@@ -0,0 +1,466 @@
<!DOCTYPE html>
<html>
{% include 'head.html' %}
<style>
body {
margin: 20px;
background: #f5f5f5;
}
.nav {
background: #fff;
padding: 15px;
margin-bottom: 20px;
box-shadow: 0 2px 4px rgba(0,0,0,0.1);
border-radius: 4px;
}
.nav a {
margin-right: 20px;
text-decoration: none;
color: #0066cc;
font-weight: 500;
}
.nav a:hover {
text-decoration: underline;
}
.nav a.active {
color: #333;
font-weight: bold;
}
.container {
max-width: 1400px;
margin: 0 auto;
}
h1 {
color: #333;
margin-bottom: 10px;
}
.subtitle {
color: #666;
margin-bottom: 30px;
}
.summary-cards {
display: grid;
grid-template-columns: repeat(auto-fit, minmax(200px, 1fr));
gap: 20px;
margin-bottom: 30px;
}
.summary-card {
background: white;
border-radius: 8px;
padding: 20px;
box-shadow: 0 2px 8px rgba(0,0,0,0.1);
text-align: center;
}
.summary-card.critical {
border-left: 5px solid #f44336;
}
.summary-card.warning {
border-left: 5px solid #ff9800;
}
.summary-card.ok {
border-left: 5px solid #4caf50;
}
.summary-number {
font-size: 3em;
font-weight: bold;
margin: 10px 0;
}
.summary-number.critical {
color: #f44336;
}
.summary-number.warning {
color: #ff9800;
}
.summary-number.ok {
color: #4caf50;
}
.summary-label {
color: #666;
text-transform: uppercase;
font-size: 0.9em;
letter-spacing: 1px;
}
.filters {
background: white;
border-radius: 8px;
padding: 15px;
margin-bottom: 20px;
box-shadow: 0 2px 8px rgba(0,0,0,0.1);
display: flex;
gap: 15px;
align-items: center;
}
.filter-label {
font-weight: bold;
color: #555;
}
.filter-button {
padding: 8px 16px;
border: 2px solid #ddd;
background: white;
border-radius: 20px;
cursor: pointer;
transition: all 0.2s;
font-size: 0.9em;
}
.filter-button:hover {
border-color: #2196f3;
}
.filter-button.active {
background: #2196f3;
color: white;
border-color: #2196f3;
}
.alerts-container {
background: white;
border-radius: 8px;
padding: 20px;
box-shadow: 0 2px 8px rgba(0,0,0,0.1);
}
.alert-item {
border-left: 5px solid #ddd;
padding: 15px;
margin-bottom: 15px;
background: #fafafa;
border-radius: 4px;
display: flex;
justify-content: space-between;
align-items: center;
transition: all 0.2s;
}
.alert-item:hover {
box-shadow: 0 2px 8px rgba(0,0,0,0.1);
transform: translateX(5px);
}
.alert-item.critical {
border-left-color: #f44336;
background: #ffebee;
}
.alert-item.warning {
border-left-color: #ff9800;
background: #fff3e0;
}
.alert-item.unknown {
border-left-color: #9e9e9e;
background: #f5f5f5;
}
.alert-main {
flex: 1;
}
.alert-header {
display: flex;
align-items: center;
gap: 15px;
margin-bottom: 8px;
}
.alert-level {
padding: 4px 12px;
border-radius: 12px;
font-size: 0.75em;
font-weight: bold;
text-transform: uppercase;
letter-spacing: 0.5px;
}
.alert-level.critical {
background: #f44336;
color: white;
}
.alert-level.warning {
background: #ff9800;
color: white;
}
.alert-level.unknown {
background: #9e9e9e;
color: white;
}
.alert-hostname {
font-weight: bold;
color: #333;
font-size: 1.1em;
}
.alert-metric {
color: #666;
font-family: 'Courier New', monospace;
font-size: 0.9em;
}
.alert-details {
display: flex;
gap: 20px;
color: #666;
font-size: 0.9em;
}
.alert-value {
font-weight: bold;
color: #333;
}
.alert-duration {
color: #999;
font-size: 0.85em;
}
.no-alerts {
text-align: center;
padding: 60px 20px;
color: #999;
}
.no-alerts-icon {
font-size: 4em;
margin-bottom: 20px;
}
.loading {
text-align: center;
padding: 40px;
color: #666;
}
.error {
background: #ffebee;
border-left: 4px solid #f44336;
padding: 20px;
margin: 20px 0;
border-radius: 4px;
color: #c62828;
}
.refresh-info {
text-align: center;
color: #999;
font-size: 0.85em;
margin-top: 20px;
padding-top: 20px;
border-top: 1px solid #e0e0e0;
}
.last-update {
color: #666;
font-size: 0.9em;
text-align: right;
margin-bottom: 15px;
}
</style>
<body>
<div class="nav">
<a href="/live">Live Dashboard</a>
<a href="/plugins">Plugin Metrics</a>
<a href="/alerts" class="active">Alerts</a>
</div>
<div class="container">
<h1>{{ header }}</h1>
<p class="subtitle">Real-time monitoring alerts and threshold violations</p>
<div class="summary-cards" id="summary-cards">
<div class="summary-card critical">
<div class="summary-label">Critical</div>
<div class="summary-number critical" id="critical-count">-</div>
</div>
<div class="summary-card warning">
<div class="summary-label">Warning</div>
<div class="summary-number warning" id="warning-count">-</div>
</div>
<div class="summary-card ok">
<div class="summary-label">Total Hosts</div>
<div class="summary-number ok" id="host-count">-</div>
</div>
</div>
<div class="filters">
<span class="filter-label">Show:</span>
<button class="filter-button active" onclick="filterAlerts('all')">All</button>
<button class="filter-button" onclick="filterAlerts('critical')">Critical Only</button>
<button class="filter-button" onclick="filterAlerts('warning')">Warning Only</button>
</div>
<div class="alerts-container">
<div class="last-update">Last updated: <span id="last-update-time">Never</span></div>
<div id="alerts-list">
<div class="loading">Loading alerts...</div>
</div>
<div class="refresh-info">
Auto-refreshing every 15 seconds
</div>
</div>
</div>
<script>
let currentFilter = 'all';
let allAlerts = [];
async function loadAlerts() {
try {
const response = await fetch('/api/0/alerts');
if (!response.ok) {
throw new Error(`HTTP ${response.status}`);
}
const data = await response.json();
allAlerts = data.alerts;
// Update summary cards
document.getElementById('critical-count').textContent = data.summary.critical || 0;
document.getElementById('warning-count').textContent = data.summary.warning || 0;
document.getElementById('host-count').textContent = data.host_count || 0;
// Update last update time
document.getElementById('last-update-time').textContent = new Date().toLocaleTimeString();
// Render alerts
renderAlerts(allAlerts);
} catch (error) {
document.getElementById('alerts-list').innerHTML =
`<div class="error">Failed to load alerts: ${error.message}</div>`;
}
}
function renderAlerts(alerts) {
const container = document.getElementById('alerts-list');
// Filter alerts based on current filter
let filteredAlerts = alerts;
if (currentFilter !== 'all') {
filteredAlerts = alerts.filter(alert =>
alert.level.toLowerCase() === currentFilter
);
}
if (filteredAlerts.length === 0) {
if (currentFilter === 'all' && alerts.length === 0) {
container.innerHTML = `
<div class="no-alerts">
<div class="no-alerts-icon">✓</div>
<h2>All Systems Normal</h2>
<p>No active alerts at this time</p>
</div>
`;
} else {
container.innerHTML = `
<div class="no-alerts">
<p>No ${currentFilter} alerts</p>
</div>
`;
}
return;
}
let html = '';
for (const alert of filteredAlerts) {
html += renderAlert(alert);
}
container.innerHTML = html;
}
function renderAlert(alert) {
const level = alert.level.toLowerCase();
const duration = getDuration(alert.since);
return `
<div class="alert-item ${level}">
<div class="alert-main">
<div class="alert-header">
<span class="alert-level ${level}">${alert.level}</span>
<span class="alert-hostname">${alert.hostname}</span>
</div>
<div class="alert-metric">${alert.metric_path}</div>
<div class="alert-details">
<span>Value: <span class="alert-value">${formatValue(alert.last_value)}</span></span>
<span class="alert-duration">Active for ${duration}</span>
</div>
</div>
</div>
`;
}
function formatValue(value) {
if (typeof value === 'number') {
if (value > 1000) {
return value.toLocaleString();
}
return value.toFixed(2);
}
return value;
}
function getDuration(timestamp) {
const now = Date.now() / 1000;
const seconds = Math.floor(now - timestamp);
if (seconds < 60) {
return `${seconds}s`;
} else if (seconds < 3600) {
return `${Math.floor(seconds / 60)}m`;
} else if (seconds < 86400) {
const hours = Math.floor(seconds / 3600);
const minutes = Math.floor((seconds % 3600) / 60);
return `${hours}h ${minutes}m`;
} else {
const days = Math.floor(seconds / 86400);
const hours = Math.floor((seconds % 86400) / 3600);
return `${days}d ${hours}h`;
}
}
function filterAlerts(filter) {
currentFilter = filter;
// Update active button
document.querySelectorAll('.filter-button').forEach(btn => {
btn.classList.remove('active');
});
event.target.classList.add('active');
// Re-render with new filter
renderAlerts(allAlerts);
}
// Auto-refresh every 15 seconds
setInterval(loadAlerts, 15000);
// Initial load
loadAlerts();
</script>
</body>
</html>
hbd/templates/foot.html → hbd/server/templates/foot.html
+1 -1
View File
@@ -1,5 +1,5 @@
<footer>
<div id="copyright">
&copy;2002-2021 <A HREF="mailto:andreas@wrede.ca">Andreas Wrede</A> All Rights Reserved.</p>
&copy;2002-2026 <A HREF="mailto:andreas@wrede.ca">Andreas Wrede</A> All Rights Reserved.</p>
</div>
</footer>
hbd/templates/head.html → hbd/server/templates/head.html
View File
hbd/templates/live.html → hbd/server/templates/live.html
+50 -1
View File
@@ -3,6 +3,30 @@
{% include 'head.html' %}
<style>
.nav {
background: #fff;
padding: 15px;
margin-bottom: 20px;
box-shadow: 0 2px 4px rgba(0,0,0,0.1);
border-radius: 4px;
}
.nav a {
margin-right: 20px;
text-decoration: none;
color: #0066cc;
font-weight: 500;
}
.nav a:hover {
text-decoration: underline;
}
.nav a.active {
color: #333;
font-weight: bold;
}
.content {
display: flex;
flex-direction: column;
@@ -235,6 +259,12 @@
WS_Connect();
</script>
<body>
<div class="nav">
<a href="/live" class="active">Live Dashboard</a>
<a href="/plugins">Plugin Metrics</a>
<a href="/alerts">Alerts</a>
</div>
{% include 'menu.html' %}
<div id="content" class="content" style="overflow: hidden">
@@ -255,7 +285,26 @@
<th style="text-align: right">Last State</th>
</tr>
</thead>
<tbody id="ntablebody"></tbody>
<tbody id="ntablebody">
{% for host in hosts %}
<tr>
<td>{{ host.name }}</td>
<td>{{ host.ver if host.ver else '' }}</td>
{% for conn in host.connections %}
<td>{{ conn.addr if conn.addr else '' }}</td>
<td>{{ conn.state if conn.state else '' }}</td>
<td style="text-align: right">{{ conn.latency if conn.latency else '' }}</td>
<td style="text-align: right">{{ conn.last_state_ts if conn.last_state_ts else '' }}</td>
{% endfor %}
{% if host.connections|length == 0 %}
<td></td><td></td><td></td><td></td>
<td></td><td></td><td></td><td></td>
{% elif host.connections|length == 1 %}
<td></td><td></td><td></td><td></td>
{% endif %}
</tr>
{% endfor %}
</tbody>
</table>
</div>
<div id="log" class="log" style="overflow: auto;">
hbd/server/templates/menu.html
+3
View File
@@ -0,0 +1,3 @@
<!-- <label for="drawer-toggle" id="drawer-toggle-label"></label>
s<header>{{ header }}</header> -->
hbd/server/templates/plugins.html
+974
View File
@@ -0,0 +1,974 @@
<!DOCTYPE html>
<html>
{% include 'head.html' %}
<style>
body {
margin: 10px;
background: #f5f5f5;
overflow: hidden;
}
.nav {
background: #fff;
padding: 10px 15px;
margin-bottom: 10px;
box-shadow: 0 2px 4px rgba(0,0,0,0.1);
border-radius: 4px;
}
.nav a {
margin-right: 20px;
text-decoration: none;
color: #0066cc;
font-weight: 500;
font-size: 0.9em;
}
.nav a:hover {
text-decoration: underline;
}
.nav a.active {
color: #333;
font-weight: bold;
}
.container {
max-width: 1400px;
margin: 0 auto;
max-height: calc(100vh - 120px);
overflow-y: auto;
padding-right: 10px;
}
h1 {
color: #333;
margin-bottom: 5px;
font-size: 1.5em;
}
.subtitle {
color: #666;
margin-bottom: 15px;
font-size: 0.9em;
}
.host-card {
background: white;
border-radius: 6px;
padding: 10px 15px;
margin-bottom: 10px;
box-shadow: 0 1px 4px rgba(0,0,0,0.1);
transition: all 0.2s;
}
.host-card.collapsed .host-body {
display: none;
}
.host-header {
display: flex;
justify-content: space-between;
align-items: center;
cursor: pointer;
user-select: none;
padding: 5px 0;
}
.host-header:hover {
background: #f9f9f9;
}
.host-title {
display: flex;
align-items: center;
gap: 10px;
}
.collapse-icon {
font-size: 1.2em;
color: #666;
transition: transform 0.2s;
min-width: 20px;
}
.host-card.collapsed .collapse-icon {
transform: rotate(-90deg);
}
.host-name {
font-size: 1.1em;
font-weight: bold;
color: #333;
}
.host-body {
padding-top: 10px;
}
.plugin-pills {
display: flex;
gap: 6px;
flex-wrap: wrap;
margin-bottom: 10px;
}
.plugin-pill {
padding: 4px 12px;
background: #e3f2fd;
border: 1px solid #90caf9;
border-radius: 15px;
cursor: pointer;
transition: all 0.2s;
font-size: 0.85em;
}
.plugin-pill:hover {
background: #90caf9;
color: white;
}
.plugin-pill.active {
background: #2196f3;
color: white;
border-color: #2196f3;
}
.plugin-content {
margin-top: 10px;
display: none;
}
.plugin-content.active {
display: block;
}
.metric-grid {
display: grid;
grid-template-columns: repeat(auto-fill, minmax(200px, 1fr));
gap: 10px;
margin-top: 10px;
}
.metric-card {
background: #fafafa;
border-left: 3px solid #2196f3;
padding: 8px 12px;
border-radius: 3px;
}
.metric-label {
font-size: 0.75em;
color: #666;
text-transform: uppercase;
letter-spacing: 0.3px;
margin-bottom: 3px;
}
.metric-value {
font-size: 1.4em;
font-weight: bold;
color: #333;
line-height: 1.2;
}
.metric-unit {
font-size: 0.6em;
color: #888;
font-weight: normal;
}
.timestamp {
color: #999;
font-size: 0.75em;
margin-top: 10px;
padding-top: 8px;
border-top: 1px solid #e0e0e0;
}
.no-data {
text-align: center;
padding: 20px;
color: #999;
font-style: italic;
font-size: 0.9em;
}
.loading {
text-align: center;
padding: 15px;
color: #666;
font-size: 0.9em;
}
.error {
background: #ffebee;
border-left: 3px solid #f44336;
padding: 10px;
margin: 10px 0;
border-radius: 3px;
color: #c62828;
font-size: 0.9em;
}
.nested-metrics {
margin-top: 8px;
padding-left: 12px;
border-left: 2px solid #ddd;
}
.nested-header {
font-weight: bold;
color: #555;
margin: 8px 0 5px 0;
font-size: 0.85em;
}
/* Scrollbar styling */
.container::-webkit-scrollbar {
width: 8px;
}
.container::-webkit-scrollbar-track {
background: #f1f1f1;
border-radius: 4px;
}
.container::-webkit-scrollbar-thumb {
background: #888;
border-radius: 4px;
}
.container::-webkit-scrollbar-thumb:hover {
background: #555;
}
/* Table styling for interface data */
.interface-table {
width: 100%;
border-collapse: collapse;
margin-top: 10px;
font-size: 0.85em;
background: #fff;
box-shadow: 0 1px 3px rgba(0,0,0,0.1);
border-radius: 4px;
overflow: hidden;
}
.interface-table thead {
background: #2196f3;
color: white;
}
.interface-table th {
padding: 8px 10px;
text-align: left;
font-weight: 600;
text-transform: uppercase;
font-size: 0.75em;
letter-spacing: 0.5px;
}
.interface-table th.number {
text-align: right;
}
.interface-table td {
padding: 6px 10px;
border-top: 1px solid #e0e0e0;
}
.interface-table td.number {
text-align: right;
font-family: 'Courier New', monospace;
}
.interface-table tbody tr:hover {
background: #f5f5f5;
}
.interface-table tbody tr:nth-child(even) {
background: #fafafa;
}
.interface-table tbody tr:nth-child(even):hover {
background: #f0f0f0;
}
.interface-name {
font-weight: bold;
color: #2196f3;
}
</style>
<body>
<div class="nav">
<a href="/live">Live Dashboard</a>
<a href="/plugins" class="active">Plugin Metrics</a>
<a href="/alerts">Alerts</a>
</div>
<div class="container">
<h1>{{ header }}</h1>
<p class="subtitle">Real-time system metrics from monitoring plugins</p>
{% if not hosts %}
<div class="no-data">
<p>No hosts with plugin data available</p>
<p style="font-size: 0.9em; margin-top: 10px;">Hosts will appear here once they start sending plugin metrics</p>
</div>
{% else %}
<div id="hosts-container">
{% for host in hosts %}
<div class="host-card" data-hostname="{{ host.name }}">
<div class="host-header" onclick="toggleHost('{{ host.name }}')">
<div class="host-title">
<span class="collapse-icon">▼</span>
<span class="host-name">{{ host.name }}</span>
</div>
</div>
<div class="host-body">
<div class="plugin-pills">
{% for plugin in host.plugins %}
<div class="plugin-pill" data-plugin="{{ plugin }}" onclick="event.stopPropagation(); showPlugin('{{ host.name }}', '{{ plugin }}')">
{{ plugin }}
</div>
{% endfor %}
</div>
{% for plugin in host.plugins %}
<div class="plugin-content" id="{{ host.name }}-{{ plugin }}" data-hostname="{{ host.name }}" data-plugin="{{ plugin }}">
<div class="loading">Loading {{ plugin }} data...</div>
</div>
{% endfor %}
</div>
</div>
{% endfor %}
</div>
{% endif %}
</div>
<script>
// Track selected plugins per host
const selectedPlugins = {};
function toggleHost(hostname) {
const card = document.querySelector(`[data-hostname="${hostname}"]`);
card.classList.toggle('collapsed');
}
function showPlugin(hostname, pluginName) {
// Update selectedPlugins tracker
selectedPlugins[hostname] = pluginName;
// Update active pill
const hostCard = document.querySelector(`[data-hostname="${hostname}"]`);
hostCard.querySelectorAll('.plugin-pill').forEach(pill => {
pill.classList.remove('active');
});
hostCard.querySelector(`[data-plugin="${pluginName}"]`).classList.add('active');
// Show plugin content
hostCard.querySelectorAll('.plugin-content').forEach(content => {
content.classList.remove('active');
});
const contentDiv = document.getElementById(`${hostname}-${pluginName}`);
contentDiv.classList.add('active');
// Load data if not already loaded
if (contentDiv.querySelector('.loading')) {
loadPluginData(hostname, pluginName);
}
}
async function loadPluginData(hostname, pluginName) {
const contentDiv = document.getElementById(`${hostname}-${pluginName}`);
try {
const response = await fetch(`/api/0/hosts/${hostname}/plugins/${pluginName}?limit=1`);
if (!response.ok) {
throw new Error(`HTTP ${response.status}`);
}
const data = await response.json();
if (data.samples && data.samples.length > 0) {
const sample = data.samples[0];
contentDiv.innerHTML = renderPluginData(sample.data, sample.timestamp);
} else {
contentDiv.innerHTML = '<div class="no-data">No data available for this plugin</div>';
}
} catch (error) {
contentDiv.innerHTML = `<div class="error">Failed to load plugin data: ${error.message}</div>`;
}
}
function renderPluginData(data, timestamp) {
let html = '<div class="metric-grid">';
for (const [key, value] of Object.entries(data)) {
// Skip nested objects for now, handle them separately
if (typeof value === 'object' && value !== null) {
continue;
}
html += renderMetric(key, value);
}
html += '</div>';
// Handle nested objects (like partitions in disk_monitor)
for (const [key, value] of Object.entries(data)) {
if (typeof value === 'object' && value !== null) {
// Check if this is interface data - render as table
if (isInterfaceData(key, value)) {
html += renderInterfaceTable(key, value);
} else if (isInterfaceStatsData(key, value)) {
html += renderInterfaceStatsTable(key, value);
} else if (isDiskPartitionData(key, value)) {
html += renderPartitionTable(key, value);
} else if (isDiskIOData(key, value)) {
html += renderDiskIOTable(key, value);
} else if (isFilesystemData(key, value)) {
html += renderFilesystemTable(key, value);
} else {
// Regular nested metrics display
html += `<div class="nested-metrics">`;
html += `<div class="nested-header">📊 ${formatLabel(key)}</div>`;
html += '<div class="metric-grid">';
if (Array.isArray(value)) {
// Handle arrays - more compact display
value.forEach((item, idx) => {
if (typeof item === 'object') {
// Add a compact separator for array items
if (idx > 0) {
html += `<div style="grid-column: 1/-1; border-top: 1px dashed #ddd; margin: 5px 0;"></div>`;
}
for (const [subKey, subValue] of Object.entries(item)) {
html += renderMetric(`${subKey}`, subValue);
}
}
});
} else {
// Handle nested objects
for (const [subKey, subValue] of Object.entries(value)) {
if (typeof subValue === 'object') {
// Another level of nesting - keep compact
html += `<div style="grid-column: 1/-1; margin-top: 5px; font-size: 0.85em; color: #666;"><strong>${subKey}:</strong></div>`;
for (const [deepKey, deepValue] of Object.entries(subValue)) {
html += renderMetric(deepKey, deepValue);
}
} else {
html += renderMetric(subKey, subValue);
}
}
}
html += '</div></div>';
}
}
}
const date = new Date(timestamp * 1000);
html += `<div class="timestamp">Last updated: ${date.toLocaleString()}</div>`;
return html;
}
function isInterfaceData(key, value) {
// Check if this is interface/network stats data (I/O counters)
if (key.toLowerCase().includes('interface') && !key.toLowerCase().includes('interface_stats')) {
// Verify it's an object with interface-like structure
if (typeof value === 'object' && !Array.isArray(value)) {
// Check if values are objects with byte/packet counters
const firstKey = Object.keys(value)[0];
if (firstKey && typeof value[firstKey] === 'object') {
const sample = value[firstKey];
return sample.hasOwnProperty('bytes_sent') ||
sample.hasOwnProperty('bytes_recv') ||
sample.hasOwnProperty('packets_sent') ||
sample.hasOwnProperty('tx_bytes') ||
sample.hasOwnProperty('rx_bytes');
}
}
}
return false;
}
function isInterfaceStatsData(key, value) {
// Check if this is interface stats data (status, speed, mtu, duplex)
if (key.toLowerCase() === 'interface_stats' || key.toLowerCase().includes('if_stats')) {
if (typeof value === 'object' && !Array.isArray(value)) {
const firstKey = Object.keys(value)[0];
if (firstKey && typeof value[firstKey] === 'object') {
const sample = value[firstKey];
return sample.hasOwnProperty('isup') ||
sample.hasOwnProperty('speed') ||
sample.hasOwnProperty('mtu') ||
sample.hasOwnProperty('duplex');
}
}
}
return false;
}
function isDiskPartitionData(key, value) {
// Check if this is disk partition data
if (key.toLowerCase() === 'partitions' || key.toLowerCase().includes('partition')) {
if (typeof value === 'object' && !Array.isArray(value)) {
const firstKey = Object.keys(value)[0];
if (firstKey && typeof value[firstKey] === 'object') {
const sample = value[firstKey];
return sample.hasOwnProperty('total') &&
sample.hasOwnProperty('used') &&
sample.hasOwnProperty('free') &&
sample.hasOwnProperty('percent');
}
}
}
return false;
}
function isDiskIOData(key, value) {
// Check if this is disk I/O counter data
if (key.toLowerCase().includes('io_counter') || key.toLowerCase().includes('disk_io')) {
if (typeof value === 'object' && !Array.isArray(value)) {
const firstKey = Object.keys(value)[0];
if (firstKey && typeof value[firstKey] === 'object') {
const sample = value[firstKey];
return sample.hasOwnProperty('read_bytes') ||
sample.hasOwnProperty('write_bytes') ||
sample.hasOwnProperty('read_count') ||
sample.hasOwnProperty('write_count');
}
}
}
return false;
}
function isFilesystemData(key, value) {
// Check if this is filesystem info data (from filesystem_info plugin)
if (key.toLowerCase() === 'filesystems' && Array.isArray(value)) {
if (value.length > 0 && typeof value[0] === 'object') {
const sample = value[0];
return sample.hasOwnProperty('device') &&
sample.hasOwnProperty('mountpoint') &&
sample.hasOwnProperty('fstype');
}
}
return false;
}
function renderInterfaceTable(key, interfaces) {
let html = `<div class="nested-metrics">`;
html += `<div class="nested-header">🌐 ${formatLabel(key)}</div>`;
html += '<table class="interface-table">';
// Determine columns based on available data
const sampleInterface = Object.values(interfaces)[0];
const hasBytes = sampleInterface.hasOwnProperty('bytes_sent') || sampleInterface.hasOwnProperty('tx_bytes');
const hasPackets = sampleInterface.hasOwnProperty('packets_sent') || sampleInterface.hasOwnProperty('tx_packets');
const hasErrors = sampleInterface.hasOwnProperty('errin') || sampleInterface.hasOwnProperty('rx_errors');
const hasDrops = sampleInterface.hasOwnProperty('dropin') || sampleInterface.hasOwnProperty('rx_dropped');
const hasDelta = sampleInterface.hasOwnProperty('bytes_sent_delta');
// Build table header
html += '<thead><tr>';
html += '<th>Interface</th>';
if (hasBytes) {
html += '<th class="number">Bytes Sent</th>';
html += '<th class="number">Bytes Recv</th>';
if (hasDelta) {
html += '<th class="number">Δ Sent</th>';
html += '<th class="number">Δ Recv</th>';
}
}
if (hasPackets) {
html += '<th class="number">Pkts Sent</th>';
html += '<th class="number">Pkts Recv</th>';
if (hasDelta) {
html += '<th class="number">Δ Pkts Sent</th>';
html += '<th class="number">Δ Pkts Recv</th>';
}
}
if (hasErrors) {
html += '<th class="number">Errors In</th>';
html += '<th class="number">Errors Out</th>';
}
if (hasDrops) {
html += '<th class="number">Drops In</th>';
html += '<th class="number">Drops Out</th>';
}
html += '</tr></thead>';
// Build table body
html += '<tbody>';
for (const [ifName, ifData] of Object.entries(interfaces)) {
html += '<tr>';
html += `<td class="interface-name">${ifName}</td>`;
if (hasBytes) {
html += `<td class="number">${formatBytes(ifData.bytes_sent || ifData.tx_bytes || 0)}</td>`;
html += `<td class="number">${formatBytes(ifData.bytes_recv || ifData.rx_bytes || 0)}</td>`;
if (hasDelta) {
html += `<td class="number">${formatBytes(ifData.bytes_sent_delta || 0)}</td>`;
html += `<td class="number">${formatBytes(ifData.bytes_recv_delta || 0)}</td>`;
}
}
if (hasPackets) {
html += `<td class="number">${(ifData.packets_sent || ifData.tx_packets || 0).toLocaleString()}</td>`;
html += `<td class="number">${(ifData.packets_recv || ifData.rx_packets || 0).toLocaleString()}</td>`;
if (hasDelta) {
html += `<td class="number">${(ifData.packets_sent_delta || 0).toLocaleString()}</td>`;
html += `<td class="number">${(ifData.packets_recv_delta || 0).toLocaleString()}</td>`;
}
}
if (hasErrors) {
html += `<td class="number">${ifData.errin || ifData.rx_errors || 0}</td>`;
html += `<td class="number">${ifData.errout || ifData.tx_errors || 0}</td>`;
}
if (hasDrops) {
html += `<td class="number">${ifData.dropin || ifData.rx_dropped || 0}</td>`;
html += `<td class="number">${ifData.dropout || ifData.tx_dropped || 0}</td>`;
}
html += '</tr>';
}
html += '</tbody>';
html += '</table>';
html += '</div>';
return html;
}
function renderInterfaceStatsTable(key, interfaces) {
let html = `<div class="nested-metrics">`;
html += `<div class="nested-header">🔌 ${formatLabel(key)}</div>`;
html += '<table class="interface-table">';
// Table header
html += '<thead><tr>';
html += '<th>Interface</th>';
html += '<th>Status</th>';
html += '<th class="number">Speed</th>';
html += '<th>Duplex</th>';
html += '<th class="number">MTU</th>';
html += '</tr></thead>';
// Table body
html += '<tbody>';
for (const [ifName, ifData] of Object.entries(interfaces)) {
html += '<tr>';
html += `<td class="interface-name">${ifName}</td>`;
// Status with color coding
const isUp = ifData.isup;
const statusColor = isUp ? '#4caf50' : '#f44336';
const statusIcon = isUp ? '✓' : '✗';
const statusText = isUp ? 'UP' : 'DOWN';
html += `<td style="color: ${statusColor}; font-weight: bold;">${statusIcon} ${statusText}</td>`;
// Speed
const speed = ifData.speed || 0;
let speedText = '-';
if (speed > 0) {
if (speed >= 1000) {
speedText = (speed / 1000).toFixed(1) + ' Gbps';
} else {
speedText = speed + ' Mbps';
}
}
html += `<td class="number">${speedText}</td>`;
// Duplex
let duplexText = ifData.duplex || '-';
if (duplexText.includes('NicDuplex.')) {
duplexText = duplexText.replace('NicDuplex.', '');
}
if (duplexText === '2') duplexText = 'FULL';
if (duplexText === '1') duplexText = 'HALF';
if (duplexText === '0') duplexText = 'UNKNOWN';
html += `<td>${duplexText}</td>`;
// MTU
html += `<td class="number">${ifData.mtu || '-'}</td>`;
html += '</tr>';
}
html += '</tbody>';
html += '</table>';
html += '</div>';
return html;
}
function renderPartitionTable(key, partitions) {
let html = `<div class="nested-metrics">`;
html += `<div class="nested-header">💾 ${formatLabel(key)}</div>`;
html += '<table class="interface-table">';
// Table header
html += '<thead><tr>';
html += '<th>Mount Point</th>';
html += '<th>Device</th>';
html += '<th>Type</th>';
html += '<th class="number">Total</th>';
html += '<th class="number">Used</th>';
html += '<th class="number">Free</th>';
html += '<th class="number">Use %</th>';
html += '</tr></thead>';
// Table body
html += '<tbody>';
for (const [mountPoint, partData] of Object.entries(partitions)) {
html += '<tr>';
html += `<td class="interface-name">${mountPoint}</td>`;
html += `<td>${partData.device || '-'}</td>`;
html += `<td>${partData.fstype || '-'}</td>`;
html += `<td class="number">${formatBytes(partData.total || 0)}</td>`;
html += `<td class="number">${formatBytes(partData.used || 0)}</td>`;
html += `<td class="number">${formatBytes(partData.free || 0)}</td>`;
// Color code the percentage
const percent = partData.percent || 0;
let percentColor = '#4caf50'; // green
if (percent > 90) percentColor = '#f44336'; // red
else if (percent > 75) percentColor = '#ff9800'; // orange
else if (percent > 50) percentColor = '#ffc107'; // yellow
html += `<td class="number" style="color: ${percentColor}; font-weight: bold;">${percent.toFixed(1)}%</td>`;
html += '</tr>';
}
html += '</tbody>';
html += '</table>';
html += '</div>';
return html;
}
function renderDiskIOTable(key, disks) {
let html = `<div class="nested-metrics">`;
html += `<div class="nested-header">📈 ${formatLabel(key)}</div>`;
html += '<table class="interface-table">';
// Determine columns based on available data
const sampleDisk = Object.values(disks)[0];
const hasDeltas = sampleDisk.hasOwnProperty('read_bytes_delta');
const hasTime = sampleDisk.hasOwnProperty('read_time');
// Table header
html += '<thead><tr>';
html += '<th>Disk</th>';
html += '<th class="number">Read Bytes</th>';
html += '<th class="number">Write Bytes</th>';
if (hasDeltas) {
html += '<th class="number">Δ Read</th>';
html += '<th class="number">Δ Write</th>';
}
html += '<th class="number">Read Count</th>';
html += '<th class="number">Write Count</th>';
if (hasDeltas) {
html += '<th class="number">Δ Reads</th>';
html += '<th class="number">Δ Writes</th>';
}
if (hasTime) {
html += '<th class="number">Read Time (ms)</th>';
html += '<th class="number">Write Time (ms)</th>';
}
html += '</tr></thead>';
// Table body
html += '<tbody>';
for (const [diskName, diskData] of Object.entries(disks)) {
html += '<tr>';
html += `<td class="interface-name">${diskName}</td>`;
html += `<td class="number">${formatBytes(diskData.read_bytes || 0)}</td>`;
html += `<td class="number">${formatBytes(diskData.write_bytes || 0)}</td>`;
if (hasDeltas) {
html += `<td class="number">${formatBytes(diskData.read_bytes_delta || 0)}</td>`;
html += `<td class="number">${formatBytes(diskData.write_bytes_delta || 0)}</td>`;
}
html += `<td class="number">${(diskData.read_count || 0).toLocaleString()}</td>`;
html += `<td class="number">${(diskData.write_count || 0).toLocaleString()}</td>`;
if (hasDeltas) {
html += `<td class="number">${(diskData.read_count_delta || 0).toLocaleString()}</td>`;
html += `<td class="number">${(diskData.write_count_delta || 0).toLocaleString()}</td>`;
}
if (hasTime) {
html += `<td class="number">${(diskData.read_time || 0).toLocaleString()}</td>`;
html += `<td class="number">${(diskData.write_time || 0).toLocaleString()}</td>`;
}
html += '</tr>';
}
html += '</tbody>';
html += '</table>';
html += '</div>';
return html;
}
function renderFilesystemTable(key, filesystems) {
let html = `<div class="nested-metrics">`;
html += `<div class="nested-header">🗄️ ${formatLabel(key)}</div>`;
html += '<table class="interface-table">';
// Table header
html += '<thead><tr>';
html += '<th>Device</th>';
html += '<th>Mount Point</th>';
html += '<th>Type</th>';
html += '<th>Options</th>';
html += '<th class="number">Max File</th>';
html += '<th class="number">Max Path</th>';
html += '</tr></thead>';
// Table body
html += '<tbody>';
for (const fs of filesystems) {
html += '<tr>';
html += `<td class="interface-name">${fs.device || '-'}</td>`;
html += `<td>${fs.mountpoint || '-'}</td>`;
html += `<td>${fs.fstype || '-'}</td>`;
// Format mount options - truncate if too long
let opts = fs.opts || '-';
if (opts.length > 40) {
opts = opts.substring(0, 37) + '...';
}
html += `<td style="font-size: 0.85em;">${opts}</td>`;
html += `<td class="number">${fs.maxfile || '-'}</td>`;
html += `<td class="number">${fs.maxpath || '-'}</td>`;
html += '</tr>';
}
html += '</tbody>';
html += '</table>';
html += '</div>';
return html;
}
function formatBytes(bytes) {
if (bytes === 0) return '0 B';
if (bytes < 1024) return bytes + ' B';
if (bytes < 1048576) return (bytes / 1024).toFixed(1) + ' KB';
if (bytes < 1073741824) return (bytes / 1048576).toFixed(1) + ' MB';
return (bytes / 1073741824).toFixed(2) + ' GB';
}
function renderMetric(key, value) {
const label = formatLabel(key);
const formattedValue = formatValue(key, value);
const unit = getUnit(key);
return `
<div class="metric-card">
<div class="metric-label">${label}</div>
<div class="metric-value">
${formattedValue}
${unit ? `<span class="metric-unit">${unit}</span>` : ''}
</div>
</div>
`;
}
function formatLabel(key) {
return key
.replace(/_/g, ' ')
.replace(/\b\w/g, l => l.toUpperCase());
}
function formatValue(key, value) {
if (typeof value === 'number') {
// Format percentages
if (key.includes('percent') || key.includes('usage')) {
return value.toFixed(1);
}
// Format bytes to MB/GB
if (key.includes('bytes') || key.includes('_mb') || key.includes('_gb')) {
if (value > 1073741824) {
return (value / 1073741824).toFixed(2);
} else if (value > 1048576) {
return (value / 1048576).toFixed(2);
}
}
// Default number formatting
if (value > 1000) {
return value.toLocaleString();
}
return value.toFixed(2);
}
return value;
}
function getUnit(key) {
if (key.includes('percent') || key.includes('usage')) return '%';
if (key.includes('_gb')) return 'GB';
if (key.includes('_mb')) return 'MB';
if (key.includes('bytes') && !key.includes('_mb') && !key.includes('_gb')) {
// Determine unit based on typical size
return 'bytes';
}
if (key.includes('count')) return '';
if (key.includes('mhz')) return 'MHz';
return '';
}
// Auto-refresh data every 30 seconds
setInterval(() => {
for (const [hostname, pluginName] of Object.entries(selectedPlugins)) {
const contentDiv = document.getElementById(`${hostname}-${pluginName}`);
if (contentDiv && contentDiv.classList.contains('active')) {
loadPluginData(hostname, pluginName);
}
}
}, 30000);
// Initialize by selecting first plugin for each host
document.addEventListener('DOMContentLoaded', () => {
const hostCards = document.querySelectorAll('.host-card');
hostCards.forEach((card, index) => {
const hostname = card.dataset.hostname;
const firstPlugin = card.querySelector('.plugin-pill');
if (firstPlugin) {
const pluginName = firstPlugin.dataset.plugin;
showPlugin(hostname, pluginName);
}
// Collapse all hosts except the first one
if (index > 0) {
card.classList.add('collapsed');
}
});
});
</script>
</body>
</html>
hbd/udp.py → hbd/server/udp.py
+53 -3
View File
@@ -4,8 +4,8 @@ import asyncio
import zlib
import logging
from .proto import stodict, oldmtodict
from hbd.utils import dur
from ..common.proto import stodict, oldmtodict
from ..common.utils import dur
logger = logging.getLogger(__name__)
@@ -72,11 +72,24 @@ def handle_datagram(msg: dict, addr, transport, ctx: dict):
- log: callable(loghost, message)
- pushmsg: callable(message)
- msg_to_websockets: callable(typ, data)
- msg_journal: MessageJournal instance for logging all messages
- DEBUG, verbose
"""
if not msg:
return
now = __import__("time").time()
# Log message to journal
msg_journal = ctx.get("msg_journal")
if msg_journal:
# Create async task to log message (non-blocking)
import asyncio
try:
loop = asyncio.get_event_loop()
loop.create_task(msg_journal.log_message(msg, addr, now))
except Exception as e:
logger.debug(f"Failed to log message to journal: {e}")
cfg = ctx.get("config", {})
hbdcls = ctx.get("hbdclass")
log = ctx.get("log")
@@ -88,7 +101,7 @@ def handle_datagram(msg: dict, addr, transport, ctx: dict):
# normalize addr (ip, port)
ip = addr[0] if isinstance(addr, (list, tuple)) else addr
name = msg.get("name", "unknown")
from hbd.utils import shortname
from ..common.utils import shortname
uname = shortname(name)
@@ -110,6 +123,43 @@ def handle_datagram(msg: dict, addr, transport, ctx: dict):
if msg.get("ID") == "HTB":
host.doesack = msg.get("acks", -1)
elif msg.get("ID") == "PLG":
# Handle plugin data message
plugin_name = msg.get("plugin")
if plugin_name:
# Extract all fields except ID and plugin name
plugin_data = {k: v for k, v in msg.items() if k not in ["ID", "plugin"]}
# Store plugin data with timestamp
host.add_plugin_data(plugin_name, plugin_data, timestamp=now)
if DEBUG > 1:
print(f"Stored plugin data for {uname}: {plugin_name}")
# Check thresholds if checker is available
threshold_checker = ctx.get("threshold_checker")
if threshold_checker:
try:
state_changes = threshold_checker.check_plugin_data(
host_name=uname,
plugin_name=plugin_name,
data=plugin_data,
alert_states=host.alert_states,
)
if DEBUG > 1 and state_changes:
print(f"Threshold state changes for {uname}: {state_changes}")
except Exception as e:
logger.error(f"Error checking thresholds for {uname}.{plugin_name}: {e}")
# Notify websockets of plugin update
if msg_to_websockets:
try:
msg_to_websockets("plugin", {
"host": uname,
"plugin": plugin_name,
"data": plugin_data,
"timestamp": now
})
except Exception:
pass
host.setcver(msg.get("ver", 0))
try:
hbd/ws.py → hbd/server/ws.py
+16 -8
View File
@@ -25,19 +25,28 @@ async def _handler(websocket, path=None):
remote_address = websocket.remote_address
if path is None:
path = getattr(websocket, "path", None)
if _verbose:
logger.info("DBG ws_serve: %s: %s", remote_address, path)
logger.info("WebSocket connection from %s: %s", remote_address, path)
try:
# send initial hosts
if _get_hosts:
for h in _get_hosts():
try:
hosts = list(_get_hosts())
logger.debug("Sending %d hosts to new WebSocket client", len(hosts))
for h in hosts:
jmsg = json.dumps({"type": "host", "data": h})
await websocket.send(jmsg)
except Exception as e:
logger.error("Error sending initial hosts: %s", e, exc_info=True)
# send recent messages
if _get_msgs:
for m in list(_get_msgs())[-100:]:
try:
msgs = list(_get_msgs())[-100:]
logger.debug("Sending %d recent messages to new WebSocket client", len(msgs))
for m in msgs:
jmsg = json.dumps({"type": "message", "data": m})
await websocket.send(jmsg)
except Exception as e:
logger.error("Error sending initial messages: %s", e, exc_info=True)
# keep connection open until client disconnects
async for _ in websocket:
@@ -50,11 +59,11 @@ async def _handler(websocket, path=None):
websockets.exceptions.ConnectionClosedOK,
websockets.exceptions.ConnectionClosedError,
) as e:
if _verbose:
logger.info("ws closed: %r", e)
logger.info("WebSocket closed from %s: %r", remote_address, e)
except Exception as e:
logger.exception("ws handler exception: %s", e)
logger.exception("WebSocket handler exception from %s: %s", remote_address, e)
finally:
logger.debug("Removing WebSocket connection from %s", remote_address)
try:
_connections.remove(websocket)
except KeyError:
@@ -101,7 +110,6 @@ async def start(
for srv in servers:
await srv
if _verbose:
logger.info(
"WebSocket server(s) started on port %s (wss %s)", ws_port, wss_port
)
hbd/templates/menu.html
-3
View File
@@ -1,3 +0,0 @@
<label for="drawer-toggle" id="drawer-toggle-label"></label>
<header>{{ header }}</header>
pyproject.toml
+26 -10
View File
@@ -5,25 +5,40 @@ build-backend = "setuptools.build_meta"
[project]
name = "hbd"
version = "5.0.5"
description = "Heartbeat daemon (hbd) — receive heartbeats and act on them"
description = "Heartbeat monitoring system — client (hbc) and server (hbd)"
readme = "README.md"
requires-python = ">=3.11"
license = "MIT"
keywords = ["heartbeat", "monitoring", "dns", "websocket"]
keywords = ["heartbeat", "monitoring", "dns", "websocket", "system-monitoring"]
authors = [
{ name = "heartbeat contributors" }
]
# Core dependencies (required for both client and server)
dependencies = [
"websockets>=13.2",
"mattermostdriver>=7.3.0",
"PyYAML>=6.0",
"aiohttp>=3.11",
"Jinja2>=3.1.6",
"fastapi>=0.128.0",
]
[project.optional-dependencies]
# Client-only dependencies (hbc - system monitoring client)
client = [
"psutil>=5.9.0",
]
# Server-only dependencies (hbd - heartbeat daemon/server)
server = [
"websockets>=13.2",
"mattermostdriver>=7.3.0",
"aiohttp>=3.11",
"Jinja2>=3.1.6",
]
# Install both client and server
all = [
"hbd[client,server]",
]
# Development dependencies
dev = [
"pytest>=7.0",
"pytest-cov>=4.0",
@@ -35,15 +50,16 @@ dev = [
]
[project.scripts]
hbd = "hbd.cli:main"
hbc = "hbd.hbc:main"
hbd = "hbd.server.cli:main"
hbc = "hbd.client.main:main"
[tool.setuptools.packages.find]
where = ["."]
include = ["hbd*"]
[tool.setuptools.package-data]
"hbd" = ["*.yaml", "static/*", "static/*/*", "templates/*"]
"hbd.server" = ["*.yaml", "static/*", "static/*/*", "templates/*"]
"hbd.client" = ["*.yaml"]
[tool.black]
scripts/demo_http_api.py
+391
View File
@@ -0,0 +1,391 @@
#!/usr/bin/env python3
"""
Demo script for HTTP API endpoints.
Tests and demonstrates the plugin data and alert APIs.
"""
import requests
import json
import sys
from datetime import datetime
from time import sleep
BASE_URL = "http://localhost:50004"
def print_section(title):
"""Print a formatted section header."""
print(f"\n{'=' * 70}")
print(f" {title}")
print('=' * 70)
def format_timestamp(timestamp):
"""Convert Unix timestamp to readable format."""
return datetime.fromtimestamp(timestamp).strftime('%Y-%m-%d %H:%M:%S')
def format_duration(seconds):
"""Format duration in human-readable format."""
if seconds < 60:
return f"{int(seconds)}s"
elif seconds < 3600:
minutes = int(seconds / 60)
secs = int(seconds % 60)
return f"{minutes}m {secs}s"
elif seconds < 86400:
hours = int(seconds / 3600)
minutes = int((seconds % 3600) / 60)
return f"{hours}h {minutes}m"
else:
days = int(seconds / 86400)
hours = int((seconds % 86400) / 3600)
return f"{days}d {hours}h"
def test_hosts_api():
"""Test GET /api/0/hosts endpoint."""
print_section("1. List All Monitored Hosts")
try:
response = requests.get(f"{BASE_URL}/api/0/hosts", timeout=5)
response.raise_for_status()
hosts = response.json()
print(f"Found {len(hosts)} hosts:\n")
for host in hosts:
name = host.get('name', 'unknown')
ver = host.get('ver', 0)
dyn = host.get('dyn', False)
conn_count = len(host.get('connections', []))
print(f" • {name}")
print(f" - Protocol: IPv{ver}")
print(f" - Dynamic: {dyn}")
print(f" - Connections: {conn_count}")
return hosts
except requests.RequestException as e:
print(f"❌ Error: {e}")
return []
def test_host_plugins_api(hostname):
"""Test GET /api/0/hosts/{hostname}/plugins endpoint."""
print_section(f"2. Get All Plugins for Host: {hostname}")
try:
response = requests.get(f"{BASE_URL}/api/0/hosts/{hostname}/plugins", timeout=5)
response.raise_for_status()
data = response.json()
plugins = data.get('plugins', {})
print(f"Found {len(plugins)} plugins:\n")
for plugin_name, plugin_data in plugins.items():
timestamp = plugin_data.get('timestamp', 0)
sample_count = plugin_data.get('sample_count', 0)
metrics = plugin_data.get('data', {})
print(f" 📦 {plugin_name}")
print(f" Last update: {format_timestamp(timestamp)}")
print(f" Samples: {sample_count}")
print(f" Metrics: {len(metrics)}")
# Show first few metrics
for i, (metric, value) in enumerate(metrics.items()):
if i < 3: # Show only first 3 metrics
if isinstance(value, float):
print(f" - {metric}: {value:.2f}")
elif isinstance(value, dict):
print(f" - {metric}: [nested data, {len(value)} keys]")
else:
print(f" - {metric}: {value}")
if len(metrics) > 3:
print(f" ... and {len(metrics) - 3} more")
print()
return list(plugins.keys())
except requests.RequestException as e:
print(f"❌ Error: {e}")
return []
def test_plugin_detail_api(hostname, plugin_name, limit=5):
"""Test GET /api/0/hosts/{hostname}/plugins/{plugin_name} endpoint."""
print_section(f"3. Get Detailed Data: {hostname}/{plugin_name}")
try:
url = f"{BASE_URL}/api/0/hosts/{hostname}/plugins/{plugin_name}"
params = {'limit': limit}
response = requests.get(url, params=params, timeout=5)
response.raise_for_status()
data = response.json()
samples = data.get('samples', [])
print(f"Retrieved {len(samples)} samples (limit={limit}):\n")
for i, sample in enumerate(samples):
timestamp = sample.get('timestamp', 0)
metrics = sample.get('data', {})
print(f" [{i+1}] {format_timestamp(timestamp)}")
for metric, value in sorted(metrics.items())[:5]: # Show first 5 metrics
if isinstance(value, float):
print(f" {metric}: {value:.2f}")
elif isinstance(value, dict):
print(f" {metric}: [nested: {len(value)} keys]")
else:
print(f" {metric}: {value}")
print()
return samples
except requests.RequestException as e:
print(f"❌ Error: {e}")
return []
def test_host_alerts_api(hostname):
"""Test GET /api/0/hosts/{hostname}/alerts endpoint."""
print_section(f"4. Get Alerts for Host: {hostname}")
try:
response = requests.get(f"{BASE_URL}/api/0/hosts/{hostname}/alerts", timeout=5)
response.raise_for_status()
data = response.json()
alerts = data.get('alerts', [])
summary = data.get('summary', {})
print(f"Summary:")
print(f" ✓ OK: {summary.get('ok', 0)}")
print(f" ⚠️ Warning: {summary.get('warning', 0)}")
print(f" 🔴 Critical: {summary.get('critical', 0)}")
print(f" ❓ Unknown: {summary.get('unknown', 0)}")
print()
# Show non-OK alerts
active_alerts = [a for a in alerts if a.get('level') != 'OK']
if active_alerts:
print(f"Active Alerts ({len(active_alerts)}):")
for alert in active_alerts:
metric = alert.get('metric_path', 'unknown')
level = alert.get('level', 'UNKNOWN')
value = alert.get('last_value', 0)
since = alert.get('since', 0)
duration = datetime.now().timestamp() - since
icon = '⚠️' if level == 'WARNING' else '🔴'
print(f" {icon} {metric}")
print(f" Level: {level}")
print(f" Value: {value:.2f}" if isinstance(value, float) else f" Value: {value}")
print(f" Duration: {format_duration(duration)}")
print()
else:
print("✓ No active alerts - all systems normal!")
return data
except requests.RequestException as e:
print(f"❌ Error: {e}")
return {}
def test_all_alerts_api():
"""Test GET /api/0/alerts endpoint."""
print_section("5. Get All Active Alerts Across All Hosts")
try:
response = requests.get(f"{BASE_URL}/api/0/alerts", timeout=5)
response.raise_for_status()
data = response.json()
alerts = data.get('alerts', [])
summary = data.get('summary', {})
host_count = data.get('host_count', 0)
print(f"Monitoring {host_count} hosts")
print(f"Active Alerts: {summary.get('total', 0)}")
print(f" 🔴 Critical: {summary.get('critical', 0)}")
print(f" ⚠️ Warning: {summary.get('warning', 0)}")
print()
if alerts:
print("Alert Details:")
for alert in alerts:
hostname = alert.get('hostname', 'unknown')
metric = alert.get('metric_path', 'unknown')
level = alert.get('level', 'UNKNOWN')
value = alert.get('last_value', 0)
since = alert.get('since', 0)
duration = datetime.now().timestamp() - since
notification_count = alert.get('notification_count', 0)
icon = '⚠️' if level == 'WARNING' else '🔴'
print(f" {icon} {hostname} / {metric}")
print(f" Level: {level}")
print(f" Value: {value:.2f}" if isinstance(value, float) else f" Value: {value}")
print(f" Duration: {format_duration(duration)}")
print(f" Notifications: {notification_count}")
print()
else:
print("✅ All systems normal - no active alerts!")
return data
except requests.RequestException as e:
print(f"❌ Error: {e}")
return {}
def test_messages_api():
"""Test GET /api/0/messages endpoint."""
print_section("6. Get Recent Messages")
try:
response = requests.get(f"{BASE_URL}/api/0/messages", timeout=5)
response.raise_for_status()
messages = response.json()
print(f"Last {len(messages)} messages:\n")
for msg in messages[-5:]: # Show last 5
timestamp = msg.get('time', 0)
host = msg.get('host', 'unknown')
text = msg.get('msg', '')
print(f" [{format_timestamp(timestamp)}] {host}: {text}")
return messages
except requests.RequestException as e:
print(f"❌ Error: {e}")
return []
def test_error_handling():
"""Test API error handling."""
print_section("7. Error Handling Tests")
# Test non-existent host
print("Testing non-existent host...")
try:
response = requests.get(f"{BASE_URL}/api/0/hosts/nonexistenthost/plugins", timeout=5)
if response.status_code == 404:
error_data = response.json()
print(f" ✓ Correctly returned 404: {error_data.get('error', 'No error message')}")
else:
print(f" ⚠️ Unexpected status code: {response.status_code}")
except Exception as e:
print(f" ❌ Error: {e}")
# Test non-existent plugin
print("\nTesting non-existent plugin...")
try:
# Get first host
hosts = requests.get(f"{BASE_URL}/api/0/hosts", timeout=5).json()
if hosts:
hostname = hosts[0]['name']
response = requests.get(
f"{BASE_URL}/api/0/hosts/{hostname}/plugins/nonexistentplugin",
timeout=5
)
if response.status_code == 404:
error_data = response.json()
print(f" ✓ Correctly returned 404: {error_data.get('error', 'No error message')}")
else:
print(f" ⚠️ Unexpected status code: {response.status_code}")
except Exception as e:
print(f" ❌ Error: {e}")
def demo_monitoring_loop():
"""Demonstrate continuous monitoring."""
print_section("8. Continuous Monitoring Demo (5 iterations)")
print("Monitoring alerts every 3 seconds (Ctrl+C to stop)...\n")
try:
for i in range(5):
response = requests.get(f"{BASE_URL}/api/0/alerts", timeout=5)
response.raise_for_status()
data = response.json()
summary = data.get('summary', {})
critical = summary.get('critical', 0)
warning = summary.get('warning', 0)
timestamp = datetime.now().strftime('%H:%M:%S')
status = "🔴 CRITICAL" if critical > 0 else "⚠️ WARNING" if warning > 0 else "✅ OK"
print(f"[{timestamp}] {status} - Critical: {critical}, Warning: {warning}")
if i < 4: # Don't sleep after last iteration
sleep(3)
except KeyboardInterrupt:
print("\n\nMonitoring stopped by user")
except Exception as e:
print(f"\n❌ Error: {e}")
def main():
"""Run all API tests."""
print("""
╔══════════════════════════════════════════════════════════════╗
║ Heartbeat Daemon HTTP API Demo & Test Suite ║
╚══════════════════════════════════════════════════════════════╝
""")
print(f"Testing API at: {BASE_URL}")
print(f"Ensure the heartbeat daemon is running!")
# Test basic connectivity
try:
response = requests.get(f"{BASE_URL}/api/0/hosts", timeout=2)
response.raise_for_status()
print("✅ API is reachable\n")
except Exception as e:
print(f"❌ Cannot connect to API: {e}")
print("\nPlease ensure:")
print(" 1. Heartbeat daemon is running")
print(" 2. HTTP server is enabled in configuration")
print(f" 3. Server is listening on port {BASE_URL.split(':')[-1]}")
sys.exit(1)
# Run test suite
hosts = test_hosts_api()
if not hosts:
print("\n⚠️ No hosts found. Ensure clients are sending heartbeats.")
return
# Pick first host for detailed testing
hostname = hosts[0].get('name', '')
if hostname:
plugins = test_host_plugins_api(hostname)
if plugins:
# Test detailed plugin data
test_plugin_detail_api(hostname, plugins[0], limit=3)
# Test alert endpoints
test_host_alerts_api(hostname)
# Test global endpoints
test_all_alerts_api()
test_messages_api()
# Test error handling
test_error_handling()
# Continuous monitoring demo
demo_monitoring_loop()
print_section("Test Suite Complete")
print("""
Next Steps:
• View the web UI at http://localhost:50004/live
• Check plugin metrics at http://localhost:50004/plugins
• Monitor alerts at http://localhost:50004/alerts
• Read API documentation: docs/HTTP_API.md
""")
if __name__ == '__main__':
try:
main()
except KeyboardInterrupt:
print("\n\nDemo interrupted by user")
sys.exit(0)
test_all_plugins.py
+99
View File
@@ -0,0 +1,99 @@
#!/usr/bin/env python3
"""
Test all plugins together.
"""
import asyncio
import logging
from pathlib import Path
# Setup path
import sys
sys.path.insert(0, str(Path(__file__).parent))
from hbd.plugin import PluginRegistry, PluginLoader
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s %(name)s: %(message)s"
)
async def test_all_plugins():
"""Test loading all plugins."""
print("=" * 70)
print("Testing All Plugins")
print("=" * 70)
# Create registry and loader
registry = PluginRegistry()
loader = PluginLoader(registry)
# Configuration for plugins
config = {
"cpu_monitor": {
"interval": 30,
"per_core": False
},
"nagios_runner": {
"interval": 60,
"commands": [
{
"name": "test_ok",
"command": "echo 'OK - test passed | metric=100%;;;0;100'"
},
{
"name": "test_warning",
"command": "echo 'WARNING - test result | value=85%;80;90;0;100' && exit 1"
}
]
}
}
# Load plugins
plugin_dir = Path(__file__).parent / "hbd" / "plugins"
print(f"\n1. Loading plugins from: {plugin_dir}")
count = await loader.load_from_directory(plugin_dir, config)
print(f" ✓ Loaded {count} plugins")
# List loaded plugins
print(f"\n2. Loaded plugins:")
for plugin in registry.get_all():
print(f" - {plugin.name} v{plugin.version}")
print(f" Type: {plugin.__class__.__name__}")
print(f" Interval: {plugin.interval}s")
print(f" Description: {plugin.description}")
# Test collection for each plugin
print(f"\n3. Testing data collection:")
for plugin in registry.get_all():
print(f"\n {plugin.name}:")
try:
data = await plugin.collect()
print(f" ✓ Collected {len(data)} fields")
# Show sample of data
sample_count = min(5, len(data))
for key, value in list(data.items())[:sample_count]:
value_str = str(value)
if len(value_str) > 50:
value_str = value_str[:47] + "..."
print(f" {key}: {value_str}")
if len(data) > sample_count:
print(f" ... and {len(data) - sample_count} more fields")
except Exception as e:
print(f" ✗ Error: {e}")
# Cleanup
print(f"\n4. Cleanup...")
await loader.unload_all()
print(f" ✓ All plugins unloaded")
print(f"\n" + "=" * 70)
print(f"Successfully tested {count} plugins!")
print("=" * 70)
if __name__ == "__main__":
asyncio.run(test_all_plugins())
test_core_plugins.py
+160
View File
@@ -0,0 +1,160 @@
#!/usr/bin/env python3
"""
Test script for all monitoring plugins.
Tests all available plugins including the new ones:
- memory_monitor
- disk_monitor
- network_monitor
- filesystem_info
"""
import asyncio
import sys
import os
import logging
# Add parent directory to path so we can import hbd
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from hbd.plugin import PluginLoader
# Configure logging
logging.basicConfig(
level=logging.INFO,
format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
)
logger = logging.getLogger(__name__)
def format_bytes(bytes_val):
"""Format bytes into human readable format."""
for unit in ['B', 'KB', 'MB', 'GB', 'TB']:
if bytes_val < 1024.0:
return f"{bytes_val:.2f} {unit}"
bytes_val /= 1024.0
return f"{bytes_val:.2f} PB"
def print_plugin_data(plugin_name, data, indent=2):
"""Pretty print plugin data."""
prefix = " " * indent
if isinstance(data, dict):
for key, value in data.items():
if isinstance(value, dict):
print(f"{prefix}{key}:")
print_plugin_data(plugin_name, value, indent + 2)
elif isinstance(value, list):
print(f"{prefix}{key}: [{len(value)} items]")
if len(value) <= 5: # Only show small lists
for item in value:
if isinstance(item, dict):
print_plugin_data(plugin_name, item, indent + 2)
else:
print(f"{prefix} - {item}")
else:
# Format output based on key name for better readability
if '_bytes' in key or key.endswith('_sent') or key.endswith('_recv') or 'memory_' in key or 'swap_' in key:
if isinstance(value, (int, float)) and value > 1024:
print(f"{prefix}{key}: {format_bytes(value)} ({value:,})")
else:
print(f"{prefix}{key}: {value}")
elif 'percent' in key:
print(f"{prefix}{key}: {value:.1f}%")
elif isinstance(value, float):
print(f"{prefix}{key}: {value:.2f}")
elif isinstance(value, int) and value > 1000:
print(f"{prefix}{key}: {value:,}")
else:
print(f"{prefix}{key}: {value}")
else:
print(f"{prefix}{data}")
async def main():
"""Main test function."""
print("="*60)
print("Plugin System Test Suite")
print("="*60)
# Load all available plugins using the plugin loader
from hbd.plugin import PluginRegistry, PluginLoader
from pathlib import Path
registry = PluginRegistry()
loader = PluginLoader(registry)
plugin_dir = Path(__file__).parent / "hbd" / "plugins"
if not plugin_dir.exists():
print(f"✗ Plugin directory not found: {plugin_dir}")
return 1
# Load plugins from directory
count = await loader.load_from_directory(plugin_dir, {})
print(f"\nLoaded {count} plugins:")
plugins = registry.get_all()
for plugin in plugins:
print(f" - {plugin.name}: {plugin.__class__.__doc__.split('.')[0] if plugin.__class__.__doc__ else 'No description'}")
# Test each plugin
results = {}
for plugin in plugins:
# Skip nagios_runner as it needs specific configuration
if plugin.name == 'nagios_runner':
print(f"\n{'='*60}")
print(f"Skipping: {plugin.name} (requires specific configuration)")
print(f"{'='*60}")
results[plugin.name] = True # Mark as success since it loaded OK
continue
print(f"\n{'='*60}")
print(f"Testing: {plugin.name}")
print(f"{'='*60}")
try:
# Collect data
data = await plugin.collect()
if data:
if 'error' in data:
print(f"✗ Collection error: {data['error']}")
results[plugin.name] = False
else:
print(f"✓ Data collected: {len(data)} top-level fields")
print_plugin_data(plugin.name, data)
results[plugin.name] = True
else:
print(f"⚠ No data collected")
results[plugin.name] = False
except Exception as e:
print(f"✗ Failed to collect data: {e}")
import traceback
traceback.print_exc()
results[plugin.name] = False
# Summary
print(f"\n{'='*60}")
print("Test Summary")
print(f"{'='*60}")
success_count = sum(1 for v in results.values() if v)
total_count = len(results)
print(f"\nResults: {success_count}/{total_count} plugins successful")
for name, success in results.items():
status = "✓" if success else "✗"
print(f" {status} {name}")
if success_count == total_count:
print("\n🎉 All plugins passed!")
return 0
else:
print(f"\n⚠ {total_count - success_count} plugin(s) failed")
return 1
if __name__ == '__main__':
exit_code = asyncio.run(main())
sys.exit(exit_code)
test_journal.py
+333
View File
@@ -0,0 +1,333 @@
#!/usr/bin/env python3
"""
Test script for message journal functionality.
Tests:
- Journal initialization
- Message logging
- File rotation based on size
- Backup management
"""
import asyncio
import sys
import os
import json
import tempfile
import shutil
from pathlib import Path
# Add parent directory to path
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from hbd.journal import MessageJournal, get_journal
async def test_basic_logging():
"""Test basic message logging."""
print("="*60)
print("Test 1: Basic Message Logging")
print("="*60)
# Create temporary directory for journal
temp_dir = tempfile.mkdtemp(prefix="journal_test_")
print(f"Using temp directory: {temp_dir}")
try:
# Create journal with config
config = {
'journal_enabled': True,
'journal_dir': temp_dir,
'journal_file': 'test.journal',
'journal_max_size': 1024, # 1KB for testing
'journal_max_backups': 3
}
journal = MessageJournal(config)
await journal.initialize()
# Log some test messages
test_messages = [
{
'ID': 'HTB',
'name': 'testhost1',
'interval': 30,
'ver': 1
},
{
'ID': 'PLG',
'plugin': 'cpu_monitor',
'cpu_percent': 45.2,
'load_1min': 1.5
},
{
'ID': 'HTB',
'name': 'testhost2',
'interval': 60,
'boot': 1
}
]
for i, msg in enumerate(test_messages):
await journal.log_message(msg, ('192.168.1.100', 50000 + i), 1000.0 + i)
print(f"✓ Logged message {i+1}: {msg['ID']}")
# Check journal file exists
journal_path = Path(temp_dir) / 'test.journal'
if journal_path.exists():
print(f"✓ Journal file created: {journal_path}")
# Read and verify content
with open(journal_path, 'r') as f:
lines = f.readlines()
print(f"✓ Journal has {len(lines)} entries")
# Parse first entry
entry = json.loads(lines[0])
print(f"✓ First entry structure: {list(entry.keys())}")
assert 'timestamp' in entry
assert 'datetime' in entry
assert 'source_ip' in entry
assert 'message' in entry
print("✓ Entry structure validated")
else:
print("✗ Journal file not created")
return False
# Get stats
stats = journal.get_stats()
print(f"\nJournal stats:")
print(f" Enabled: {stats['enabled']}")
print(f" Current size: {stats['current_size']} bytes")
print(f" Max size: {stats['max_size']} bytes")
print(f" Rotation threshold: {stats['rotation_threshold']}")
await journal.close()
print("\n✅ Test 1 PASSED")
return True
except Exception as e:
print(f"\n✗ Test 1 FAILED: {e}")
import traceback
traceback.print_exc()
return False
finally:
# Cleanup
shutil.rmtree(temp_dir, ignore_errors=True)
async def test_rotation():
"""Test log rotation based on size."""
print("\n" + "="*60)
print("Test 2: Log Rotation")
print("="*60)
# Create temporary directory for journal
temp_dir = tempfile.mkdtemp(prefix="journal_test_")
print(f"Using temp directory: {temp_dir}")
try:
# Create journal with small max size
config = {
'journal_enabled': True,
'journal_dir': temp_dir,
'journal_file': 'test.journal',
'journal_max_size': 500, # 500 bytes - very small for testing
'journal_max_backups': 3
}
journal = MessageJournal(config)
await journal.initialize()
# Log many messages to trigger rotation
print("Logging messages to trigger rotation...")
for i in range(20):
msg = {
'ID': 'HTB',
'name': f'testhost{i}',
'interval': 30,
'ver': 1,
'data': 'x' * 50 # Add some padding
}
await journal.log_message(msg, ('192.168.1.100', 50000 + i), 1000.0 + i)
# Give rotation time to complete
await asyncio.sleep(0.01)
print(f"✓ Logged 20 messages")
# Check for rotated files
journal_dir = Path(temp_dir)
all_files = list(journal_dir.glob('test.journal*'))
print(f"✓ Found {len(all_files)} journal files")
for f in sorted(all_files):
size = f.stat().st_size
print(f" - {f.name}: {size} bytes")
# Should have current file + some backups
if len(all_files) > 1:
print(f"✓ Rotation occurred ({len(all_files) - 1} backup files)")
else:
print("⚠ No rotation occurred (may not have reached threshold)")
# Check max backups limit
backup_files = [f for f in all_files if f.name != 'test.journal']
if len(backup_files) <= config['journal_max_backups']:
print(f"✓ Backup count within limit: {len(backup_files)} <= {config['journal_max_backups']}")
else:
print(f"✗ Too many backups: {len(backup_files)} > {config['journal_max_backups']}")
return False
await journal.close()
print("\n✅ Test 2 PASSED")
return True
except Exception as e:
print(f"\n✗ Test 2 FAILED: {e}")
import traceback
traceback.print_exc()
return False
finally:
# Cleanup
shutil.rmtree(temp_dir, ignore_errors=True)
async def test_disabled_journal():
"""Test that disabled journal doesn't write anything."""
print("\n" + "="*60)
print("Test 3: Disabled Journal")
print("="*60)
temp_dir = tempfile.mkdtemp(prefix="journal_test_")
print(f"Using temp directory: {temp_dir}")
try:
config = {
'journal_enabled': False,
'journal_dir': temp_dir,
'journal_file': 'test.journal'
}
journal = MessageJournal(config)
await journal.initialize()
# Try to log a message
msg = {'ID': 'HTB', 'name': 'testhost'}
await journal.log_message(msg, ('192.168.1.100', 50000), 1000.0)
# Check that no file was created
journal_path = Path(temp_dir) / 'test.journal'
if not journal_path.exists():
print("✓ No journal file created (as expected)")
else:
print("✗ Journal file was created despite being disabled")
return False
await journal.close()
print("\n✅ Test 3 PASSED")
return True
except Exception as e:
print(f"\n✗ Test 3 FAILED: {e}")
import traceback
traceback.print_exc()
return False
finally:
shutil.rmtree(temp_dir, ignore_errors=True)
async def test_global_instance():
"""Test global journal instance."""
print("\n" + "="*60)
print("Test 4: Global Journal Instance")
print("="*60)
temp_dir = tempfile.mkdtemp(prefix="journal_test_")
try:
config = {
'journal_enabled': True,
'journal_dir': temp_dir,
'journal_file': 'global.journal'
}
# Get global instance
journal1 = get_journal(config)
journal2 = get_journal() # Should return same instance
if journal1 is journal2:
print("✓ Global instance returns same object")
else:
print("✗ Global instance returns different objects")
return False
await journal1.initialize()
# Log through convenience function
from hbd.journal import log_message
msg = {'ID': 'HTB', 'name': 'testhost'}
await log_message(msg, ('192.168.1.100', 50000))
journal_path = Path(temp_dir) / 'global.journal'
if journal_path.exists():
print("✓ Global journal logged message")
else:
print("✗ Global journal did not log message")
return False
await journal1.close()
print("\n✅ Test 4 PASSED")
return True
except Exception as e:
print(f"\n✗ Test 4 FAILED: {e}")
import traceback
traceback.print_exc()
return False
finally:
shutil.rmtree(temp_dir, ignore_errors=True)
async def main():
"""Run all tests."""
print("Message Journal Test Suite")
print("="*60)
tests = [
test_basic_logging,
test_rotation,
test_disabled_journal,
test_global_instance
]
results = []
for test in tests:
result = await test()
results.append(result)
# Summary
print("\n" + "="*60)
print("Test Summary")
print("="*60)
passed = sum(results)
total = len(results)
print(f"Passed: {passed}/{total}")
for i, (test, result) in enumerate(zip(tests, results), 1):
status = "✅ PASS" if result else "❌ FAIL"
print(f" {status} - Test {i}: {test.__name__}")
if passed == total:
print("\n🎉 All tests passed!")
return 0
else:
print(f"\n⚠ {total - passed} test(s) failed")
return 1
if __name__ == '__main__':
exit_code = asyncio.run(main())
sys.exit(exit_code)
test_nagios.py
+150
View File
@@ -0,0 +1,150 @@
#!/usr/bin/env python3
"""
Test the Nagios Runner Plugin.
"""
import asyncio
import logging
from pathlib import Path
# Setup path
import sys
sys.path.insert(0, str(Path(__file__).parent))
from hbd.plugins.nagios_runner import NagiosRunnerPlugin, NAGIOS_OK, NAGIOS_WARNING
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s %(name)s %(levelname)s: %(message)s"
)
async def test_nagios_runner():
"""Test Nagios runner plugin."""
print("=" * 70)
print("Testing Nagios Runner Plugin")
print("=" * 70)
# Create test configuration with simple shell commands
# These mimic Nagios plugin output format
config = {
"interval": 60,
"timeout": 10,
"commands": [
{
"name": "check_uptime",
"command": "echo 'OK - uptime is 5 days | uptime=432000s;;;0'"
},
{
"name": "check_memory",
"command": "echo 'OK - Memory usage 45% | memory=45%;80;90;0;100'"
},
{
"name": "check_cpu",
"command": "echo 'WARNING - CPU load high | load1=5.2;5.0;10.0;0 load5=4.8;4.0;8.0;0 load15=3.2;3.0;6.0;0' && exit 1"
},
{
"name": "check_disk",
"command": "echo 'OK - Disk usage 62% | /=62%;80;90;0;100 /home=45%;80;90;0;100'"
}
]
}
print("\n1. Creating Nagios Runner plugin with test configuration")
print(f" Configured {len(config['commands'])} test commands")
plugin = NagiosRunnerPlugin(config)
print(f"\n2. Initializing plugin...")
initialized = await plugin.initialize()
print(f" Initialized: {initialized}")
if not initialized:
print(" ERROR: Plugin failed to initialize!")
return
print(f"\n3. Collecting metrics from Nagios plugins...")
data = await plugin.collect()
print(f" ✓ Collected {len(data)} data points")
print(f"\n4. Results:")
print(f" Overall Status: {data.get('overall_status')} (code: {data.get('overall_status_code')})")
print(f" Plugins Executed: {data.get('plugin_count')}")
# Show individual plugin results
print(f"\n5. Individual Plugin Results:")
for cmd_config in config["commands"]:
name = cmd_config["name"]
status = data.get(f"{name}_status", "N/A")
status_code = data.get(f"{name}_status_code", "N/A")
output = data.get(f"{name}_output", "N/A")
print(f"\n {name}:")
print(f" Status: {status} (code: {status_code})")
print(f" Output: {output}")
# Show performance data if present
perf_keys = [k for k in data.keys() if k.startswith(f"{name}_") and
k not in [f"{name}_status", f"{name}_status_code", f"{name}_output"]]
if perf_keys:
print(f" Performance Data:")
for key in perf_keys:
metric_name = key.replace(f"{name}_", "")
print(f" {metric_name}: {data[key]}")
print(f"\n6. Testing Nagios plugin detection (if available)...")
# Try to find actual Nagios plugins on the system
common_nagios_paths = [
"/usr/lib/nagios/plugins",
"/usr/local/nagios/libexec",
"/usr/lib64/nagios/plugins"
]
nagios_plugin_dir = None
for path in common_nagios_paths:
if Path(path).exists():
nagios_plugin_dir = Path(path)
print(f" ✓ Found Nagios plugins at: {nagios_plugin_dir}")
break
if nagios_plugin_dir:
# Try check_users if it exists
check_users = nagios_plugin_dir / "check_users"
if check_users.exists():
print(f"\n Testing real Nagios plugin: check_users")
real_config = {
"commands": [{
"name": "users",
"command": f"{check_users} -w 10 -c 20"
}]
}
real_plugin = NagiosRunnerPlugin(real_config)
await real_plugin.initialize()
real_data = await real_plugin.collect()
print(f" Status: {real_data.get('users_status')}")
print(f" Output: {real_data.get('users_output')}")
# Show any performance data
for key in real_data:
if key.startswith("users_") and "status" not in key and "output" not in key:
print(f" {key}: {real_data[key]}")
else:
print(f" check_users not found at {check_users}")
else:
print(f" No Nagios plugins directory found")
print(f" Install nagios-plugins to test with real plugins:")
print(f" sudo apt-get install nagios-plugins # Debian/Ubuntu")
print(f" sudo yum install nagios-plugins-all # RHEL/CentOS")
print(f"\n7. Cleanup...")
await plugin.cleanup()
print(f" ✓ Cleanup complete")
print(f"\n" + "=" * 70)
print("Test complete!")
print("=" * 70)
if __name__ == "__main__":
asyncio.run(test_nagios_runner())
test_plugins.py
+119
View File
@@ -0,0 +1,119 @@
#!/usr/bin/env python3
"""
Test script for plugin system.
"""
import asyncio
import logging
from pathlib import Path
# Setup path
import sys
sys.path.insert(0, str(Path(__file__).parent))
from hbd.plugin import PluginRegistry, PluginLoader
logging.basicConfig(
level=logging.DEBUG,
format="%(asctime)s %(name)s %(levelname)s: %(message)s"
)
async def test_plugins():
"""Test plugin loading and collection."""
print("=" * 60)
print("Testing Plugin System")
print("=" * 60)
# Create registry and loader
registry = PluginRegistry()
loader = PluginLoader(registry)
# Load plugins
plugin_dir = Path(__file__).parent / "hbd" / "plugins"
print(f"\n1. Loading plugins from: {plugin_dir}")
if not plugin_dir.exists():
print(f" ERROR: Plugin directory does not exist!")
return
count = await loader.load_from_directory(plugin_dir)
print(f" Loaded {count} plugins")
# List loaded plugins
print(f"\n2. Loaded plugins:")
for plugin in registry.get_all():
print(f" - {plugin.name} v{plugin.version} ({plugin.__class__.__name__})")
print(f" Description: {plugin.description}")
print(f" Interval: {plugin.interval}s")
# Test InfoPlugins
print(f"\n3. Testing InfoPlugins (collect once):")
from hbd.plugin import InfoPlugin
for plugin in registry.get_by_type(InfoPlugin):
print(f"\n Collecting {plugin.name}...")
try:
data = await plugin.collect()
print(f" ✓ Success! Got {len(data)} fields")
for key, value in list(data.items())[:5]: # Show first 5 fields
print(f" {key}: {value}")
if len(data) > 5:
print(f" ... and {len(data) - 5} more fields")
except Exception as e:
print(f" ✗ Error: {e}")
# Test MonitorPlugins
print(f"\n4. Testing MonitorPlugins (periodic collection):")
from hbd.plugin import MonitorPlugin
for plugin in registry.get_by_type(MonitorPlugin):
print(f"\n Collecting {plugin.name}...")
try:
data = await plugin.collect()
print(f" ✓ Success! Got {len(data)} fields")
for key, value in list(data.items())[:8]: # Show first 8 fields
print(f" {key}: {value}")
if len(data) > 8:
print(f" ... and {len(data) - 8} more fields")
except Exception as e:
print(f" ✗ Error: {e}")
# Test protocol encoding
print(f"\n5. Testing protocol encoding:")
from hbd.proto import dicttos, stodict
# Create sample plugin data
test_data = {
"plugin": "test_plugin",
"cpu_percent": 42.5,
"memory_mb": 1024,
"processes": 156,
"load_avg": [1.2, 0.8, 0.5],
"disk_info": {"sda": {"used": 50, "total": 100}}
}
print(f" Original data: {test_data}")
# Encode
encoded = dicttos("PLG", test_data, compress=True)
print(f" Encoded ({len(encoded)} bytes): {encoded[:50]}...")
# Decode
decoded = stodict(encoded)
print(f" Decoded: {decoded}")
# Verify
if decoded.get("ID") == "PLG" and decoded.get("plugin") == "test_plugin":
print(f" ✓ Protocol encoding/decoding works!")
else:
print(f" ✗ Protocol encoding/decoding failed!")
# Cleanup
print(f"\n6. Cleaning up...")
await loader.unload_all()
print(f" ✓ Cleanup complete")
print(f"\n" + "=" * 60)
print("Test complete!")
print("=" * 60)
if __name__ == "__main__":
asyncio.run(test_plugins())
test_plugins_manual.py
+60
View File
@@ -0,0 +1,60 @@
#!/usr/bin/env python3
"""
Debug plugin loading.
"""
import asyncio
import logging
from pathlib import Path
# Setup path
import sys
sys.path.insert(0, str(Path(__file__).parent))
from hbd.plugin import PluginRegistry, PluginLoader
logging.basicConfig(
level=logging.DEBUG,
format="%(asctime)s %(name)s %(levelname)s: %(message)s"
)
async def test_manual_load():
"""Test manual plugin loading."""
print("Testing manual plugin import...")
# Import plugins directly
from hbd.plugins.os_info import OSInfoPlugin
from hbd.plugins.cpu_monitor import CPUMonitorPlugin
# Create instances
os_plugin = OSInfoPlugin()
cpu_plugin = CPUMonitorPlugin()
print(f"OS Plugin: {os_plugin.name} v{os_plugin.version}")
print(f"CPU Plugin: {cpu_plugin.name} v{cpu_plugin.version}")
# Initialize
print("\nInitializing plugins...")
os_init = await os_plugin.initialize()
cpu_init = await cpu_plugin.initialize()
print(f"OS plugin initialized: {os_init}")
print(f"CPU plugin initialized: {cpu_init}")
# Collect data
if os_init:
print("\nCollecting OS info...")
os_data = await os_plugin.collect()
print(f"Got {len(os_data)} fields:")
for k, v in list(os_data.items())[:10]:
print(f" {k}: {v}")
if cpu_init:
print("\nCollecting CPU info...")
cpu_data = await cpu_plugin.collect()
print(f"Got {len(cpu_data)} fields:")
for k, v in cpu_data.items():
print(f" {k}: {v}")
if __name__ == "__main__":
asyncio.run(test_manual_load())
test_threshold.py
+495
View File
@@ -0,0 +1,495 @@
#!/usr/bin/env python3
"""
Test suite for the threshold checking and alerting system.
Tests cover:
- Threshold configuration parsing
- Threshold evaluation (all operators)
- Hysteresis functionality
- Alert state tracking
- State change detection
- Notification triggering
- Re-notification logic
"""
import sys
import time
from pathlib import Path
# Add parent directory to path for imports
sys.path.insert(0, str(Path(__file__).parent.parent))
from hbd.threshold import (
ThresholdChecker,
ThresholdConfig,
AlertLevel,
AlertState,
ComparisonOperator,
)
def test_threshold_config_basic():
"""Test basic threshold configuration."""
print("Test 1: Basic threshold configuration...")
config = ThresholdConfig(
metric_path="cpu_monitor.cpu_percent",
warning=80.0,
critical=90.0,
operator=">",
)
# Test below warning
result = config.evaluate(50.0)
assert result == AlertLevel.OK, f"Expected OK, got {result}"
# Test at warning
result = config.evaluate(80.0)
assert result == AlertLevel.OK, f"Expected OK at boundary, got {result}"
# Test above warning but below critical
result = config.evaluate(85.0)
assert result == AlertLevel.WARNING, f"Expected WARNING, got {result}"
# Test above critical
result = config.evaluate(95.0)
assert result == AlertLevel.CRITICAL, f"Expected CRITICAL, got {result}"
print(" ✓ Basic threshold configuration works")
def test_threshold_operators():
"""Test all comparison operators."""
print("\nTest 2: Comparison operators...")
# Greater than operator
config_gt = ThresholdConfig(
metric_path="test.metric",
warning=80.0,
critical=90.0,
operator=">",
)
assert config_gt.evaluate(85.0) == AlertLevel.WARNING
assert config_gt.evaluate(75.0) == AlertLevel.OK
# Less than operator (for inverse thresholds like available memory)
config_lt = ThresholdConfig(
metric_path="memory.available_mb",
warning=1000,
critical=500,
operator="<",
)
assert config_lt.evaluate(800) == AlertLevel.WARNING, "Should warn when below 1000"
assert config_lt.evaluate(400) == AlertLevel.CRITICAL, "Should be critical when below 500"
assert config_lt.evaluate(1500) == AlertLevel.OK, "Should be OK when above 1000"
# Greater than or equal
config_gte = ThresholdConfig(
metric_path="test.metric",
warning=80.0,
operator=">=",
)
assert config_gte.evaluate(80.0) == AlertLevel.WARNING
assert config_gte.evaluate(79.9) == AlertLevel.OK
# Less than or equal
config_lte = ThresholdConfig(
metric_path="test.metric",
warning=20.0,
operator="<=",
)
assert config_lte.evaluate(20.0) == AlertLevel.WARNING
assert config_lte.evaluate(20.1) == AlertLevel.OK
print(" ✓ All comparison operators work correctly")
def test_hysteresis():
"""Test hysteresis to prevent flapping."""
print("\nTest 3: Hysteresis...")
config = ThresholdConfig(
metric_path="cpu_monitor.cpu_percent",
warning=80.0,
critical=90.0,
operator=">",
hysteresis=0.1, # 10% hysteresis
)
# Start at OK, go to WARNING
result = config.evaluate_with_hysteresis(85.0, AlertLevel.OK)
assert result == AlertLevel.WARNING, "Should enter WARNING state"
# Try to recover with insufficient improvement (within hysteresis)
# Warning threshold is 80, hysteresis is 10%, so need to go below 80 - 8 = 72
result = config.evaluate_with_hysteresis(77.0, AlertLevel.WARNING)
assert result == AlertLevel.WARNING, "Should stay in WARNING (hysteresis)"
# Recover with sufficient improvement
result = config.evaluate_with_hysteresis(70.0, AlertLevel.WARNING)
assert result == AlertLevel.OK, "Should recover to OK"
# Test critical hysteresis
result = config.evaluate_with_hysteresis(95.0, AlertLevel.WARNING)
assert result == AlertLevel.CRITICAL, "Should escalate to CRITICAL"
# Try to recover from critical with insufficient improvement
# Critical threshold is 90, hysteresis is 10%, so need to go below 90 - 9 = 81
result = config.evaluate_with_hysteresis(85.0, AlertLevel.CRITICAL)
assert result == AlertLevel.CRITICAL, "Should stay CRITICAL (hysteresis, still above 81)"
# Sufficient improvement to drop from CRITICAL (below 81)
result = config.evaluate_with_hysteresis(75.0, AlertLevel.CRITICAL)
assert result == AlertLevel.OK, "Should drop to OK (below warning threshold)"
# Now test dropping from CRITICAL to WARNING
result = config.evaluate_with_hysteresis(95.0, AlertLevel.OK)
assert result == AlertLevel.CRITICAL, "Should go to CRITICAL"
# Drop to between warning and critical, but still in hysteresis zone
result = config.evaluate_with_hysteresis(82.0, AlertLevel.CRITICAL)
assert result == AlertLevel.CRITICAL, "Should stay CRITICAL (in hysteresis)"
# Drop below critical hysteresis but still above warning threshold
# At 80.1, we're above WARNING (80) so should evaluate to WARNING
result = config.evaluate_with_hysteresis(80.5, AlertLevel.CRITICAL)
assert result == AlertLevel.WARNING, "Should drop to WARNING when below critical hysteresis"
print(" ✓ Hysteresis prevents flapping")
def test_alert_state():
"""Test alert state tracking."""
print("\nTest 4: Alert state tracking...")
alert = AlertState("cpu_monitor.cpu_percent")
# Initial state
assert alert.level == AlertLevel.OK
assert alert.notification_count == 0
# Update to WARNING - should trigger notification
changed = alert.update(AlertLevel.WARNING, 85.0)
assert changed == True, "State change should return True"
assert alert.level == AlertLevel.WARNING
assert alert.last_value == 85.0
# Update with same level - no notification
changed = alert.update(AlertLevel.WARNING, 86.0)
assert changed == False, "No state change should return False"
assert alert.last_value == 86.0
# Escalate to CRITICAL
changed = alert.update(AlertLevel.CRITICAL, 95.0)
assert changed == True, "Escalation should trigger notification"
assert alert.level == AlertLevel.CRITICAL
# Recover to OK
changed = alert.update(AlertLevel.OK, 50.0)
assert changed == True, "Recovery should trigger notification"
assert alert.level == AlertLevel.OK
print(" ✓ Alert state tracking works correctly")
def test_threshold_checker_parsing():
"""Test parsing threshold configuration from YAML structure."""
print("\nTest 5: Configuration parsing...")
config = {
"thresholds": {
"cpu_monitor": {
"cpu_percent": {
"warning": 80.0,
"critical": 90.0,
"operator": ">",
"hysteresis": 0.1,
},
"load_1min": {
"warning": 4.0,
"critical": 8.0,
},
},
"memory_monitor": {
"percent": {
"warning": 85.0,
"critical": 95.0,
},
"available_mb": {
"warning": 1000,
"critical": 500,
"operator": "<",
},
},
"disk_monitor": {
"partitions": {
"/": {
"percent": {
"warning": 80.0,
"critical": 90.0,
},
},
"/home": {
"percent": {
"warning": 85.0,
"critical": 95.0,
},
},
},
},
}
}
checker = ThresholdChecker(config)
# Verify thresholds were parsed
assert "cpu_monitor.cpu_percent" in checker.thresholds
assert "cpu_monitor.load_1min" in checker.thresholds
assert "memory_monitor.percent" in checker.thresholds
assert "memory_monitor.available_mb" in checker.thresholds
assert "disk_monitor./.percent" in checker.thresholds
assert "disk_monitor./home.percent" in checker.thresholds
# Verify operators were parsed correctly
assert checker.thresholds["cpu_monitor.cpu_percent"].operator == ComparisonOperator.GT
assert checker.thresholds["memory_monitor.available_mb"].operator == ComparisonOperator.LT
print(f" ✓ Parsed {len(checker.thresholds)} thresholds correctly")
def test_check_plugin_data():
"""Test checking plugin data against thresholds."""
print("\nTest 6: Plugin data checking...")
config = {
"thresholds": {
"cpu_monitor": {
"cpu_percent": {
"warning": 80.0,
"critical": 90.0,
},
"load_1min": {
"warning": 4.0,
"critical": 8.0,
},
},
}
}
notifications = []
def notification_callback(msg):
notifications.append(msg)
checker = ThresholdChecker(config, notification_callback=notification_callback)
alert_states = {}
# First check - OK
plugin_data = {
"cpu_percent": 50.0,
"load_1min": 2.0,
}
state_changes = checker.check_plugin_data(
host_name="testhost",
plugin_name="cpu_monitor",
data=plugin_data,
alert_states=alert_states,
)
assert len(state_changes) == 0, "No thresholds violated, no state changes"
assert len(notifications) == 0, "No notifications should be sent"
# Second check - WARNING
plugin_data = {
"cpu_percent": 85.0,
"load_1min": 2.0,
}
state_changes = checker.check_plugin_data(
host_name="testhost",
plugin_name="cpu_monitor",
data=plugin_data,
alert_states=alert_states,
)
assert len(state_changes) == 1, "One metric should change state"
assert state_changes[0][0] == "cpu_monitor.cpu_percent"
assert state_changes[0][2] == AlertLevel.WARNING
assert len(notifications) == 1, "One notification should be sent"
assert "WARNING" in notifications[0]
assert "testhost" in notifications[0]
# Third check - CRITICAL
plugin_data = {
"cpu_percent": 95.0,
"load_1min": 9.0,
}
notifications.clear()
state_changes = checker.check_plugin_data(
host_name="testhost",
plugin_name="cpu_monitor",
data=plugin_data,
alert_states=alert_states,
)
assert len(state_changes) == 2, "Two metrics should change state"
assert len(notifications) == 2, "Two notifications should be sent"
# Fourth check - Recovery
plugin_data = {
"cpu_percent": 50.0,
"load_1min": 1.0,
}
notifications.clear()
state_changes = checker.check_plugin_data(
host_name="testhost",
plugin_name="cpu_monitor",
data=plugin_data,
alert_states=alert_states,
)
assert len(state_changes) == 2, "Two metrics should recover"
assert len(notifications) == 2, "Two recovery notifications"
assert any("RECOVERED" in n for n in notifications), "Should have recovery notification"
print(" ✓ Plugin data checking and notifications work")
def test_nested_metrics():
"""Test checking nested metrics like disk partitions."""
print("\nTest 7: Nested metrics (partitions)...")
config = {
"thresholds": {
"disk_monitor": {
"partitions": {
"/": {
"percent": {
"warning": 80.0,
"critical": 90.0,
},
},
"/home": {
"percent": {
"warning": 85.0,
"critical": 95.0,
},
},
},
},
}
}
notifications = []
checker = ThresholdChecker(config, notification_callback=lambda m: notifications.append(m))
alert_states = {}
plugin_data = {
"partitions": {
"/": {
"percent": 75.0,
"free_gb": 50.0,
},
"/home": {
"percent": 88.0, # Should trigger WARNING
"free_gb": 100.0,
},
},
}
state_changes = checker.check_plugin_data(
host_name="testhost",
plugin_name="disk_monitor",
data=plugin_data,
alert_states=alert_states,
)
assert len(state_changes) == 1, "One partition should trigger alert"
assert "/home" in state_changes[0][0], "Should be /home partition"
assert state_changes[0][2] == AlertLevel.WARNING
assert len(notifications) == 1
print(" ✓ Nested metric checking works")
def test_alert_summary():
"""Test getting alert summaries."""
print("\nTest 8: Alert summaries...")
config = {
"thresholds": {
"cpu_monitor": {
"cpu_percent": {"warning": 80.0, "critical": 90.0},
"load_1min": {"warning": 4.0, "critical": 8.0},
},
"memory_monitor": {
"percent": {"warning": 85.0, "critical": 95.0},
},
}
}
checker = ThresholdChecker(config)
alert_states = {}
# Create some alert states
plugin_data = {
"cpu_percent": 85.0, # WARNING
"load_1min": 9.0, # CRITICAL
}
checker.check_plugin_data("testhost", "cpu_monitor", plugin_data, alert_states)
plugin_data = {
"percent": 96.0, # CRITICAL
}
checker.check_plugin_data("testhost", "memory_monitor", plugin_data, alert_states)
# Get summary
summary = checker.get_alert_summary(alert_states)
assert summary["warning"] == 1, "Should have 1 warning"
assert summary["critical"] == 2, "Should have 2 critical"
# Get active alerts
active = checker.get_active_alerts(alert_states)
assert len(active) == 3, "Should have 3 active alerts"
print(" ✓ Alert summaries work correctly")
def run_all_tests():
"""Run all tests."""
print("=" * 70)
print("THRESHOLD SYSTEM TEST SUITE")
print("=" * 70)
try:
test_threshold_config_basic()
test_threshold_operators()
test_hysteresis()
test_alert_state()
test_threshold_checker_parsing()
test_check_plugin_data()
test_nested_metrics()
test_alert_summary()
print("\n" + "=" * 70)
print("✓ ALL TESTS PASSED")
print("=" * 70)
return 0
except AssertionError as e:
print(f"\n✗ TEST FAILED: {e}")
import traceback
traceback.print_exc()
return 1
except Exception as e:
print(f"\n✗ UNEXPECTED ERROR: {e}")
import traceback
traceback.print_exc()
return 1
if __name__ == "__main__":
sys.exit(run_all_tests())