heartbeat/README.md

Heartbeat Daemon (hbd) ✅

A lightweight daemon that listens for UDP heartbeat messages and acts on them: keeps host state, optionally updates DNS records via nsupdate, forwards messages to WebSocket clients, and sends notifications (email, Pushover, Mattermost, Signal). It is a refactor of a previously monolithic script into a modular Python package (hbd).

---

📌 Features

• Receive and parse heartbeat datagrams (text or zlib-compressed) ✅
• Maintain host state and detect up/down transitions ✅
• Queue DNS updates via nsupdate and run them in a background thread ✅
• WebSocket API for live updates (hosts & messages) ✅
• Notification pipeline (email, Pushover, Mattermost, Signal) ✅
• HTTP API & Web UI ✅
  • REST API for plugin data, alerts, and host information
  • Live dashboard with WebSocket updates
  • Interactive plugin metrics visualization
  • Alerts dashboard with filtering and summaries
• Message journal with automatic log rotation ✅
  • Logs all received messages in JSON format
  • Size-based automatic rotation
  • Configurable retention and backup management
• Plugin system for extensible monitoring ✅
  • Collect system metrics (CPU, memory, disk, network)
  • Execute existing Nagios monitoring plugins
  • Create custom plugins with simple Python classes
• Threshold alerting system ✅
  • Monitor metrics against configurable WARNING/CRITICAL thresholds
  • Hysteresis to prevent alert flapping
  • Automatic notifications on state changes
  • Re-notification for ongoing alerts
• Modular codebase suitable for unit testing and CI ✅

---

🔌 Plugin System

Heartbeat includes a comprehensive plugin architecture that extends monitoring beyond simple heartbeats. The plugin system allows you to:

• Collect system information: OS details, hardware info, system configuration
• Monitor resources: CPU usage, memory, disk space, network statistics
• Run Nagios plugins: Execute thousands of existing Nagios monitoring plugins without modification
• Create custom plugins: Build your own monitoring logic with simple Python classes

Plugin Types

• InfoPlugin: Collects static information once (e.g., OS version, hardware specs)
• MonitorPlugin: Collects metrics periodically (e.g., CPU usage every 30 seconds)

Built-in Plugins

• os_info: Collects OS, kernel, distribution, and architecture information
• cpu_monitor: Monitors CPU usage, load average, frequency, and process counts
• memory_monitor: Monitors RAM and swap usage, available memory
• disk_monitor: Monitors disk usage, I/O statistics, and filesystem metrics
• network_monitor: Monitors network interface statistics, bandwidth, and connections
• filesystem_info: Collects mounted filesystem information (physical filesystems only by default)
• nagios_runner: Executes Nagios monitoring plugins (check_disk, check_load, check_http, etc.)

Nagios Integration

The nagios_runner plugin provides seamless integration with the vast Nagios plugin ecosystem. You can run any Nagios-compatible plugin and have the results automatically parsed and stored:

• Executes plugins via subprocess with timeout protection
• Parses exit codes (OK/WARNING/CRITICAL/UNKNOWN)
• Extracts performance data with thresholds
• Reports aggregated status across all configured checks

See docs/NAGIOS_INTEGRATION.md for complete integration guide including configuration examples and custom plugin development.

Creating Custom Plugins

from hbd.plugin import MonitorPlugin

class DiskMonitorPlugin(MonitorPlugin):
    name = "disk_monitor"
    interval = 60  # Run every 60 seconds
    
    async def collect(self):
        return {
            "disk_usage": get_disk_usage(),
            "timestamp": time.time()
        }

Place plugins in hbd/plugins/ and they'll be automatically discovered and loaded by the client.

---

📝 Message Journal

Heartbeat includes a message journal that logs all received messages with automatic rotation.

Features

• JSON Format: All messages logged in JSONL (JSON Lines) format for easy parsing
• Automatic Rotation: Size-based rotation with configurable thresholds
• Backup Management: Keeps configurable number of rotated log files
• Non-blocking: Async logging with minimal performance impact

Configuration

# Message journal settings
journal_enabled: true                    # Enable/disable journaling
journal_dir: /var/log/heartbeat         # Journal directory
journal_file: messages.journal           # Base filename
journal_max_size: 104857600             # Max size (100MB default)
journal_max_backups: 10                 # Number of backups to keep

Example Journal Entry

{"timestamp":1711234567.123,"datetime":"2026-03-28T12:34:56","source_ip":"192.168.1.100","source_port":50003,"message":{"ID":"HTB","name":"webserver1","interval":30}}

Analyzing Journal Files

# View recent messages
tail -100 /var/log/heartbeat/messages.journal | jq .

# Count messages by type
cat /var/log/heartbeat/messages.journal | jq -r '.message.ID' | sort | uniq -c

# Filter by hostname
cat /var/log/heartbeat/messages.journal | jq 'select(.message.name == "webserver1")'

See docs/MESSAGE_JOURNAL.md for complete documentation including rotation behavior, integration with log management systems, and analysis examples.

---

🚨 Threshold Alerting

Heartbeat includes a sophisticated threshold alerting system that monitors plugin metrics and triggers notifications when values exceed configured limits.

Features

• Multi-level alerts: WARNING and CRITICAL severity levels
• Flexible operators: Support for >, >=, <, <=, ==, != comparisons
• Hysteresis: Prevents alert flapping with configurable recovery thresholds
• Smart notifications: Alerts only on state changes, not every check
• Re-notifications: Periodic reminders for ongoing alerts
• Journal integration: All threshold events logged for audit trail

Configuration

thresholds:
  cpu_monitor:
    cpu_percent:
      warning: 80.0      # Warn when CPU > 80%
      critical: 90.0     # Critical when CPU > 90%
      operator: ">"
      hysteresis: 0.1    # 10% hysteresis to prevent flapping
  
  memory_monitor:
    percent:
      warning: 85.0
      critical: 95.0
  
  disk_monitor:
    partitions:
      /:
        percent:
          warning: 80.0
          critical: 90.0
        free_gb:
          warning: 10.0   # Alert when < 10GB free
          critical: 5.0
          operator: "<"   # Inverse threshold

# Global settings
threshold_renotify_interval: 3600  # Re-notify every hour for ongoing alerts

Alert Behavior

1. State Changes: Notifications sent when crossing thresholds

   • OK → WARNING: Early notification
   • WARNING → CRITICAL: Escalation
   • CRITICAL → OK: Recovery

2. Hysteresis: Prevents rapid state transitions

   Critical threshold: 90%
   Hysteresis: 10%
   Recovery threshold: 81% (90 - 10% of 90)
   
   Value 91% → CRITICAL (threshold crossed)
   Value 85% → CRITICAL (still above 81%)
   Value 79% → OK (below recovery threshold)
   

3. Re-notifications: Periodic reminders for ongoing alerts

   • Default: Every 60 minutes
   • Configurable via threshold_renotify_interval

Example Notifications

WARNING: webserver01 - cpu_monitor.cpu_percent = 85.0
CRITICAL: webserver01 - memory_monitor.percent = 96.0
RECOVERED: database01 - disk_monitor./.percent = 75.0 (WARNING -> OK)
REMINDER (CRITICAL): mailserver - cpu_monitor.load_1min = 12.5 (ongoing for 3600s)

Supported Metrics

All plugin metrics can be thresholded:

• CPU: cpu_percent, load_1min, load_5min, load_15min
• Memory: percent, available_mb, swap_percent
• Disk: Per-partition percent, free_gb, free_mb
• Network: errors_total, dropped packets, connection counts
• Nagios: exit_code mapping (0=OK, 1=WARNING, 2=CRITICAL)

See docs/THRESHOLD_ALERTING.md for comprehensive documentation including best practices, troubleshooting, and advanced configuration.

---

🌐 HTTP API & Web UI

Heartbeat includes a built-in HTTP/WebSocket server that provides both a REST API and web-based dashboards for monitoring and visualization.

Features

• REST API: JSON endpoints for accessing plugin data, alerts, and host information
• Live Dashboard: Real-time WebSocket-powered host status view
• Plugin Metrics: Interactive visualization of all plugin data with auto-refresh
• Alerts Dashboard: Comprehensive alert monitoring with filtering and summaries
• CORS Support: Configurable for integration with external applications

Web Dashboards

• Live View (/live): Real-time host connectivity, latency, and messages
• Plugin Metrics (/plugins): Browse and visualize metrics from all plugins
• Alerts Dashboard (/alerts): Monitor active alerts with severity filtering

API Endpoints

# List all monitored hosts
curl http://localhost:50004/api/0/hosts

# Get all plugin data for a host
curl http://localhost:50004/api/0/hosts/webserver01/plugins

# Get detailed plugin history (last 50 samples)
curl http://localhost:50004/api/0/hosts/webserver01/plugins/cpu_monitor?limit=50

# Get alert states for a specific host
curl http://localhost:50004/api/0/hosts/webserver01/alerts

# Get all active alerts across all hosts
curl http://localhost:50004/api/0/alerts

Integration Examples

Python Client:

import requests

# Monitor for critical alerts
response = requests.get('http://localhost:50004/api/0/alerts')
alerts = response.json()

if alerts['summary']['critical'] > 0:
    print(f"⚠️ {alerts['summary']['critical']} CRITICAL alerts!")
    for alert in alerts['alerts']:
        if alert['level'] == 'CRITICAL':
            print(f"  {alert['hostname']}: {alert['metric_path']} = {alert['last_value']}")

Bash Monitoring Script:

#!/bin/bash
# Check for critical alerts
CRITICAL=$(curl -s http://localhost:50004/api/0/alerts | jq '.summary.critical')
if [ "$CRITICAL" -gt 0 ]; then
    echo "CRITICAL: $CRITICAL critical alerts detected!"
    # Send notification
fi

Demo & Testing

Run the API demo script to test all endpoints:

python3 scripts/demo_http_api.py

See docs/HTTP_API.md for complete API documentation including response formats, error handling, and integration examples.

---

⚙️ Quickstart

Prerequisites:

• Python 3.10+ (project uses language features from recent Python)
• nsupdate (for DNS updates) if using dynamic DNS

Install dependencies (recommended into a venv):

This project now declares its dependencies in pyproject.toml. Instead of the old requirements.txt flow, install the package into a virtualenv using pip:

See scripts/install.sh for a way to install.

Run the daemon (example):

# run with default config lookup (~/.hb.yaml)
hbd -c .hb.yaml -f -v

You can also run it directly via the package entrypoint after installation:

python -m hbd.cli -c /path/to/config.yaml

Running the Client

The heartbeat client (hbc) sends periodic heartbeats and plugin data to the server:

# Basic usage pointing to server
python -m hbd.hbc --server your-server.example.com

# With custom configuration
python -m hbd.hbc --server 192.168.1.100 --port 50003 --interval 30

# Run with specific plugins enabled/disabled
python -m hbd.hbc --server hbd.local --disable-plugin os_info

Client configuration can also be specified in YAML:

server: hbd.example.com
port: 50003
interval: 30
plugins:
  cpu_monitor:
    interval: 300      # Check every 5 minutes (default)
    per_core: true
  memory_monitor:
    interval: 300      # Check every 5 minutes (default)
  disk_monitor:
    interval: 300      # Check every 5 minutes (default)
  network_monitor:
    interval: 300      # Check every 5 minutes (default)
  nagios_runner:
    interval: 300      # Check every 5 minutes (default)
    commands:
      - /usr/lib/nagios/plugins/check_load -w 5,4,3 -c 10,8,6
      - /usr/lib/nagios/plugins/check_disk -w 20% -c 10% -p /

All monitoring plugins default to 5-minute (300 second) intervals, but can be customized as needed.

🐞 Debugging in VS Code

This repository includes a ready-to-use .vscode/launch.json with configurations to run or attach the VS Code debugger to hbd.

• Ensure the Python extension is installed and select the project .venv as the interpreter (bottom-left of VS Code).
• Use F5 and pick one of these configurations from the Run view:
  • Python: Run hbd (module) — runs hbd.cli as a module and sets PYTHONPATH to the workspace root (recommended).
  • Python: Run hbd with debugpy (listen) — launches debugpy and hbd together; useful when you want the process to listen for a debugger.
  • Python: Attach (localhost:5678) — attach the debugger to a running process started with debugpy.

To start hbd manually and wait for the debugger to attach, run:

PYTHONPATH=. python -m debugpy --listen 5678 --wait-for-client -m hbd.cli -c .hb.yaml -f -v

Set breakpoints in modules such as hbd/udp.py, hbd/dns.py, or hbd/server.py, and use the Attach configuration to connect. Use justMyCode: false if you need to step into third-party code.

---

🛠 Configuration

hbd reads YAML configuration (optional). If PyYAML is not installed, built-in defaults are used. Example configuration keys (see hbd/config.py):

• hb_port: UDP port to listen for heartbeats (default: 50003)
• hbd_port: internal control port (default: 50004)
• hbd_host: bind address for HTTP/WSS
• pickfile: path for persisted state
• logfile: path to log file
• logfmt: text or msg
• pushsrv: push service (pushover|mattermost|all)
• interval / grace: heartbeat timing configuration
• dyndomains: list of dyndomains to update via nsupdate
• nsupdate_bin: path to nsupdate binary
• ws_port: port for plain WebSocket connections (default: 50005)
• wss_port: port for secure WebSocket (WSS) connections (default: none). If set, hbd will attempt to serve WSS on this port when wss_pem and wss_key SSL files are available under cert_path (see below).
• cert_path: directory where TLS certificate and key are looked up (default: /usr/local/etc/ssl/)
• wss_pem: filename for the certificate chain (default: fullchain.pem)
• wss_key: filename for the private key (default: privkey.pem)

Example .hb.yaml (minimal):

hbd_host: 0.0.0.0
hbd_port: 50004
dyndomains:
  - example.com
nsupdate_bin: /usr/bin/nsupdate
pushsrv: pushover

Tip: config.DEFAULTS in hbd/config.py contains the canonical defaults and accepted configuration keys.

---

🔧 Architecture & Modules

• hbd.proto — serialization/deserialization of heartbeat messages (supports compressed payloads and plugin data)
• hbd.udp — UDP parsing and handle_datagram implementation (main state machine)
• hbd.dns — create_nsupdate_payload, nsupdate, and an asyncio DNS worker (start_dns_worker). The DNS worker now runs as an asyncio task and the package exposes a small thread-safe bridge so legacy synchronous code can put() updates into the queue; there is no longer a permanently-blocking background threading.Thread.
• hbd.notify — email and push notification helpers
• hbd.ws — WebSocket server and thread-safe broadcast helpers
• hbd.http — HTTP handler factory for the status UI/API
• hbd.journal — message journal with size-based log rotation and backup management
• hbd.plugin — plugin framework with base classes, registry, and dynamic loader
• hbd.plugins/ — built-in plugins (os_info, cpu_monitor, memory_monitor, disk_monitor, network_monitor, filesystem_info, nagios_runner)
• hbd.hbc — heartbeat client that sends heartbeats and plugin data to server
• hbd.utils — small utility helpers (shortname, dur, initlog)
• hbd.cli — CLI entrypoint and argument parsing
• hbd.server — async orchestration to run UDP/HTTP/WSS components

This modular layout makes the code easier to test and maintain.

Runtime & Shutdown

• The main runtime is asyncio-based. Services (UDP listener, HTTP server, WebSocket server, monitor, and DNS worker) run as asyncio tasks.
• On SIGINT/SIGTERM the server triggers a graceful shutdown: it cancels active tasks, signals the DNS worker via a sentinel, and cleans up resources before exit.
• The DNS update worker is implemented as an asyncio task; synchronous producers can still enqueue DNS updates via a small thread-safe bridge available at hbd.hbdclass.Host.dnsQ.

Templates & Static Files

• Template files are located under hbd/templates by default. The HTTP server resolves templates relative to the hbd package but the path can be overridden with the templates_dir config key.
• Static assets (CSS/JS/images) are served from hbd/static via the /static/<path> HTTP route. Place your static files in that directory or configure the HTTP server as needed.

---

🧪 Testing & Dev

Tests are implemented using unittest and additional tests rely on pytest if you prefer. To run tests locally without installing anything beyond the dev requirements:

# with project root on PYTHONPATH
PYTHONPATH=. python -m unittest discover -v
# or with pytest if installed
pytest -q

Developer tooling included:

• pyproject.toml — project metadata and dependencies
• tox.ini — convenience wrappers for running tests, lint, and mypy

To run linters and type checks locally:

# after installing dev deps
tox -e lint
tox -e mypy

---

🚀 Running in production

• Use your system service manager (systemd, launchd, etc.) to run hbd in the background.
• Ensure nsupdate and necessary credentials are available for dynamic DNS updates.
• Configure TLS for WSS if you enable secure websockets.

Note: The project contains a small example for obtaining DNS-verified certs (certbot with RFC2136) — see earlier commit history or ask me to re-add the example to this README if you want it documented here.

---

🤝 Contributing

Contributions welcome! Please:

1. Open an issue to discuss larger changes.
2. Create a topic branch and a clear PR.
3. Add tests for new features and run linters.
4. Keep changes focused and documented.

---

📜 License

This repository is licensed under the MIT license. See LICENSE for details.

---

If you'd like, I can also:

• add a GitHub Actions workflow that runs tests and lint on push/PR 🔁
• add a CONTRIBUTING.md template for PRs and code style 💬

Which one should I do next? ✨