heartbeat/docs/NOTIFICATIONS.md

Notification System

Overview

Notifications are dispatched to the owner and managers of a host, each via their own configured notification channels. Channel definitions are global; users reference them by name. No users configured → no notifications sent.

Architecture

Alert event (udp.py / threshold.py)
  └─ notify.send_notification(host_name, Notification)
       ├─ look up host.owner + host.managers
       ├─ for each user → user.notification_channels
       └─ for each channel → _dispatch_to_channel (filtered by min_level)

Every notification carries:

• title — [LEVEL] hostname (e.g. [CRITICAL] webserver01)
• body — detail message (metric value, threshold, duration)
• url — link to the plugin metrics page ({base_url}/plugins#{hostname})
• level — RECOVER | WARNING | CRITICAL | INFO

Configuration

Base URL

Set base_url so notification links point to your hbd instance:

base_url: https://hbd.example.com

Channel definitions

Channels are defined under notification_channels. Each entry specifies a delivery type and its credentials. Two optional metadata fields control visibility:

Field	Default	Description
owner	(absent)	Username who created/owns this channel. Absent = admin-created.
private	false	When true, only the owner can see and select this channel.
min_level	WARNING	Minimum alert level this channel receives.

Admin-created channels (set in the config file or via the admin settings UI) are public by default — all users can select them:

notification_channels:

  pushover_ops:
    type: pushover
    token: your-app-token
    user: your-user-key
    min_level: WARNING

  email_ops:
    type: email
    recipients: [ops@example.com]
    sender: hbd@example.com
    smtp_server: smtp.example.com
    smtp_port: 587
    smtp_user: hbd@example.com
    smtp_password: secret
    min_level: WARNING

  matrix_oncall:
    type: matrix
    homeserver: https://matrix.example.org
    access_token: syt_xxx
    room_id: "!abc:matrix.example.org"
    min_level: CRITICAL

  sms_oncall:
    type: sms_voipms
    api_user: me@example.com
    api_password: secret
    did: "5551234567"
    dst: "5559876543"
    min_level: CRITICAL

  signal_ops:
    type: signal
    cli_path: /usr/local/bin/signal-cli
    user: +12025551234
    recipient: +12025559999

  mattermost_devops:
    type: mattermost
    host: mattermost.example.com
    token: webhook-token
    channel: devops-alerts
    username: heartbeat-bot

User-created channels are written by authenticated users through the API or their profile page. They carry an owner field and optionally private: true:

notification_channels:

  alice_personal:
    type: pushover
    token: personal-token
    user: personal-key
    owner: alice          # created by alice
    private: true         # only alice can see this channel

Channel visibility

Channel	Who can see / select it
No private field (or private: false)	All users
private: true	Only the owner
Any channel	Admins always see everything

Users with notification channels

Each user lists which channels they receive notifications on. Users can manage their own selection from the profile page:

users:
  alice:
    full_name: Alice Smith
    password: pbkdf2:sha256:...
    admin: true
    notification_channels: [pushover_ops, email_ops]

  bob:
    full_name: Bob Jones
    password: pbkdf2:sha256:...
    notification_channels: [sms_oncall, matrix_oncall]

Host access — owner and managers

Notifications for a host go to its owner and all managers:

hosts:
  webserver01:
    owner: alice             # receives all notifications for this host
    managers: [bob]          # also receives notifications
    threshold_config: default
    watch: true              # bold in dashboard (cosmetic only)
    dyndns: false

  dbserver01:
    owner: alice
    managers: [bob]
    threshold_config: database
    dyndns: false

watch: true only affects display (bold name in the live dashboard). Notifications are now controlled entirely by owner/managers.

Channel Types

min_level filtering

Every channel accepts an optional min_level field:

Value	Channels receive
WARNING (default)	WARNING, CRITICAL, RECOVER
CRITICAL	CRITICAL only (and RECOVER)

RECOVER is always passed through — you don't want to miss a recovery.

pushover

Sends push notifications via Pushover. Includes title, body, and a clickable URL.

type: pushover
token: your-app-token     # Required: Pushover application token
user: your-user-key       # Required: Recipient's user key
min_level: WARNING

email

Sends via SMTP. Subject = title, body = message + URL on final line.

type: email
recipients: [ops@example.com, oncall@example.com]
sender: hbd@example.com
smtp_server: smtp.example.com
smtp_port: 587             # 587 = STARTTLS (default), 465 = SSL
smtp_user: hbd@example.com
smtp_password: secret
min_level: WARNING

matrix

Sends a formatted HTML message to a Matrix room via matrix-nio.

type: matrix
homeserver: https://matrix.example.org
access_token: syt_xxx      # Bot account access token
room_id: "!abc:matrix.example.org"
min_level: WARNING

Setup:

1. Create a bot Matrix account
2. Obtain its access token (Element → Settings → Help & About → Access Token)
3. Invite the bot to the target room and note the room ID

sms_voipms

Sends SMS via the voip.ms REST API. Message is truncated to 160 characters.

type: sms_voipms
api_user: me@example.com   # voip.ms account email
api_password: secret       # voip.ms API password
did: "5551234567"          # Your voip.ms DID (sending number)
dst: "5559876543"          # Destination number
min_level: CRITICAL

signal

Sends via signal-cli.

type: signal
cli_path: /usr/local/bin/signal-cli
user: +12025551234         # Your registered Signal number
recipient: +12025559999    # Recipient number
min_level: WARNING

Setup:

signal-cli -u +12025551234 register
signal-cli -u +12025551234 verify CODE

mattermost

Sends via Mattermost incoming webhook. Message is formatted as Markdown.

type: mattermost
host: mattermost.example.com
token: your-webhook-token
channel: devops-alerts
username: heartbeat-bot    # Optional: display name
icon: https://…/icon.png   # Optional: bot icon URL
min_level: WARNING

Notification events

Source	Level	Title example	Body example
Host overdue	CRITICAL	[CRITICAL] webserver01	IPv4 overdue
Host recover	RECOVER	[RECOVER] webserver01	IPv4 back after being overdue for 5:23
Host boot	INFO	[INFO] webserver01	webserver01 booted
Host shutdown	INFO	[INFO] webserver01	IPv4 shutdown
Threshold breach	WARNING/CRITICAL	[CRITICAL] webserver01	cpu_percent = 95.2 (threshold: > 90.0)
Threshold reminder	CRITICAL	[REMINDER/CRITICAL] webserver01	REMINDER (CRITICAL): … ongoing for 3600s
Connection issue	WARNING	[WARNING] webserver01	new address detected …

Reminder notifications (re-notify) are sent only for CRITICAL level alerts.

API reference

send_notification(host_name, notif) -> dict

Main entry point. Dispatches to owner + managers.

from hbd.server.notify import send_notification, Notification

send_notification(
    "webserver01",
    Notification(
        title="[CRITICAL] webserver01",
        body="cpu_percent = 95.2 (threshold: > 90.0)",
        level="CRITICAL",
        url="https://hbd.example.com/plugins#webserver01",
    ),
)

Returns {channel_name: bool} for each channel dispatched.

setup(cfg, loop=None)

Called once at startup from main.py. Pass the running asyncio event loop so Matrix sends work correctly.

Troubleshooting

No notifications sent:

• Check that users are configured (users: section in yaml)
• Check that the host has an owner or managers set
• Check that users have notification_channels listed
• Check that the channel names in user config match keys under notification_channels:
• If a user can't select a channel, check whether it is private: true and owned by someone else

min_level filtering too aggressive:

• Default is WARNING — both WARNING and CRITICAL are sent
• Set min_level: WARNING explicitly if you were expecting warnings but set CRITICAL

Matrix sends time out:

• Verify the access token is valid and the bot is in the room
• matrix-nio must be installed: pip install matrix-nio

voip.ms SMS fails:

• Enable the API in your voip.ms account (Account → API)
• Verify the DID is SMS-capable in your voip.ms account

Signal not found:

• Specify full cli_path
• Run signal-cli -u +NUMBER receive to sync trust store

Email authentication failed:

• Use app-specific passwords for Gmail/Fastmail
• Verify port: 587 for STARTTLS, 465 for SSL

Pushover 400 errors:

• Double-check token (app) and user (user key) — they are different values