Fix rtt, including bug in time compute

2026-04-01 19:41:53 -04:00
parent 090d341244
commit 460d2be9e9
13 changed files with 1366 additions and 372 deletions
@@ -335,43 +335,111 @@ threshold_renotify_interval: 3600  # Re-notify every hour for ongoing alerts

 ### Notification Channels

-Thresholds use the same notification infrastructure as heartbeat monitoring:
+The system supports centralized notification channel definitions, allowing different hosts to use different notification providers and credentials. This provides fine-grained control over who gets notified about what.
+
+#### Supported Channel Types

 - **Email** (via SMTP)
 - **Pushover** (mobile notifications)
- **Mattermost** (team chat)
- **Custom webhooks**
+- **Signal** (via signal-cli)
+- **Mattermost** (team chat webhooks)

-Configuration:
+#### Centralized Channel Configuration
+
+Define notification channels once in the configuration file:

 ```yaml
-# Email
-toemail:
-  - admin@example.com
-  - oncall@example.com
-fromemail: heartbeat@example.com
-smtpserver: smtp.example.com
-smtpport: 587
-smtpuser: heartbeat@example.com
-smtppassword: your-password
+notification_channels:
+  # Signal notifications
+  signal_ops:
+    type: signal
+    cli_path: /usr/local/bin/signal-cli
+    user: +1234567890
+    recipient: +1234567890
+  
+  # Email notifications
+  email_ops:
+    type: email
+    recipients: [ops@example.com, alerts@example.com]
+    sender: heartbeat@example.com
+    smtp_server: smtp.example.com
+    smtp_port: 587
+    smtp_user: heartbeat@example.com
+    smtp_password: your-smtp-password
+  
+  # Pushover notifications
+  pushover_urgent:
+    type: pushover
+    token: your-pushover-app-token
+    user: your-pushover-user-key
+  
+  # Mattermost notifications
+  mattermost_devops:
+    type: mattermost
+    host: mattermost.example.com
+    token: your-webhook-token
+    channel: devops-alerts
+    username: heartbeat-bot
+    icon: https://example.com/heartbeat-icon.png

-# Pushover
-pushover_token: your-app-token
-pushover_user: your-user-key
+# Default channels for hosts that don't specify channels
+default_notification_channels: [email_ops]
+```
+
+#### Per-Host Channel Assignment
+
+Assign notification channels to specific hosts in the `hosts` section:
+
+```yaml
+hosts:
+  # Critical server - multiple notification channels
+  prod-web-01:
+    threshold_config: high_sensitivity
+    watch: true
+    notification_channels: [signal_ops, pushover_urgent, email_ops]
+    dyndns: false
+  
+  # Database server - ops team only
+  prod-db-01:
+    threshold_config: database
+    watch: true
+    notification_channels: [signal_ops, email_ops]
+    dyndns: false
+  
+  # Development server - email only
+  dev-server-01:
+    threshold_config: low_sensitivity
+    watch: false
+    notification_channels: [email_ops]
+    dyndns: false
+  
+  # Uses default_notification_channels if not specified
+  test-server-01:
+    threshold_config: default
+    watch: false
+    dyndns: false
 ```

 ### Watched Hosts

-Only hosts in the `watchhosts` list will trigger notifications:
+Only hosts with `watch: true` in the `hosts` section will trigger notifications:

 ```yaml
-watchhosts:
-  - webserver01
-  - database01
-  - mailserver
+hosts:
+  webserver01:
+    watch: true
+    notification_channels: [email_ops]
+  
+  database01:
+    watch: true
+    notification_channels: [signal_ops, email_ops]
+  
+  mailserver:
+    watch: true
+    notification_channels: [pushover_urgent]
 ```

-Hosts not in this list will still have thresholds checked and alert states tracked, but won't send notifications.
+Hosts not marked for watching will still have thresholds checked and alert states tracked, but won't send notifications.

 ## Alert State Tracking