Compare commits

...
51 Commits
Author SHA1 Message Date
andreasandClaude Sonnet 5 00d6ba607e docs: complete the 5.4.4 changelog entry
Release / release (push) Successful in 45s
The release script only picks up unscoped feat:/fix: subjects, so the
presence/simplex feature was missing from the generated entry.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uc9V1v7iXruubpoLiSE24M
2026-09-21 10:01:52 -04:00
andreas cd5de7f205 version 5.4.4
Release / release (push) Successful in 48s
2026-09-21 10:00:43 -04:00
andreasandClaude Sonnet 5 95355374f8 fix: never park the last present family as absent
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uc9V1v7iXruubpoLiSE24M
2026-09-21 09:50:43 -04:00
andreasandClaude Sonnet 5 8122c8d7dc feat(ui): show host status, simplex and absent protocols
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uc9V1v7iXruubpoLiSE24M
2026-09-21 09:36:25 -04:00
andreasandClaude Sonnet 5 3970f4cdde fix(server): restore keeps connectivity alert for connections dropped to unknown
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uc9V1v7iXruubpoLiSE24M
2026-09-21 09:34:38 -04:00
andreasandClaude Sonnet 5 c3736f47ee feat(server): monitor only present protocols; host-level overdue and simplex alerts
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uc9V1v7iXruubpoLiSE24M
2026-09-21 09:28:26 -04:00
andreasandClaude Sonnet 5 2f1cad9156 feat(server): host presence, absent state and host-level status
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uc9V1v7iXruubpoLiSE24M
2026-09-21 09:25:33 -04:00
andreasandClaude Sonnet 5 1898bc52e5 feat(client): report per-protocol presence in HTB and network_monitor
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uc9V1v7iXruubpoLiSE24M
2026-09-21 09:23:33 -04:00
andreasandClaude Sonnet 5 c61a571995 fix(client): keep ping-less family unknown after route returns
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uc9V1v7iXruubpoLiSE24M
2026-09-21 09:22:03 -04:00
andreasandClaude Sonnet 5 9e8d68e712 feat(client): per-protocol connectivity probe
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uc9V1v7iXruubpoLiSE24M
2026-09-21 09:19:48 -04:00
andreasandClaude Sonnet 5 ad19d2596a fix(client): default_gateway never raises
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uc9V1v7iXruubpoLiSE24M
2026-09-21 09:17:48 -04:00
andreasandClaude Sonnet 5 82ba69d827 feat(client): best-effort default gateway lookup
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uc9V1v7iXruubpoLiSE24M
2026-09-21 09:15:17 -04:00
andreasandClaude Sonnet 5 9519169b47 refactor: extract ping helpers into pingutil
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uc9V1v7iXruubpoLiSE24M
2026-09-21 09:12:53 -04:00
andreas f838920f7c version 5.4.3
Release / release (push) Successful in 1m16s
2026-08-17 13:54:02 -04:00
andreasandClaude Sonnet 5 45d11a66b5 fix: fix RTT y-axis label clipping, add x-axis tickmarks, enlarge charts
Y-axis labels on the RTT chart were truncated above 99ms because the
left margin was a fixed guess; it's now sized to the widest tick
label. The x-axis only showed start/end timestamps; added evenly
spaced intermediate tickmarks and labels. Plot area enlarged 15%.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-08-17 13:53:57 -04:00
andreas a42d783c0c version 5.4.2
Release / release (push) Successful in 1m10s
2026-08-17 09:24:23 -04:00
andreasandClaude Sonnet 5 18e33656c0 fix: preserve chart history across reconnects, show gaps for missing data
Only wipe real (non-RTT) plugin data on an actual client reboot (boot
flag), not on every ordinary OVERDUE/DOWN -> UP recovery. A transient
network blip no longer erases CPU/memory/etc. history.

Also split the shared time-series chart into separate line/area segments
wherever the gap between samples is much larger than the typical spacing,
so missing data (host overdue, or history that simply hasn't accumulated
across a drop) renders as a visual gap instead of an interpolated line.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DEimzMv4Q5EjFg3hoiZ69T
2026-08-17 09:13:49 -04:00
andreas 4ac5a34a43 version 5.4.1
Release / release (push) Successful in 1m19s
2026-08-17 08:16:25 -04:00
andreasandClaude Sonnet 5 6d39c5be47 fix: correct empty-state copy on Host Overview page
Since RTT history keys make host.plugin_data non-empty after the very
first heartbeat, the "no hosts" empty state can now only trigger when
zero hosts have ever connected, not when hosts exist but haven't sent
plugin metrics yet (that in-between state no longer exists). Update
the copy to describe the actual current trigger condition.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DEimzMv4Q5EjFg3hoiZ69T
2026-08-17 08:07:23 -04:00
andreasandClaude Sonnet 5 cdb2183989 fix: stop synthetic rtt_* keys from masking real plugin-data checks
host.plugin_data now always holds rtt_ipv4/rtt_ipv6 history after the
first heartbeat, which broke two pre-existing behaviors that assumed
plugin_data reflects only real client-collected data:

- The request_update ACK gate in handle_datagram() never fired again
  after a host's first heartbeat, so stale OS/agent-version info was
  never refreshed after a reconnect (only a client restart fixed it).
- plugin_data.clear() on every UP transition wiped rtt_* history too,
  so a flaky host could never accumulate a useful RTT graph.

Add _is_rtt_key()/_has_real_plugin_data() helpers so both spots treat
rtt_* keys as synthetic: the gate now looks only at real plugin data,
and the recovery clear only drops real plugin keys, preserving RTT
history across reconnects.

Also fixes two pre-existing E127 continuation-indent flake8 issues in
tests/test_udp_rtt_history.py.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DEimzMv4Q5EjFg3hoiZ69T
2026-08-17 08:07:19 -04:00
andreasandClaude Sonnet 5 41c78e8e34 feat: live-refresh Connectivity section and RTT charts every 30s
Add fetchHostInfo() call to the 30-second auto-refresh setInterval loop
for expanded host cards. This ensures the Connectivity table and RTT
charts update in real-time without requiring the user to collapse and
re-expand the host card.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DEimzMv4Q5EjFg3hoiZ69T
2026-08-17 07:44:14 -04:00
andreas acf1eb8804 feat: render RTT history chart per connection in Connectivity section 2026-08-17 07:39:17 -04:00
andreasandClaude Sonnet 5 49b631f02d refactor: extract shared time-series chart renderer from CPU chart
Pulls the CPU chart's SVG-drawing logic into a parameterized
renderTimeSeriesChart(elId, pts, opts) helper (yDomain, colorFor,
clipId, unitSuffix), keeping renderCpuChart's signature and behavior
unchanged. Task 5 will reuse this for the RTT chart.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DEimzMv4Q5EjFg3hoiZ69T
2026-08-17 07:33:22 -04:00
andreas 8204737ea6 fix: use the latest RTT sample, not the oldest, in host info API 2026-08-17 07:28:36 -04:00
andreas 5b2fc8fd46 fix: hide synthetic rtt history keys from plugin accordion list 2026-08-17 07:25:17 -04:00
andreasandClaude Sonnet 5 70c3e93684 fix: remove extra blank line to pass flake8 E303 check
Removed extraneous blank line after conndata() try/except block
that was left during code relocation. Fixes E303 linting error.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DEimzMv4Q5EjFg3hoiZ69T
2026-08-17 07:22:33 -04:00
andreasandClaude Sonnet 5 52968201fa feat: record RTT history per heartbeat for charting
Capture RTT on every heartbeat using the existing plugin_data history
mechanism, recording under rtt_ipv4 or rtt_ipv6 keys. Maintains 100-sample
retention per host and address family.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DEimzMv4Q5EjFg3hoiZ69T
2026-08-17 07:18:01 -04:00
andreasandClaude Sonnet 5 a7b698e6b9 fix: clear flap state when a host is dropped
flap._state persists at module level keyed by (host, service) and only
clears via a RECOVER-triggered quiet window. A host dropped mid-flap with
no RECOVER ever received leaves ok_since permanently None, so the
flapping flag could never clear on its own.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-08-07 08:35:10 -04:00
andreas 136b10239f update 2026-08-07 07:30:41 -04:00
Andreas WredeandClaude Sonnet 5 1e46f33648 fix: correct plugin import path in docs, wire up orphaned footer template
docs/PLUGIN_DEVELOPMENT.md referenced a nonexistent hbd/plugin.py and
hbd/plugins/ directory; the real module is hbd/client/plugin.py with
plugins under hbd/client/plugins/, as README.md and the actual code
already use. Anyone following the doc would hit ModuleNotFoundError.

foot.html was never included by any template (confirmed via git grep
and template audit) despite being actively maintained; wired it into
about.html. menu.html was fully commented out and unreferenced, so
removed it.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-31 09:51:36 -04:00
andreas 4c3a63e645 update claude guideline 2026-07-31 09:22:57 -04:00
Andreas Wrede 1f4dd7cb1f version 5.4.0
Release / release (push) Successful in 44s
2026-07-23 13:17:33 -07:00
Andreas WredeandClaude Opus 4.8 ae3f2fc70f feat: gate remote command execution behind allow_remote_command
CMD packets arrive as unauthenticated UDP datagrams, yet every hbc client
executed the shell command they carry without any opt-in. Add an
allow_remote_command config key, default false: when off, the command is
logged and refused with "Refused: allow_remote_command is false" (visible in
the server event log under the command service), and subprocess is never
reached. When on, the client warns at startup that it will execute CMD
packets.

Applied to all four clients that handle CMD — hbc, hbc_windows.py,
hbc_mini.py, and the C hbc_mini — since gating only one leaves the others
wide open. The C client reads the same key from ~/.hbc.json and needs a
rebuild to pick it up. UPD (self-update) is unchanged.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-23 13:10:41 -07:00
Andreas WredeandClaude Opus 4.8 4414967bdc feat: flapping detection suppresses notification storms
A (host, service) pair is flapping once it exceeds flap_count warning or
critical notifications within flap_interval minutes. The notification that
trips the state carries "Now flapping!! No more messages!" and every later
one is dropped, including RECOVER. The state ends silently flap_interval
minutes after a RECOVER, provided no further alert arrived meanwhile.

Hooked into notify.send_notification, the single choke point for channel
delivery, so only outbound notifications are suppressed — eventlog keeps
recording, leaving the journal and /log with the full history of the flap.

Threshold alerts key on their metric path, so a flapping disk check cannot
silence a CPU alert; connectivity, boot and shutdown events key on the host
itself. State lives at module level in flap.py and is never pickled.

Flapping pairs surface in Host.stateinfo() and render as an amber badge on
the live dashboard. Config: flap_count (5), flap_interval (10 minutes),
0 in either disables; both editable on the settings page.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-23 09:53:46 -07:00
Andreas Wrede e3b0e5041f add: info on deplyment 2026-07-23 09:51:32 -07:00
andreasandClaude Fable 5 92982c0ca7 fix: harden events log — non-dict ring entries, off-loop journal reads, no default-config singleton
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-10 15:51:34 -04:00
andreasandClaude Fable 5 00fdbcaa5b fix: skip journal scheduling when the event loop is not running
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-10 15:39:42 -04:00
andreasandClaude Fable 5 600e68b509 feat: remove log section from live dashboard (moved to /log)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-10 15:29:39 -04:00
andreasandClaude Fable 5 b26e853a1c fix: dedup log rows between API seed and websocket tail
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-10 15:25:01 -04:00
andreasandClaude Fable 5 36adad5b3b feat: dedicated /log page with journal-backed history and nav item
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-10 15:20:17 -04:00
andreasandClaude Fable 5 7591268a89 feat: GET /api/0/log — paged, filtered events journal API
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-10 15:15:46 -04:00
andreasandClaude Fable 5 7afc06a329 fix: journal startup/shutdown lifecycle events (fired outside journal lifetime)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-10 15:11:30 -04:00
andreasandClaude Fable 5 a3f303e6ba feat: eventlog writes to dedicated events journal; init/backfill/close wiring
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-10 15:03:48 -04:00
andreasandClaude Fable 5 16c2922bea feat: events journal read path with filters and backward pagination
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-10 14:57:32 -04:00
andreasandClaude Fable 5 0343eb4a24 fix: drop typing names not yet used from journal.py import
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-10 14:53:24 -04:00
andreasandClaude Fable 5 fb5570155b feat: events journal write primitives (log_event, backfill, get_events_journal)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-10 14:47:42 -04:00
andreasandClaude Fable 5 8d268adf70 feat: Live Dashboard redesign — 6 columns, hover details, last-alert column
Replaces the 11-column table with severity-sorted record rows: host,
combined status chip, latency, alert chips, time-in-state, and the most
recent alert message. Per-family address/state/latency/last-change move
to a hover card on the status chip and permanently to a new Connectivity
table in the Host Overview info section (host info API now includes
connections). Last-alert is seeded from /api/0/alerts and kept fresh
from the event stream; the event log below uses the same row idiom with
its filters intact. WS reconnect state shows in the toolbar instead of
a modal.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NfPpSpccTWBfZg1FTveyaU
2026-07-10 13:45:03 -04:00
andreasandClaude Fable 5 5d9f161706 feat: extend the record-row design system to Host Overview, Alerts, and About
Extract the settings redesign's tokens and components into shared
static/hbd-ui.css + hbd-ui.js and dedupe settings.html against them.
About becomes yaml-key sections with kv rows; Alerts gets stat tiles,
chip filters, and alert record rows (same fetch/ack logic); Host
Overview keeps its DOM and live-update JS but is re-skinned to the
token system with a page toolbar. Live Dashboard intentionally
untouched pending its own redesign.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NfPpSpccTWBfZg1FTveyaU
2026-07-10 09:37:58 -04:00
andreasandClaude Fable 5 067feacd59 fix: settings toolbar sticks below the site nav instead of scrolling away
Sticky offsets (toolbar, rail, mobile chips, anchor margins) derive from
the measured nav/toolbar heights via CSS variables, since the fixed nav's
height varies with viewport width.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NfPpSpccTWBfZg1FTveyaU
2026-07-10 09:19:21 -04:00
andreasandClaude Fable 5 c564fafe73 fix: settings page not scrollable — restore html/body overflow override
static/style.css pins html/body to height:100% with overflow:hidden for
the dashboard pages; the redesign dropped the counter-override the old
template carried.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NfPpSpccTWBfZg1FTveyaU
2026-07-10 08:20:21 -04:00
andreasandClaude Fable 5 8eeae48964 feat: unified record-row redesign of the settings page
One idiom for every section: mono identifier, badges, fact chips, inline
accordion editor. Scalar config sections merge into one grouped 'server'
section. Sidebar becomes YAML-key nav (mobile: horizontal chips); sticky
toolbar carries pending-changes state and Publish. Staging, publish,
channel CRUD, and permission gating are unchanged — same payload shapes
and endpoints.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NfPpSpccTWBfZg1FTveyaU
2026-07-10 08:08:58 -04:00
53 changed files with 5343 additions and 3265 deletions
+1
View File
@@ -16,3 +16,4 @@ uv.lock
.superpowers/ .superpowers/
rndc-key rndc-key
docs/superpowers/ docs/superpowers/
graphify-out/
+77
View File
@@ -2,6 +2,83 @@
All notable changes to this project are documented here, organized by release. All notable changes to this project are documented here, organized by release.
## [5.4.4]
### Added
- client reports per-protocol (IPv4/IPv6) internet presence: a connectivity probe (route check every ~5s, ICMP echo every ~60s, debounced) sends `p4`/`p6` flags on every heartbeat and a `connectivity` metric (presence, own address, gateway, RTT) via `network_monitor`
- server monitors only the protocols a client reports as present; a protocol reported absent is shown as `absent` and raises no alert
- host status (`up`, `simplex`, `overdue`, `down`, `unknown`) computed on the server: a host is overdue only when every present protocol is overdue; `simplex` (both present, one silent) has the same severity as `up` and is logged without notification
- live and plugins pages show the host status, `simplex`/`absent` protocols and a Connectivity table
- `hbc` config keys under `network_monitor`: `probe_enabled`, `probe_v4`, `probe_v6`, `probe_interval`
### Changed
- overdue and recovery alerts are raised per host, not per protocol: a single CRITICAL when the last present protocol goes overdue; a protocol coming back while another stayed up is logged as INFO instead of a RECOVER notification (also applies to clients that do not report presence)
### Fixed
- never park the last present family as absent, so a host that reports no protocol as present still alerts when it dies
- keep the connectivity alert for a host overdue past the drop window across an `hbd` restart
- `default_gateway` never raises; a protocol whose probe cannot run stays unknown rather than absent
---
## [5.4.3]
### Fixed
- fix RTT y-axis label clipping, add x-axis tickmarks, enlarge charts
---
## [5.4.2]
### Fixed
- preserve chart history across reconnects, show gaps for missing data
---
## [5.4.1]
### Added
- live-refresh Connectivity section and RTT charts every 30s
- render RTT history chart per connection in Connectivity section
- record RTT history per heartbeat for charting
### Fixed
- correct empty-state copy on Host Overview page
- stop synthetic rtt_* keys from masking real plugin-data checks
- use the latest RTT sample, not the oldest, in host info API
- hide synthetic rtt history keys from plugin accordion list
- remove extra blank line to pass flake8 E303 check
- clear flap state when a host is dropped
- correct plugin import path in docs, wire up orphaned footer template
---
## [5.4.0]
### Added
- gate remote command execution behind allow_remote_command
- flapping detection suppresses notification storms
- remove log section from live dashboard (moved to /log)
- dedicated /log page with journal-backed history and nav item
- GET /api/0/log — paged, filtered events journal API
- eventlog writes to dedicated events journal; init/backfill/close wiring
- events journal read path with filters and backward pagination
- events journal write primitives (log_event, backfill, get_events_journal)
- Live Dashboard redesign — 6 columns, hover details, last-alert column
- extend the record-row design system to Host Overview, Alerts, and About
- unified record-row redesign of the settings page
### Fixed
- harden events log — non-dict ring entries, off-loop journal reads, no default-config singleton
- skip journal scheduling when the event loop is not running
- dedup log rows between API seed and websocket tail
- journal startup/shutdown lifecycle events (fired outside journal lifetime)
- drop typing names not yet used from journal.py import
- settings toolbar sticks below the site nav instead of scrolling away
- settings page not scrollable — restore html/body overflow override
---
## [5.3.12] ## [5.3.12]
### Added ### Added
+94 -1
View File
@@ -1,4 +1,97 @@
# CLAUDE.md
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
## Working principles
1. Don't assume. Don't hide confusion. Surface tradeoffs. 1. Don't assume. Don't hide confusion. Surface tradeoffs.
2. Minimum code that solves the problem. Nothing speculative. 2. Minimum code that solves the problem. Nothing speculative.
3. Touch only what you must. Clean up only your own mess. 3. Touch only what you must. Clean up only your own mess.
4. Define success criteria. Loop until verified. 4. Define success criteria. Loop until verified.
## Commands
```bash
# Run tests
pytest -q # all tests
pytest tests/test_threshold.py # single file
pytest -k test_name # single test
# Lint and type check
tox -e lint # flake8 over hbd/ and tests/
tox -e mypy # mypy over hbd/
# Install for development
pip install -e ".[all,dev]"
# Run server
python -m hbd.server.cli serve -c .hb.yaml -f -v
# Generate a password hash
hbd passwd <username>
```
Tests live in `tests/` (pytest, imported as a package). A second directory `test/` contains only TLS fixtures (`.pem` files), not test code.
Line length: 111 characters (`black`, `flake8`, both configured).
## Architecture
The system has two components: `hbc` (client) and `hbd` (server), both in the `hbd` Python package.
```
hbd/
common/ # proto.py (encode/decode), utils.py
client/ # hbc: plugins/, main.py, config.py, plugin.py
server/ # hbd: all server modules
```
### Server module map
| Module | Role |
|---|---|
| `cli.py` | Argument parsing, entry point, daemon startup |
| `main.py` | Asyncio runtime: starts UDP, HTTP, WebSocket; pickle save/load |
| `hbdclass.py` | `Host` and `Connection` domain objects; DNS queue |
| `udp.py` | UDP datagram listener; processes `HTB` and `PLG` messages; sets/resets overdue timers |
| `http.py` | aiohttp web server; all REST API routes (`/api/0/`) and page routes |
| `ws.py` | WebSocket broadcast to connected dashboard clients |
| `notify.py` | Notification dispatch (Pushover, email, Mattermost, Matrix, Signal, SMS); `eventlog()` is the single choke-point for all alert events |
| `threshold.py` | Threshold evaluation against plugin metrics; state transitions (OK/WARNING/CRITICAL/UNKNOWN) |
| `monitor.py` | Timer cleanup on shutdown (reachability is event-driven via timers in `udp.py`, not polled) |
| `data.py` | Shared in-memory message ring buffer |
| `journal.py` | JSONL message journal with size-based rotation |
| `users.py` | Session management, password hashing (PBKDF2), role checks |
| `config.py` | Config loading and defaults |
| `configio.py` | Config file read/write via `ruamel.yaml` (preserves comments) |
| `settings.py` | Settings sections for the web UI settings page |
| `dns.py` | `nsupdate` integration for dynamic DNS |
| `oauth.py` | OAuth2 login (Gitea) |
### Key data flows
**Heartbeat received:** `udp.py` decodes the `HTB` datagram → updates `Connection` state in `hbdclass.py` → resets overdue asyncio timer → broadcasts via `ws.py` → `notify.py` fires connectivity alerts on state change.
**Plugin data received:** `udp.py` decodes `PLG` datagram → stores on `Host` → `threshold.py` evaluates against configured thresholds → `notify.py` fires threshold alerts on state transitions.
**State persistence:** `Host.hosts` dict + `data.msgs` ring + active sessions are pickled every 5 minutes and on clean shutdown. Asyncio timers are stripped before pickling (`Connection.__getstate__`).
**Config reload:** SIGHUP → `configio.py` re-reads YAML → live-updates hosts, thresholds, users, notification channels. Port/cert/pickle/journal changes require a full restart.
### Client plugin system
Plugins in `hbd/client/plugins/` subclass `InfoPlugin` (collected once, on demand) or `MonitorPlugin` (periodic). `initialize()` returns `False` to self-disable. Data is sent as `PLG` UDP messages.
`hbc_mini.py` (scripts/) and `hbc_mini.c` (scripts/c/) are standalone single-file clients with no external dependencies.
### Protocol
All UDP messages: `!<ID>: <zlib-compressed key=value payload>`. Encoding in `hbd/common/proto.py`. Lists/dicts encoded as JSON with `@` prefix; booleans as `1`/`0`.
### Web UI
Jinja2 templates in `hbd/server/templates/`. Static assets in `hbd/server/static/`. Live pages (`/live`, `/plugins`) use WebSocket connections for real-time push.
### CI
Gitea Actions workflow at `.gitea/workflows/release.yml`.
+33 -2
View File
@@ -20,7 +20,7 @@ A lightweight UDP-based host monitoring system. Monitored hosts run a client (`h
└────────────────────┘ └────────────────────────────┘ └────────────────────┘ └────────────────────────────┘
``` ```
**Package:** `hbd` v5.3.12 **Package:** `hbd` v5.4.4
**Python:** 3.11+ **Python:** 3.11+
### Subpackages ### Subpackages
@@ -116,6 +116,11 @@ dyndomains:
# Threshold alert re-notification interval (seconds) # Threshold alert re-notification interval (seconds)
threshold_renotify_interval: 3600 threshold_renotify_interval: 3600
# Flap detection — silence a service/host after flap_count warning or critical
# notifications within flap_interval minutes (flap_count: 0 disables)
flap_count: 5
flap_interval: 10
# Notification channels # Notification channels
notification_channels: notification_channels:
pushover_ops: pushover_ops:
@@ -207,6 +212,9 @@ hb_port: 50003 # Server UDP port
interval: 10 # Heartbeat interval (seconds) interval: 10 # Heartbeat interval (seconds)
owner: alice # Optional: claim ownership of this host owner: alice # Optional: claim ownership of this host
allow_remote_command: false # Execute shell commands sent by the server in CMD packets.
# Default false — see Remote command execution below.
plugins: plugins:
cpu_monitor: cpu_monitor:
interval: 300 # Override collection interval interval: 300 # Override collection interval
@@ -237,6 +245,27 @@ plugins:
- If a connection fails to open at startup, IPv6 connections are dropped after 3 consecutive failures. IPv4 connections retry indefinitely. - If a connection fails to open at startup, IPv6 connections are dropped after 3 consecutive failures. IPv4 connections retry indefinitely.
- In daemon mode (`-d`), all log output goes to syslog (`LOG_DAEMON` facility). - In daemon mode (`-d`), all log output goes to syslog (`LOG_DAEMON` facility).
### Remote command execution
The server can queue a shell command for a host, delivered in a `CMD` packet. Because
heartbeat packets are unauthenticated UDP, the client only runs those commands when the
host opts in:
```yaml
allow_remote_command: true
```
With the default `false`, the command is logged and refused, and the client replies
`Refused: allow_remote_command is false` — visible in the server's event log under the
`command` service, so a queued command never fails silently. When enabled, `hbc` logs a
warning at startup naming the risk. `SIGHUP` re-execs the client, so a config change takes
effect on reload.
All four clients enforce this: `hbc`, `hbc_windows.py`, `hbc_mini.py`, and the C
`hbc_mini` (which reads the same key from `~/.hbc.json` and needs a rebuild to pick the
change up). It does not gate `UPD` (self-update via `hb_install.sh`), which remains
ungated.
--- ---
## UDP Protocol ## UDP Protocol
@@ -254,7 +283,7 @@ Payload format: `key=value;key=value;...`
| `HTB` | client → server | Heartbeat (name, timestamp, RTT, acks, interval) | | `HTB` | client → server | Heartbeat (name, timestamp, RTT, acks, interval) |
| `PLG` | client → server | Plugin data (plugin name + metrics) | | `PLG` | client → server | Plugin data (plugin name + metrics) |
| `ACK` | server → client | Acknowledgment | | `ACK` | server → client | Acknowledgment |
| `CMD` | server → client | Execute a shell command on the client | | `CMD` | server → client | Execute a shell command on the client (requires `allow_remote_command`) |
| `UPD` | server → client | Trigger self-update via `hb_install.sh` | | `UPD` | server → client | Trigger self-update via `hb_install.sh` |
Value encoding: Value encoding:
@@ -400,6 +429,8 @@ hosts:
Notifications are sent on state transitions (OK → WARNING, WARNING → CRITICAL, CRITICAL → OK). De-escalations (CRITICAL → WARNING) do not trigger a notification. Ongoing alerts generate a re-notification every `threshold_renotify_interval` seconds (default: 3600). Alerts can be acknowledged via the web UI or API to suppress re-notifications. Notifications are sent on state transitions (OK → WARNING, WARNING → CRITICAL, CRITICAL → OK). De-escalations (CRITICAL → WARNING) do not trigger a notification. Ongoing alerts generate a re-notification every `threshold_renotify_interval` seconds (default: 3600). Alerts can be acknowledged via the web UI or API to suppress re-notifications.
A service or host that exceeds `flap_count` warning/critical notifications within `flap_interval` minutes is marked **flapping**: the tripping notification carries `Now flapping!! No more messages!` and further notifications are suppressed until it stays OK for `flap_interval` minutes. Flapping hosts are badged on the live dashboard; the event log keeps recording throughout. See [docs/NOTIFICATIONS.md](docs/NOTIFICATIONS.md#flap-detection).
### RTT thresholds ### RTT thresholds
The server measures heartbeat round-trip time and supports RTT thresholds using the same format: The server measures heartbeat round-trip time and supports RTT thresholds using the same format:
+15
View File
@@ -406,6 +406,21 @@ Potential improvements for future versions:
- Journal file encryption - Journal file encryption
- Signed journal entries - Signed journal entries
## Events journal
Alert/connectivity events shown on the `/log` page are written to a second,
dedicated journal (one JSON object per line, same rotation mechanism):
| Key | Default | Meaning |
|---|---|---|
| `events_journal_file` | `events.journal` | Filename inside `journal_dir` |
| `events_journal_max_size` | 10 MB | Rotation threshold |
| `events_journal_max_backups` | 10 | Rotated files kept |
`journal_dir` and `journal_enabled` are shared with the message journal.
On startup, if the events journal file is empty, it is seeded once from the
pickled in-memory message ring so history survives the upgrade.
## See Also ## See Also
- [Configuration Guide](../hbd/config.py) - Full configuration options - [Configuration Guide](../hbd/config.py) - Full configuration options
+39
View File
@@ -9,6 +9,7 @@ Notifications are dispatched to the **owner and managers** of a host, each via t
``` ```
Alert event (udp.py / threshold.py) Alert event (udp.py / threshold.py)
└─ notify.send_notification(host_name, Notification) └─ notify.send_notification(host_name, Notification)
├─ flap.observe(host, service, level) → pass | trip | suppress
├─ look up host.owner + host.managers ├─ look up host.owner + host.managers
├─ for each user → user.notification_channels ├─ for each user → user.notification_channels
└─ for each channel → _dispatch_to_channel (filtered by min_level) └─ for each channel → _dispatch_to_channel (filtered by min_level)
@@ -19,6 +20,7 @@ Every notification carries:
- **body** — detail message (metric value, threshold, duration) - **body** — detail message (metric value, threshold, duration)
- **url** — link to the plugin metrics page (`{base_url}/plugins#{hostname}`) - **url** — link to the plugin metrics page (`{base_url}/plugins#{hostname}`)
- **level** — `RECOVER | WARNING | CRITICAL | INFO` - **level** — `RECOVER | WARNING | CRITICAL | INFO`
- **service** — flap-detection key within the host (empty = the host itself)
## Configuration ## Configuration
@@ -268,6 +270,43 @@ min_level: WARNING
Reminder notifications (re-notify) are sent only for CRITICAL level alerts. Reminder notifications (re-notify) are sent only for CRITICAL level alerts.
## Flap detection
A check that toggles between OK and alerting produces a notification per swing. Flap
detection silences it after the first few.
A **`(host, service)`** pair is flapping once it exceeds `flap_count` WARNING/CRITICAL
notifications within `flap_interval` minutes. Threshold alerts key on their metric path,
so a flapping disk check does not silence an unrelated CPU alert; connectivity, boot and
shutdown events key on the host itself.
```yaml
flap_count: 5 # notifications within the window that trip flapping (0 disables)
flap_interval: 10 # minutes — both the counting window and the quiet window
```
Lifecycle:
| Event | Effect |
|---|---|
| Alerts 1..`flap_count` within the window | Delivered normally |
| Alert `flap_count + 1` | Delivered with ` Now flapping!! No more messages!` appended to the body |
| Every notification after that | Dropped — including RECOVER and INFO |
| RECOVER while flapping | Dropped, and starts the `flap_interval` quiet window |
| WARNING/CRITICAL during the quiet window | Restarts the quiet window; still flapping |
| Quiet window elapses | Flapping ends **silently** — no notification |
Only outbound notifications are suppressed. `notify.eventlog` keeps recording every event,
so the journal and the `/log` page retain the full history of the flap.
Flapping pairs appear in each host's `stateinfo()` under `flapping` (a list of service
keys; `""` means the host itself) and render as an amber **flapping** badge next to the
host name on the live dashboard, with the affected services in its tooltip.
State lives at module level in `hbd/server/flap.py` and is never pickled — a server restart
starts every check with a clean slate. Hosts with `watch: false` never reach the
notification path, so they never flap.
## API reference ## API reference
### `send_notification(host_name, notif) -> dict` ### `send_notification(host_name, notif) -> dict`
+10 -10
View File
@@ -21,7 +21,7 @@ Heartbeat's plugin system is designed to be simple yet powerful. Plugins are Pyt
### Key Concepts ### Key Concepts
- **Plugin Registry**: Central registry that manages all loaded plugins - **Plugin Registry**: Central registry that manages all loaded plugins
- **Plugin Loader**: Automatically discovers and loads plugins from the `hbd/plugins/` directory - **Plugin Loader**: Automatically discovers and loads plugins from the `hbd/client/plugins/` directory
- **Plugin Types**: InfoPlugin (static data) and MonitorPlugin (periodic metrics) - **Plugin Types**: InfoPlugin (static data) and MonitorPlugin (periodic metrics)
- **Async/Await**: All plugin methods are async for non-blocking operation - **Async/Await**: All plugin methods are async for non-blocking operation
@@ -64,7 +64,7 @@ Decide whether your plugin collects static information (InfoPlugin) or dynamic m
### Step 2: Create Plugin File ### Step 2: Create Plugin File
Create a new Python file in `hbd/plugins/` directory: Create a new Python file in `hbd/client/plugins/` directory:
```python ```python
""" """
@@ -82,7 +82,7 @@ try:
except ImportError: except ImportError:
psutil = None psutil = None
from hbd.plugin import MonitorPlugin # or InfoPlugin from hbd.client.plugin import MonitorPlugin # or InfoPlugin
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
@@ -193,7 +193,7 @@ from pathlib import Path
# Add parent directory to path # Add parent directory to path
sys.path.insert(0, str(Path(__file__).parent)) sys.path.insert(0, str(Path(__file__).parent))
from hbd.plugins.my_awesome_plugin import MyAwesomePlugin from hbd.client.plugins.my_awesome_plugin import MyAwesomePlugin
async def test(): async def test():
# Create plugin instance # Create plugin instance
@@ -224,7 +224,7 @@ Understanding the plugin lifecycle helps you implement plugins correctly:
``` ```
1. Plugin Discovery 1. Plugin Discovery
└─> Loader scans hbd/plugins/ directory └─> Loader scans hbd/client/plugins/ directory
└─> Finds Python files (except those starting with _) └─> Finds Python files (except those starting with _)
└─> Imports modules └─> Imports modules
@@ -378,7 +378,7 @@ Document your plugin thoroughly:
### Example 1: Simple InfoPlugin ### Example 1: Simple InfoPlugin
```python ```python
from hbd.plugin import InfoPlugin from hbd.client.plugin import InfoPlugin
import platform import platform
class SimpleInfoPlugin(InfoPlugin): class SimpleInfoPlugin(InfoPlugin):
@@ -406,7 +406,7 @@ plugin = SimpleInfoPlugin
### Example 2: MonitorPlugin with State ### Example 2: MonitorPlugin with State
```python ```python
from hbd.plugin import MonitorPlugin from hbd.client.plugin import MonitorPlugin
import time import time
class CounterPlugin(MonitorPlugin): class CounterPlugin(MonitorPlugin):
@@ -442,7 +442,7 @@ plugin = CounterPlugin
### Example 3: Plugin with External Command ### Example 3: Plugin with External Command
```python ```python
from hbd.plugin import MonitorPlugin from hbd.client.plugin import MonitorPlugin
import asyncio import asyncio
class CommandPlugin(MonitorPlugin): class CommandPlugin(MonitorPlugin):
@@ -561,7 +561,7 @@ python -m hbd.hbc -c test_config.yaml --verbose
## Further Reading ## Further Reading
- [Plugin Framework Source](../hbd/plugin.py) - Core plugin implementation - [Plugin Framework Source](../hbd/client/plugin.py) - Core plugin implementation
- [Built-in Plugins](../hbd/plugins/) - Examples of working plugins - [Built-in Plugins](../hbd/client/plugins/) - Examples of working plugins
- [Nagios Integration](NAGIOS_INTEGRATION.md) - Running external plugins - [Nagios Integration](NAGIOS_INTEGRATION.md) - Running external plugins
- [Configuration Guide](../hbd/config_example.yaml) - Full configuration reference - [Configuration Guide](../hbd/config_example.yaml) - Full configuration reference
+1 -1
View File
@@ -14,4 +14,4 @@ Install options:
""" """
__all__ = ["__version__"] __all__ = ["__version__"]
__version__ = "5.3.12" __version__ = "5.4.4"
+3
View File
@@ -19,6 +19,9 @@ CLIENT_DEFAULTS = {
# Host identity # Host identity
"owner": None, # Optional username to set as this host's owner on the server "owner": None, # Optional username to set as this host's owner on the server
# Security
"allow_remote_command": False, # Execute shell commands received in CMD packets from the server
# Runtime flags # Runtime flags
"foreground": False, "foreground": False,
"verbose": False, "verbose": False,
+204
View File
@@ -0,0 +1,204 @@
"""Per-protocol internet presence probe.
Tier 1 (every tier1_interval seconds, sends no packets): a UDP connect() to the probe
address reveals whether a route exists and which source address would be used.
Tier 2 (every probe_interval seconds, sooner after a failure or a change of address or
gateway): one ICMP echo to the probe address. Absence needs fail_limit consecutive
failures (or no route at all); presence needs one success.
The result is read by the heartbeat sender (p4/p6 flags on every HTB) and by the
network_monitor plugin (connectivity metric) via the shared instance.
"""
import asyncio
import logging
import socket
import sys
import time
from dataclasses import asdict, dataclass, replace
from typing import Any, Awaitable, Callable, Dict, Optional, Set, Tuple
from hbd.client import gateway as gateway_mod
from hbd.client.pingutil import PingUnavailable, run_ping
logger = logging.getLogger("hbc.connectivity")
FAMILIES = ("ipv4", "ipv6")
FLAG_KEYS = {"ipv4": "p4", "ipv6": "p6"}
DEFAULT_PROBES = {"ipv4": "1.1.1.1", "ipv6": "2606:4700:4700::1111"}
class ProbeUnavailable(Exception):
"""The address family cannot be probed on this host (e.g. no IPv6 support)."""
@dataclass
class FamilyState:
present: Optional[bool] = None # None = unknown (probe cannot run yet/at all)
addr: Optional[str] = None
gateway: Optional[str] = None
rtt: Optional[float] = None
def local_address(family: str, target: str) -> Optional[str]:
"""Source address the kernel would use to reach target, or None if there is no route.
A UDP connect() sends nothing. Raises ProbeUnavailable if the family is unsupported.
"""
af = socket.AF_INET if family == "ipv4" else socket.AF_INET6
try:
sock = socket.socket(af, socket.SOCK_DGRAM)
except OSError as e:
raise ProbeUnavailable(str(e)) from e
try:
sock.connect((target, 9))
return str(sock.getsockname()[0])
except OSError:
return None
finally:
sock.close()
async def ping_rtt(target: str, family: str) -> Optional[float]:
"""One ICMP echo; returns RTT in ms or None if no reply."""
result = await run_ping(target, 1, 2, 4 if family == "ipv4" else 6)
rtt = result["rtt_avg"]
return None if rtt == float("inf") else rtt
class Connectivity:
def __init__(
self,
probes: Optional[Dict[str, str]] = None,
probe_interval: float = 60.0,
retry_interval: float = 10.0,
tier1_interval: float = 5.0,
fail_limit: int = 2,
local_address_fn: Callable[[str, str], Optional[str]] = local_address,
ping_fn: Callable[[str, str], Awaitable[Optional[float]]] = ping_rtt,
gateway_fn: Callable[[str], Awaitable[Optional[str]]] = gateway_mod.default_gateway,
clock: Callable[[], float] = time.monotonic,
):
self.probes = {**DEFAULT_PROBES, **(probes or {})}
self.probe_interval = probe_interval
self.retry_interval = retry_interval
self.tier1_interval = tier1_interval
self.fail_limit = fail_limit
self._local_address = local_address_fn
self._ping = ping_fn
self._gateway_fn = gateway_fn
self._clock = clock
# Reading the gateway is a cheap /proc read on Linux but a subprocess elsewhere,
# so elsewhere it is refreshed only when a tier-2 probe is due.
self._cheap_gateway = sys.platform.startswith("linux")
self.states: Dict[str, FamilyState] = {f: FamilyState() for f in FAMILIES}
self._fails: Dict[str, int] = {f: 0 for f in FAMILIES}
self._next_ping: Dict[str, float] = {f: 0.0 for f in FAMILIES}
self._last_key: Dict[str, Optional[Tuple[Optional[str], Optional[str]]]] = {
f: None for f in FAMILIES}
self._ping_disabled: Set[str] = set()
@classmethod
def from_config(cls, cfg: Optional[Dict[str, Any]]) -> "Connectivity":
cfg = cfg or {}
probes = {}
if cfg.get("probe_v4"):
probes["ipv4"] = str(cfg["probe_v4"])
if cfg.get("probe_v6"):
probes["ipv6"] = str(cfg["probe_v6"])
return cls(probes=probes, probe_interval=float(cfg.get("probe_interval", 60)))
# ---- readers (cheap; safe to call on every heartbeat) --------------------------
def snapshot(self) -> Dict[str, FamilyState]:
return {f: replace(s) for f, s in self.states.items()}
def flags(self) -> Dict[str, bool]:
"""HTB flags for families whose presence is known."""
return {FLAG_KEYS[f]: s.present for f, s in self.states.items() if s.present is not None}
def as_metric(self) -> Dict[str, Dict[str, Any]]:
return {f: asdict(s) for f, s in self.states.items()}
# ---- probing -------------------------------------------------------------------
async def refresh(self) -> None:
for fam in FAMILIES:
try:
await self._refresh_family(fam)
except Exception:
logger.exception("connectivity probe failed for %s", fam)
async def run(self, stop: asyncio.Event) -> None:
while not stop.is_set():
await self.refresh()
try:
await asyncio.wait_for(stop.wait(), timeout=self.tier1_interval)
except asyncio.TimeoutError:
pass
async def _gateway(self, fam: str) -> Optional[str]:
try:
return await self._gateway_fn(fam)
except Exception:
return None
async def _refresh_family(self, fam: str) -> None:
st = self.states[fam]
try:
addr = self._local_address(fam, self.probes[fam])
except ProbeUnavailable:
st.present = None
return
if addr is None: # tier 1: no route => absent immediately
self._fails[fam] = 0
self._next_ping[fam] = 0.0
self._last_key[fam] = None
st.present, st.addr, st.gateway, st.rtt = False, None, None, None
return
now = self._clock()
gw = await self._gateway(fam) if self._cheap_gateway else st.gateway
key = (addr, gw)
due = key != self._last_key[fam] or now >= self._next_ping[fam]
if due and not self._cheap_gateway:
gw = await self._gateway(fam)
key = (addr, gw)
st.addr, st.gateway = addr, gw
self._last_key[fam] = key
if fam in self._ping_disabled:
st.present = None # cannot probe: unknown, never absent, while a route exists
elif due:
await self._tier2(fam, st, now)
async def _tier2(self, fam: str, st: FamilyState, now: float) -> None:
try:
rtt = await self._ping(self.probes[fam], fam)
except PingUnavailable:
logger.warning("ping unavailable; %s presence stays unknown", fam)
self._ping_disabled.add(fam)
st.present = None
return
except (asyncio.TimeoutError, OSError):
rtt = None
if rtt is None:
self._fails[fam] += 1
self._next_ping[fam] = now + self.retry_interval
if self._fails[fam] >= self.fail_limit:
st.present, st.rtt = False, None
else:
self._fails[fam] = 0
self._next_ping[fam] = now + self.probe_interval
st.present, st.rtt = True, rtt
# ---- shared instance (heartbeat sender and network_monitor plugin) -----------------
_shared: Optional[Connectivity] = None
def set_shared(conn: Optional[Connectivity]) -> None:
global _shared
_shared = conn
def get_shared() -> Optional[Connectivity]:
return _shared
+103
View File
@@ -0,0 +1,103 @@
"""Best-effort default-gateway lookup (informational only; never affects presence)."""
import asyncio
import re
import socket
import struct
import sys
from typing import List, Optional, Tuple
_RTF_GATEWAY = 0x2
_RTF_REJECT = 0x200
def parse_proc_route_v4(text: str) -> Optional[str]:
"""Default gateway from /proc/net/route (lowest metric wins)."""
best: Optional[Tuple[int, str]] = None
for line in text.splitlines()[1:]:
f = line.split()
if len(f) < 8:
continue
_iface, dest, gw, flags = f[0], f[1], f[2], f[3] # noqa: F841
if dest != "00000000" or not int(flags, 16) & _RTF_GATEWAY:
continue
metric = int(f[6])
if best is None or metric < best[0]:
best = (metric, socket.inet_ntoa(struct.pack("<L", int(gw, 16))))
return best[1] if best else None
def parse_proc_route_v6(text: str) -> Optional[str]:
"""Default gateway from /proc/net/ipv6_route (lowest metric wins)."""
best: Optional[Tuple[int, str]] = None
for line in text.splitlines():
f = line.split()
if len(f) < 10:
continue
dest, plen, nexthop, metric, flags, iface = f[0], f[1], f[4], f[5], f[8], f[9]
if dest != "0" * 32 or plen != "00" or nexthop == "0" * 32 or iface == "lo":
continue
fl = int(flags, 16)
if not fl & _RTF_GATEWAY or fl & _RTF_REJECT:
continue
m = int(metric, 16)
if best is None or m < best[0]:
best = (m, socket.inet_ntop(socket.AF_INET6, bytes.fromhex(nexthop)))
return best[1] if best else None
def parse_macos_route(text: str) -> Optional[str]:
"""Gateway from `route -n get [-inet6] default`."""
m = re.search(r"^\s*gateway:\s*(\S+)", text, re.MULTILINE)
return m.group(1) if m else None
def parse_windows_route(text: str, family: str) -> Optional[str]:
"""Default gateway from `route print -4` / `route print -6` (lowest metric wins)."""
best: Optional[Tuple[int, str]] = None
for line in text.splitlines():
if family == "ipv4":
f = line.split()
if len(f) >= 5 and f[0] == "0.0.0.0" and f[1] == "0.0.0.0" and f[2] != "On-link":
cand = (int(f[4]), f[2])
else:
continue
else:
m = re.match(r"^\s*\d+\s+(\d+)\s+::/0\s+(\S+)\s*$", line)
if not m or m.group(2) == "On-link":
continue
cand = (int(m.group(1)), m.group(2))
if best is None or cand[0] < best[0]:
best = cand
return best[1] if best else None
async def _run(cmd: List[str]) -> str:
proc = await asyncio.create_subprocess_exec(
*cmd, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.DEVNULL,
)
try:
out, _ = await asyncio.wait_for(proc.communicate(), timeout=5)
except asyncio.TimeoutError:
proc.kill()
await proc.wait()
raise
return out.decode(errors="replace")
async def default_gateway(family: str) -> Optional[str]:
"""Return the default gateway address for "ipv4"/"ipv6", or None on any failure."""
v4 = family == "ipv4"
try:
if sys.platform.startswith("linux"):
with open("/proc/net/route" if v4 else "/proc/net/ipv6_route") as fh:
text = fh.read()
return parse_proc_route_v4(text) if v4 else parse_proc_route_v6(text)
if sys.platform == "darwin":
cmd = ["route", "-n", "get"] + ([] if v4 else ["-inet6"]) + ["default"]
return parse_macos_route(await _run(cmd))
if sys.platform == "win32":
return parse_windows_route(await _run(["route", "print", "-4" if v4 else "-6"]), family)
except Exception:
return None
return None
+45 -13
View File
@@ -25,6 +25,7 @@ from .. import __version__
# Import plugin system # Import plugin system
from .plugin import PluginRegistry, PluginLoader, InfoPlugin, MonitorPlugin from .plugin import PluginRegistry, PluginLoader, InfoPlugin, MonitorPlugin
from .connectivity import Connectivity, set_shared
# Constants # Constants
PORT = 50003 PORT = 50003
@@ -37,6 +38,10 @@ dorestart = False
shutdown_event: Optional[asyncio.Event] = None shutdown_event: Optional[asyncio.Event] = None
active_tasks: List[asyncio.Task] = [] active_tasks: List[asyncio.Task] = []
# Set from config in async_main. Off by default: a CMD packet is an unauthenticated
# UDP datagram, so executing one must be opted into per host.
allow_remote_command = False
class AsyncConnection: class AsyncConnection:
"""Async UDP connection to a heartbeat server.""" """Async UDP connection to a heartbeat server."""
@@ -183,16 +188,25 @@ class HeartbeatProtocol(asyncio.DatagramProtocol):
async def handle_command(conn: AsyncConnection, msg: dict): async def handle_command(conn: AsyncConnection, msg: dict):
"""Execute a command received from server.""" """Execute a command received from server, if allow_remote_command is set."""
import subprocess import subprocess
cmd = msg.get("cmd", "") cmd = msg.get("cmd", "")
if not cmd: if not cmd:
return return
logger = logging.getLogger("hbc.command") logger = logging.getLogger("hbc.command")
if not allow_remote_command:
logger.warning(f"Refused command (allow_remote_command is false): {cmd}")
await conn.sendto({
"service": "command",
"msg": "Refused: allow_remote_command is false",
})
return
logger.info(f"Executing command: {cmd}") logger.info(f"Executing command: {cmd}")
try: try:
result = subprocess.check_output( result = subprocess.check_output(
cmd, shell=True, stderr=subprocess.STDOUT, timeout=30 cmd, shell=True, stderr=subprocess.STDOUT, timeout=30
@@ -268,7 +282,16 @@ async def handle_update(conn: AsyncConnection, _msg: dict): # pyright: ignore[r
stop() stop()
async def heartbeat_sender(conn: AsyncConnection, interval: int): def build_heartbeat_msg(conn, interval: int, connectivity: Optional[Connectivity] = None) -> dict:
"""HTB payload; carries p4/p6 presence flags when the probe knows them."""
msg = {"acks": conn.ackcount, "rtt": conn.rtts[-1], "interval": interval}
if connectivity is not None:
msg.update(connectivity.flags())
return msg
async def heartbeat_sender(conn: AsyncConnection, interval: int,
connectivity: Optional[Connectivity] = None):
"""Send periodic heartbeats, retrying the connection if it is not open. """Send periodic heartbeats, retrying the connection if it is not open.
IPv6 connections that fail to open before their first successful send are IPv6 connections that fail to open before their first successful send are
@@ -278,6 +301,7 @@ async def heartbeat_sender(conn: AsyncConnection, interval: int):
Args: Args:
conn: Connection to send on conn: Connection to send on
interval: Heartbeat interval in seconds interval: Heartbeat interval in seconds
connectivity: Optional probe whose presence flags are attached to every heartbeat
""" """
logger = logging.getLogger("hbc.heartbeat") logger = logging.getLogger("hbc.heartbeat")
IPV6_EARLY_FAIL_LIMIT = 3 IPV6_EARLY_FAIL_LIMIT = 3
@@ -315,11 +339,7 @@ async def heartbeat_sender(conn: AsyncConnection, interval: int):
continue continue
try: try:
msg = { msg = build_heartbeat_msg(conn, interval, connectivity)
"acks": conn.ackcount,
"rtt": conn.rtts[-1],
"interval": interval
}
await conn.sendto(msg, "HTB") await conn.sendto(msg, "HTB")
except asyncio.CancelledError: except asyncio.CancelledError:
@@ -501,11 +521,12 @@ async def cleanup(connections: List[AsyncConnection]):
async def async_main(args, config): async def async_main(args, config):
"""Async main function.""" """Async main function."""
global running, shutdown_event, active_tasks, send_shutdown global running, shutdown_event, active_tasks, send_shutdown, allow_remote_command
# Create shutdown event # Create shutdown event
shutdown_event = asyncio.Event() shutdown_event = asyncio.Event()
active_tasks = [] active_tasks = []
allow_remote_command = bool(config.get("allow_remote_command", False))
logger = logging.getLogger("hbc.main") logger = logging.getLogger("hbc.main")
@@ -519,6 +540,8 @@ async def async_main(args, config):
interval = config.get("interval", INTERVAL) interval = config.get("interval", INTERVAL)
logger.info(f"hbc {__version__} on {iam} -> {hb_hosts} port={hb_port}, interval={interval}s") logger.info(f"hbc {__version__} on {iam} -> {hb_hosts} port={hb_port}, interval={interval}s")
if allow_remote_command:
logger.warning("allow_remote_command is true — CMD packets from the server will be executed")
af_filter = (socket.AF_INET if getattr(args, "ipv4_only", False) af_filter = (socket.AF_INET if getattr(args, "ipv4_only", False)
else socket.AF_INET6 if getattr(args, "ipv6_only", False) else socket.AF_INET6 if getattr(args, "ipv6_only", False)
@@ -603,9 +626,18 @@ async def async_main(args, config):
loop.add_signal_handler(signal.SIGHUP, _sighup) loop.add_signal_handler(signal.SIGHUP, _sighup)
# Start async tasks # Start async tasks
# Per-protocol internet presence probe (shared with the network_monitor plugin)
net_cfg = dict((config.get("plugins") or {}).get("network_monitor")
or config.get("network_monitor") or {})
connectivity = None
if net_cfg.get("probe_enabled", True):
connectivity = Connectivity.from_config(net_cfg)
set_shared(connectivity)
active_tasks.append(asyncio.create_task(connectivity.run(shutdown_event)))
# Heartbeat senders (one per connection) # Heartbeat senders (one per connection)
for conn in connections: for conn in connections:
task = asyncio.create_task(heartbeat_sender(conn, interval)) task = asyncio.create_task(heartbeat_sender(conn, interval, connectivity))
active_tasks.append(task) active_tasks.append(task)
# Plugin collector (uses all connections, but we'll use first one) # Plugin collector (uses all connections, but we'll use first one)
+73
View File
@@ -0,0 +1,73 @@
"""System ping helpers shared by ping_monitor and the connectivity probe."""
import asyncio
import re
import sys
from typing import Dict, List, Optional
_INF = float("inf")
class PingUnavailable(Exception):
"""The system ping binary could not be executed."""
def ping_command(host: str, count: int, timeout: int, family: Optional[int] = None) -> List[str]:
"""Build the platform ping command. family is 4, 6 or None (let ping decide)."""
if sys.platform == "win32":
cmd = ["ping", "-n", str(count), "-w", str(timeout * 1000)]
if family:
cmd.append(f"-{family}")
return cmd + [host]
if sys.platform == "darwin" and family:
if family == 6:
return ["ping6", "-c", str(count), host]
return ["ping", "-c", str(count), "-W", str(timeout * 1000), host]
cmd = ["ping", "-c", str(count), "-W", str(timeout)]
if family:
cmd.append(f"-{family}")
return cmd + [host]
def parse_ping_output(output: str) -> Dict[str, float]:
"""Parse rtt_min/avg/max/loss from ping output (Linux, macOS, BSD)."""
loss = 100.0
loss_match = re.search(r"(\d+(?:\.\d+)?)\s*%\s*packet\s*loss", output)
if loss_match:
loss = float(loss_match.group(1))
# Linux: "rtt min/avg/max/mdev = x/x/x/x ms"; macOS: "round-trip min/avg/max/stddev = ..."
rtt_match = re.search(
r"(?:rtt|round-trip)\s+min/avg/max/\S+\s*=\s*([\d.]+)/([\d.]+)/([\d.]+)",
output,
)
if rtt_match:
return {
"rtt_min": float(rtt_match.group(1)),
"rtt_avg": float(rtt_match.group(2)),
"rtt_max": float(rtt_match.group(3)),
"loss": loss,
}
return {"rtt_min": _INF, "rtt_avg": _INF, "rtt_max": _INF, "loss": loss}
async def run_ping(host: str, count: int, timeout: int, family: Optional[int] = None) -> Dict[str, float]:
"""Run ping and return parsed metrics.
Raises PingUnavailable if the ping binary is missing and asyncio.TimeoutError
(after killing the process) if it hangs.
"""
cmd = ping_command(host, count, timeout, family)
try:
proc = await asyncio.create_subprocess_exec(
*cmd, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE,
)
except FileNotFoundError as e:
raise PingUnavailable(str(e)) from e
try:
stdout, _ = await asyncio.wait_for(proc.communicate(), timeout=timeout * count + 2)
except asyncio.TimeoutError:
proc.kill()
await proc.wait()
raise
return parse_ping_output(stdout.decode(errors="replace"))
+11
View File
@@ -13,6 +13,7 @@ except ImportError:
psutil = None psutil = None
from hbd.client.plugin import MonitorPlugin from hbd.client.plugin import MonitorPlugin
from hbd.client.connectivity import get_shared
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
@@ -32,6 +33,10 @@ class NetworkMonitorPlugin(MonitorPlugin):
interfaces: List of interfaces to monitor (default: all) interfaces: List of interfaces to monitor (default: all)
include_connections: Include connection statistics (default: True) include_connections: Include connection statistics (default: True)
include_addresses: Include interface addresses (default: False) include_addresses: Include interface addresses (default: False)
probe_enabled: Run the per-protocol internet presence probe (default: True)
probe_v4: IPv4 address to ping (default: 1.1.1.1)
probe_v6: IPv6 address to ping (default: 2606:4700:4700::1111)
probe_interval: Seconds between ICMP probes while healthy (default: 60)
""" """
name = "network_monitor" name = "network_monitor"
@@ -103,6 +108,7 @@ class NetworkMonitorPlugin(MonitorPlugin):
- etc. - etc.
- addresses: Interface address information (if include_addresses) - addresses: Interface address information (if include_addresses)
- Dict keyed by interface name with address details - Dict keyed by interface name with address details
- connectivity: per-protocol presence {ipv4, ipv6} -> {present, addr, gateway, rtt}
""" """
if psutil is None: if psutil is None:
logger.error("psutil not available") logger.error("psutil not available")
@@ -228,6 +234,11 @@ class NetworkMonitorPlugin(MonitorPlugin):
except Exception as e: except Exception as e:
logger.warning(f"Could not collect interface stats: {e}") logger.warning(f"Could not collect interface stats: {e}")
# Per-protocol internet presence, address and gateway (from the shared probe)
shared = get_shared()
if shared is not None:
metrics['connectivity'] = shared.as_metric()
return metrics return metrics
+3 -41
View File
@@ -40,9 +40,9 @@ threshold_configs:
import asyncio import asyncio
import re import re
import sys
from typing import Any, Dict, Optional from typing import Any, Dict, Optional
from hbd.client.pingutil import PingUnavailable, run_ping
from hbd.client.plugin import MonitorPlugin from hbd.client.plugin import MonitorPlugin
@@ -84,51 +84,13 @@ class PingMonitorPlugin(MonitorPlugin):
async def _ping(self, host: str) -> Dict[str, float]: async def _ping(self, host: str) -> Dict[str, float]:
"""Run a system ping command and return rtt_min/avg/max/loss.""" """Run a system ping command and return rtt_min/avg/max/loss."""
if sys.platform == "win32":
cmd = ["ping", "-n", str(self.count), "-w", str(self.timeout * 1000), host]
else:
cmd = ["ping", "-c", str(self.count), "-W", str(self.timeout), host]
try: try:
proc = await asyncio.create_subprocess_exec( return await run_ping(host, self.count, self.timeout)
*cmd, except (asyncio.TimeoutError, PingUnavailable, OSError) as e:
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.PIPE,
)
stdout, _ = await asyncio.wait_for(
proc.communicate(),
timeout=self.timeout * self.count + 2,
)
output = stdout.decode(errors="replace")
except (asyncio.TimeoutError, FileNotFoundError, OSError) as e:
self.logger.warning("ping_monitor: ping failed for %s: %s", host, e) self.logger.warning("ping_monitor: ping failed for %s: %s", host, e)
return {"rtt_min": float("inf"), "rtt_avg": float("inf"), return {"rtt_min": float("inf"), "rtt_avg": float("inf"),
"rtt_max": float("inf"), "loss": 100.0} "rtt_max": float("inf"), "loss": 100.0}
# Parse packet loss
loss = 100.0
loss_match = re.search(r"(\d+(?:\.\d+)?)\s*%\s*packet\s*loss", output)
if loss_match:
loss = float(loss_match.group(1))
# Parse rtt min/avg/max — Linux: "rtt min/avg/max/mdev = x/x/x/x ms"
# macOS: "round-trip min/avg/max/stddev = x/x/x/x ms"
rtt_match = re.search(
r"(?:rtt|round-trip)\s+min/avg/max/\S+\s*=\s*([\d.]+)/([\d.]+)/([\d.]+)",
output,
)
if rtt_match:
return {
"rtt_min": float(rtt_match.group(1)),
"rtt_avg": float(rtt_match.group(2)),
"rtt_max": float(rtt_match.group(3)),
"loss": loss,
}
# Host unreachable or all packets lost
return {"rtt_min": float("inf"), "rtt_avg": float("inf"),
"rtt_max": float("inf"), "loss": loss}
async def _collect_metrics(self) -> Dict[str, Any]: async def _collect_metrics(self) -> Dict[str, Any]:
data: Dict[str, Any] = {} data: Dict[str, Any] = {}
tasks = {host: asyncio.create_task(self._ping(host)) for host in self.hosts} tasks = {host: asyncio.create_task(self._ping(host)) for host in self.hosts}
+4
View File
@@ -30,6 +30,10 @@ SERVER_DEFAULTS = {
"grace": 2, # Grace period (extra seconds before notifying after a missed heartbeat) "grace": 2, # Grace period (extra seconds before notifying after a missed heartbeat)
"threshold_renotify_interval": 3600, # Seconds between threshold re-notifications "threshold_renotify_interval": 3600, # Seconds between threshold re-notifications
# Flap detection (0 in either key disables it)
"flap_count": 5, # Warning/critical notifications within flap_interval that trip flapping
"flap_interval": 10, # Minutes: the counting window, and the quiet window after an OK
# User management # User management
"users": {}, # username -> {full_name, avatar, password, admin, notification_channels} "users": {}, # username -> {full_name, avatar, password, admin, notification_channels}
"default_owner": None, # Username that owns hosts with no explicit owner "default_owner": None, # Username that owns hosts with no explicit owner
+1
View File
@@ -19,6 +19,7 @@ def _make_yaml() -> YAML:
_SERVER_KEYS = [ _SERVER_KEYS = [
"hbd_port", "hbd_host", "ws_port", "wss_port", "hb_port", "hbd_port", "hbd_host", "ws_port", "wss_port", "hb_port",
"interval", "grace", "base_url", "threshold_renotify_interval", "interval", "grace", "base_url", "threshold_renotify_interval",
"flap_count", "flap_interval",
"logfile", "pidfile", "pickfile", "journal_enabled", "journal_dir", "logfile", "pidfile", "pickfile", "journal_enabled", "journal_dir",
"journal_max_size", "journal_max_backups", "default_owner", "journal_max_size", "journal_max_backups", "default_owner",
"default_threshold_config", "default_threshold_config",
+130
View File
@@ -0,0 +1,130 @@
"""Flap detection: silence checks that toggle faster than they are useful.
A ``(host, service)`` pair is *flapping* once it exceeds ``flap_count``
WARNING/CRITICAL notifications within ``flap_interval`` minutes. The
notification that trips the threshold is delivered with ``FLAP_MARKER``
appended; every notification for that pair afterwards is dropped. The state
clears silently ``flap_interval`` minutes after a RECOVER, provided no further
WARNING/CRITICAL arrived in the meantime.
Only outbound notifications are affected — ``notify.eventlog`` keeps recording
every event, so the journal and ``/log`` retain the full history of the flap.
State lives here at module level rather than on ``Host`` so it is never
pickled: a restart starts every check with a clean slate.
"""
import logging
import time
logger = logging.getLogger(__name__)
FLAP_MARKER = "Now flapping!! No more messages!"
# Actions returned by observe()
PASS = "pass" # deliver unchanged
TRIP = "trip" # deliver with FLAP_MARKER appended
SUPPRESS = "suppress" # drop
_count = 0 # flap_count: alerts within the window needed to trip
_window = 0.0 # flap_interval in seconds
# {(host, service): {"events": [ts, ...], "flapping": bool, "ok_since": float|None}}
_state: dict = {}
def setup(cfg) -> None:
"""Read flap_count / flap_interval from *cfg* (also called on reload)."""
global _count, _window
_count = int(cfg.get("flap_count", 0) or 0)
_window = float(cfg.get("flap_interval", 0) or 0) * 60.0
def _enabled() -> bool:
return _count > 0 and _window > 0
def _label(key) -> str:
host, service = key
return f"{host}/{service}" if service else host
def _sweep(st: dict, now: float) -> bool:
"""Clear the flapping flag once the post-RECOVER quiet window has elapsed."""
if st["flapping"] and st["ok_since"] is not None and now - st["ok_since"] >= _window:
st["flapping"] = False
st["ok_since"] = None
st["events"].clear()
return True
return False
def observe(host: str, service: str, level: str) -> str:
"""Record a notification for *host*/*service* and return what to do with it.
Returns PASS, TRIP or SUPPRESS.
"""
if not _enabled():
return PASS
now = time.time()
key = (host or "", service or "")
level = (level or "").upper()
st = _state.get(key)
if st is not None and _sweep(st, now):
logger.info("flapping cleared for %s", _label(key))
if level in ("WARNING", "CRITICAL"):
if st is None:
st = _state[key] = {"events": [], "flapping": False, "ok_since": None}
# An alert inside the quiet window means it never settled.
st["ok_since"] = None
st["events"] = [t for t in st["events"] if now - t < _window]
st["events"].append(now)
if st["flapping"]:
return SUPPRESS
if len(st["events"]) > _count:
st["flapping"] = True
logger.info(
"flapping detected for %s (%d alerts in %.0f min)",
_label(key), len(st["events"]), _window / 60,
)
return TRIP
return PASS
if st is None or not st["flapping"]:
return PASS
if level == "RECOVER":
# Starts the quiet window; the state ends silently when it elapses.
st["ok_since"] = now
return SUPPRESS
def clear_host(host: str) -> None:
"""Discard all flap state for *host* (called when a host is dropped).
A dropped host may be mid-flap with no RECOVER ever received, in which
case ``ok_since`` stays ``None`` and ``_sweep`` can never clear it on its
own — the state would otherwise persist forever.
"""
for key in [k for k in _state if k[0] == host]:
del _state[key]
def flapping_services(host: str) -> list:
"""Return the services of *host* that are currently flapping.
An empty string in the result means the host itself (connectivity, boot,
shutdown) rather than a named service.
"""
if not _enabled():
return []
now = time.time()
flapping = []
for (h, service), st in _state.items():
if h != host:
continue
_sweep(st, now)
if st["flapping"]:
flapping.append(service)
return sorted(flapping)
+55 -1
View File
@@ -29,6 +29,7 @@ class Connection:
UP = "up" UP = "up"
DOWN = "down" DOWN = "down"
OVERDUE = "overdue" OVERDUE = "overdue"
ABSENT = "absent"
def __init__(self, host, cid, addr, afam): def __init__(self, host, cid, addr, afam):
self.host = host self.host = host
@@ -94,7 +95,9 @@ class Connection:
now = time.time() now = time.time()
if not Null: if not Null:
d["addr"] = self.addr d["addr"] = self.addr
if self.rtts[-1]: if self.state == Connection.ABSENT:
d["rtt"] = ""
elif self.rtts[-1]:
d["rtt"] = "%d" % round(self.rtts[-1]) d["rtt"] = "%d" % round(self.rtts[-1])
elif self.state == Connection.UNKNOWN: elif self.state == Connection.UNKNOWN:
d["rtt"] = "" d["rtt"] = ""
@@ -299,6 +302,8 @@ class Host:
self.alert_states = {} self.alert_states = {}
# Stale-data timers: {plugin_name: asyncio.TimerHandle} # Stale-data timers: {plugin_name: asyncio.TimerHandle}
self.plugin_timers = {} self.plugin_timers = {}
# Client-reported protocol presence: {afam: {"present": bool, "ts": float}}
self.presence: dict = {}
# User access control # User access control
self.owner: str | None = None # username of owner self.owner: str | None = None # username of owner
self.managers: list = [] # usernames with manager role self.managers: list = [] # usernames with manager role
@@ -322,6 +327,7 @@ class Host:
d["name"] = "<b>%s</b>" % d["name"] d["name"] = "<b>%s</b>" % d["name"]
d["dyn"] = str(self.dyn) d["dyn"] = str(self.dyn)
d["num"] = self.num d["num"] = self.num
d["status"] = self.status()
# Add alert counts (split by acknowledged status) # Add alert counts (split by acknowledged status)
warning_unacked = 0 warning_unacked = 0
@@ -428,6 +434,10 @@ class Host:
ddict["alert_critical_unacked"] = critical_unacked ddict["alert_critical_unacked"] = critical_unacked
ddict["alert_critical_acked"] = critical_acked ddict["alert_critical_acked"] = critical_acked
# Flap detection state (module-level in flap.py, never pickled)
from . import flap
ddict["flapping"] = flap.flapping_services(self.name)
# User access # User access
ddict["owner"] = getattr(self, "owner", None) ddict["owner"] = getattr(self, "owner", None)
ddict["managers"] = list(getattr(self, "managers", [])) ddict["managers"] = list(getattr(self, "managers", []))
@@ -440,6 +450,7 @@ class Host:
_, os_data = latest_os _, os_data = latest_os
hbc_version = os_data.get("hbc_version") hbc_version = os_data.get("hbc_version")
ddict["hbc_version"] = hbc_version ddict["hbc_version"] = hbc_version
ddict["status"] = self.status()
return ddict return ddict
@@ -455,6 +466,47 @@ class Host:
else: else:
return addr.find(".") > 0 return addr.find(".") > 0
def set_presence(self, afam, present, now):
"""Record the client's report of whether it has internet connectivity on afam."""
self.presence[afam] = {"present": bool(present), "ts": now}
def flag_present(self, afam):
"""Client-reported presence; True when the client never reported (legacy clients)."""
entry = getattr(self, "presence", {}).get(afam)
return True if entry is None else entry["present"]
def family_present(self, afam):
"""True if afam should be monitored: the client says it is present, or heartbeats
are demonstrably arriving on it (real traffic overrides a false "absent")."""
conn = self.connections.get(afam)
return self.flag_present(afam) or (conn is not None and conn.state == Connection.UP)
def status(self):
"""Host-level status: down, overdue, simplex, up or unknown.
overdue: every present family is overdue. simplex: both present, one up, one overdue
(same severity as up). Families the client reports absent are not considered,
unless none is present: then every connection not parked as ABSENT is considered.
"""
conns = self.connections
if any(c.state == Connection.DOWN for c in conns.values()):
return "down"
present = [c for afam, c in conns.items() if self.family_present(afam)]
if not present:
# The client reports no protocol present (e.g. its internet probe fails on every
# family) yet may still be heard: monitor whatever is not parked as absent, so a
# host that then dies is "overdue", not silently "unknown".
present = [c for c in conns.values() if c.state != Connection.ABSENT]
up = sum(c.state == Connection.UP for c in present)
overdue = sum(c.state == Connection.OVERDUE for c in present)
if up and overdue:
return "simplex"
if up:
return "up"
if overdue:
return "overdue"
return "unknown"
def conndata(self, cid, addr, rtt, now): def conndata(self, cid, addr, rtt, now):
if addr[0:7] == "::ffff:": if addr[0:7] == "::ffff:":
addr = addr[7:] addr = addr[7:]
@@ -495,6 +547,8 @@ class Host:
self.monitors = [] self.monitors = []
if not hasattr(self, "plugin_timers"): if not hasattr(self, "plugin_timers"):
self.plugin_timers = {} self.plugin_timers = {}
if not hasattr(self, "presence"):
self.presence = {}
pass pass
+94 -2
View File
@@ -14,12 +14,14 @@ import logging
from aiohttp import web from aiohttp import web
import jinja2 import jinja2
from . import data from . import data
from . import flap as flap_mod
from . import notify as notify_mod from . import notify as notify_mod
from . import settings as settings_mod from . import settings as settings_mod
from . import users as users_mod from . import users as users_mod
from . import oauth as oauth_mod from . import oauth as oauth_mod
from . import ws as ws_mod from . import ws as ws_mod
from . import configio as configio_mod from . import configio as configio_mod
from . import journal as journal_mod
from . import config_access from . import config_access
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
@@ -208,6 +210,16 @@ def _mask_config_for_api(config) -> dict:
return result return result
def _visible_plugin_names(plugin_data: dict) -> list:
"""Plugin names to render as accordion sections.
Excludes synthetic rtt_ipv4 / rtt_ipv6 history streams (written by
udp.py for charting) which aren't real client plugins and have no
accordion renderer.
"""
return [p for p in plugin_data.keys() if not p.startswith("rtt_")]
def _build_host_info(host, threshold_checker=None) -> dict: def _build_host_info(host, threshold_checker=None) -> dict:
"""Assemble the info payload for GET /api/0/hosts/{hostname}/info.""" """Assemble the info payload for GET /api/0/hosts/{hostname}/info."""
hbc_version = None hbc_version = None
@@ -258,12 +270,25 @@ def _build_host_info(host, threshold_checker=None) -> dict:
key=lambda x: x["metric"], key=lambda x: x["metric"],
) )
connections = [
{
"family": getattr(conn, "afam", family),
"addr": getattr(conn, "addr", ""),
"state": getattr(conn, "state", ""),
"rtt": (getattr(conn, "rtts", None) or [None])[-1],
"statetime": getattr(conn, "statetime", None),
"lastbeat": getattr(conn, "lastbeat", None),
}
for family, conn in sorted(host.connections.items())
]
return { return {
"owner": getattr(host, "owner", None), "owner": getattr(host, "owner", None),
"managers": list(getattr(host, "managers", [])), "managers": list(getattr(host, "managers", [])),
"hbc_version": hbc_version, "hbc_version": hbc_version,
"hbc_type": hbc_type, "hbc_type": hbc_type,
"last_packet": last_packet, "last_packet": last_packet,
"connections": connections,
"thresholds": thresholds, "thresholds": thresholds,
} }
@@ -349,6 +374,50 @@ async def start(
lst = data.msgs[-30:] lst = data.msgs[-30:]
return web.json_response(lst) return web.json_response(lst)
async def api_log(request):
"""Paged, filtered read of the events journal (newest first)."""
user, err = _require_auth(request)
if err:
return err
qa = request.rel_url.query
try:
limit = max(1, min(int(qa.get("limit", "100")), 1000))
except ValueError:
limit = 100
before = None
if qa.get("before"):
try:
before = float(qa["before"])
except ValueError:
return web.json_response({"error": "invalid before"}, status=400)
host_f = qa.get("host") or None
level_f = qa.get("level") or None
q_f = qa.get("q") or None
def visible(ev):
h = ev.get("host")
return not h or ws_mod._user_can_see_host(user, h)
ej = journal_mod.get_events_journal(config)
if ej.enabled and ej.journal_path.is_file():
# File scan can be large; keep it off the loop that services UDP/WS
events, more = await asyncio.get_running_loop().run_in_executor(
None,
lambda: journal_mod.read_events(
ej.journal_dir, ej.journal_file,
limit=limit, before=before, host=host_f, level=level_f, q=q_f,
predicate=visible,
),
)
else:
# Journal disabled or not yet written: serve the in-memory ring
events, more = journal_mod.filter_events(
reversed(data.msgs),
limit=limit, before=before, host=host_f, level=level_f, q=q_f,
predicate=visible,
)
return web.json_response({"events": events, "more": more})
async def cmd(request): async def cmd(request):
user, err = _require_auth(request) user, err = _require_auth(request)
if err: if err:
@@ -381,6 +450,7 @@ async def start(
return web.json_response({"error": "Forbidden"}, status=403) return web.json_response({"error": "Forbidden"}, status=403)
eventlog(uname, "INFO", "dropped") eventlog(uname, "INFO", "dropped")
del hbdclass.Host.hosts[uname] del hbdclass.Host.hosts[uname]
flap_mod.clear_host(uname)
return web.Response(text="Done") return web.Response(text="Done")
async def register(request): async def register(request):
@@ -453,7 +523,6 @@ async def start(
for h in sorted(hbdclass.Host.hosts) for h in sorted(hbdclass.Host.hosts)
if _can_operate_host(current_user, hbdclass.Host.hosts[h]) if _can_operate_host(current_user, hbdclass.Host.hosts[h])
], ],
messages=data.msgs[-30:],
current_user=current_user.to_dict() if current_user else None, current_user=current_user.to_dict() if current_user else None,
active_page="live", active_page="live",
) )
@@ -708,7 +777,7 @@ async def start(
if host.plugin_data: if host.plugin_data:
hosts_with_plugins.append({ hosts_with_plugins.append({
"name": hostname, "name": hostname,
"plugins": list(host.plugin_data.keys()), "plugins": _visible_plugin_names(host.plugin_data),
"is_owner": _can_own_host(current_user, host), "is_owner": _can_own_host(current_user, host),
"owner": host.owner, "owner": host.owner,
}) })
@@ -739,6 +808,27 @@ async def start(
) )
return web.Response(text=body, content_type="text/html") return web.Response(text=body, content_type="text/html")
async def log_page(request):
"""Render the event log page."""
current_user, _ = _require_auth_redirect(request)
pkg_dir = os.path.dirname(__file__)
templates_dir = config.get("templates_dir", os.path.join(pkg_dir, "templates"))
env = jinja2.Environment(loader=jinja2.FileSystemLoader(templates_dir), autoescape=True)
host = request.host # includes port if non-standard
forwarded_proto = request.headers.get("X-Forwarded-Proto", "")
is_secure = request.secure or forwarded_proto.lower() == "https"
scheme = "wss" if is_secure else "ws"
heartbeat_ws_url = f"{scheme}://{host}/ws"
tmpl = env.get_template("log.html")
body = tmpl.render(
title="Log - Heartbeat",
header="Log",
heartbeat_ws_url=heartbeat_ws_url,
current_user=current_user.to_dict() if current_user else None,
active_page="log",
)
return web.Response(text=body, content_type="text/html")
# ------------------------------------------------------------------------- # -------------------------------------------------------------------------
# Auth endpoints # Auth endpoints
# ------------------------------------------------------------------------- # -------------------------------------------------------------------------
@@ -1766,6 +1856,7 @@ async def start(
web.get("/api/0/hosts", api_hosts), web.get("/api/0/hosts", api_hosts),
web.get("/api/0/alert_summary", api_alert_summary), web.get("/api/0/alert_summary", api_alert_summary),
web.get("/api/0/messages", api_messages), web.get("/api/0/messages", api_messages),
web.get("/api/0/log", api_log),
web.get("/api/0/hosts/{hostname}/plugins", api_host_plugins), web.get("/api/0/hosts/{hostname}/plugins", api_host_plugins),
web.get("/api/0/hosts/{hostname}/plugins/{plugin_name}", api_host_plugin_detail), web.get("/api/0/hosts/{hostname}/plugins/{plugin_name}", api_host_plugin_detail),
web.get("/api/0/hosts/{hostname}/alerts", api_host_alerts), web.get("/api/0/hosts/{hostname}/alerts", api_host_alerts),
@@ -1781,6 +1872,7 @@ async def start(
web.get("/live", live), web.get("/live", live),
web.get("/plugins", plugins_page), web.get("/plugins", plugins_page),
web.get("/alerts", alerts_page), web.get("/alerts", alerts_page),
web.get("/log", log_page),
web.get("/about", about_page), web.get("/about", about_page),
web.get("/profile", profile_page), web.get("/profile", profile_page),
web.get("/settings", settings_page), web.get("/settings", settings_page),
+136 -4
View File
@@ -11,7 +11,7 @@ import os
import asyncio import asyncio
from datetime import datetime from datetime import datetime
from pathlib import Path from pathlib import Path
from typing import Dict, Any, Optional from typing import Dict, Any, Optional, Callable, Iterable, List, Tuple, Union
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
@@ -149,10 +149,35 @@ class MessageJournal:
self._current_size += len(json_bytes) self._current_size += len(json_bytes)
logger.debug(f"Logged message from {addr[0]}: {msg.get('ID', 'UNKNOWN')}") logger.debug(f"Logged message from {addr[0]}: {msg.get('ID', 'UNKNOWN')}")
except Exception as e: except Exception as e:
logger.error(f"Error writing to journal: {e}") logger.error(f"Error writing to journal: {e}")
async def log_event(self, event: Dict[str, Any]):
"""Write a caller-provided dict verbatim as one JSONL line (with rotation)."""
if not self.enabled or not self._initialized:
return
async with self._lock:
try:
line = json.dumps(event, separators=(',', ':')) + '\n'
nbytes = len(line.encode('utf-8'))
if self._current_size + nbytes > self.max_size:
await self._rotate()
if self._file_handle:
self._file_handle.write(line)
self._file_handle.flush()
self._current_size += nbytes
except Exception as e:
logger.error(f"Error writing event to journal: {e}")
async def backfill(self, events: Iterable[Dict[str, Any]]):
"""One-time seed: write *events* only if the journal file is currently empty."""
if not self.enabled or not self._initialized or self._current_size > 0:
return
for ev in events:
if isinstance(ev, dict):
await self.log_event(ev)
async def _rotate(self): async def _rotate(self):
""" """
Rotate the journal file. Rotate the journal file.
@@ -309,6 +334,88 @@ class MessageJournal:
} }
def _iter_journal_events(journal_dir: Union[str, Path], journal_file: str) -> Iterable[Dict[str, Any]]:
"""Yield event dicts from the journal files, newest event first.
Reads the current file, then rotated backups newest-to-oldest (backup
names embed rotation timestamps, so filename sort is chronological).
"""
dirp = Path(journal_dir)
files = [dirp / journal_file]
files.extend(sorted(dirp.glob(journal_file + '.*'), reverse=True))
for f in files:
if not f.is_file():
continue
try:
lines = f.read_text(encoding='utf-8', errors='replace').splitlines()
except OSError as e:
logger.warning(f"Cannot read journal file {f}: {e}")
continue
for line in reversed(lines):
try:
ev = json.loads(line)
except ValueError:
continue
if isinstance(ev, dict):
yield ev
def filter_events(
events: Iterable[Dict[str, Any]],
limit: int = 100,
before: Optional[float] = None,
host: Optional[str] = None,
level: Optional[str] = None,
q: Optional[str] = None,
predicate: Optional[Callable[[Dict[str, Any]], bool]] = None,
) -> Tuple[List[Dict[str, Any]], bool]:
"""Filter an iterable of event dicts already ordered newest-first.
Returns (events, more): up to *limit* matching events, and whether at
least one further matching event exists beyond the limit.
"""
host_l = host.lower() if host else None
level_l = level.lower() if level else None
q_l = q.lower() if q else None
out: List[Dict[str, Any]] = []
for ev in events:
if not isinstance(ev, dict):
continue
ts = ev.get('ts')
if before is not None and (not isinstance(ts, (int, float)) or ts >= before):
continue
if host_l and host_l not in str(ev.get('host') or '').lower():
continue
if level_l and str(ev.get('level') or '').lower() != level_l:
continue
if q_l and q_l not in str(ev.get('message') or '').lower():
continue
if predicate is not None and not predicate(ev):
continue
if len(out) >= limit:
return out, True
out.append(ev)
return out, False
def read_events(
journal_dir: Union[str, Path],
journal_file: str = 'events.journal',
*,
limit: int = 100,
before: Optional[float] = None,
host: Optional[str] = None,
level: Optional[str] = None,
q: Optional[str] = None,
predicate: Optional[Callable[[Dict[str, Any]], bool]] = None,
) -> Tuple[List[Dict[str, Any]], bool]:
"""Read filtered events newest-first from the events journal files."""
return filter_events(
_iter_journal_events(journal_dir, journal_file),
limit=limit, before=before, host=host, level=level, q=q, predicate=predicate,
)
# Global journal instance # Global journal instance
_journal_instance: Optional[MessageJournal] = None _journal_instance: Optional[MessageJournal] = None
@@ -332,7 +439,7 @@ def get_journal(config: Optional[Dict[str, Any]] = None) -> MessageJournal:
async def log_message(msg: Dict[str, Any], addr: tuple, timestamp: Optional[float] = None): async def log_message(msg: Dict[str, Any], addr: tuple, timestamp: Optional[float] = None):
""" """
Convenience function to log a message using the global journal. Convenience function to log a message using the global journal.
Args: Args:
msg: Parsed message dictionary msg: Parsed message dictionary
addr: Source address (ip, port) tuple addr: Source address (ip, port) tuple
@@ -340,3 +447,28 @@ async def log_message(msg: Dict[str, Any], addr: tuple, timestamp: Optional[floa
""" """
journal = get_journal() journal = get_journal()
await journal.log_message(msg, addr, timestamp) await journal.log_message(msg, addr, timestamp)
# Global events journal instance (human-readable event log, written by notify.eventlog)
_events_journal_instance: Optional[MessageJournal] = None
def get_events_journal(config: Optional[Dict[str, Any]] = None) -> MessageJournal:
"""Get or create the global events journal instance.
Uses the events_journal_* config keys; shares journal_dir and
journal_enabled with the raw-datagram journal.
"""
global _events_journal_instance
if _events_journal_instance is None:
cfg = config or {}
_events_journal_instance = MessageJournal(
{
'journal_dir': cfg.get('journal_dir', '/var/log/heartbeat'),
'journal_file': cfg.get('events_journal_file', 'events.journal'),
'journal_max_size': cfg.get('events_journal_max_size', 10 * 1024 * 1024),
'journal_max_backups': cfg.get('events_journal_max_backups', 10),
'journal_enabled': cfg.get('journal_enabled', True),
}
)
return _events_journal_instance
+31 -4
View File
@@ -160,11 +160,20 @@ async def _run_async(config, config_path=None):
from . import threshold as threshold_mod from . import threshold as threshold_mod
notify_mod.setup(config, loop=loop) notify_mod.setup(config, loop=loop)
# Initialize message journal # Initialize message journal
msg_journal = journal_mod.get_journal(config) msg_journal = journal_mod.get_journal(config)
await msg_journal.initialize() await msg_journal.initialize()
# Initialize events journal (human-readable event log for the /log page)
events_journal = journal_mod.get_events_journal(config)
await events_journal.initialize()
if data.msgs:
# One-time seed on upgrade: only writes when the journal file is empty
await events_journal.backfill(data.msgs)
notify_mod.eventlog(None, "INFO", f"hbd version {__version__} starting up")
# Initialize threshold checker # Initialize threshold checker
threshold_checker = threshold_mod.ThresholdChecker( threshold_checker = threshold_mod.ThresholdChecker(
config=config, config=config,
@@ -379,6 +388,26 @@ async def _run_async(config, config_path=None):
except Exception as e: except Exception as e:
logger.warning("Error closing message journal: %s", e) logger.warning("Error closing message journal: %s", e)
# Journal the shutdown event now; run()'s shutdown eventlog fires after this journal closes
try:
await events_journal.log_event(
{
"ts": time.time(),
"host": None,
"level": "INFO",
"service": None,
"message": f"hbd version {__version__} shutdown",
}
)
except Exception as e:
logger.debug("Could not journal shutdown event: %s", e)
# Close events journal
try:
await events_journal.close()
except Exception as e:
logger.warning("Error closing events journal: %s", e)
# Signal DNS worker to exit and await it # Signal DNS worker to exit and await it
try: try:
if "dns_task" in locals() and dns_task: if "dns_task" in locals() and dns_task:
@@ -487,8 +516,6 @@ def run(config, config_path=None):
except Exception as e: except Exception as e:
logger.warning("Failed to write pidfile %s: %s", pidfile, e) logger.warning("Failed to write pidfile %s: %s", pidfile, e)
eventlog(None, "INFO", f"hbd version {__version__} starting up")
if config_path: if config_path:
logger.info(f"Config file: {config_path} (reload with SIGHUP)") logger.info(f"Config file: {config_path} (reload with SIGHUP)")
else: else:
+34 -1
View File
@@ -25,6 +25,7 @@ from dataclasses import dataclass, field
from typing import Optional from typing import Optional
from . import data from . import data
from . import flap as flap_mod
from . import ws as ws_mod from . import ws as ws_mod
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
@@ -33,6 +34,7 @@ msg_to_websockets = ws_mod.broadcast
# Module-level state set via setup() # Module-level state set via setup()
_config: dict = {} _config: dict = {}
_loop: Optional[asyncio.AbstractEventLoop] = None
# Tracks which channels fired a WARNING/CRITICAL per host. # Tracks which channels fired a WARNING/CRITICAL per host.
# {host_name: set of channel_names} — used to route RECOVER to the same channels. # {host_name: set of channel_names} — used to route RECOVER to the same channels.
@@ -62,6 +64,7 @@ class Notification:
body: str # detail message body: str # detail message
level: str # RECOVER | WARNING | CRITICAL | INFO level: str # RECOVER | WARNING | CRITICAL | INFO
url: str = "" # link to plugin metrics page url: str = "" # link to plugin metrics page
service: str = "" # flap-detection key within the host ("" = the host itself)
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
@@ -70,14 +73,18 @@ class Notification:
def setup(cfg: dict, loop: Optional[asyncio.AbstractEventLoop] = None): def setup(cfg: dict, loop: Optional[asyncio.AbstractEventLoop] = None):
"""Initialize notifier from configuration dict.""" """Initialize notifier from configuration dict."""
global _config global _config, _loop
_config = dict(cfg) _config = dict(cfg)
flap_mod.setup(_config)
if loop is not None:
_loop = loop
def reload_config(cfg: dict): def reload_config(cfg: dict):
"""Reload notification configuration on SIGHUP.""" """Reload notification configuration on SIGHUP."""
global _config global _config
_config = dict(cfg) _config = dict(cfg)
flap_mod.setup(_config)
logger.info("Notification configuration reloaded") logger.info("Notification configuration reloaded")
@@ -131,6 +138,21 @@ def eventlog(host, lvl, m, service=None):
except Exception as e: except Exception as e:
logger.warning("failed to write to logfile: %s", e) logger.warning("failed to write to logfile: %s", e)
msg_to_websockets("message", msg) msg_to_websockets("message", msg)
_journal_event(msg)
def _journal_event(msg: dict):
"""Schedule an async write of *msg* to the events journal (no-op without a loop)."""
if _loop is None or not _loop.is_running():
return
from . import journal as journal_mod
ej = journal_mod._events_journal_instance
if ej is None or not ej.enabled:
return
try:
asyncio.run_coroutine_threadsafe(ej.log_event(msg), _loop)
except Exception as e:
logger.warning("failed to schedule events journal write: %s", e)
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
@@ -421,11 +443,22 @@ async def send_notification(host_name: str, notif: Notification) -> dict:
notification_channels, and dispatches. Silently does nothing if notification_channels, and dispatches. Silently does nothing if
no users are configured. no users are configured.
Flap detection runs first: once *host_name*/*notif.service* is flapping the
notification is dropped, and the one that trips the state carries
``flap.FLAP_MARKER``.
Returns a dict of {channel_name: bool} results. Returns a dict of {channel_name: bool} results.
""" """
from . import users as users_mod from . import users as users_mod
from . import hbdclass from . import hbdclass
action = flap_mod.observe(host_name, notif.service, notif.level)
if action == flap_mod.SUPPRESS:
logger.debug("flapping: suppressed %s notification for %s", notif.level, host_name)
return {}
if action == flap_mod.TRIP:
notif.body = f"{notif.body} {flap_mod.FLAP_MARKER}"
if not users_mod.users_enabled(): if not users_mod.users_enabled():
return {} return {}
+6
View File
@@ -386,6 +386,12 @@ def get_settings_sections(config: dict, threshold_checker=None, user=None) -> li
"Extra seconds to wait after a missed heartbeat before sending notifications.", editable=True), "Extra seconds to wait after a missed heartbeat before sending notifications.", editable=True),
field("threshold_renotify_interval", "Re-notify interval", "duration", field("threshold_renotify_interval", "Re-notify interval", "duration",
"How often to re-send notifications for ongoing threshold alerts.", editable=True), "How often to re-send notifications for ongoing threshold alerts.", editable=True),
field("flap_count", "Flap count", "number",
"Warning/critical notifications within the flap interval that mark a "
"service as flapping and silence it. 0 disables flap detection.", editable=True),
field("flap_interval", "Flap interval", "number",
"Minutes: the window flap count is measured over, and how long a service "
"must stay OK before flapping ends.", editable=True),
field("autosave_interval", "Autosave interval", "duration", field("autosave_interval", "Autosave interval", "duration",
"How often the server saves its state to disk."), "How often the server saves its state to disk."),
field("base_url", "Base URL", "text", field("base_url", "Base URL", "text",
+155
View File
@@ -0,0 +1,155 @@
/* hbd-ui.css — shared design system for hbd pages.
*
* Green-biased neutrals, teal accent, mono identifiers. Pages link this
* after static/style.css and build on the tokens + components below.
* Introduced with the settings-page redesign; see settings.html for the
* fullest use of the row/editor idiom.
*/
:root {
--st-paper: #f4f6f4;
--st-surface: #ffffff;
--st-surface-2: #eef1ee;
--st-ink: #1d2420;
--st-muted: #5f6a63;
--st-faint: #8a948d;
--st-line: #dde3de;
--st-line-soft: #e8ece8;
--st-accent: #0e7280;
--st-accent-ink: #ffffff;
--st-accent-soft: #e0eef0;
--st-ok: #3f9c5a;
--st-ok-soft: #e2f2e7;
--st-warn: #d9982a;
--st-warn-soft: #f9efdc;
--st-crit: #d64545;
--st-crit-soft: #f9e3e3;
--st-badge-owner: #ede7f6; --st-badge-owner-ink: #5e35b1;
--st-badge-mgr: #e3f2fd; --st-badge-mgr-ink: #1565c0;
--st-mono: ui-monospace, "SF Mono", "Cascadia Mono", Menlo, Consolas, monospace;
}
html[data-theme="dark"] {
--st-paper: #131816;
--st-surface: #1b211e;
--st-surface-2: #222925;
--st-ink: #e4eae6;
--st-muted: #9aa69e;
--st-faint: #6e7a72;
--st-line: #2c342f;
--st-line-soft: #252c28;
--st-accent: #4cc4d4;
--st-accent-ink: #0d2326;
--st-accent-soft: #143b40;
--st-ok: #5cba77;
--st-ok-soft: #1c2f22;
--st-warn: #e0ad4e;
--st-warn-soft: #33290f;
--st-crit: #e06c6c;
--st-crit-soft: #3a1a1a;
--st-badge-owner: #33294a; --st-badge-owner-ink: #b39ddb;
--st-badge-mgr: #1b3350; --st-badge-mgr-ink: #90caf9;
}
/* ── sticky toolbar under the site nav ── */
.st-toolbar {
position: sticky; top: var(--nav-h, 48px); z-index: 90;
display: flex; align-items: center; gap: 12px; flex-wrap: wrap;
padding: 9px 20px;
background: var(--st-surface); border-bottom: 1px solid var(--st-line);
}
.st-toolbar .brand { font-family: var(--st-mono); font-size: 15px; font-weight: 700; letter-spacing: -.02em; color: var(--st-ink); }
.st-toolbar .brand .tld { color: var(--st-accent); }
.st-toolbar .hintline { font-size: 12px; color: var(--st-faint); }
.st-toolbar .spacer { flex: 1; }
/* ── ECG pulse divider ── */
.pulse { display: block; width: 100%; height: 14px; }
.pulse polyline { fill: none; stroke: var(--st-accent); stroke-width: 1.5; opacity: .55; }
/* ── buttons ── */
.btn {
font: 600 13px/1 inherit; border: 1px solid var(--st-line); border-radius: 6px;
background: var(--st-surface); color: var(--st-ink); padding: 7px 14px; cursor: pointer;
}
.btn:focus-visible, a:focus-visible { outline: 2px solid var(--st-accent); outline-offset: 2px; }
.btn.primary { background: var(--st-accent); border-color: var(--st-accent); color: var(--st-accent-ink); }
.btn.quiet { border-color: transparent; background: transparent; color: var(--st-muted); }
.btn.small { padding: 4px 10px; font-size: 12px; }
.btn.danger { color: var(--st-crit); }
.btn:disabled { opacity: .6; cursor: default; }
/* ── section headers as config keys ── */
section.st { margin-top: 26px; scroll-margin-top: calc(var(--bars-h, 94px) + 12px); }
.sec-head { display: flex; align-items: baseline; gap: 10px; margin-bottom: 8px; flex-wrap: wrap; }
.sec-head h2 { margin: 0; font-family: var(--st-mono); font-size: 15px; font-weight: 700; color: var(--st-ink); }
.sec-head h2 .colon { color: var(--st-accent); }
.sec-head .count { font-family: var(--st-mono); font-size: 11.5px; color: var(--st-faint); }
.sec-head .sub { font-size: 12px; color: var(--st-muted); }
.sec-head .add { margin-left: auto; }
/* ── record list + row ── */
.list { background: var(--st-surface); border: 1px solid var(--st-line); border-radius: 8px; overflow: hidden; }
.row {
display: grid; grid-template-columns: 190px minmax(0,1fr) auto;
gap: 4px 14px; align-items: center;
padding: 8px 14px; border-bottom: 1px solid var(--st-line-soft);
transition: background .12s ease;
}
@media (prefers-reduced-motion: reduce) { .row { transition: none; } }
.row:last-child { border-bottom: none; }
.row:hover { background: var(--st-surface-2); }
.row .id { font-family: var(--st-mono); font-size: 13px; font-weight: 600; color: var(--st-ink); display: flex; align-items: center; gap: 8px; min-width: 0; }
.row .id .name { overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }
.row .act { display: flex; gap: 4px; justify-content: flex-end; }
.dot { width: 8px; height: 8px; border-radius: 50%; flex: none; }
.dot.ok { background: var(--st-ok); }
.dot.warn { background: var(--st-warn); }
.dot.crit { background: var(--st-crit); }
.dot.off { background: transparent; border: 1.5px solid var(--st-faint); }
.facts { display: flex; flex-wrap: wrap; gap: 4px 6px; align-items: center; min-width: 0; }
.chip {
display: inline-flex; align-items: center; gap: 4px;
font-family: var(--st-mono); font-size: 11px; line-height: 1.3;
padding: 3px 7px; border-radius: 4px;
background: var(--st-surface-2); color: var(--st-muted);
}
.chip.owner { background: var(--st-badge-owner); color: var(--st-badge-owner-ink); }
.chip.mgr { background: var(--st-badge-mgr); color: var(--st-badge-mgr-ink); }
.chip.level { background: transparent; border: 1px solid var(--st-line); }
.chip.private { background: transparent; border: 1px dashed var(--st-badge-owner-ink); color: var(--st-badge-owner-ink); }
.chip.ok { background: var(--st-ok-soft); color: var(--st-ok); }
.chip.warn { background: var(--st-warn-soft); color: var(--st-warn); }
.chip.crit { background: var(--st-crit-soft); color: var(--st-crit); }
.chip .k { color: var(--st-faint); }
.chip .w { color: var(--st-warn); font-weight: 700; }
.chip .c { color: var(--st-crit); font-weight: 700; }
/* ── key/value rows (server groups, about page) ── */
.kv .row { grid-template-columns: 230px minmax(0,1fr); padding: 6px 14px; }
.group {
display: flex; align-items: center; gap: 10px;
padding: 7px 14px 5px; font-family: var(--st-mono); font-size: 11px; color: var(--st-faint);
background: var(--st-surface); border-bottom: 1px solid var(--st-line-soft);
}
.group .gbtn { margin-left: auto; }
.kv .val { font-family: var(--st-mono); font-size: 12.5px; color: var(--st-muted); min-width: 0; }
.kv .val a { color: var(--st-accent); text-decoration: none; }
.kv .val a:hover { text-decoration: underline; }
.kv .desc { display: block; font-size: 11px; color: var(--st-faint); margin-top: 2px; }
.val-masked { color: var(--st-faint); letter-spacing: 2px; }
.note { margin: 8px 2px 0; font-size: 12px; color: var(--st-faint); }
/* ── handheld defaults for rows ── */
@media (max-width: 760px) {
.st-toolbar { padding: 8px 10px; }
.st-toolbar .hintline { display: none; }
.row { grid-template-columns: minmax(0,1fr) auto; padding: 9px 12px; }
.row .id { grid-row: 1; grid-column: 1; }
.row .act { grid-row: 1; grid-column: 2; }
.row .facts { grid-column: 1 / -1; }
.kv .row { grid-template-columns: 1fr; }
section.st { scroll-margin-top: calc(var(--bars-h, 94px) + 50px); }
}
+16
View File
@@ -0,0 +1,16 @@
/* hbd-ui.js — shared helpers for pages using hbd-ui.css.
*
* Keeps sticky toolbars stuck just below the fixed site nav, whose height
* varies with viewport width and wrapping.
*/
function _setStickyOffsets() {
const nav = document.querySelector('.nav');
const tb = document.querySelector('.st-toolbar');
const navH = nav ? nav.offsetHeight : 48;
const barsH = navH + (tb ? tb.offsetHeight : 46);
document.documentElement.style.setProperty('--nav-h', navH + 'px');
document.documentElement.style.setProperty('--bars-h', barsH + 'px');
}
window.addEventListener('resize', _setStickyOffsets);
document.addEventListener('DOMContentLoaded', _setStickyOffsets);
+96 -173
View File
@@ -1,190 +1,113 @@
<!DOCTYPE html> <!DOCTYPE html>
<html> <html>
{% include 'head.html' %} {% include 'head.html' %}
<link rel="stylesheet" href="/static/hbd-ui.css">
<script src="/static/hbd-ui.js"></script>
<style> <style>
html, body { overflow: visible; } html, body { height: auto; overflow: visible; }
body { background: var(--st-paper); padding: 60px 0 0; }
.container { .about-main { max-width: 700px; margin: 0 auto; padding: 0 20px 60px; }
max-width: 700px;
margin: 0 auto; .hero {
display: flex; align-items: baseline; gap: 14px; flex-wrap: wrap;
padding: 26px 2px 0;
} }
.hero .hb-logo {
h1 { font-family: var(--st-mono); font-size: 30px; font-weight: 700;
color: #333; letter-spacing: -.03em; color: var(--st-ink);
margin-bottom: 4px;
font-size: 1.5em;
} }
.hero .hb-logo .tld { color: var(--st-accent); }
.subtitle { .hero .version { font-family: var(--st-mono); font-size: 13px; color: var(--st-accent);
color: #666; background: var(--st-accent-soft); border-radius: 999px; padding: 3px 12px; }
margin-bottom: 24px; .hero .tagline { flex-basis: 100%; font-size: 13px; color: var(--st-muted); }
font-size: 0.9em;
}
.section {
background: #fff;
border-radius: 8px;
box-shadow: 0 1px 6px rgba(0,0,0,0.1);
padding: 20px 24px;
margin-bottom: 20px;
}
.section h2 {
font-size: 1em;
font-weight: 700;
color: #333;
margin: 0 0 16px;
padding-bottom: 10px;
border-bottom: 1px solid #eee;
text-transform: uppercase;
letter-spacing: 0.5px;
}
.info-row {
display: flex;
align-items: baseline;
padding: 8px 0;
border-bottom: 1px solid #f5f5f5;
font-size: 0.9em;
}
.info-row:last-child { border-bottom: none; }
.info-label {
width: 160px;
flex-shrink: 0;
color: #666;
font-size: 0.88em;
}
.info-value {
color: #222;
word-break: break-all;
}
.info-value a {
color: #0066cc;
text-decoration: none;
}
.info-value a:hover { text-decoration: underline; }
.version-badge {
display: inline-block;
padding: 3px 12px;
background: #e8f0fe;
color: #1a73e8;
border-radius: 12px;
font-size: 1.00em;
font-weight: 600;
font-family: monospace;
}
.hb-logo {
font-size: 2.5em;
font-weight: 700;
color: #0066cc;
letter-spacing: -1px;
margin-bottom: 6px;
}
.hb-tagline {
color: #555;
font-size: 0.95em;
}
.logo-section {
display: flex;
align-items: center;
gap: 20px;
padding: 8px 0 4px;
}
.logo-text { flex: 1; }
/* ── Dark mode ── */
html[data-theme="dark"] h1 { color: var(--text); }
html[data-theme="dark"] .subtitle { color: var(--text-sec); }
html[data-theme="dark"] .section { background: var(--surface); box-shadow: 0 1px 6px var(--shadow); }
html[data-theme="dark"] .section h2 { color: var(--text); border-bottom-color: var(--border); }
html[data-theme="dark"] .info-row { border-bottom-color: var(--border-4); }
html[data-theme="dark"] .info-label { color: var(--text-sec); }
html[data-theme="dark"] .info-value { color: var(--text); }
html[data-theme="dark"] .info-value a { color: var(--link); }
html[data-theme="dark"] .hb-logo { color: var(--link); }
html[data-theme="dark"] .hb-tagline { color: var(--text-sec); }
html[data-theme="dark"] .version-badge { background: #1a3255; color: #60a5fa; }
</style> </style>
<body> <body>
{% include 'nav.html' %} {% include 'nav.html' %}
<div class="container"> <div class="st-toolbar">
<h1>{{ header }}</h1> <span class="brand">hbd<span class="tld">·about</span></span>
<p class="subtitle">Heartbeat monitoring system</p> <span class="hintline">heartbeat monitoring system</span>
<div class="section">
<div class="logo-section">
<div class="logo-text">
<div class="hb-logo">Heartbeat</div>
<div class="hb-tagline">Lightweight host monitoring over UDP</div>
</div>
<span class="version-badge">v{{ hbd_version }}</span>
</div>
</div>
<div class="section">
<h2>Version</h2>
<div class="info-row">
<span class="info-label">Server version</span>
<span class="info-value">{{ hbd_version }}</span>
</div>
<div class="info-row">
<span class="info-label">Python</span>
<span class="info-value">{{ python_version }}</span>
</div>
<div class="info-row">
<span class="info-label">License</span>
<span class="info-value">MIT</span>
</div>
</div>
<div class="section">
<h2>Runtime</h2>
<div class="info-row">
<span class="info-label">Host</span>
<span class="info-value">{{ server_hostname }}</span>
</div>
<div class="info-row">
<span class="info-label">Started</span>
<span class="info-value">{{ start_time_str }}</span>
</div>
<div class="info-row">
<span class="info-label">Uptime</span>
<span class="info-value" id="uptime-value">{{ uptime_str }}</span>
</div>
<div class="info-row">
<span class="info-label">Hosts monitored</span>
<span class="info-value">{{ host_count }}</span>
</div>
</div>
<div class="section">
<h2>Contact &amp; Source</h2>
<div class="info-row">
<span class="info-label">Author</span>
<span class="info-value">Andreas Wrede</span>
</div>
<div class="info-row">
<span class="info-label">Email</span>
<span class="info-value"><a href="mailto:aew.hbd@wrede.ca">aew.hbd@wrede.ca</a></span>
</div>
<div class="info-row">
<span class="info-label">Repository</span>
<span class="info-value"><a href="https://git.wrede.ca/andreas/heartbeat" target="_blank" rel="noopener">git.wrede.ca/andreas/heartbeat</a></span>
</div>
</div>
</div> </div>
<svg class="pulse" viewBox="0 0 1200 14" preserveAspectRatio="none" aria-hidden="true">
<polyline points="0,10 340,10 352,10 358,3 364,13 370,1 378,12 384,10 560,10 572,10 578,3 584,13 590,1 598,12 604,10 1200,10"/>
</svg>
<div class="about-main">
<div class="hero">
<span class="hb-logo">heart<span class="tld">beat</span></span>
<span class="version">v{{ hbd_version }}</span>
<span class="tagline">Lightweight host monitoring over UDP</span>
</div>
<section class="st">
<div class="sec-head">
<h2>version<span class="colon">:</span></h2>
</div>
<div class="list kv">
<div class="row">
<span class="id"><span class="name">server</span></span>
<span class="val">{{ hbd_version }}</span>
</div>
<div class="row">
<span class="id"><span class="name">python</span></span>
<span class="val">{{ python_version }}</span>
</div>
<div class="row">
<span class="id"><span class="name">license</span></span>
<span class="val">MIT</span>
</div>
</div>
</section>
<section class="st">
<div class="sec-head">
<h2>runtime<span class="colon">:</span></h2>
</div>
<div class="list kv">
<div class="row">
<span class="id"><span class="name">host</span></span>
<span class="val">{{ server_hostname }}</span>
</div>
<div class="row">
<span class="id"><span class="name">started</span></span>
<span class="val">{{ start_time_str }}</span>
</div>
<div class="row">
<span class="id"><span class="name">uptime</span></span>
<span class="val" id="uptime-value">{{ uptime_str }}</span>
</div>
<div class="row">
<span class="id"><span class="name">hosts_monitored</span></span>
<span class="val">{{ host_count }}</span>
</div>
</div>
</section>
<section class="st">
<div class="sec-head">
<h2>contact<span class="colon">:</span></h2>
</div>
<div class="list kv">
<div class="row">
<span class="id"><span class="name">author</span></span>
<span class="val">Andreas Wrede</span>
</div>
<div class="row">
<span class="id"><span class="name">email</span></span>
<span class="val"><a href="mailto:aew.hbd@wrede.ca">aew.hbd@wrede.ca</a></span>
</div>
<div class="row">
<span class="id"><span class="name">repository</span></span>
<span class="val"><a href="https://git.wrede.ca/andreas/heartbeat" target="_blank" rel="noopener">git.wrede.ca/andreas/heartbeat</a></span>
</div>
</div>
</section>
</div>
{% include 'foot.html' %}
<script> <script>
(function() { (function() {
+142 -493
View File
@@ -1,376 +1,97 @@
<!DOCTYPE html> <!DOCTYPE html>
<html> <html>
{% include 'head.html' %} {% include 'head.html' %}
<link rel="stylesheet" href="/static/hbd-ui.css">
<script src="/static/hbd-ui.js"></script>
<style> <style>
html, body { height: auto; overflow: visible; }
body { background: var(--st-paper); padding: 60px 0 0; }
html, body { .alerts-main { max-width: 1100px; margin: 0 auto; padding: 0 20px 60px; }
height: auto;
overflow-y: auto;
}
.container { /* summary tiles */
max-width: 1400px; .stats { display: flex; gap: 10px; flex-wrap: wrap; padding-top: 18px; }
margin: 0 auto; .stat {
flex: 1 1 140px; display: flex; align-items: baseline; gap: 10px;
background: var(--st-surface); border: 1px solid var(--st-line); border-radius: 8px;
padding: 10px 14px;
} }
.stat .num { font-family: var(--st-mono); font-size: 24px; font-weight: 700; font-variant-numeric: tabular-nums; }
.stat .lbl { font-family: var(--st-mono); font-size: 11.5px; color: var(--st-faint); }
.stat.crit .num { color: var(--st-crit); }
.stat.warn .num { color: var(--st-warn); }
.stat.ok .num { color: var(--st-ok); }
h1 { color: #333; margin-bottom: 5px; margin-top: 15px; font-size: 1.5em; } /* filter bar */
.filterbar { display: flex; gap: 6px; align-items: center; flex-wrap: wrap; margin: 14px 0 0; }
.filterbar .fbtn {
font-family: var(--st-mono); font-size: 12px; cursor: pointer;
color: var(--st-muted); background: var(--st-surface); border: 1px solid var(--st-line);
border-radius: 999px; padding: 5px 12px;
}
.filterbar .fbtn.active { background: var(--st-accent-soft); border-color: var(--st-accent); color: var(--st-accent); font-weight: 600; }
.filterbar input {
font: 12px var(--st-mono); color: var(--st-ink);
background: var(--st-surface); border: 1px solid var(--st-line); border-radius: 999px;
padding: 5px 12px; min-width: 170px;
}
.filterbar input.invalid { border-color: var(--st-crit); }
.filterbar .upd { margin-left: auto; font-size: 11.5px; color: var(--st-faint); }
.subtitle { /* alert row specifics */
color: #666; .row.alert-crit { border-left: 3px solid var(--st-crit); padding-left: 11px; }
margin-bottom: 30px; .row.alert-warn { border-left: 3px solid var(--st-warn); padding-left: 11px; }
} .row.acked { opacity: .55; border-left-style: dashed; }
.row .id a { color: inherit; text-decoration: none; }
.row .id a:hover { color: var(--st-accent); }
.acked-chip { color: var(--st-ok); font-size: 12px; white-space: nowrap; }
.summary-cards { .empty {
display: flex; padding: 36px 14px; text-align: center; color: var(--st-faint); font-size: 13px;
flex-wrap: wrap;
gap: 10px;
margin-bottom: 16px;
} }
.empty .big { font-size: 26px; color: var(--st-ok); display: block; margin-bottom: 6px; }
.summary-card { .errorbox {
background: white; background: var(--st-crit-soft); color: var(--st-crit);
border-radius: 6px; border-radius: 8px; padding: 14px 16px; font-size: 13px;
padding: 6px 14px;
box-shadow: 0 1px 4px rgba(0,0,0,0.1);
display: flex;
align-items: center;
gap: 8px;
border-left: 4px solid #ddd;
} }
.summary-card.critical { border-left-color: #ea1e0f; }
.summary-card.warning { border-left-color: #ff9800; }
.summary-card.ok { border-left-color: #4caf50; }
.summary-number {
font-size: 1.4em;
font-weight: bold;
line-height: 1;
}
.summary-number.critical { color: #ea1e0f; }
.summary-number.warning { color: #ff9800; }
.summary-number.ok { color: #4caf50; }
.summary-label {
color: #666;
font-size: 1.00em;
}
.filters {
background: white;
border-radius: 8px;
padding: 15px;
margin-bottom: 20px;
box-shadow: 0 2px 8px rgba(0,0,0,0.1);
display: flex;
gap: 15px;
align-items: center;
}
.filter-label {
font-weight: bold;
color: #555;
}
.filter-button {
padding: 8px 16px;
border: 2px solid #ddd;
background: white;
border-radius: 20px;
cursor: pointer;
transition: all 0.2s;
font-size: 0.9em;
}
.filter-button:hover {
border-color: #2196f3;
}
.filter-button.active {
background: #2196f3;
color: white;
border-color: #2196f3;
}
.filter-input {
padding: 7px 12px;
border: 2px solid #ddd;
border-radius: 20px;
font-size: 0.9em;
outline: none;
width: 200px;
transition: border-color 0.2s;
}
.filter-input:focus {
border-color: #2196f3;
}
.filter-input.invalid {
border-color: #f44336;
}
.alerts-container {
background: white;
border-radius: 8px;
padding: 20px;
box-shadow: 0 2px 8px rgba(0,0,0,0.1);
}
.alert-item {
border-left: 5px solid #ddd;
padding: 15px;
margin-bottom: 15px;
background: #fafafa;
border-radius: 4px;
display: flex;
justify-content: space-between;
align-items: center;
transition: all 0.2s;
}
.alert-item.acknowledged {
opacity: 0.8;
background: #f0f0f0;
}
.alert-item:hover {
box-shadow: 0 2px 8px rgba(0,0,0,0.1);
transform: translateX(5px);
}
.alert-item.critical {
border-left-color: #f44336;
background: #ffebee;
}
.alert-item.warning {
border-left-color: #ff9800;
background: #fff3e0;
}
.alert-item.unknown {
border-left-color: #9e9e9e;
background: #f5f5f5;
}
.alert-main {
flex: 1;
}
.alert-header {
display: flex;
align-items: center;
gap: 15px;
margin-bottom: 8px;
}
.alert-level {
padding: 4px 12px;
border-radius: 12px;
font-size: 0.75em;
font-weight: bold;
text-transform: uppercase;
letter-spacing: 0.5px;
}
.alert-level.critical {
background: #f44336;
color: white;
}
.alert-level.warning {
background: #ff9800;
color: white;
}
.alert-level.unknown {
background: #9e9e9e;
color: white;
}
.alert-hostname {
font-weight: bold;
color: #0066cc;
font-size: 1.1em;
text-decoration: none;
}
.alert-hostname:hover {
text-decoration: underline;
}
.alert-metric {
color: #0066cc;
font-size: 1.1em;
font-weight: normal;
}
.alert-details {
display: flex;
gap: 20px;
color: #666;
font-size: 0.9em;
}
.alert-value {
font-weight: bold;
color: #333;
}
.alert-duration {
color: #999;
font-size: 1.00em;
}
.alert-actions {
display: flex;
flex-direction: column;
gap: 8px;
margin-left: 15px;
}
.acknowledge-btn {
padding: 8px 16px;
background: #2196f3;
color: white;
border: none;
border-radius: 4px;
cursor: pointer;
font-size: 1.00em;
transition: all 0.2s;
white-space: nowrap;
}
.acknowledge-btn:hover {
background: #1976d2;
transform: scale(1.05);
}
.acknowledge-btn:disabled {
background: #ccc;
cursor: not-allowed;
transform: none;
}
.acknowledged-badge {
padding: 4px 8px;
background: #4caf50;
color: white;
border-radius: 4px;
font-size: 0.75em;
text-align: center;
white-space: nowrap;
}
.no-alerts {
text-align: center;
padding: 60px 20px;
color: #999;
}
.no-alerts-icon {
font-size: 4em;
margin-bottom: 20px;
}
.loading {
text-align: center;
padding: 40px;
color: #666;
}
.error {
background: #ffebee;
border-left: 4px solid #f44336;
padding: 20px;
margin: 20px 0;
border-radius: 4px;
color: #c62828;
}
.refresh-info {
text-align: center;
color: #999;
font-size: 1.00em;
margin-top: 20px;
padding-top: 20px;
border-top: 1px solid #e0e0e0;
}
.last-update {
color: #666;
font-size: 0.9em;
text-align: right;
margin-bottom: 15px;
}
/* ── Dark mode ── */
html[data-theme="dark"] h1 { color: var(--text); }
html[data-theme="dark"] .subtitle { color: var(--text-sec); }
html[data-theme="dark"] .summary-card { background: var(--surface); }
html[data-theme="dark"] .summary-label { color: var(--text-sec); }
html[data-theme="dark"] .filters { background: var(--surface); }
html[data-theme="dark"] .filter-label { color: var(--text-sec); }
html[data-theme="dark"] .filter-button { background: var(--surface-2); border-color: var(--border); color: var(--text); }
html[data-theme="dark"] .filter-button.active { background: #2196f3; color: #fff; border-color: #2196f3; }
html[data-theme="dark"] .filter-input { background: var(--input-bg); border-color: var(--input-border); color: var(--text); }
html[data-theme="dark"] .alerts-container { background: var(--surface); }
html[data-theme="dark"] .alert-item { background: var(--surface-2); }
html[data-theme="dark"] .alert-item.acknowledged { background: var(--surface-3); }
html[data-theme="dark"] .alert-item.critical { background: #2e0a0a; border-left-color: #f44336; }
html[data-theme="dark"] .alert-item.warning { background: #2e1a00; border-left-color: #ff9800; }
html[data-theme="dark"] .alert-item.unknown { background: var(--surface-2); }
html[data-theme="dark"] .alert-hostname { color: var(--link); }
html[data-theme="dark"] .alert-details { color: var(--text-sec); }
html[data-theme="dark"] .alert-value { color: var(--text); }
html[data-theme="dark"] .alert-duration { color: var(--text-muted); }
html[data-theme="dark"] .last-update { color: var(--text-sec); }
html[data-theme="dark"] .refresh-info { color: var(--text-muted); border-top-color: var(--border); }
html[data-theme="dark"] .no-alerts,
html[data-theme="dark"] .loading { color: var(--text-muted); }
</style> </style>
<body> <body>
{% include 'nav.html' %} {% include 'nav.html' %}
<div class="container"> <div class="st-toolbar">
<h1>{{ header }}</h1> <span class="brand">hbd<span class="tld">·alerts</span></span>
<p class="subtitle">Real-time monitoring alerts and threshold violations</p> <span class="hintline">threshold violations and reachability — refreshes every 15 s</span>
</div>
<svg class="pulse" viewBox="0 0 1200 14" preserveAspectRatio="none" aria-hidden="true">
<polyline points="0,10 340,10 352,10 358,3 364,13 370,1 378,12 384,10 560,10 572,10 578,3 584,13 590,1 598,12 604,10 1200,10"/>
</svg>
<div class="summary-cards" id="summary-cards"> <div class="alerts-main">
<div class="summary-card critical"> <div class="stats">
<div class="summary-label">Critical</div> <div class="stat crit"><span class="num" id="critical-count">–</span><span class="lbl">critical</span></div>
<div class="summary-number critical" id="critical-count">-</div> <div class="stat warn"><span class="num" id="warning-count">–</span><span class="lbl">warning</span></div>
</div> <div class="stat ok"><span class="num" id="host-count">–</span><span class="lbl">hosts</span></div>
<div class="summary-card warning">
<div class="summary-label">Warning</div>
<div class="summary-number warning" id="warning-count">-</div>
</div>
<div class="summary-card ok">
<div class="summary-label">Total Hosts</div>
<div class="summary-number ok" id="host-count">-</div>
</div>
</div> </div>
<div class="filters"> <div class="filterbar">
<span class="filter-label">Show:</span> <button class="fbtn active" onclick="filterAlerts('all', this)">all</button>
<button class="filter-button active" onclick="filterAlerts('all')">All</button> <button class="fbtn" onclick="filterAlerts('critical', this)">critical</button>
<button class="filter-button" onclick="filterAlerts('critical')">Critical Only</button> <button class="fbtn" onclick="filterAlerts('warning', this)">warning</button>
<button class="filter-button" onclick="filterAlerts('warning')">Warning Only</button> <input id="host-filter" type="text" placeholder="host filter (regex)" oninput="onHostFilterInput(this)">
<input id="host-filter" class="filter-input" type="text" placeholder="host filter (regex)" oninput="onHostFilterInput(this)"> <span class="upd">updated <span id="last-update-time">never</span></span>
</div> </div>
<div class="alerts-container"> <section class="st" id="alerts">
<div class="last-update">Last updated: <span id="last-update-time">Never</span></div> <div class="sec-head">
<div id="alerts-list"> <h2>alerts<span class="colon">:</span></h2>
<div class="loading">Loading alerts...</div> <span class="count" id="alert-count"></span>
</div> </div>
<div class="refresh-info"> <div class="list" id="alerts-list">
Auto-refreshing every 15 seconds <div class="empty">Loading alerts…</div>
</div> </div>
</div> </section>
</div> </div>
<script> <script>
@@ -378,207 +99,138 @@
let allAlerts = []; let allAlerts = [];
let hostFilterRe = null; let hostFilterRe = null;
function escHtml(s) {
return String(s).replace(/&/g,'&amp;').replace(/</g,'&lt;').replace(/>/g,'&gt;').replace(/"/g,'&quot;').replace(/'/g,'&#39;');
}
async function loadAlerts() { async function loadAlerts() {
try { try {
const response = await fetch('/api/0/alerts'); const response = await fetch('/api/0/alerts');
if (!response.ok) { if (!response.ok) throw new Error(`HTTP ${response.status}`);
throw new Error(`HTTP ${response.status}`);
}
const data = await response.json(); const data = await response.json();
allAlerts = data.alerts; allAlerts = data.alerts;
// Update summary cards
document.getElementById('critical-count').textContent = data.summary.critical || 0; document.getElementById('critical-count').textContent = data.summary.critical || 0;
document.getElementById('warning-count').textContent = data.summary.warning || 0; document.getElementById('warning-count').textContent = data.summary.warning || 0;
document.getElementById('host-count').textContent = data.host_count || 0; document.getElementById('host-count').textContent = data.host_count || 0;
// Update last update time
document.getElementById('last-update-time').textContent = new Date().toLocaleTimeString(); document.getElementById('last-update-time').textContent = new Date().toLocaleTimeString();
// Render alerts
renderAlerts(allAlerts); renderAlerts(allAlerts);
} catch (error) { } catch (error) {
document.getElementById('alerts-list').innerHTML = document.getElementById('alerts-list').innerHTML =
`<div class="error">Failed to load alerts: ${error.message}</div>`; `<div class="errorbox">Failed to load alerts: ${escHtml(error.message)}. Retrying automatically.</div>`;
} }
} }
function renderAlerts(alerts) { function renderAlerts(alerts) {
const container = document.getElementById('alerts-list'); const container = document.getElementById('alerts-list');
// Filter alerts based on current filter let filtered = alerts;
let filteredAlerts = alerts;
if (currentFilter !== 'all') { if (currentFilter !== 'all') {
filteredAlerts = filteredAlerts.filter(alert => filtered = filtered.filter(a => a.level.toLowerCase() === currentFilter);
alert.level.toLowerCase() === currentFilter
);
} }
if (hostFilterRe) { if (hostFilterRe) {
filteredAlerts = filteredAlerts.filter(alert => hostFilterRe.test(alert.hostname)); filtered = filtered.filter(a => hostFilterRe.test(a.hostname));
} }
if (filteredAlerts.length === 0) { document.getElementById('alert-count').textContent =
if (currentFilter === 'all' && alerts.length === 0) { filtered.length === alerts.length ? alerts.length : `${filtered.length} of ${alerts.length}`;
container.innerHTML = `
<div class="no-alerts"> if (filtered.length === 0) {
<div class="no-alerts-icon">✓</div> container.innerHTML = (currentFilter === 'all' && !hostFilterRe && alerts.length === 0)
<h2>All Systems Normal</h2> ? `<div class="empty"><span class="big">✓</span>All systems normal — no active alerts.</div>`
<p>No active alerts at this time</p> : `<div class="empty">No matching alerts.</div>`;
</div>
`;
} else {
container.innerHTML = `
<div class="no-alerts">
<p>No ${currentFilter} alerts</p>
</div>
`;
}
return; return;
} }
let html = ''; container.innerHTML = filtered.map(renderAlert).join('');
for (const alert of filteredAlerts) {
html += renderAlert(alert);
}
container.innerHTML = html;
} }
function renderAlert(alert) { function renderAlert(alert) {
const level = alert.level.toLowerCase(); const level = alert.level.toLowerCase();
const cls = level === 'critical' ? 'crit' : (level === 'warning' ? 'warn' : 'level');
const duration = getDuration(alert.since); const duration = getDuration(alert.since);
const acknowledged = alert.acknowledged || false; const acked = alert.acknowledged || false;
// Use formatted message if available, otherwise build from individual fields const metric = (alert.metric_path.includes('.')
let valueText = `Value: <span class="alert-value">${formatValue(alert.last_value)}</span>`; ? alert.metric_path.slice(alert.metric_path.indexOf('.') + 1)
: alert.metric_path).replace(/_status_code$/, '');
let chips = `<span class="chip ${cls}">${escHtml(alert.level)}</span>`;
chips += `<span class="chip"><span class="k">metric</span> ${escHtml(metric)}</span>`;
if (alert.formatted_message) { if (alert.formatted_message) {
valueText += ` <span class="threshold-info">${alert.formatted_message}</span>`; chips += `<span class="chip">${escHtml(alert.formatted_message)}</span>`;
} else if (alert.threshold_value !== undefined && alert.threshold_value !== null && alert.operator) { } else {
valueText += ` <span class="threshold-info">(threshold: ${alert.operator} ${formatValue(alert.threshold_value)})</span>`; chips += `<span class="chip"><span class="k">value</span> ${escHtml(formatValue(alert.last_value))}</span>`;
if (alert.threshold_value !== undefined && alert.threshold_value !== null && alert.operator) {
chips += `<span class="chip level">${escHtml(alert.operator)} ${escHtml(formatValue(alert.threshold_value))}</span>`;
}
} }
if (alert.recovery_threshold !== undefined && alert.recovery_threshold !== null) { if (alert.recovery_threshold !== undefined && alert.recovery_threshold !== null) {
const recOp = (alert.operator === '>' || alert.operator === '>=') ? '<' : '>'; const recOp = (alert.operator === '>' || alert.operator === '>=') ? '<' : '>';
valueText += ` <span class="threshold-info" style="color:#888">(recovers ${recOp} ${formatValue(alert.recovery_threshold)})</span>`; chips += `<span class="chip level">recovers ${recOp} ${escHtml(formatValue(alert.recovery_threshold))}</span>`;
} }
chips += `<span class="chip"><span class="k">for</span> ${duration}</span>`;
// Build actions section
let actionsHtml = ''; const act = acked
if (acknowledged) { ? `<span class="acked-chip">✓ acknowledged</span>`
actionsHtml = ` : `<button class="btn small" onclick="acknowledgeAlert('${escHtml(alert.hostname)}', '${escHtml(alert.metric_path)}', event)">Acknowledge</button>`;
<div class="alert-actions">
<div class="acknowledged-badge">✓ Acknowledged</div>
</div>
`;
} else {
actionsHtml = `
<div class="alert-actions">
<button class="acknowledge-btn" onclick="acknowledgeAlert('${alert.hostname}', '${alert.metric_path}', event)">
Acknowledge
</button>
</div>
`;
}
return ` return `
<div class="alert-item ${level} ${acknowledged ? 'acknowledged' : ''}"> <div class="row alert-${cls === 'level' ? 'warn' : cls}${acked ? ' acked' : ''}">
<div class="alert-main"> <span class="id">
<div class="alert-header"> <span class="dot ${cls === 'crit' ? 'crit' : 'warn'}"></span>
<span class="alert-level ${level}">${alert.level}</span> <a class="name" href="/plugins#${encodeURIComponent(alert.hostname)}">${escHtml(alert.hostname)}</a>
<a class="alert-hostname" href="/plugins#${alert.hostname}">${alert.hostname}</a> </span>
<span class="alert-metric">${(alert.metric_path.includes('.') ? alert.metric_path.slice(alert.metric_path.indexOf('.') + 1) : alert.metric_path).replace(/_status_code$/, '')}</span> <span class="facts">${chips}</span>
</div> <span class="act">${act}</span>
<div class="alert-details"> </div>`;
<span>${valueText}</span>
<span class="alert-duration">Active for ${duration}</span>
</div>
</div>
${actionsHtml}
</div>
`;
} }
function formatValue(value) { function formatValue(value) {
if (typeof value === 'number') { if (typeof value === 'number') {
if (value > 1000) { if (value > 1000) return value.toLocaleString();
return value.toLocaleString();
}
return value.toFixed(2); return value.toFixed(2);
} }
return value; return value;
} }
function getDuration(timestamp) { function getDuration(timestamp) {
const now = Date.now() / 1000; const seconds = Math.floor(Date.now() / 1000 - timestamp);
const seconds = Math.floor(now - timestamp); if (seconds < 60) return `${seconds}s`;
if (seconds < 3600) return `${Math.floor(seconds / 60)}m`;
if (seconds < 60) { if (seconds < 86400) {
return `${seconds}s`; return `${Math.floor(seconds / 3600)}h ${Math.floor((seconds % 3600) / 60)}m`;
} else if (seconds < 3600) {
return `${Math.floor(seconds / 60)}m`;
} else if (seconds < 86400) {
const hours = Math.floor(seconds / 3600);
const minutes = Math.floor((seconds % 3600) / 60);
return `${hours}h ${minutes}m`;
} else {
const days = Math.floor(seconds / 86400);
const hours = Math.floor((seconds % 86400) / 3600);
return `${days}d ${hours}h`;
} }
return `${Math.floor(seconds / 86400)}d ${Math.floor((seconds % 86400) / 3600)}h`;
} }
function filterAlerts(filter) { function filterAlerts(filter, btn) {
currentFilter = filter; currentFilter = filter;
document.querySelectorAll('.filterbar .fbtn').forEach(b => b.classList.remove('active'));
// Update active button btn.classList.add('active');
document.querySelectorAll('.filter-button').forEach(btn => {
btn.classList.remove('active');
});
event.target.classList.add('active');
// Re-render with new filter
renderAlerts(allAlerts); renderAlerts(allAlerts);
} }
async function acknowledgeAlert(hostname, metricPath, event) { async function acknowledgeAlert(hostname, metricPath, event) {
// Prevent event bubbling if (event) event.stopPropagation();
if (event) {
event.stopPropagation();
}
// Disable the button
const button = event.target; const button = event.target;
button.disabled = true; button.disabled = true;
button.textContent = 'Acknowledging...'; button.textContent = 'Acknowledging…';
try { try {
const response = await fetch('/api/0/alerts/acknowledge', { const response = await fetch('/api/0/alerts/acknowledge', {
method: 'POST', method: 'POST',
headers: { headers: {'Content-Type': 'application/json'},
'Content-Type': 'application/json', body: JSON.stringify({hostname: hostname, metric_path: metricPath}),
},
body: JSON.stringify({
hostname: hostname,
metric_path: metricPath,
}),
}); });
if (!response.ok) throw new Error(`HTTP ${response.status}`);
if (!response.ok) {
throw new Error(`HTTP ${response.status}`);
}
const result = await response.json(); const result = await response.json();
const a = allAlerts.find(x => x.hostname === hostname && x.metric_path === metricPath);
// Update the alert in our local data if (a) {
const alert = allAlerts.find(a => a.hostname === hostname && a.metric_path === metricPath); a.acknowledged = true;
if (alert) { a.acknowledged_at = result.acknowledged_at;
alert.acknowledged = true;
alert.acknowledged_at = result.acknowledged_at;
} }
// Re-render alerts
renderAlerts(allAlerts); renderAlerts(allAlerts);
} catch (error) { } catch (error) {
alert(`Failed to acknowledge alert: ${error.message}`); alert(`Failed to acknowledge alert: ${error.message}`);
button.disabled = false; button.disabled = false;
@@ -603,10 +255,8 @@
renderAlerts(allAlerts); renderAlerts(allAlerts);
} }
// Auto-refresh every 15 seconds
setInterval(loadAlerts, 15000); setInterval(loadAlerts, 15000);
// Initialise filter from URL query string (?filter=...)
(function () { (function () {
const param = new URLSearchParams(window.location.search).get('filter'); const param = new URLSearchParams(window.location.search).get('filter');
if (param) { if (param) {
@@ -616,7 +266,6 @@
} }
})(); })();
// Initial load
loadAlerts(); loadAlerts();
</script> </script>
</body> </body>
File diff suppressed because it is too large Load Diff
+228
View File
@@ -0,0 +1,228 @@
<!DOCTYPE html>
<html>
{% include 'head.html' %}
<link rel="stylesheet" href="/static/hbd-ui.css">
<script src="/static/hbd-ui.js"></script>
<style>
html, body { height: auto; overflow: visible; }
body { background: var(--st-paper); padding: 60px 0 0; }
.log-main { max-width: 1100px; margin: 0 auto; padding: 0 20px 60px; }
.st-toolbar input, .st-toolbar select {
font: 12px var(--st-mono); color: var(--st-ink);
background: var(--st-surface); border: 1px solid var(--st-line); border-radius: 999px;
padding: 5px 12px;
}
.live-dot { display: inline-flex; align-items: center; gap: 6px; font-size: 12px; font-weight: 600; color: var(--st-ok); }
.live-dot::before { content: ""; width: 7px; height: 7px; border-radius: 50%; background: var(--st-ok); }
.live-dot.reconnecting { color: var(--st-crit); }
.live-dot.reconnecting::before { background: var(--st-crit); }
.logrow {
display: grid; grid-template-columns: 130px 74px 150px minmax(0,1fr);
gap: 4px 14px; align-items: baseline;
padding: 4px 14px; border-bottom: 1px solid var(--st-line-soft);
font-family: var(--st-mono); font-size: 11.5px;
}
.logrow:last-child { border-bottom: none; }
.logrow .t { color: var(--st-faint); font-variant-numeric: tabular-nums; white-space: nowrap; }
.logrow .h { color: var(--st-ink); font-weight: 600; overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }
.logrow .m { color: var(--st-muted); overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }
.empty { padding: 24px 14px; text-align: center; color: var(--st-faint); font-size: 12.5px; }
.loadmore {
display: block; margin: 14px auto 0; cursor: pointer;
font-family: var(--st-mono); font-size: 12px; color: var(--st-muted);
background: var(--st-surface); border: 1px solid var(--st-line); border-radius: 999px;
padding: 6px 16px;
}
.loadmore:hover { color: var(--st-accent); border-color: var(--st-accent); }
.loadmore:disabled { opacity: .5; cursor: default; }
@media (max-width: 760px) {
.logrow { grid-template-columns: 66px minmax(0,1fr); }
.logrow .lvlc, .logrow .h { display: none; }
}
</style>
<body>
{% include 'nav.html' %}
<div class="st-toolbar">
<span class="brand">hbd<span class="tld">·log</span></span>
<span class="hintline">connectivity and alert events — full journal history</span>
<span class="spacer"></span>
<input type="text" id="filter-host" placeholder="host">
<select id="filter-level">
<option value="">level</option>
<option value="info">INFO</option>
<option value="warning">WARNING</option>
<option value="critical">CRITICAL</option>
<option value="recover">RECOVER</option>
<option value="unknown">UNKNOWN</option>
</select>
<input type="text" id="filter-msg" placeholder="message">
<span class="live-dot reconnecting" id="ws-status">connecting…</span>
</div>
<svg class="pulse" viewBox="0 0 1200 14" preserveAspectRatio="none" aria-hidden="true">
<polyline points="0,10 340,10 352,10 358,3 364,13 370,1 378,12 384,10 560,10 572,10 578,3 584,13 590,1 598,12 604,10 1200,10"/>
</svg>
<div class="log-main">
<section class="st">
<div class="sec-head">
<h2>log<span class="colon">:</span></h2>
<span class="count" id="log-count"></span>
</div>
<div class="list" id="messages">
<div class="empty">Loading events…</div>
</div>
<button class="loadmore" id="load-older" style="display:none">load older</button>
</section>
</div>
<script>
var PAGE_SIZE = 200;
var oldestTs = null; // ts of the oldest loaded row (pagination cursor)
var seq = 0; // guards against out-of-order fetch responses
var newestTs = null; // ts of the newest loaded row (dedup guard for the WS tail)
function escHtml(s) {
return String(s).replace(/&/g,'&amp;').replace(/</g,'&lt;').replace(/>/g,'&gt;').replace(/"/g,'&quot;').replace(/'/g,'&#39;');
}
function fmtClock(ts) {
var d = new Date(ts * 1000);
function p(n) { return n < 10 ? '0' + n : '' + n; }
var t = p(d.getHours()) + ':' + p(d.getMinutes()) + ':' + p(d.getSeconds());
return (d.toDateString() === new Date().toDateString())
? t
: d.getFullYear() + '-' + p(d.getMonth() + 1) + '-' + p(d.getDate()) + ' ' + t;
}
function logRowHtml(msg) {
var lvl = (msg.level || 'INFO');
var lc = lvl.toLowerCase();
var chipCls = lc === 'critical' ? 'crit' : (lc === 'warning' ? 'warn' : (lc === 'recover' ? 'ok' : ''));
var short = lc === 'critical' ? 'CRIT' : (lc === 'warning' ? 'WARN' : escHtml(lvl));
return '<div class="logrow">'
+ '<span class="t">' + fmtClock(msg.ts) + '</span>'
+ '<span class="lvlc"><span class="chip ' + chipCls + '">' + short + '</span></span>'
+ '<span class="h">' + escHtml(msg.host || '') + (msg.service ? ' · ' + escHtml(msg.service) : '') + '</span>'
+ '<span class="m">' + escHtml(msg.message || '') + '</span>'
+ '</div>';
}
function currentFilters() {
return {
host: document.getElementById('filter-host').value.trim(),
level: document.getElementById('filter-level').value,
q: document.getElementById('filter-msg').value.trim()
};
}
function matchesFilters(msg) {
var f = currentFilters();
if (f.host && !(msg.host || '').toLowerCase().includes(f.host.toLowerCase())) return false;
if (f.level && (msg.level || '').toLowerCase() !== f.level) return false;
if (f.q && !(msg.message || '').toLowerCase().includes(f.q.toLowerCase())) return false;
return true;
}
function updateCount() {
var n = document.querySelectorAll('#messages .logrow').length;
document.getElementById('log-count').textContent = n ? n + ' events loaded' : '';
}
async function fetchLog(reset) {
var mySeq = ++seq;
var box = document.getElementById('messages');
var btn = document.getElementById('load-older');
btn.disabled = true;
var f = currentFilters();
var params = new URLSearchParams();
params.set('limit', String(PAGE_SIZE));
if (f.host) params.set('host', f.host);
if (f.level) params.set('level', f.level);
if (f.q) params.set('q', f.q);
if (!reset && oldestTs != null) params.set('before', String(oldestTs));
var data;
try {
var r = await fetch('/api/0/log?' + params.toString());
if (!r.ok) throw new Error('HTTP ' + r.status);
data = await r.json();
} catch (e) {
if (mySeq === seq && reset) {
box.innerHTML = '<div class="empty">Failed to load events: ' + escHtml(e.message) + '</div>';
}
btn.disabled = false;
return;
}
if (mySeq !== seq) return; // a newer request superseded this one
if (reset) { box.innerHTML = ''; oldestTs = null; }
var html = '';
data.events.forEach(function (ev) {
html += logRowHtml(ev);
if (ev.ts != null && (oldestTs == null || ev.ts < oldestTs)) oldestTs = ev.ts;
if (ev.ts != null && (newestTs == null || ev.ts > newestTs)) newestTs = ev.ts;
});
box.insertAdjacentHTML('beforeend', html);
if (!box.children.length) box.innerHTML = '<div class="empty">No events found.</div>';
btn.style.display = data.more ? '' : 'none';
btn.disabled = false;
updateCount();
}
function onFilterChange() { fetchLog(true); }
function debounce(fn, ms) {
var t;
return function () { clearTimeout(t); t = setTimeout(fn, ms); };
}
document.getElementById('filter-host').addEventListener('input', debounce(onFilterChange, 300));
document.getElementById('filter-msg').addEventListener('input', debounce(onFilterChange, 300));
document.getElementById('filter-level').addEventListener('change', onFilterChange);
document.getElementById('load-older').addEventListener('click', function () { fetchLog(false); });
// ---- live tail over the shared websocket --------------------------------
function setWsStatus(ok) {
var el = document.getElementById('ws-status');
el.classList.toggle('reconnecting', !ok);
el.textContent = ok ? 'live' : 'reconnecting…';
}
function WS_Connect() {
if (!("WebSocket" in window)) return;
var ws_hbd = new WebSocket("{{ heartbeat_ws_url }}");
ws_hbd.onopen = function () {
setWsStatus(true);
ws_hbd.send("heartbeat_web");
};
ws_hbd.onmessage = function (event) {
var state = JSON.parse(event.data);
// history replays are covered by the API seed; only tail new events
if (state.type !== "message" || state.history) return;
var msg = state.data;
if (!matchesFilters(msg)) return;
if (msg.ts != null && newestTs != null && msg.ts <= newestTs) return; // already loaded via the API
if (msg.ts != null && (newestTs == null || msg.ts > newestTs)) newestTs = msg.ts;
var box = document.getElementById('messages');
var placeholder = box.querySelector('.empty');
if (placeholder) placeholder.remove();
box.insertAdjacentHTML('afterbegin', logRowHtml(msg));
updateCount();
};
ws_hbd.onclose = function () {
setWsStatus(false);
setTimeout(WS_Connect, 3000);
};
}
// ---- boot ----------------------------------------------------------------
fetchLog(true);
WS_Connect();
</script>
</body>
</html>
-2
View File
@@ -1,2 +0,0 @@
<!-- <label for="drawer-toggle" id="drawer-toggle-label"></label>
s<header>{{ header }}</header> -->
+1
View File
@@ -6,6 +6,7 @@
<a href="/live"{% if active_page == "live" %} class="active"{% endif %}>Live Dashboard</a> <a href="/live"{% if active_page == "live" %} class="active"{% endif %}>Live Dashboard</a>
<a href="/plugins"{% if active_page == "plugins" %} class="active"{% endif %}>Host Overview</a> <a href="/plugins"{% if active_page == "plugins" %} class="active"{% endif %}>Host Overview</a>
<a href="/alerts"{% if active_page == "alerts" %} class="active"{% endif %}>Alerts</a> <a href="/alerts"{% if active_page == "alerts" %} class="active"{% endif %}>Alerts</a>
<a href="/log"{% if active_page == "log" %} class="active"{% endif %}>Log</a>
{% if current_user %} {% if current_user %}
<a href="/settings"{% if active_page == "settings" %} class="active"{% endif %}>Settings</a> <a href="/settings"{% if active_page == "settings" %} class="active"{% endif %}>Settings</a>
{% endif %} {% endif %}
+335 -369
View File
@@ -2,38 +2,30 @@
<html> <html>
{% include 'head.html' %} {% include 'head.html' %}
<link rel="stylesheet" href="/static/hbd-ui.css">
<script src="/static/hbd-ui.js"></script>
<style> <style>
body { overflow: hidden; } body { overflow: hidden; background: var(--st-paper); }
/* This page scrolls inside .container, not the body — the toolbar is
always in view, and sticky misbehaves inside an overflow:hidden body. */
.st-toolbar { position: static; }
.container { .container {
max-width: 1400px; max-width: 1400px;
margin: 0 auto; margin: 0 auto;
max-height: calc(100vh - 120px); max-height: calc(100vh - var(--bars-h, 94px) - 30px);
overflow-y: auto; overflow-y: auto;
padding-right: 10px; padding: 0 10px 20px;
} }
h1 { /* ── Host cards: one record list per host ───────────────────── */
color: #333;
margin-bottom: 5px;
margin-top: 15px;
font-size: 1.5em;
}
.subtitle {
color: #666;
margin-bottom: 15px;
font-size: 0.9em;
}
/* ── Host cards ─────────────────────────────────────────────── */
.host-card { .host-card {
background: white; background: var(--st-surface);
border-radius: 6px; border: 1px solid var(--st-line);
padding: 0; border-radius: 8px;
margin-bottom: 10px; margin-bottom: 8px;
box-shadow: 0 1px 4px rgba(0,0,0,0.1);
} }
.host-header { .host-header {
@@ -42,115 +34,74 @@
gap: 10px; gap: 10px;
cursor: pointer; cursor: pointer;
user-select: none; user-select: none;
padding: 10px 15px; padding: 8px 14px;
border-radius: 6px; border-radius: 8px;
} }
.host-header:hover { background: var(--st-surface-2); border-radius: 8px 8px 0 0; }
.host-card.collapsed .host-header:hover { border-radius: 8px; }
.host-header:hover { background: #f9f9f9; border-radius: 6px 6px 0 0; } .host-left { display: flex; align-items: center; gap: 8px; flex-shrink: 0; }
.host-card.collapsed .host-header:hover { border-radius: 6px; }
.host-left {
display: flex;
align-items: center;
gap: 8px;
flex-shrink: 0;
}
.collapse-icon { .collapse-icon {
font-size: 1em; font-size: .85em;
color: #888; color: var(--st-faint);
transition: transform 0.2s; transition: transform 0.2s;
min-width: 16px; min-width: 14px;
} }
@media (prefers-reduced-motion: reduce) { .collapse-icon, .acc-icon { transition: none; } }
.host-card.collapsed .collapse-icon { transform: rotate(-90deg); } .host-card.collapsed .collapse-icon { transform: rotate(-90deg); }
.host-name { .host-name {
font-size: 1.05em; font-family: var(--st-mono);
font-weight: bold; font-size: 13.5px;
color: #333; font-weight: 600;
color: var(--st-ink);
white-space: nowrap; white-space: nowrap;
} }
/* ── Glance strip ───────────────────────────────────────────── */ /* ── Glance strip ───────────────────────────────────────────── */
.glance-strip { display: flex; align-items: center; gap: 5px; flex: 1; flex-wrap: wrap; padding: 0 8px; }
.glance-strip {
display: flex;
align-items: center;
gap: 8px;
flex: 1;
flex-wrap: wrap;
padding: 0 12px;
}
.glance-chip { .glance-chip {
font-size: 0.78em; display: inline-flex; align-items: center;
padding: 2px 9px; font-family: var(--st-mono); font-size: 11px; line-height: 1.3;
border-radius: 10px; padding: 3px 7px; border-radius: 4px; white-space: nowrap;
font-weight: 500; background: var(--st-ok-soft); color: var(--st-ok);
white-space: nowrap;
background: #e8f5e9;
color: #2e7d32;
} }
.glance-chip.warn { background: var(--st-warn-soft); color: var(--st-warn); }
.glance-chip.warn { background: #fff3e0; color: #e65100; } .glance-chip.crit { background: var(--st-crit-soft); color: var(--st-crit); }
.glance-chip.crit { background: #ffebee; color: #b71c1c; } .glance-chip.neutral { background: var(--st-surface-2); color: var(--st-muted); }
.glance-chip.neutral { background: #f5f5f5; color: #555; } .glance-loading { font-size: 11.5px; color: var(--st-faint); font-style: italic; }
.glance-loading { font-size: 0.8em; color: #bbb; font-style: italic; }
/* ── Host right zone ────────────────────────────────────────── */ /* ── Host right zone ────────────────────────────────────────── */
.host-right { display: flex; align-items: center; gap: 8px; flex-shrink: 0; }
.host-right {
display: flex;
align-items: center;
gap: 10px;
flex-shrink: 0;
}
.nagios-badge { .nagios-badge {
font-size: 0.75em; font-family: var(--st-mono); font-size: 10.5px; font-weight: 700;
font-weight: bold; padding: 3px 9px; border-radius: 4px;
padding: 2px 10px; background: var(--st-surface-2); color: var(--st-muted);
border-radius: 10px; text-transform: uppercase; white-space: nowrap;
background: #9e9e9e;
color: white;
text-transform: uppercase;
white-space: nowrap;
} }
.nagios-badge.ok { background: var(--st-ok-soft); color: var(--st-ok); }
.nagios-badge.ok { background: #4caf50; } .nagios-badge.warning { background: var(--st-warn-soft); color: var(--st-warn); }
.nagios-badge.warning { background: #ff9800; } .nagios-badge.critical { background: var(--st-crit-soft); color: var(--st-crit); }
.nagios-badge.critical { background: #f44336; }
.os-label { .os-label {
font-size: 0.75em; font-size: 11px; color: var(--st-faint); white-space: nowrap;
color: #999; max-width: 200px; overflow: hidden; text-overflow: ellipsis;
white-space: nowrap;
max-width: 200px;
overflow: hidden;
text-overflow: ellipsis;
} }
.host-action-btn { .host-action-btn {
font-size: 0.75em; font: 600 12px/1 inherit;
font-weight: bold; padding: 4px 10px; border-radius: 6px;
padding: 3px 10px; border: 1px solid var(--st-line);
border-radius: 4px; background: var(--st-surface);
border: none; cursor: pointer; text-decoration: none; white-space: nowrap;
cursor: pointer;
text-decoration: none;
white-space: nowrap;
} }
.host-action-btn.update-btn { .host-action-btn.update-btn { color: var(--st-accent); }
background: #e3f2fd; .host-action-btn.update-btn:hover { background: var(--st-accent-soft); border-color: var(--st-accent); }
color: #1565c0; .host-action-btn.delete-btn { color: var(--st-crit); }
} .host-action-btn.delete-btn:hover { background: var(--st-crit-soft); border-color: var(--st-crit); }
.host-action-btn.update-btn:hover { background: #bbdefb; }
.host-action-btn.delete-btn {
background: #ffebee;
color: #c62828;
}
.host-action-btn.delete-btn:hover { background: #ffcdd2; }
/* ── Action result toast ───────────────────────────────────── */ /* ── Action result toast ───────────────────────────────────── */
#action-toast { #action-toast {
@@ -158,8 +109,8 @@
bottom: 24px; bottom: 24px;
left: 50%; left: 50%;
transform: translateX(-50%) translateY(20px); transform: translateX(-50%) translateY(20px);
background: #323232; background: var(--st-ink);
color: #fff; color: var(--st-paper);
padding: 12px 22px; padding: 12px 22px;
border-radius: 6px; border-radius: 6px;
font-size: 0.9em; font-size: 0.9em;
@@ -171,301 +122,176 @@
z-index: 9000; z-index: 9000;
white-space: pre-wrap; white-space: pre-wrap;
} }
#action-toast.show { #action-toast.show { opacity: 1; transform: translateX(-50%) translateY(0); }
opacity: 1; #action-toast.error { background: var(--st-crit); color: #fff; }
transform: translateX(-50%) translateY(0);
}
#action-toast.error { background: #c62828; }
/* ── Host body ──────────────────────────────────────────────── */ /* ── Host body ──────────────────────────────────────────────── */
.host-body { padding: 8px 14px 12px; border-top: 1px solid var(--st-line-soft); }
.host-body {
padding: 8px 15px 12px;
border-top: 1px solid #f0f0f0;
}
.host-card.collapsed .host-body { display: none; } .host-card.collapsed .host-body { display: none; }
/* ── Plugin accordions ──────────────────────────────────────── */ /* ── Plugin accordions ──────────────────────────────────────── */
.plugin-accordion { .plugin-accordion {
border: 1px solid #e8e8e8; border: 1px solid var(--st-line-soft);
border-radius: 4px; border-radius: 6px;
margin-bottom: 5px; margin-bottom: 5px;
overflow: hidden; overflow: hidden;
} }
.plugin-acc-header { .plugin-acc-header {
display: flex; display: flex; align-items: center; gap: 10px;
align-items: center; padding: 6px 12px; cursor: pointer;
gap: 10px; background: var(--st-surface-2);
padding: 7px 12px;
cursor: pointer;
background: #fafafa;
user-select: none; user-select: none;
} }
.plugin-acc-header:hover { background: var(--st-accent-soft); }
.plugin-acc-header:hover { background: #f0f4ff; } .acc-icon { font-size: 0.7em; color: var(--st-faint); transition: transform 0.15s; min-width: 12px; }
.acc-icon {
font-size: 0.7em;
color: #999;
transition: transform 0.15s;
min-width: 12px;
}
.plugin-accordion:not(.collapsed) .acc-icon { transform: rotate(90deg); } .plugin-accordion:not(.collapsed) .acc-icon { transform: rotate(90deg); }
.plugin-label { .plugin-label {
font-weight: 600; font-family: var(--st-mono);
font-size: 1.00em; font-weight: 600; font-size: 12.5px;
color: #444; color: var(--st-ink);
min-width: 140px; min-width: 140px;
} }
.plugin-summary { font-family: var(--st-mono); font-size: 11.5px; color: var(--st-faint); flex: 1; }
.plugin-summary {
font-size: 0.82em;
color: #888;
flex: 1;
}
.plugin-accordion.collapsed .plugin-acc-body { display: none; } .plugin-accordion.collapsed .plugin-acc-body { display: none; }
.plugin-acc-body { padding: 10px 12px; } .plugin-acc-body { padding: 10px 12px; }
/* ── Tables ─────────────────────────────────────────────────── */ /* ── Tables ─────────────────────────────────────────────────── */
.data-table { .data-table {
width: 100%; width: 100%;
border-collapse: collapse; border-collapse: collapse;
font-size: 1.00em; font-size: 12.5px;
background: #fff; background: var(--st-surface);
box-shadow: 0 1px 3px rgba(0,0,0,0.08); border: 1px solid var(--st-line-soft);
border-radius: 4px; border-radius: 6px;
overflow: hidden; overflow: hidden;
margin-bottom: 10px; margin-bottom: 10px;
font-variant-numeric: tabular-nums;
} }
.data-table thead { background: var(--st-surface-2); }
.data-table thead { background: #2196f3; color: white; }
.data-table th { .data-table th {
padding: 7px 10px; font-family: var(--st-mono);
text-align: left; font-size: 10.5px; text-transform: uppercase; letter-spacing: .05em;
font-weight: 600; color: var(--st-faint); font-weight: 600;
text-transform: uppercase; text-align: left; padding: 5px 8px;
font-size: 0.75em; border-bottom: 1px solid var(--st-line);
letter-spacing: 0.4px;
} }
.data-table th.num { text-align: right; } .data-table th.num { text-align: right; }
.data-table th.center { text-align: center; } .data-table th.center { text-align: center; }
.data-table td { .data-table td {
/* padding: 6px 10px; */ padding: 4px 8px;
border-top: 1px solid #e8e8e8; border-top: 1px solid var(--st-line-soft);
color: #333; color: var(--st-ink);
} }
.data-table td.num { text-align: right; font-family: var(--st-mono); }
.data-table td.num {
text-align: right;
font-family: 'Courier New', monospace;
font-size: 0.95em;
}
.data-table td.center { text-align: center; } .data-table td.center { text-align: center; }
.data-table td.key { color: #666; font-weight: 500; width: 38%; } .data-table td.key { color: var(--st-muted); font-weight: 500; width: 38%; }
.data-table tbody tr:hover { background: var(--st-surface-2); }
.data-table tbody tr:nth-child(even) { background: #fafafa; } .iface-name { font-family: var(--st-mono); font-weight: 600; color: var(--st-accent); }
.data-table tbody tr:hover { background: #f0f4ff; }
.iface-name { font-weight: bold; color: #2196f3; }
/* ── Percent bars ───────────────────────────────────────────── */ /* ── Percent bars ───────────────────────────────────────────── */
.bar-wrap { display: flex; align-items: center; gap: 8px; min-width: 120px; }
.bar-wrap {
display: flex;
align-items: center;
gap: 8px;
}
.bar-track { .bar-track {
display: inline-block; flex: 1; height: 7px; border-radius: 4px;
width: 70px; background: var(--st-line-soft); overflow: hidden; min-width: 60px;
height: 6px;
background: #e0e0e0;
border-radius: 3px;
vertical-align: middle;
flex-shrink: 0;
} }
.bar-fill { height: 100%; border-radius: 4px; background: var(--st-ok); }
.bar-fill { .bar-fill.warn { background: var(--st-warn); }
height: 6px; .bar-fill.crit { background: var(--st-crit); }
border-radius: 3px;
background: #4caf50;
max-width: 100%;
}
.bar-fill.warn { background: #ff9800; }
.bar-fill.crit { background: #f44336; }
/* ── Disk two-table layout ──────────────────────────────────── */ /* ── Disk two-table layout ──────────────────────────────────── */
.flex-tables { display: flex; gap: 14px; flex-wrap: wrap; }
.flex-tables {
display: flex;
gap: 14px;
flex-wrap: wrap;
}
.flex-tables > div { flex: 1 1 380px; } .flex-tables > div { flex: 1 1 380px; }
.table-section-label { .table-section-label {
font-size: 0.78em; font-family: var(--st-mono);
font-weight: 600; font-size: 11px; font-weight: 600; color: var(--st-faint);
text-transform: uppercase; text-transform: uppercase; letter-spacing: .05em;
color: #888; margin: 4px 0 6px;
letter-spacing: 0.4px;
margin-bottom: 4px;
} }
/* ── Status / misc ──────────────────────────────────────────── */ /* ── Status / misc ──────────────────────────────────────────── */
.status-up { color: var(--st-ok); font-weight: bold; }
.status-down { color: var(--st-crit); font-weight: bold; }
.status-up { color: #4caf50; font-weight: bold; } .pct-ok { color: var(--st-ok); font-weight: bold; }
.status-down { color: #f44336; font-weight: bold; } .pct-warn { color: var(--st-warn); font-weight: bold; }
.pct-crit { color: var(--st-crit); font-weight: bold; }
.pct-ok { color: #2e7d32; font-weight: bold; } .check-ok { background: var(--st-ok-soft); }
.pct-warn { color: #e65100; font-weight: bold; } .check-warning { background: var(--st-warn-soft); }
.pct-crit { color: #b71c1c; font-weight: bold; } .check-critical { background: var(--st-crit-soft); }
.check-unknown { background: var(--st-surface-2); }
.check-ok { background: #e8f5e9; } .check-status-ok { color: var(--st-ok); font-weight: bold; }
.check-warning { background: #fff3e0; } .check-status-warning { color: var(--st-warn); font-weight: bold; }
.check-critical { background: #ffebee; } .check-status-critical { color: var(--st-crit); font-weight: bold; }
.check-unknown { background: #f5f5f5; } .check-status-unknown { color: var(--st-faint); font-weight: bold; }
.check-status-ok { color: #2e7d32; font-weight: bold; } .check-output { font-size: 0.9em; color: var(--st-muted); }
.check-status-warning { color: #e65100; font-weight: bold; }
.check-status-critical { color: #b71c1c; font-weight: bold; }
.check-status-unknown { color: #777; font-weight: bold; }
.check-output { font-size: 0.9em; color: #555; }
.timestamp { .timestamp {
color: #bbb; font-size: 11px; color: var(--st-faint);
font-size: 0.75em; margin-top: 6px; padding-top: 6px;
margin-top: 8px; border-top: 1px solid var(--st-line-soft);
padding-top: 6px;
border-top: 1px solid #f0f0f0;
text-align: right;
}
.no-data {
text-align: center;
padding: 20px;
color: #aaa;
font-style: italic;
font-size: 0.9em;
}
.loading {
text-align: center;
padding: 12px;
color: #aaa;
font-size: 1.00em;
} }
.no-data { text-align: center; padding: 40px; color: var(--st-faint); }
.loading { text-align: center; padding: 12px; color: var(--st-faint); font-style: italic; }
.error { .error {
background: #ffebee; background: var(--st-crit-soft);
border-left: 3px solid #f44336; border-left: 3px solid var(--st-crit);
padding: 8px 12px; padding: 12px 16px; margin: 10px 0;
margin: 8px 0; border-radius: 6px; color: var(--st-crit);
border-radius: 3px;
color: #c62828;
font-size: 1.00em;
} }
/* ── Scrollbar ──────────────────────────────────────────────── */ /* ── Scrollbar ──────────────────────────────────────────────── */
.container::-webkit-scrollbar { width: 8px; } .container::-webkit-scrollbar { width: 8px; }
.container::-webkit-scrollbar-track { background: #f1f1f1; border-radius: 4px; } .container::-webkit-scrollbar-track { background: var(--st-surface-2); border-radius: 4px; }
.container::-webkit-scrollbar-thumb { background: #ccc; border-radius: 4px; } .container::-webkit-scrollbar-thumb { background: var(--st-line); border-radius: 4px; }
.container::-webkit-scrollbar-thumb:hover { background: #999; } .container::-webkit-scrollbar-thumb:hover { background: var(--st-faint); }
/* ── Host info section ──────────────────────────────────────────────────── */ /* ── Host info section ──────────────────────────────────────── */
.host-info-section { .host-info-section {
padding: 12px 16px; background: var(--st-surface-2);
background: #fafafa; border-radius: 6px;
border-bottom: 1px solid #e0e0e0; padding: 8px 12px;
font-size: 1.00em; margin-bottom: 8px;
font-size: 12.5px;
} }
.info-meta { .info-meta { display: flex; flex-wrap: wrap; gap: 4px 18px; }
display: grid; .info-label { font-family: var(--st-mono); font-size: 11px; font-weight: 600; color: var(--st-faint); white-space: nowrap; }
grid-template-columns: max-content 1fr; .info-value { color: var(--st-ink); }
gap: 3px 14px;
margin-bottom: 10px;
}
.info-label { font-weight: 600; color: #555; white-space: nowrap; }
.info-value { color: #222; }
.info-thresholds-title { .info-thresholds-title {
font-weight: 600; font-family: var(--st-mono);
color: #555; font-size: 11px; font-weight: 600; color: var(--st-faint);
margin-bottom: 6px; text-transform: uppercase; letter-spacing: .05em;
margin: 8px 0 4px;
} }
.info-note { color: #888; font-style: italic; } .info-note { color: var(--st-faint); font-style: italic; }
.info-loading { color: #bbb; font-style: italic; } .info-loading { color: var(--st-faint); font-style: italic; }
.threshold-covers { font-size: 1.00em; color: #777; font-style: italic; } .threshold-covers { font-size: 11.5px; color: var(--st-faint); font-style: italic; }
/* ── Dark mode ── */
html[data-theme="dark"] h1 { color: var(--text); }
html[data-theme="dark"] .subtitle { color: var(--text-sec); }
html[data-theme="dark"] .host-card { background: var(--surface); }
html[data-theme="dark"] .host-header:hover { background: var(--surface-2); }
html[data-theme="dark"] .host-name { color: var(--text); }
html[data-theme="dark"] .collapse-icon,
html[data-theme="dark"] .acc-icon { color: var(--text-muted); }
html[data-theme="dark"] .host-body { border-top-color: var(--border-3); }
html[data-theme="dark"] .plugin-accordion { border-color: var(--border); }
html[data-theme="dark"] .plugin-acc-header { background: var(--surface-2); }
html[data-theme="dark"] .plugin-acc-header:hover { background: var(--surface-3); }
html[data-theme="dark"] .plugin-label { color: var(--text-2); }
html[data-theme="dark"] .plugin-summary { color: var(--text-muted); }
html[data-theme="dark"] .data-table { background: var(--surface); }
html[data-theme="dark"] .data-table td { border-top-color: var(--border); color: var(--text); }
html[data-theme="dark"] .data-table td.key { color: var(--text-sec); }
html[data-theme="dark"] .data-table tbody tr:nth-child(even) { background: var(--surface-2); }
html[data-theme="dark"] .data-table tbody tr:hover { background: #1e3a5f; }
html[data-theme="dark"] .bar-track { background: var(--border); }
html[data-theme="dark"] .table-section-label { color: var(--text-muted); }
html[data-theme="dark"] .no-data,
html[data-theme="dark"] .loading { color: var(--text-dim); }
html[data-theme="dark"] .timestamp { color: var(--text-dim); border-top-color: var(--border-3); }
html[data-theme="dark"] .glance-chip.neutral { background: var(--surface-3); color: var(--text-sec); }
html[data-theme="dark"] .os-label { color: var(--text-muted); }
html[data-theme="dark"] .host-info-section { background: var(--surface-2); border-bottom-color: var(--border); }
html[data-theme="dark"] .info-label { color: var(--text-3); }
html[data-theme="dark"] .info-value { color: var(--text); }
html[data-theme="dark"] .info-thresholds-title { color: var(--text-3); }
html[data-theme="dark"] .info-note,
html[data-theme="dark"] .info-loading,
html[data-theme="dark"] .threshold-covers { color: var(--text-muted); }
html[data-theme="dark"] .check-ok { background: #0d2e17; }
html[data-theme="dark"] .check-warning { background: #2e1a00; }
html[data-theme="dark"] .check-critical { background: #2e0a0a; }
html[data-theme="dark"] .check-unknown { background: var(--surface-2); }
html[data-theme="dark"] .check-output { color: var(--text-sec); }
html[data-theme="dark"] .container::-webkit-scrollbar-track { background: var(--surface-2); }
html[data-theme="dark"] .container::-webkit-scrollbar-thumb { background: var(--border); }
</style> </style>
<body> <body>
{% include 'nav.html' %} {% include 'nav.html' %}
<div class="st-toolbar">
<span class="brand">hbd<span class="tld">·overview</span></span>
<span class="hintline">per-host metrics — expand a host for plugin details</span>
</div>
<svg class="pulse" viewBox="0 0 1200 14" preserveAspectRatio="none" aria-hidden="true">
<polyline points="0,10 340,10 352,10 358,3 364,13 370,1 378,12 384,10 560,10 572,10 578,3 584,13 590,1 598,12 604,10 1200,10"/>
</svg>
<div class="container"> <div class="container">
<h1>{{ header }}</h1>
<p class="subtitle">Per-host system metrics — expand a host to see plugin details</p>
{% if not hosts %} {% if not hosts %}
<div class="no-data"> <div class="no-data">
<p>No hosts with plugin data available</p> <p>No hosts have connected yet</p>
<p style="font-size:0.9em;margin-top:10px;">Hosts will appear here once they start sending plugin metrics</p> <p style="font-size:0.9em;margin-top:10px;">Hosts will appear here once they start sending heartbeats</p>
</div> </div>
{% else %} {% else %}
<div id="hosts-container"> <div id="hosts-container">
@@ -618,6 +444,34 @@
<span class="info-label">Last Packet</span><span class="info-value">${lastPkt}</span> <span class="info-label">Last Packet</span><span class="info-value">${lastPkt}</span>
</div>`; </div>`;
if (data.connections && data.connections.length) {
html += `<div class="info-thresholds-title">Connectivity</div>`;
for (const c of data.connections) {
html += `<div class="info-note">${escHtml(c.family)} &middot; ${escHtml(c.addr || '—')} RTT</div>
<div id="rtt-chart-${hostname}-${escHtml(c.family)}" style="margin-bottom:8px;"></div>`;
}
html += `<table class="data-table"><thead><tr>
<th>Family</th><th>Address</th><th>State</th>
<th class="num">RTT</th><th>Last Change</th><th>Last Packet</th>
</tr></thead><tbody>`;
for (const c of data.connections) {
const st = escHtml(c.state || '—');
const stCls = c.state === 'up' ? 'status-up' : (c.state === 'absent' ? '' : (c.state ? 'status-down' : ''));
const rtt = (c.state === 'up' && c.rtt) ? Math.round(c.rtt) + ' ms' : '—';
const chg = c.statetime ? new Date(c.statetime * 1000).toLocaleString() : '—';
const seen = c.lastbeat ? new Date(c.lastbeat * 1000).toLocaleString() : '—';
html += `<tr>
<td class="key">${escHtml(c.family)}</td>
<td>${escHtml(c.addr || '—')}</td>
<td><span class="${stCls}">${st}</span></td>
<td class="num">${rtt}</td>
<td>${chg}</td>
<td>${seen}</td>
</tr>`;
}
html += `</tbody></table>`;
}
if (data.thresholds === null) { if (data.thresholds === null) {
html += `<div class="info-note">Threshold alerting not configured.</div>`; html += `<div class="info-note">Threshold alerting not configured.</div>`;
} else if (data.thresholds.length === 0) { } else if (data.thresholds.length === 0) {
@@ -645,6 +499,15 @@
} }
el.innerHTML = html; el.innerHTML = html;
if (data.connections && data.connections.length) {
const rttThresholds = (data.thresholds || []).find(t => t.metric === 'rtt') || null;
for (const c of data.connections) {
fetchRttHistory(hostname, c.family)
.then(samples => renderRttChart(hostname, c.family, samples, rttThresholds))
.catch(() => {});
}
}
} }
async function fetchHostGlance(hostname) { async function fetchHostGlance(hostname) {
@@ -959,84 +822,164 @@
return json.samples || []; return json.samples || [];
} }
function renderCpuChart(hostname, samples) { function renderTimeSeriesChart(elId, pts, opts) {
const el = document.getElementById(`cpu-chart-${hostname}`); const el = document.getElementById(elId);
if (!el || !samples.length) return; if (!el || pts.length < 2) { if (el) el.style.display = 'none'; return; }
const pts = samples const unitSuffix = opts.unitSuffix || '';
.filter(s => s.data.cpu_percent != null) const W = 690, H = 92, PAD = { top: 6, right: 8, bottom: 18, left: 28 };
.map(s => ({ t: s.timestamp, v: s.data.cpu_percent }));
if (pts.length < 2) { el.style.display = 'none'; return; }
const W = 600, H = 80, PAD = { top: 6, right: 8, bottom: 18, left: 28 };
const cW = W - PAD.left - PAD.right;
const cH = H - PAD.top - PAD.bottom;
const tMin = pts[0].t, tMax = pts[pts.length - 1].t; const tMin = pts[0].t, tMax = pts[pts.length - 1].t;
const tRange = tMax - tMin || 1; const tRange = tMax - tMin || 1;
const x = t => PAD.left + ((t - tMin) / tRange) * cW;
// Auto-scale Y axis with 10% padding, clamped to [0, 100] // Auto-scale Y axis with 10% padding, optionally clamped to opts.yDomain
const vMin = Math.min(...pts.map(p => p.v)); const vMin = Math.min(...pts.map(p => p.v));
const vMax = Math.max(...pts.map(p => p.v)); const vMax = Math.max(...pts.map(p => p.v));
const vRange = vMax - vMin || 1; const vRange = vMax - vMin || 1;
const vPad = Math.max(vRange * 0.1, 1); const vPad = Math.max(vRange * 0.1, 1);
const yLow = Math.max(0, vMin - vPad); const domainLow = Array.isArray(opts.yDomain) ? opts.yDomain[0] : 0;
const yHigh = Math.min(100, vMax + vPad); const domainHigh = Array.isArray(opts.yDomain) ? opts.yDomain[1] : Infinity;
const yLow = Math.max(domainLow, vMin - vPad);
const yHigh = Math.min(domainHigh, vMax + vPad);
const yRange = yHigh - yLow || 1; const yRange = yHigh - yLow || 1;
const y = v => PAD.top + cH - ((v - yLow) / yRange) * cH;
// Build polyline points and filled area path // Compute nice tick step for ~3-5 grid lines, then size the left
const linePoints = pts.map(p => `${x(p.t).toFixed(1)},${y(p.v).toFixed(1)}`).join(' '); // margin to fit the widest label (values/units can run to 3+ digits,
const areaPath = `M${x(pts[0].t).toFixed(1)},${(PAD.top + cH).toFixed(1)} ` + // e.g. RTT samples above 99ms) instead of a fixed guess that clips them.
pts.map(p => `L${x(p.t).toFixed(1)},${y(p.v).toFixed(1)}`).join(' ') +
` L${x(pts[pts.length-1].t).toFixed(1)},${(PAD.top + cH).toFixed(1)} Z`;
// Color based on latest absolute CPU %
const latest = pts[pts.length - 1].v;
const strokeColor = latest > 90 ? '#e53935' : latest > 70 ? '#fb8c00' : '#43a047';
const fillColor = latest > 90 ? '#ffcdd2' : latest > 70 ? '#ffe0b2' : '#c8e6c9';
// Compute nice tick step for ~3-5 grid lines
const rawStep = yRange / 4; const rawStep = yRange / 4;
const mag = Math.pow(10, Math.floor(Math.log10(rawStep || 1))); const mag = Math.pow(10, Math.floor(Math.log10(rawStep || 1)));
const niceStep = [1, 2, 5, 10].map(f => f * mag).find(s => yRange / s <= 5) || mag * 10; const niceStep = [1, 2, 5, 10].map(f => f * mag).find(s => yRange / s <= 5) || mag * 10;
const tickStart = Math.ceil(yLow / niceStep) * niceStep; const tickStart = Math.ceil(yLow / niceStep) * niceStep;
let gridLines = ''; const yTicks = [];
for (let v = tickStart; v <= yHigh + 0.001; v += niceStep) { for (let v = tickStart; v <= yHigh + 0.001; v += niceStep) yTicks.push(v);
const yy = y(v).toFixed(1); const yTickLabels = yTicks.map(v => (Number.isInteger(v) ? v : v.toFixed(1)) + unitSuffix);
const label = Number.isInteger(v) ? v : v.toFixed(1); const maxLabelLen = yTickLabels.reduce((m, s) => Math.max(m, s.length), 0);
gridLines += `<line x1="${PAD.left}" y1="${yy}" x2="${PAD.left + cW}" y2="${yy}" stroke="#e0e0e0" stroke-width="1"/>`; PAD.left = Math.max(28, Math.ceil(maxLabelLen * 5.5) + 10);
gridLines += `<text x="${(PAD.left - 3).toFixed(1)}" y="${yy}" text-anchor="end" dominant-baseline="middle" font-size="8" fill="#999">${label}</text>`;
const cW = W - PAD.left - PAD.right;
const cH = H - PAD.top - PAD.bottom;
const x = t => PAD.left + ((t - tMin) / tRange) * cW;
const y = v => PAD.top + cH - ((v - yLow) / yRange) * cH;
// Color based on latest value
const latest = pts[pts.length - 1].v;
const { stroke: strokeColor, fill: fillColor } = opts.colorFor(latest);
// Split into segments wherever the gap between consecutive samples is
// much larger than the typical spacing (e.g. the host was overdue/down
// for a while) — draw each segment separately so missing data reads as
// a visual gap instead of an interpolated line across dead time.
const deltas = [];
for (let i = 1; i < pts.length; i++) deltas.push(pts[i].t - pts[i - 1].t);
deltas.sort((a, b) => a - b);
const medianDelta = deltas[Math.floor(deltas.length / 2)];
const gapThreshold = medianDelta * 2.5;
const segments = [[pts[0]]];
for (let i = 1; i < pts.length; i++) {
if (pts[i].t - pts[i - 1].t > gapThreshold) segments.push([]);
segments[segments.length - 1].push(pts[i]);
} }
// X-axis time labels let linePolylines = '';
let areaPaths = '';
for (const seg of segments) {
if (seg.length < 2) continue;
const segPoints = seg.map(p => `${x(p.t).toFixed(1)},${y(p.v).toFixed(1)}`).join(' ');
linePolylines += `<polyline points="${segPoints}" fill="none" stroke="${strokeColor}" stroke-width="1.5" stroke-linejoin="round"/>`;
const segArea = `M${x(seg[0].t).toFixed(1)},${(PAD.top + cH).toFixed(1)} ` +
seg.map(p => `L${x(p.t).toFixed(1)},${y(p.v).toFixed(1)}`).join(' ') +
` L${x(seg[seg.length-1].t).toFixed(1)},${(PAD.top + cH).toFixed(1)} Z`;
areaPaths += `<path d="${segArea}" fill="${fillColor}" opacity="0.6"/>`;
}
let gridLines = '';
yTicks.forEach((v, i) => {
const yy = y(v).toFixed(1);
gridLines += `<line x1="${PAD.left}" y1="${yy}" x2="${PAD.left + cW}" y2="${yy}" stroke="#e0e0e0" stroke-width="1"/>`;
gridLines += `<text x="${(PAD.left - 3).toFixed(1)}" y="${yy}" text-anchor="end" dominant-baseline="middle" font-size="8" fill="#999">${yTickLabels[i]}</text>`;
});
// X-axis: start/end plus evenly spaced intermediate tickmarks + labels
const fmt = ts => { const fmt = ts => {
const d = new Date(ts * 1000); const d = new Date(ts * 1000);
return d.toLocaleTimeString([], { hour: '2-digit', minute: '2-digit' }); return d.toLocaleTimeString([], { hour: '2-digit', minute: '2-digit' });
}; };
const xLabels = ` const xTickCount = 5;
<text x="${PAD.left}" y="${H - 2}" text-anchor="start" font-size="8" fill="#999">${fmt(pts[0].t)}</text> const xAxisY = (PAD.top + cH).toFixed(1);
<text x="${PAD.left + cW}" y="${H - 2}" text-anchor="end" font-size="8" fill="#999">${fmt(pts[pts.length-1].t)}</text>`; let xAxisMarks = '';
let xLabels = '';
for (let i = 0; i < xTickCount; i++) {
const t = tMin + (tRange * i) / (xTickCount - 1);
const xx = x(t).toFixed(1);
const anchor = i === 0 ? 'start' : (i === xTickCount - 1 ? 'end' : 'middle');
xAxisMarks += `<line x1="${xx}" y1="${xAxisY}" x2="${xx}" y2="${(PAD.top + cH + 3).toFixed(1)}" stroke="#ccc" stroke-width="1"/>`;
xLabels += `<text x="${xx}" y="${H - 2}" text-anchor="${anchor}" font-size="8" fill="#999">${fmt(t)}</text>`;
}
el.innerHTML = `<svg viewBox="0 0 ${W} ${H}" preserveAspectRatio="none" el.innerHTML = `<svg viewBox="0 0 ${W} ${H}" preserveAspectRatio="none"
style="width:100%;height:${H}px;display:block;"> style="width:100%;height:${H}px;display:block;">
<defs> <defs>
<clipPath id="cpu-clip-${hostname}"> <clipPath id="${opts.clipId}">
<rect x="${PAD.left}" y="${PAD.top}" width="${cW}" height="${cH}"/> <rect x="${PAD.left}" y="${PAD.top}" width="${cW}" height="${cH}"/>
</clipPath> </clipPath>
</defs> </defs>
${gridLines} ${gridLines}
<line x1="${PAD.left}" y1="${PAD.top}" x2="${PAD.left}" y2="${PAD.top + cH}" stroke="#ccc" stroke-width="1"/> <line x1="${PAD.left}" y1="${PAD.top}" x2="${PAD.left}" y2="${PAD.top + cH}" stroke="#ccc" stroke-width="1"/>
<line x1="${PAD.left}" y1="${PAD.top + cH}" x2="${PAD.left + cW}" y2="${PAD.top + cH}" stroke="#ccc" stroke-width="1"/> <line x1="${PAD.left}" y1="${PAD.top + cH}" x2="${PAD.left + cW}" y2="${PAD.top + cH}" stroke="#ccc" stroke-width="1"/>
<g clip-path="url(#cpu-clip-${hostname})"> <g clip-path="url(#${opts.clipId})">
<path d="${areaPath}" fill="${fillColor}" opacity="0.6"/> ${areaPaths}
<polyline points="${linePoints}" fill="none" stroke="${strokeColor}" stroke-width="1.5" stroke-linejoin="round"/> ${linePolylines}
</g> </g>
${xAxisMarks}
${xLabels} ${xLabels}
</svg>`; </svg>`;
} }
function renderCpuChart(hostname, samples) {
const pts = samples
.filter(s => s.data.cpu_percent != null)
.map(s => ({ t: s.timestamp, v: s.data.cpu_percent }));
renderTimeSeriesChart(`cpu-chart-${hostname}`, pts, {
yDomain: [0, 100],
clipId: `cpu-clip-${hostname}`,
colorFor: (latest) => ({
stroke: latest > 90 ? '#e53935' : latest > 70 ? '#fb8c00' : '#43a047',
fill: latest > 90 ? '#ffcdd2' : latest > 70 ? '#ffe0b2' : '#c8e6c9',
}),
});
}
async function fetchRttHistory(hostname, family) {
const plugin = `rtt_${family.toLowerCase()}`;
const r = await fetch(`/api/0/hosts/${encodeURIComponent(hostname)}/plugins/${plugin}?limit=100`);
if (!r.ok) return [];
const json = await r.json();
return json.samples || [];
}
function renderRttChart(hostname, family, samples, rttThresholds) {
const pts = samples
.filter(s => s.data.rtt != null && s.data.rtt > 0)
.map(s => ({ t: s.timestamp, v: s.data.rtt }));
renderTimeSeriesChart(`rtt-chart-${hostname}-${family}`, pts, {
yDomain: 'auto',
clipId: `rtt-clip-${hostname}-${family}`,
unitSuffix: ' ms',
colorFor: (latest) => {
if (rttThresholds?.critical != null && latest > rttThresholds.critical) {
return { stroke: '#e53935', fill: '#ffcdd2' };
}
if (rttThresholds?.warning != null && latest > rttThresholds.warning) {
return { stroke: '#fb8c00', fill: '#ffe0b2' };
}
return { stroke: '#1976d2', fill: '#bbdefb' };
},
});
}
function renderCpuTable(hostname, d) { function renderCpuTable(hostname, d) {
const KEYS = [ const KEYS = [
['cpu_percent', 'CPU Usage', 'bar'], ['cpu_percent', 'CPU Usage', 'bar'],
@@ -1200,6 +1143,24 @@
function renderNetworkTables(d) { function renderNetworkTables(d) {
let html = ''; let html = '';
if (d.connectivity) {
let t = '<table class="data-table"><thead><tr><th>Protocol</th><th>Internet</th>'
+ '<th>Address</th><th>Gateway</th><th class="num">RTT</th></tr></thead><tbody>';
for (const fam of ['ipv4', 'ipv6']) {
const c = d.connectivity[fam];
if (!c) continue;
const known = c.present === true || c.present === false;
const cls = c.present === true ? 'status-up' : '';
const txt = c.present === true ? '✓ present' : (c.present === false ? 'absent' : '?');
t += `<tr><td>${fam === 'ipv4' ? 'IPv4' : 'IPv6'}</td>
<td class="${cls}">${txt}</td>
<td>${escHtml(c.addr || '—')}</td><td>${escHtml(c.gateway || '—')}</td>
<td class="num">${known && c.rtt != null ? Math.round(c.rtt) + ' ms' : '—'}</td></tr>`;
}
t += '</tbody></table>';
html += `<div class="table-section-label">Connectivity</div>${t}`;
}
if (d.interfaces && Object.keys(d.interfaces).length) { if (d.interfaces && Object.keys(d.interfaces).length) {
const sample = Object.values(d.interfaces)[0]; const sample = Object.values(d.interfaces)[0];
const hasDelta = 'bytes_sent_delta' in sample; const hasDelta = 'bytes_sent_delta' in sample;
@@ -1443,6 +1404,11 @@
document.querySelectorAll('.host-card:not(.collapsed)').forEach(card => { document.querySelectorAll('.host-card:not(.collapsed)').forEach(card => {
const hostname = card.dataset.hostname; const hostname = card.dataset.hostname;
fetchHostInfo(hostname).then(data => {
infoCache[hostname] = data;
renderInfoSection(hostname, data);
}).catch(() => {});
card.querySelectorAll('.plugin-accordion:not(.collapsed)').forEach(acc => { card.querySelectorAll('.plugin-accordion:not(.collapsed)').forEach(acc => {
const pname = acc.dataset.plugin; const pname = acc.dataset.plugin;
if (!GLANCE_PLUGINS.includes(pname)) { if (!GLANCE_PLUGINS.includes(pname)) {
File diff suppressed because it is too large Load Diff
+2
View File
@@ -1251,6 +1251,7 @@ class ThresholdChecker:
title=title, title=title,
body=body, body=body,
level=lvl, level=lvl,
service=short_path,
), ),
)) ))
@@ -1534,6 +1535,7 @@ class ThresholdChecker:
title=f"[REMINDER/{alert_state.level.name}] {host_name} {short_path}", title=f"[REMINDER/{alert_state.level.name}] {host_name} {short_path}",
body=body, body=body,
level=alert_state.level.name, level=alert_state.level.name,
service=short_path,
), ),
)) ))
logger.info("Re-notification sent: %s", message) logger.info("Re-notification sent: %s", message)
+125 -29
View File
@@ -194,6 +194,40 @@ def _set_connectivity_alert(host, afam, level_name):
state.update(level, level_name) state.update(level, level_name)
_PRESENCE_KEYS = {"p4": "IPv4", "p6": "IPv6"}
def _apply_presence(host, msg, now, uname, ctx, skip_conn):
"""Record client-reported protocol presence and settle the affected connections.
A family reported absent whose connection is already overdue/unknown is parked as
ABSENT (timer cancelled, connectivity alert cleared). One still UP is left alone; its
timer expiry parks it. A family reported present again while ABSENT restarts a grace
timer. The connection that carried this very heartbeat (skip_conn) is never touched.
"""
cfg = ctx.get("config", {})
for key, afam in _PRESENCE_KEYS.items():
if key not in msg:
continue
present = bool(msg[key])
host.set_presence(afam, present, now)
conn = host.connections.get(afam)
if conn is None or conn is skip_conn:
continue
cls = conn.__class__
if not present:
if conn.getstate() in (cls.OVERDUE, cls.UNKNOWN):
conn.cancel_overdue_timer()
conn.newstate(cls.ABSENT, now)
_set_connectivity_alert(host, afam, "OK")
elif conn.getstate() == cls.ABSENT:
conn.newstate(cls.UNKNOWN, now)
interval = int(msg.get("interval", 0) or 0) or host.interval
if interval > 0:
on_overdue, _ = _make_timer_callbacks(uname, host, ctx)
conn.reset_overdue_timer(interval + cfg.get("grace", 2), on_overdue)
def _make_timer_callbacks(uname, host, ctx): def _make_timer_callbacks(uname, host, ctx):
"""Return (on_overdue, on_unknown) async callbacks for connection timer logic. """Return (on_overdue, on_unknown) async callbacks for connection timer logic.
@@ -210,26 +244,52 @@ def _make_timer_callbacks(uname, host, ctx):
msg_to_websockets("host", host.stateinfo()) msg_to_websockets("host", host.stateinfo())
async def on_overdue(connection): async def on_overdue(connection):
if connection.getstate() != connection.__class__.UP: cls = connection.__class__
if connection.getstate() not in (cls.UP, cls.UNKNOWN):
return return
now = time.time() now = time.time()
connection.newstate(connection.__class__.OVERDUE, now, cfg.get("grace", 2)) afam = connection.afam
msg = f"{connection.afam} overdue" # Park only while some other family is still present: the last monitored family
eventlog(uname, "CRITICAL", msg) # must never be parked silently, or a host whose client reports no internet on any
if host.watched: # protocol would die without an alert.
asyncio.create_task(notify_mod.send_notification( if not host.flag_present(afam) and any(
uname, af != afam and host.family_present(af) for af in host.connections):
notify_mod.Notification(title=f"[CRITICAL] {uname}", body=msg, level="CRITICAL"), # The client says this protocol has no connectivity: not an outage.
)) connection.newstate(cls.ABSENT, now)
# Track in alert_states so the Alerts Dashboard shows this connection.cancel_overdue_timer()
_set_connectivity_alert(host, connection.afam, "CRITICAL") _set_connectivity_alert(host, afam, "OK")
if threshold_checker: if msg_to_websockets:
threshold_checker.check_value( msg_to_websockets("host", host.stateinfo())
host_name=uname, return
metric_path="rtt",
value=float("inf"), # If any connection is DOWN (clean shutdown reported) Host.status() is "down", so a
alert_states=host.alert_states, # later overdue of the other family raises no separate event/alert; the shutdown did.
) prev = host.status()
connection.newstate(cls.OVERDUE, now, cfg.get("grace", 2))
new = host.status()
if new == "overdue" and prev != "overdue":
names = ", ".join(sorted(af for af, c in host.connections.items()
if c.getstate() == cls.OVERDUE))
msg = f"{names} overdue"
eventlog(uname, "CRITICAL", msg)
if host.watched:
asyncio.create_task(notify_mod.send_notification(
uname,
notify_mod.Notification(title=f"[CRITICAL] {uname}", body=msg, level="CRITICAL"),
))
# Track in alert_states so the Alerts Dashboard shows this
for af, c in host.connections.items():
if c.getstate() == cls.OVERDUE:
_set_connectivity_alert(host, af, "CRITICAL")
if threshold_checker:
threshold_checker.check_value(
host_name=uname,
metric_path="rtt",
value=float("inf"),
alert_states=host.alert_states,
)
elif new == "simplex":
eventlog(uname, "INFO", f"{afam} overdue (simplex)")
if msg_to_websockets: if msg_to_websockets:
msg_to_websockets("host", host.stateinfo()) msg_to_websockets("host", host.stateinfo())
connection.reset_overdue_timer(DROPOVERDUE, on_unknown) connection.reset_overdue_timer(DROPOVERDUE, on_unknown)
@@ -270,6 +330,8 @@ def restore_connection_timers(hbdclass, ctx):
interval = host.interval interval = host.interval
for afam, conn in list(host.connections.items()): for afam, conn in list(host.connections.items()):
state = conn.getstate() state = conn.getstate()
if state == hbdclass.Connection.ABSENT:
continue
if state == hbdclass.Connection.DOWN: if state == hbdclass.Connection.DOWN:
_set_connectivity_alert(host, afam, "CRITICAL") _set_connectivity_alert(host, afam, "CRITICAL")
continue continue
@@ -295,6 +357,7 @@ def restore_connection_timers(hbdclass, ctx):
restored += 1 restored += 1
elif state == hbdclass.Connection.OVERDUE: elif state == hbdclass.Connection.OVERDUE:
host_overdue = host.status() == "overdue"
elapsed_overdue = now - conn.statetime elapsed_overdue = now - conn.statetime
remaining = DROPOVERDUE - elapsed_overdue remaining = DROPOVERDUE - elapsed_overdue
if remaining <= 1: if remaining <= 1:
@@ -304,6 +367,9 @@ def restore_connection_timers(hbdclass, ctx):
"Marking %s/%s UNKNOWN (overdue %.1f days)", "Marking %s/%s UNKNOWN (overdue %.1f days)",
uname, afam, elapsed_overdue / 86400, uname, afam, elapsed_overdue / 86400,
) )
# A connection dropped to UNKNOWN keeps its alert, like the
# UNKNOWN branch above and on_unknown do.
host_overdue = True
else: else:
conn.reset_overdue_timer(remaining, on_unknown) conn.reset_overdue_timer(remaining, on_unknown)
logger.debug( logger.debug(
@@ -313,12 +379,23 @@ def restore_connection_timers(hbdclass, ctx):
# Ensure the connectivity alert is set — it may be missing if # Ensure the connectivity alert is set — it may be missing if
# hbd was shut down before the on_overdue callback had a chance # hbd was shut down before the on_overdue callback had a chance
# to record it. # to record it.
_set_connectivity_alert(host, afam, "CRITICAL") if host_overdue:
_set_connectivity_alert(host, afam, "CRITICAL")
restored += 1 restored += 1
logger.info("Restored timers for %d connection(s)", restored) logger.info("Restored timers for %d connection(s)", restored)
def _is_rtt_key(plugin_name: str) -> bool:
"""True for synthetic RTT-history keys (rtt_ipv4/rtt_ipv6), not real plugin data."""
return plugin_name.startswith("rtt_")
def _has_real_plugin_data(plugin_data: dict) -> bool:
"""True if plugin_data holds any real (non-RTT) client-collected plugin data."""
return any(not _is_rtt_key(k) for k in plugin_data)
def handle_datagram(msg: dict, addr, transport, ctx: dict): def handle_datagram(msg: dict, addr, transport, ctx: dict):
"""Handle a parsed datagram message. """Handle a parsed datagram message.
@@ -385,7 +462,7 @@ def handle_datagram(msg: dict, addr, transport, ctx: dict):
host.doesack = msg.get("acks", -1) host.doesack = msg.get("acks", -1)
# send ACK back; ask client to resend plugin info when we have none yet # send ACK back; ask client to resend plugin info when we have none yet
rmsg = {"time": time.time()} rmsg = {"time": time.time()}
if not host.plugin_data: if not _has_real_plugin_data(host.plugin_data):
rmsg["request_update"] = 1 rmsg["request_update"] = 1
opkt = dicttos("ACK", rmsg) opkt = dicttos("ACK", rmsg)
try: try:
@@ -507,24 +584,36 @@ def handle_datagram(msg: dict, addr, transport, ctx: dict):
if message: if message:
eventlog(uname, "INFO", message, service=service) eventlog(uname, "INFO", message, service=service)
_apply_presence(host, msg, now, uname, ctx, conn)
prev_status = host.status()
if conn.getstate() != hbdcls.Connection.UP: if conn.getstate() != hbdcls.Connection.UP:
# Transition to UP and log/notify if appropriate # Transition to UP and log/notify if appropriate
lasts = conn.state lasts = conn.state
d = conn.newstate(hbdcls.Connection.UP, now) d = conn.newstate(hbdcls.Connection.UP, now)
# On reboot, pre-boot plugin data and derived alerts are stale. if boot:
# Cancel all plugin timers and wipe plugin state so timers restart # On reboot, pre-boot plugin data and derived alerts are stale.
# cleanly from the first two post-boot samples. # Cancel all plugin timers and wipe plugin state so timers restart
for pname in list(host.plugin_timers): # cleanly from the first two post-boot samples. An ordinary
host.cancel_plugin_timer(pname) # reconnect (no boot flag) doesn't invalidate the client's
host.plugin_data.clear() # already-collected data, so it's left alone — this keeps chart
# history intact across a transient network blip.
for pname in list(host.plugin_timers):
host.cancel_plugin_timer(pname)
for pname in [k for k in host.plugin_data if not _is_rtt_key(k)]:
del host.plugin_data[pname]
stale_plugin_keys = [ stale_plugin_keys = [
k for k in host.alert_states k for k in host.alert_states
if k not in ("rtt",) and not k.startswith("connectivity.") if k not in ("rtt",) and not k.startswith("connectivity.")
] ]
for k in stale_plugin_keys: for k in stale_plugin_keys:
del host.alert_states[k] del host.alert_states[k]
# Clear connectivity alert now that the host is back up # Clear connectivity alert now that the host is back up; once the host is no longer
# overdue no family may keep a connectivity alert (e.g. the simplex case).
_set_connectivity_alert(host, conn.afam, "OK") _set_connectivity_alert(host, conn.afam, "OK")
if host.status() in ("up", "simplex"):
for af in host.connections:
_set_connectivity_alert(host, af, "OK")
# Don't log/notify RECOVER for a brand-new host seen for the first time — # Don't log/notify RECOVER for a brand-new host seen for the first time —
# it was never down, it just hasn't been seen before. # it was never down, it just hasn't been seen before.
if not newh: if not newh:
@@ -536,8 +625,11 @@ def handle_datagram(msg: dict, addr, transport, ctx: dict):
else: else:
m = "%s back after being %s for %s" % (conn.afam, lasts, dur(d)) m = "%s back after being %s for %s" % (conn.afam, lasts, dur(d))
if m: if m:
eventlog(uname, "RECOVER", m) # RECOVER (and a notification) only if the host itself was overdue/down;
if host.watched: # a single family coming back while another stayed up is informational.
level = "RECOVER" if prev_status in ("overdue", "down", "unknown") else "INFO"
eventlog(uname, level, m)
if host.watched and level == "RECOVER":
asyncio.create_task(notify_mod.send_notification( asyncio.create_task(notify_mod.send_notification(
uname, uname,
notify_mod.Notification(title=f"[RECOVER] {uname}", body=m, level="RECOVER"), notify_mod.Notification(title=f"[RECOVER] {uname}", body=m, level="RECOVER"),
@@ -562,6 +654,10 @@ def handle_datagram(msg: dict, addr, transport, ctx: dict):
if interval > 0: if interval > 0:
host.interval = interval host.interval = interval
# Record RTT history for charting
if rtt is not None:
host.add_plugin_data(f"rtt_{conn.afam.lower()}", {"rtt": rtt}, timestamp=now)
# Timer-based reachability monitoring # Timer-based reachability monitoring
# Reset overdue timer on every heartbeat # Reset overdue timer on every heartbeat
if interval > 0 and conn.getstate() != hbdcls.Connection.DOWN: if interval > 0 and conn.getstate() != hbdcls.Connection.DOWN:
+1 -1
View File
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
[project] [project]
name = "hbd" name = "hbd"
version = "5.3.12" version = "5.4.4"
description = "Heartbeat monitoring system — client (hbc) and server (hbd)" description = "Heartbeat monitoring system — client (hbc) and server (hbd)"
readme = "README.md" readme = "README.md"
requires-python = ">=3.11" requires-python = ">=3.11"
+14 -1
View File
@@ -375,6 +375,10 @@ static const char *jstr(const jval_t *v, const char *def) {
* Config * Config
* ============================================================ */ * ============================================================ */
/* Set from config in cfg_load. Off by default: a CMD packet is an unauthenticated
* UDP datagram, so executing one must be opted into per host. */
static bool g_allow_remote_command = false;
typedef struct { typedef struct {
int hb_port, interval; int hb_port, interval;
char owner[256]; char owner[256];
@@ -429,6 +433,9 @@ static void config_load(config_t *cfg, const char *path) {
if ((v = jget(root, "hb_port"))) cfg->hb_port = jint(v, cfg->hb_port); if ((v = jget(root, "hb_port"))) cfg->hb_port = jint(v, cfg->hb_port);
if ((v = jget(root, "interval"))) cfg->interval = jint(v, cfg->interval); if ((v = jget(root, "interval"))) cfg->interval = jint(v, cfg->interval);
if ((v = jget(root, "owner"))) snprintf(cfg->owner, sizeof(cfg->owner), "%s", jstr(v, "")); if ((v = jget(root, "owner"))) snprintf(cfg->owner, sizeof(cfg->owner), "%s", jstr(v, ""));
if ((v = jget(root, "allow_remote_command"))) g_allow_remote_command = jint(v, 0) != 0;
if (g_allow_remote_command)
LOGI("allow_remote_command is true - CMD packets from the server will be executed");
jval_t *plugins = jget(root, "plugins"); jval_t *plugins = jget(root, "plugins");
@@ -591,7 +598,13 @@ static void conn_recv(conn_t *c) {
LOGD("ACK rtt=%.1fms", c->rtt); LOGD("ACK rtt=%.1fms", c->rtt);
} else if (strcmp(id, "CMD") == 0) { } else if (strcmp(id, "CMD") == 0) {
const char *cmd = kv_get(&msg, "cmd"); const char *cmd = kv_get(&msg, "cmd");
if (cmd) { if (cmd && !g_allow_remote_command) {
LOGI("refused command (allow_remote_command is false): %s", cmd);
kvdict_t rep; kv_clear(&rep);
kv_set(&rep, "service", "command");
kv_set(&rep, "msg", "Refused: allow_remote_command is false");
conn_send(c, "HTB", &rep);
} else if (cmd) {
LOGI("CMD: %s", cmd); LOGI("CMD: %s", cmd);
char out[4096] = ""; char out[4096] = "";
FILE *p = popen(cmd, "r"); FILE *p = popen(cmd, "r");
+15 -1
View File
@@ -41,7 +41,7 @@ from pathlib import Path
from typing import Any, Dict, List, Optional, Tuple from typing import Any, Dict, List, Optional, Tuple
# updated by scripts/bumpminor.sh # updated by scripts/bumpminor.sh
__version__ = "5.3.12" __version__ = "5.4.4"
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# Protocol (mirrors hbd/common/proto.py) # Protocol (mirrors hbd/common/proto.py)
@@ -115,9 +115,14 @@ _DEFAULTS: Dict[str, Any] = {
"hb_port": 50003, "hb_port": 50003,
"interval": 10, "interval": 10,
"owner": None, "owner": None,
"allow_remote_command": False, # Execute shell commands received in CMD packets
"plugins": {}, "plugins": {},
} }
# Set from config in main(). Off by default: a CMD packet is an unauthenticated
# UDP datagram, so executing one must be opted into per host.
_allow_remote_command = False
def _load_config(path: Optional[str] = None) -> Dict[str, Any]: def _load_config(path: Optional[str] = None) -> Dict[str, Any]:
cfg = dict(_DEFAULTS) cfg = dict(_DEFAULTS)
@@ -870,6 +875,10 @@ async def _handle_command(conn: AsyncConnection, msg: Dict[str, Any]):
if not cmd: if not cmd:
return return
log = logging.getLogger("hbc.cmd") log = logging.getLogger("hbc.cmd")
if not _allow_remote_command:
log.warning("refused command (allow_remote_command is false): %s", cmd)
await conn.sendto({"service": "command", "msg": "Refused: allow_remote_command is false"})
return
log.info("exec: %s", cmd) log.info("exec: %s", cmd)
try: try:
out = subprocess.check_output( out = subprocess.check_output(
@@ -1180,6 +1189,11 @@ def main(argv=None):
cfg = _load_config(args.configfile) cfg = _load_config(args.configfile)
global _allow_remote_command
_allow_remote_command = bool(cfg.get("allow_remote_command", False))
if _allow_remote_command:
logging.warning("allow_remote_command is true — CMD packets from the server will be executed")
if args.daemon: if args.daemon:
_daemonize() _daemonize()
_reconfigure_syslog(level) _reconfigure_syslog(level)
+14
View File
@@ -112,9 +112,14 @@ _DEFAULTS: Dict[str, Any] = {
"hb_port": 50003, "hb_port": 50003,
"interval": 10, "interval": 10,
"owner": None, "owner": None,
"allow_remote_command": False, # Execute shell commands received in CMD packets
"plugins": {}, "plugins": {},
} }
# Set from config in main(). Off by default: a CMD packet is an unauthenticated
# UDP datagram, so executing one must be opted into per host.
_allow_remote_command = False
def _load_config(path: Optional[str] = None) -> Dict[str, Any]: def _load_config(path: Optional[str] = None) -> Dict[str, Any]:
cfg = dict(_DEFAULTS) cfg = dict(_DEFAULTS)
@@ -886,6 +891,10 @@ async def _handle_command(conn: AsyncConnection, msg: Dict[str, Any]):
if not cmd: if not cmd:
return return
log = logging.getLogger("hbc.cmd") log = logging.getLogger("hbc.cmd")
if not _allow_remote_command:
log.warning("refused command (allow_remote_command is false): %s", cmd)
await conn.sendto({"service": "command", "msg": "Refused: allow_remote_command is false"})
return
log.info("exec: %s", cmd) log.info("exec: %s", cmd)
try: try:
out = subprocess.check_output( out = subprocess.check_output(
@@ -1180,6 +1189,11 @@ def main(argv=None):
cfg = _load_config(args.configfile) cfg = _load_config(args.configfile)
global _allow_remote_command
_allow_remote_command = bool(cfg.get("allow_remote_command", False))
if _allow_remote_command:
logging.warning("allow_remote_command is true — CMD packets from the server will be executed")
try: try:
rc = asyncio.run(_async_main(args, cfg)) rc = asyncio.run(_async_main(args, cfg))
except KeyboardInterrupt: except KeyboardInterrupt:
+103
View File
@@ -0,0 +1,103 @@
"""Tests for the allow_remote_command gate on CMD packets in hbc."""
import asyncio
import subprocess
import pytest
from hbd.client import main as client_main
from hbd.client.config import CLIENT_DEFAULTS, load_config
class FakeConn:
"""Stand-in for AsyncConnection that records what the client sends back."""
def __init__(self):
self.sent = []
async def sendto(self, msg, msg_id="HTB"):
self.sent.append(msg)
@pytest.fixture
def conn(monkeypatch):
"""A fake connection with remote commands disabled and subprocess booby-trapped."""
monkeypatch.setattr(client_main, "allow_remote_command", False)
def explode(*a, **kw):
raise AssertionError("subprocess must not run when the command is refused")
monkeypatch.setattr(subprocess, "check_output", explode)
return FakeConn()
def test_default_is_false():
assert CLIENT_DEFAULTS["allow_remote_command"] is False
assert load_config("/nonexistent/hbc.yaml")["allow_remote_command"] is False
assert client_main.allow_remote_command is False
def test_refused_when_disabled(conn):
asyncio.run(client_main.handle_command(conn, {"cmd": "id"}))
assert conn.sent == [{
"service": "command",
"msg": "Refused: allow_remote_command is false",
}]
def test_refusal_reported_for_every_attempt(conn):
for _ in range(3):
asyncio.run(client_main.handle_command(conn, {"cmd": "rm -rf /"}))
assert len(conn.sent) == 3
def test_empty_command_is_ignored_either_way(conn, monkeypatch):
asyncio.run(client_main.handle_command(conn, {}))
monkeypatch.setattr(client_main, "allow_remote_command", True)
asyncio.run(client_main.handle_command(conn, {"cmd": ""}))
assert conn.sent == []
def test_executed_when_enabled(conn, monkeypatch):
monkeypatch.setattr(client_main, "allow_remote_command", True)
monkeypatch.setattr(subprocess, "check_output", lambda *a, **kw: b"uid=0(root)")
asyncio.run(client_main.handle_command(conn, {"cmd": "id"}))
assert conn.sent == [{"service": "command", "msg": "OK uid=0(root)"}]
def test_failure_reported_when_enabled(conn, monkeypatch):
monkeypatch.setattr(client_main, "allow_remote_command", True)
def fail(*a, **kw):
raise subprocess.CalledProcessError(1, "false")
monkeypatch.setattr(subprocess, "check_output", fail)
asyncio.run(client_main.handle_command(conn, {"cmd": "false"}))
assert len(conn.sent) == 1
assert conn.sent[0]["msg"].startswith("CalledProcessError ")
def test_config_file_can_opt_in(tmp_path):
cfg_file = tmp_path / "hbc.yaml"
cfg_file.write_text("allow_remote_command: true\n", encoding="utf-8")
assert load_config(str(cfg_file))["allow_remote_command"] is True
@pytest.mark.parametrize("configured, expected", [(True, True), (False, False), (None, False)])
def test_async_main_sets_the_flag_from_config(monkeypatch, configured, expected):
"""async_main must apply the config value before any CMD packet can arrive."""
monkeypatch.setattr(client_main, "allow_remote_command", not expected)
class Bail(Exception):
pass
# gethostname() is the first call after the flag is assigned — bail there so the
# test never opens a socket.
def bail():
raise Bail
monkeypatch.setattr(client_main.socket, "gethostname", bail)
cfg = {} if configured is None else {"allow_remote_command": configured}
with pytest.raises(Bail):
asyncio.run(client_main.async_main(object(), cfg))
assert client_main.allow_remote_command is expected
+28
View File
@@ -0,0 +1,28 @@
from hbd.client.main import build_heartbeat_msg
class _Conn:
ackcount = 3
rtts = [0, 12.5]
class _Conn2:
def __init__(self, flags):
self._flags = flags
def flags(self):
return self._flags
def test_heartbeat_without_connectivity_is_unchanged():
assert build_heartbeat_msg(_Conn(), 60) == {"acks": 3, "rtt": 12.5, "interval": 60}
def test_heartbeat_carries_presence_flags():
msg = build_heartbeat_msg(_Conn(), 60, _Conn2({"p4": True, "p6": False}))
assert msg == {"acks": 3, "rtt": 12.5, "interval": 60, "p4": True, "p6": False}
def test_heartbeat_omits_unknown_flags():
msg = build_heartbeat_msg(_Conn(), 60, _Conn2({}))
assert "p4" not in msg and "p6" not in msg
+173
View File
@@ -0,0 +1,173 @@
import asyncio
from hbd.client.connectivity import Connectivity, ProbeUnavailable
from hbd.client.pingutil import PingUnavailable
class Net:
"""Scriptable network: addr None = no route, 'unavailable' = family unsupported."""
def __init__(self):
self.addr = {"ipv4": "192.0.2.5", "ipv6": "2001:db8::5"}
self.gw = {"ipv4": "192.0.2.1", "ipv6": "fe80::1"}
self.rtt = {"ipv4": 10.0, "ipv6": 20.0} # None = ping fails, 'unavailable' = no ping binary
self.pings = []
self.now = 1000.0
def local(self, fam, target):
a = self.addr[fam]
if a == "unavailable":
raise ProbeUnavailable("no such family")
return a
async def ping(self, target, fam):
self.pings.append(fam)
r = self.rtt[fam]
if r == "unavailable":
raise PingUnavailable("no ping")
return r
async def gateway(self, fam):
return self.gw[fam]
def clock(self):
return self.now
def make(net, **kw):
c = Connectivity(local_address_fn=net.local, ping_fn=net.ping, gateway_fn=net.gateway,
clock=net.clock, **kw)
c._cheap_gateway = True
return c
def refresh(c):
asyncio.run(c.refresh())
def test_present_with_addr_gateway_rtt():
net = Net()
c = make(net)
refresh(c)
s = c.snapshot()["ipv4"]
assert (s.present, s.addr, s.gateway, s.rtt) == (True, "192.0.2.5", "192.0.2.1", 10.0)
assert c.flags() == {"p4": True, "p6": True}
assert c.as_metric()["ipv6"]["gateway"] == "fe80::1"
def test_no_route_is_immediately_absent():
net = Net()
net.addr["ipv6"] = None
c = make(net)
refresh(c)
assert c.flags() == {"p4": True, "p6": False}
assert net.pings == ["ipv4"] # no ping wasted on a family without a route
assert c.snapshot()["ipv6"].addr is None
def test_ping_is_debounced_and_rate_limited():
net = Net()
c = make(net, probe_interval=60.0, retry_interval=10.0)
refresh(c)
refresh(c) # same instant: cached, no second ping
assert net.pings == ["ipv4", "ipv6"]
net.now += 60
net.rtt["ipv6"] = None # first failure: still present (debounce)
refresh(c)
assert c.flags()["p6"] is True
net.now += 10 # retry_interval elapsed: second failure flips it
refresh(c)
assert c.flags()["p6"] is False
def test_first_probe_failing_once_is_unknown():
net = Net()
net.rtt["ipv4"] = None
c = make(net)
refresh(c)
assert "p4" not in c.flags() and c.flags()["p6"] is True
def test_recovers_on_first_success():
net = Net()
net.rtt["ipv6"] = None
c = make(net, retry_interval=10.0)
refresh(c)
net.now += 10
refresh(c)
assert c.flags()["p6"] is False
net.rtt["ipv6"] = 25.0
net.now += 10
refresh(c)
assert c.flags()["p6"] is True
def test_address_change_invalidates_cache():
net = Net()
c = make(net)
refresh(c)
net.addr["ipv4"] = "198.51.100.7" # moved networks, no time has passed
refresh(c)
assert net.pings.count("ipv4") == 2
assert c.snapshot()["ipv4"].addr == "198.51.100.7"
def test_gateway_change_invalidates_cache():
net = Net()
c = make(net)
refresh(c)
net.gw["ipv4"] = "192.0.2.254"
refresh(c)
assert net.pings.count("ipv4") == 2
def test_route_loss_then_return_pings_immediately():
net = Net()
c = make(net)
refresh(c)
net.addr["ipv6"] = None
refresh(c)
net.addr["ipv6"] = "2001:db8::5"
refresh(c)
assert net.pings.count("ipv6") == 2 and c.flags()["p6"] is True
def test_missing_ping_binary_leaves_presence_unknown_and_stops_retrying():
net = Net()
net.rtt["ipv4"] = "unavailable"
c = make(net)
refresh(c)
net.now += 600
refresh(c)
assert "p4" not in c.flags()
assert net.pings.count("ipv4") == 1
def test_unsupported_family_is_unknown():
net = Net()
net.addr["ipv6"] = "unavailable"
c = make(net)
refresh(c)
assert "p6" not in c.flags() and c.flags()["p4"] is True
def test_from_config():
c = Connectivity.from_config({"probe_v4": "9.9.9.9", "probe_interval": 30})
assert c.probes["ipv4"] == "9.9.9.9"
assert c.probes["ipv6"] == "2606:4700:4700::1111"
assert c.probe_interval == 30.0
def test_ping_less_family_stays_unknown_after_route_returns():
net = Net()
net.rtt["ipv4"] = "unavailable"
c = make(net)
refresh(c)
assert "p4" not in c.flags()
net.addr["ipv4"] = None
refresh(c)
assert c.flags()["p4"] is False # no route is definitive
net.addr["ipv4"] = "192.0.2.5"
refresh(c)
assert "p4" not in c.flags()
assert net.pings.count("ipv4") == 1 # missing binary is not retried
+265
View File
@@ -0,0 +1,265 @@
"""Tests for the dedicated events journal (write path and read path)."""
import asyncio
import json
from hbd.server import journal
def _make_journal(tmp_path, **overrides):
cfg = {"journal_dir": str(tmp_path), "journal_file": "events.journal"}
cfg.update(overrides)
j = journal.MessageJournal(cfg)
assert asyncio.run(j.initialize())
return j
def _read_lines(tmp_path, name="events.journal"):
return (tmp_path / name).read_text(encoding="utf-8").splitlines()
EV1 = {"ts": 1000.0, "host": "h1", "level": "INFO", "service": None, "message": "host up"}
EV2 = {"ts": 2000.0, "host": "h2", "level": "CRITICAL", "service": "cpu", "message": "cpu high"}
def test_log_event_writes_one_json_line(tmp_path):
j = _make_journal(tmp_path)
asyncio.run(j.log_event(EV1))
asyncio.run(j.close())
lines = _read_lines(tmp_path)
assert len(lines) == 1
assert json.loads(lines[0]) == EV1
def test_log_event_appends_in_order(tmp_path):
j = _make_journal(tmp_path)
asyncio.run(j.log_event(EV1))
asyncio.run(j.log_event(EV2))
asyncio.run(j.close())
lines = _read_lines(tmp_path)
assert [json.loads(ln)["ts"] for ln in lines] == [1000.0, 2000.0]
def test_log_event_rotates_at_max_size(tmp_path):
# max_size fits one serialized event line (~77 bytes) but not two, so the
# second write triggers exactly one rotation
j = _make_journal(tmp_path, journal_max_size=120)
asyncio.run(j.log_event(EV1))
asyncio.run(j.log_event(EV2))
asyncio.run(j.close())
backups = list(tmp_path.glob("events.journal.*"))
assert len(backups) == 1
assert json.loads(backups[0].read_text().splitlines()[0]) == EV1
assert json.loads(_read_lines(tmp_path)[0]) == EV2
def test_log_event_noop_when_disabled(tmp_path):
j = journal.MessageJournal(
{"journal_dir": str(tmp_path), "journal_file": "events.journal", "journal_enabled": False}
)
asyncio.run(j.initialize())
asyncio.run(j.log_event(EV1))
assert not (tmp_path / "events.journal").exists()
def test_backfill_seeds_empty_journal(tmp_path):
j = _make_journal(tmp_path)
asyncio.run(j.backfill([EV1, EV2]))
asyncio.run(j.close())
lines = _read_lines(tmp_path)
assert len(lines) == 2
assert json.loads(lines[0]) == EV1
def test_backfill_skipped_when_journal_nonempty(tmp_path):
(tmp_path / "events.journal").write_text(json.dumps(EV1) + "\n")
j = _make_journal(tmp_path) # initialize() picks up the existing size
asyncio.run(j.backfill([EV2]))
asyncio.run(j.close())
assert len(_read_lines(tmp_path)) == 1
def test_get_events_journal_uses_events_config_keys(tmp_path):
journal._events_journal_instance = None
try:
ej = journal.get_events_journal(
{
"journal_dir": str(tmp_path),
"events_journal_file": "ev.jsonl",
"events_journal_max_size": 12345,
"events_journal_max_backups": 3,
}
)
assert ej.journal_file == "ev.jsonl"
assert ej.max_size == 12345
assert ej.max_backups == 3
assert ej.journal_dir == tmp_path
# singleton: second call returns the same instance
assert journal.get_events_journal() is ej
finally:
journal._events_journal_instance = None
def test_get_events_journal_defaults(tmp_path):
journal._events_journal_instance = None
try:
ej = journal.get_events_journal({"journal_dir": str(tmp_path)})
assert ej.journal_file == "events.journal"
assert ej.max_size == 10 * 1024 * 1024
assert ej.max_backups == 10
finally:
journal._events_journal_instance = None
# ---- read path -------------------------------------------------------------
def _write_journal(path, events):
path.write_text("".join(json.dumps(e) + "\n" for e in events), encoding="utf-8")
def _evts(*ts_list):
return [
{"ts": float(t), "host": f"host{i}", "level": "INFO", "service": None, "message": f"msg {t}"}
for i, t in enumerate(ts_list)
]
def test_read_events_newest_first(tmp_path):
_write_journal(tmp_path / "events.journal", _evts(1, 2, 3))
events, more = journal.read_events(tmp_path)
assert [e["ts"] for e in events] == [3.0, 2.0, 1.0]
assert more is False
def test_read_events_limit_and_more(tmp_path):
_write_journal(tmp_path / "events.journal", _evts(1, 2, 3))
events, more = journal.read_events(tmp_path, limit=2)
assert [e["ts"] for e in events] == [3.0, 2.0]
assert more is True
def test_read_events_before_cursor(tmp_path):
_write_journal(tmp_path / "events.journal", _evts(1, 2, 3))
events, _ = journal.read_events(tmp_path, before=3.0)
assert [e["ts"] for e in events] == [2.0, 1.0]
def test_read_events_spans_rotated_files(tmp_path):
# rotated backup holds the oldest events; current file the newest
_write_journal(tmp_path / "events.journal.20260101-000000", _evts(1, 2))
_write_journal(tmp_path / "events.journal.20260201-000000", _evts(3, 4))
_write_journal(tmp_path / "events.journal", _evts(5, 6))
events, more = journal.read_events(tmp_path, limit=10)
assert [e["ts"] for e in events] == [6.0, 5.0, 4.0, 3.0, 2.0, 1.0]
assert more is False
def test_read_events_pagination_across_files(tmp_path):
_write_journal(tmp_path / "events.journal.20260101-000000", _evts(1, 2))
_write_journal(tmp_path / "events.journal", _evts(3, 4))
page1, more1 = journal.read_events(tmp_path, limit=3)
assert [e["ts"] for e in page1] == [4.0, 3.0, 2.0]
assert more1 is True
page2, more2 = journal.read_events(tmp_path, limit=3, before=page1[-1]["ts"])
assert [e["ts"] for e in page2] == [1.0]
assert more2 is False
def test_read_events_host_filter_substring_case_insensitive(tmp_path):
evs = [
{"ts": 1.0, "host": "Wentworth", "level": "INFO", "service": None, "message": "a"},
{"ts": 2.0, "host": "winter", "level": "INFO", "service": None, "message": "b"},
]
_write_journal(tmp_path / "events.journal", evs)
events, _ = journal.read_events(tmp_path, host="went")
assert [e["host"] for e in events] == ["Wentworth"]
def test_read_events_level_filter_exact_case_insensitive(tmp_path):
evs = [
{"ts": 1.0, "host": "h", "level": "CRITICAL", "service": None, "message": "a"},
{"ts": 2.0, "host": "h", "level": "INFO", "service": None, "message": "b"},
]
_write_journal(tmp_path / "events.journal", evs)
events, _ = journal.read_events(tmp_path, level="critical")
assert [e["level"] for e in events] == ["CRITICAL"]
def test_read_events_message_filter(tmp_path):
evs = [
{"ts": 1.0, "host": "h", "level": "INFO", "service": None, "message": "disk almost full"},
{"ts": 2.0, "host": "h", "level": "INFO", "service": None, "message": "all quiet"},
]
_write_journal(tmp_path / "events.journal", evs)
events, _ = journal.read_events(tmp_path, q="Disk")
assert [e["ts"] for e in events] == [1.0]
def test_read_events_skips_malformed_lines(tmp_path):
p = tmp_path / "events.journal"
p.write_text('{"ts": 1.0, "host": "h", "level": "INFO", "message": "ok"}\nnot json\n[1,2]\n')
events, _ = journal.read_events(tmp_path)
assert [e["ts"] for e in events] == [1.0]
def test_read_events_predicate(tmp_path):
_write_journal(tmp_path / "events.journal", _evts(1, 2))
events, _ = journal.read_events(tmp_path, predicate=lambda e: e["host"] == "host0")
assert [e["host"] for e in events] == ["host0"]
def test_read_events_missing_dir(tmp_path):
events, more = journal.read_events(tmp_path / "nope")
assert events == [] and more is False
def test_filter_events_over_in_memory_ring():
ring = _evts(1, 2, 3) # oldest-first, like data.msgs
events, more = journal.filter_events(reversed(ring), limit=2)
assert [e["ts"] for e in events] == [3.0, 2.0]
assert more is True
def test_filter_events_skips_non_dict_entries():
ring = [
{"ts": 1.0, "host": "h", "level": "INFO", "service": None, "message": "a"},
"legacy string entry",
None,
{"ts": 2.0, "host": "h", "level": "INFO", "service": None, "message": "b"},
]
events, more = journal.filter_events(reversed(ring), limit=10)
assert [e["ts"] for e in events] == [2.0, 1.0]
assert more is False
# ---- eventlog wiring --------------------------------------------------------
def test_eventlog_writes_to_events_journal(tmp_path):
from hbd.server import data, notify
journal._events_journal_instance = None
saved_msgs = data.msgs
data.msgs = []
try:
ej = journal.get_events_journal({"journal_dir": str(tmp_path)})
async def scenario():
await ej.initialize()
notify.setup({}, loop=asyncio.get_running_loop())
notify.eventlog("h1", "INFO", "hello world")
await asyncio.sleep(0.05) # let the scheduled journal write run
await ej.close()
asyncio.run(scenario())
lines = _read_lines(tmp_path)
assert len(lines) == 1
ev = json.loads(lines[0])
assert ev["host"] == "h1"
assert ev["level"] == "INFO"
assert ev["message"] == "hello world"
assert isinstance(ev["ts"], float)
finally:
journal._events_journal_instance = None
data.msgs = saved_msgs
notify._loop = None
+217
View File
@@ -0,0 +1,217 @@
"""Tests for flap detection (hbd.server.flap) and its hook in notify.send_notification."""
import asyncio
import json
import pytest
from hbd.server import flap, hbdclass, notify, users as users_mod
@pytest.fixture(autouse=True)
def clean_flap_state():
"""Every test starts with an empty state and a 3-in-10-minutes config."""
flap._state.clear()
flap.setup({"flap_count": 3, "flap_interval": 10})
yield
flap._state.clear()
flap.setup({})
def alerts(n, host="h1", service="cpu", level="CRITICAL"):
return [flap.observe(host, service, level) for _ in range(n)]
def advance(seconds, host="h1", service="cpu"):
"""Backdate a key's recorded timestamps to simulate the clock moving on."""
st = flap._state[(host, service)]
st["events"] = [t - seconds for t in st["events"]]
if st["ok_since"] is not None:
st["ok_since"] -= seconds
# --- tripping ---------------------------------------------------------------
def test_alerts_up_to_flap_count_pass():
assert alerts(3) == [flap.PASS] * 3
assert flap.flapping_services("h1") == []
def test_exceeding_flap_count_trips_then_suppresses():
assert alerts(5) == [flap.PASS, flap.PASS, flap.PASS, flap.TRIP, flap.SUPPRESS]
assert flap.flapping_services("h1") == ["cpu"]
def test_warning_and_critical_both_count():
flap.observe("h1", "cpu", "WARNING")
flap.observe("h1", "cpu", "CRITICAL")
flap.observe("h1", "cpu", "WARNING")
assert flap.observe("h1", "cpu", "CRITICAL") == flap.TRIP
def test_alerts_outside_the_window_do_not_count():
alerts(3)
advance(11 * 60) # older than flap_interval
assert alerts(3) == [flap.PASS] * 3
assert flap.flapping_services("h1") == []
def test_recover_alone_never_trips():
assert [flap.observe("h1", "cpu", "RECOVER") for _ in range(5)] == [flap.PASS] * 5
# --- suppression while flapping --------------------------------------------
def test_recover_and_info_are_suppressed_while_flapping():
alerts(4)
assert flap.observe("h1", "cpu", "RECOVER") == flap.SUPPRESS
assert flap.observe("h1", "cpu", "INFO") == flap.SUPPRESS
def test_info_passes_when_not_flapping():
assert flap.observe("h1", "cpu", "INFO") == flap.PASS
# --- clearing ---------------------------------------------------------------
def test_clears_silently_one_interval_after_recover():
alerts(4)
flap.observe("h1", "cpu", "RECOVER")
advance(10 * 60)
assert flap.flapping_services("h1") == []
assert flap.observe("h1", "cpu", "CRITICAL") == flap.PASS
def test_still_flapping_before_the_interval_elapses():
alerts(4)
flap.observe("h1", "cpu", "RECOVER")
advance(9 * 60)
assert flap.flapping_services("h1") == ["cpu"]
assert flap.observe("h1", "cpu", "CRITICAL") == flap.SUPPRESS
def test_alert_during_the_quiet_window_keeps_it_flapping():
alerts(4)
flap.observe("h1", "cpu", "RECOVER")
advance(9 * 60)
flap.observe("h1", "cpu", "CRITICAL") # resets the quiet window
advance(2 * 60)
assert flap.flapping_services("h1") == ["cpu"]
# --- host removal ------------------------------------------------------------
def test_clear_host_drops_flapping_state_even_without_a_recover():
# A host dropped mid-flap (no RECOVER ever received) must not stay
# flapping forever: nothing will ever set ok_since for it again.
alerts(4)
assert flap.flapping_services("h1") == ["cpu"]
flap.clear_host("h1")
assert flap.flapping_services("h1") == []
assert flap.observe("h1", "cpu", "CRITICAL") == flap.PASS
def test_clear_host_only_affects_the_named_host():
alerts(4, host="h1")
alerts(4, host="h2")
flap.clear_host("h1")
assert flap.flapping_services("h1") == []
assert flap.flapping_services("h2") == ["cpu"]
# --- keying -----------------------------------------------------------------
def test_services_and_hosts_are_tracked_independently():
alerts(4, service="cpu")
assert flap.observe("h1", "disk", "CRITICAL") == flap.PASS
assert flap.observe("h2", "cpu", "CRITICAL") == flap.PASS
assert flap.flapping_services("h1") == ["cpu"]
assert flap.flapping_services("h2") == []
def test_host_level_events_use_the_empty_service_key():
alerts(4, service="")
assert flap.flapping_services("h1") == [""]
# --- disabled ---------------------------------------------------------------
@pytest.mark.parametrize("cfg", [
{"flap_count": 0, "flap_interval": 10},
{"flap_count": 3, "flap_interval": 0},
{},
])
def test_disabled_never_suppresses(cfg):
flap.setup(cfg)
assert alerts(20) == [flap.PASS] * 20
assert flap.flapping_services("h1") == []
# --- integration through the real dispatch path -----------------------------
NOTIFY_CFG = {
"flap_count": 3,
"flap_interval": 10,
"notification_channels": {"ch1": {"type": "pushover", "token": "t", "user": "u"}},
"users": {"alice": {"notification_channels": ["ch1"]}},
}
@pytest.fixture
def delivered(monkeypatch):
"""Wire up a host, a user and a stub channel driver; collect what gets delivered."""
sent = []
monkeypatch.setattr(notify, "_config", dict(notify._config)) # restored on teardown
notify.setup(NOTIFY_CFG)
users_mod.load_users(NOTIFY_CFG)
monkeypatch.setitem(notify._DRIVERS, "pushover", lambda cfg, n: sent.append(n.body) or True)
host = hbdclass.Host("flaphost")
host.watched = True
host.owner = "alice"
yield sent, host
hbdclass.Host.hosts.pop("flaphost", None)
users_mod.load_users({})
def _notify(level, service, body):
n = notify.Notification(title=f"[{level}] flaphost", body=body, level=level, service=service)
asyncio.run(notify.send_notification("flaphost", n))
def test_marker_on_the_tripping_notification_then_nothing(delivered):
sent, host = delivered
for i in range(5):
_notify("CRITICAL", "cpu", f"cpu = 9{i}")
_notify("RECOVER", "cpu", "cpu = 12")
assert sent == ["cpu = 90", "cpu = 91", "cpu = 92", f"cpu = 93 {flap.FLAP_MARKER}"]
assert host.stateinfo()["flapping"] == ["cpu"]
def test_a_flapping_service_does_not_silence_another(delivered):
sent, host = delivered
for i in range(5):
_notify("CRITICAL", "cpu", "cpu = 99")
sent.clear()
_notify("CRITICAL", "disk", "disk = 91")
assert sent == ["disk = 91"]
def test_notifications_resume_after_the_state_clears(delivered):
sent, host = delivered
for i in range(5):
_notify("CRITICAL", "cpu", "cpu = 99")
_notify("RECOVER", "cpu", "cpu = 12")
advance(10 * 60, host="flaphost")
assert host.stateinfo()["flapping"] == []
sent.clear()
_notify("CRITICAL", "cpu", "cpu = 99")
assert sent == ["cpu = 99"]
def test_stateinfo_flapping_survives_json_encoding(delivered):
sent, host = delivered
for i in range(5):
_notify("CRITICAL", "", "IPv4 overdue") # host-level, empty service key
info = host.stateinfo()
assert info["flapping"] == [""]
assert json.loads(json.dumps(info))["flapping"] == [""]
+75
View File
@@ -0,0 +1,75 @@
import asyncio
import struct
from hbd.client import gateway
PROC_V4 = (
"Iface\tDestination\tGateway \tFlags\tRefCnt\tUse\tMetric\tMask\t\tMTU\tWindow\tIRTT\n"
"wlan0\t00000000\t0100A8C0\t0003\t0\t0\t600\t00000000\t0\t0\t0\n"
"eth0\t00000000\t0101A8C0\t0003\t0\t0\t100\t00000000\t0\t0\t0\n"
"wlan0\t0000A8C0\t00000000\t0001\t0\t0\t600\t00FFFFFF\t0\t0\t0\n"
)
PROC_V6 = (
"00000000000000000000000000000000 00 00000000000000000000000000000000 00 "
"fe800000000000000000000000000001 00000400 00000001 00000000 00000003 wlan0\n"
"00000000000000000000000000000000 00 00000000000000000000000000000000 00 "
"00000000000000000000000000000000 ffffffff 00000001 00000000 00200200 lo\n"
"fe800000000000000000000000000000 40 00000000000000000000000000000000 00 "
"00000000000000000000000000000000 00000100 00000001 00000000 00000001 wlan0\n"
)
MACOS = """ route to: default
destination: default
mask: default
gateway: 192.168.1.1
interface: en0
"""
WIN_V4 = """IPv4 Route Table
Active Routes:
Network Destination Netmask Gateway Interface Metric
0.0.0.0 0.0.0.0 192.168.1.1 192.168.1.50 25
0.0.0.0 0.0.0.0 10.0.0.1 10.0.0.9 50
127.0.0.0 255.0.0.0 On-link 127.0.0.1 331
"""
WIN_V6 = """IPv6 Route Table
Active Routes:
If Metric Network Destination Gateway
12 281 ::/0 fe80::1
1 331 ::1/128 On-link
"""
def test_proc_v4_picks_lowest_metric_gateway():
assert gateway.parse_proc_route_v4(PROC_V4) == "192.168.1.1" # eth0, metric 100
def test_proc_v4_none_without_default_route():
assert gateway.parse_proc_route_v4(PROC_V4.splitlines()[0] + "\n") is None
def test_proc_v6_skips_reject_and_non_default():
assert gateway.parse_proc_route_v6(PROC_V6) == "fe80::1"
def test_macos():
assert gateway.parse_macos_route(MACOS) == "192.168.1.1"
assert gateway.parse_macos_route("route: writing to routing socket: not in table") is None
def test_windows_v4_lowest_metric():
assert gateway.parse_windows_route(WIN_V4, "ipv4") == "192.168.1.1"
def test_windows_v6():
assert gateway.parse_windows_route(WIN_V6, "ipv6") == "fe80::1"
def test_default_gateway_never_raises_on_parser_error(monkeypatch):
"""default_gateway catches all exceptions; never raises even if parser fails."""
monkeypatch.setattr("sys.platform", "linux")
monkeypatch.setattr(gateway, "parse_proc_route_v4", lambda _: (_ for _ in ()).throw(struct.error))
result = asyncio.run(gateway.default_gateway("ipv4"))
assert result is None
+120
View File
@@ -0,0 +1,120 @@
import time
import pytest
from hbd.server import hbdclass
from hbd.server.hbdclass import Connection as C
def _host(name):
hbdclass.Host.hosts.pop(name, None)
return hbdclass.Host(name)
def _conn(host, afam, state):
addr = "192.0.2.1" if afam == "IPv4" else "2001:db8::1"
conn, _ = host.conndata(0, addr, None, 1000.0)
conn.state = state
return conn
CASES = [
# v4 state, v6 state, presence flags, expected status
(C.UP, C.UP, {}, "up"),
(C.UP, C.OVERDUE, {}, "simplex"), # both present (legacy) => simplex
(C.UP, C.OVERDUE, {"IPv6": True}, "simplex"),
(C.UP, C.OVERDUE, {"IPv6": False}, "up"), # v6 not present => not monitored
(C.OVERDUE, C.OVERDUE, {}, "overdue"),
(C.OVERDUE, C.ABSENT, {"IPv6": False}, "overdue"), # only present family is overdue
(C.UP, C.ABSENT, {"IPv6": False}, "up"),
(C.DOWN, C.UP, {}, "down"),
(C.UNKNOWN, C.UNKNOWN, {}, "unknown"),
(C.OVERDUE, C.UNKNOWN, {}, "overdue"),
(C.UP, C.UNKNOWN, {}, "up"),
(C.OVERDUE, C.UP, {"IPv6": False}, "simplex"), # HTB arriving => present despite flag
]
@pytest.mark.parametrize("s4,s6,flags,expected", CASES)
def test_status_matrix(s4, s6, flags, expected):
h = _host("status-matrix")
_conn(h, "IPv4", s4)
_conn(h, "IPv6", s6)
for afam, present in flags.items():
h.set_presence(afam, present, time.time())
assert h.status() == expected
def test_status_single_family_host():
h = _host("status-v4only")
_conn(h, "IPv4", C.UP)
assert h.status() == "up"
h.connections["IPv4"].state = C.OVERDUE
assert h.status() == "overdue"
def test_status_no_connections_is_unknown():
assert _host("status-empty").status() == "unknown"
def test_family_present_rules():
h = _host("fp")
v6 = _conn(h, "IPv6", C.OVERDUE)
assert h.family_present("IPv6") is True # never reported => present
h.set_presence("IPv6", False, 1.0)
assert h.family_present("IPv6") is False
v6.state = C.UP # real traffic overrides the flag
assert h.family_present("IPv6") is True
assert h.flag_present("IPv6") is False
def test_stateinfo_and_statedict_expose_status():
h = _host("status-exposed")
_conn(h, "IPv4", C.UP)
assert h.stateinfo()["status"] == "up"
assert h.statedict()["status"] == "up"
def test_absent_connection_statedict_is_neutral():
h = _host("absent-dict")
conn = _conn(h, "IPv6", C.ABSENT)
conn.rtts.append(42.0)
d = conn.statedict()
assert d["state"] == "absent" and d["rtt"] == "" and d["rttstate"] == "absent"
def test_presence_survives_pickle_and_old_pickles_get_default():
import pickle
h = _host("pickled")
h.set_presence("IPv6", False, 5.0)
h2 = pickle.loads(pickle.dumps(h))
assert h2.presence == {"IPv6": {"present": False, "ts": 5.0}}
del h2.__dict__["presence"] # simulate a pre-feature pickle
h2.fixup()
assert h2.presence == {}
def _all_absent(name, s4, s6=None):
h = _host(name)
_conn(h, "IPv4", s4)
h.set_presence("IPv4", False, time.time())
if s6 is not None:
_conn(h, "IPv6", s6)
h.set_presence("IPv6", False, time.time())
return h
def test_all_absent_single_family_overdue_is_overdue():
assert _all_absent("abs-single", C.OVERDUE).status() == "overdue"
def test_all_absent_dual_stack_one_absent_one_overdue_is_overdue():
assert _all_absent("abs-dual-od", C.ABSENT, C.OVERDUE).status() == "overdue"
def test_all_absent_dual_stack_v6_up_is_up():
assert _all_absent("abs-dual-up", C.ABSENT, C.UP).status() == "up"
def test_all_absent_all_unknown_is_unknown():
assert _all_absent("abs-unknown", C.UNKNOWN, C.UNKNOWN).status() == "unknown"
+39 -1
View File
@@ -1,5 +1,4 @@
"""Tests for _build_host_info helper in http.py.""" """Tests for _build_host_info helper in http.py."""
import pytest
from unittest.mock import MagicMock from unittest.mock import MagicMock
from hbd.server.http import _build_host_info from hbd.server.http import _build_host_info
@@ -172,3 +171,42 @@ def test_build_host_info_covers_empty_when_exact_matches_only():
result = _build_host_info(host, threshold_checker=checker) result = _build_host_info(host, threshold_checker=checker)
t = result["thresholds"][0] t = result["thresholds"][0]
assert t["covers"] == [] assert t["covers"] == []
class _FakeConnFull:
def __init__(self, afam, addr, state, rtt, statetime, lastbeat, rtts=None):
self.afam = afam
self.addr = addr
self.state = state
self.rtts = rtts if rtts is not None else [rtt]
self.statetime = statetime
self.lastbeat = lastbeat
def test_build_host_info_includes_connections():
host = _FakeHost(connections={
"IPv4": _FakeConnFull("IPv4", "10.0.0.5", "up", 14.2, 1000.0, 2000.0),
"IPv6": _FakeConnFull("IPv6", "fd00::5", "overdue", 0, 1500.0, 1900.0),
})
result = _build_host_info(host)
conns = {c["family"]: c for c in result["connections"]}
assert conns["IPv4"] == {"family": "IPv4", "addr": "10.0.0.5", "state": "up",
"rtt": 14.2, "statetime": 1000.0, "lastbeat": 2000.0}
assert conns["IPv6"]["state"] == "overdue"
def test_build_host_info_connections_empty():
host = _FakeHost()
result = _build_host_info(host)
assert result["connections"] == []
def test_build_host_info_connection_rtt_uses_latest_sample():
"""rtt should reflect the most recent sample, not the oldest retained one."""
host = _FakeHost(connections={
"IPv4": _FakeConnFull("IPv4", "10.0.0.5", "up", None, 1000.0, 2000.0,
rtts=[100.0, 50.0, 14.2]),
})
result = _build_host_info(host)
conns = {c["family"]: c for c in result["connections"]}
assert conns["IPv4"]["rtt"] == 14.2
+23
View File
@@ -0,0 +1,23 @@
"""Tests for the plugin-accordion visibility filter in http.py."""
from hbd.server.http import _visible_plugin_names
def test_visible_plugin_names_excludes_rtt_history_keys():
plugin_data = {
"cpu_monitor": [],
"rtt_ipv4": [],
"rtt_ipv6": [],
"network_monitor": [],
}
result = _visible_plugin_names(plugin_data)
assert sorted(result) == ["cpu_monitor", "network_monitor"]
def test_visible_plugin_names_empty_when_no_plugins():
assert _visible_plugin_names({}) == []
def test_visible_plugin_names_keeps_names_not_prefixed_with_rtt_():
plugin_data = {"nagios_runner": [], "os_info": []}
result = _visible_plugin_names(plugin_data)
assert sorted(result) == ["nagios_runner", "os_info"]
@@ -0,0 +1,26 @@
import asyncio
from hbd.client import connectivity
from hbd.client.plugins.network_monitor import NetworkMonitorPlugin
class _FakeShared:
def as_metric(self):
return {"ipv4": {"present": True, "addr": "192.0.2.5", "gateway": "192.0.2.1", "rtt": 9.0},
"ipv6": {"present": False, "addr": None, "gateway": None, "rtt": None}}
def test_collect_includes_connectivity_when_probe_running():
connectivity.set_shared(_FakeShared())
try:
data = asyncio.run(NetworkMonitorPlugin()._collect_metrics())
finally:
connectivity.set_shared(None)
assert data["connectivity"]["ipv4"]["present"] is True
assert data["connectivity"]["ipv6"]["present"] is False
def test_collect_omits_connectivity_without_probe():
connectivity.set_shared(None)
data = asyncio.run(NetworkMonitorPlugin()._collect_metrics())
assert "connectivity" not in data
+60
View File
@@ -0,0 +1,60 @@
import sys
import pytest
from hbd.client import pingutil
LINUX_OK = """PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
64 bytes from 1.1.1.1: icmp_seq=1 ttl=57 time=11.2 ms
--- 1.1.1.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 11.234/11.234/11.234/0.000 ms
"""
MACOS_OK = """PING 1.1.1.1 (1.1.1.1): 56 data bytes
64 bytes from 1.1.1.1: icmp_seq=0 ttl=57 time=12.3 ms
--- 1.1.1.1 ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 12.3/12.3/12.3/0.000 ms
"""
LOST = """--- 1.1.1.1 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms
"""
def test_parse_linux():
r = pingutil.parse_ping_output(LINUX_OK)
assert r == {"rtt_min": 11.234, "rtt_avg": 11.234, "rtt_max": 11.234, "loss": 0.0}
def test_parse_macos():
r = pingutil.parse_ping_output(MACOS_OK)
assert r["rtt_avg"] == 12.3 and r["loss"] == 0.0
def test_parse_lost():
r = pingutil.parse_ping_output(LOST)
assert r["rtt_avg"] == float("inf") and r["loss"] == 100.0
def test_parse_garbage_is_unreachable():
r = pingutil.parse_ping_output("ping: connect: Network is unreachable")
assert r["rtt_avg"] == float("inf") and r["loss"] == 100.0
@pytest.mark.parametrize("platform,family,expected", [
("linux", None, ["ping", "-c", "3", "-W", "5", "h"]),
("linux", 6, ["ping", "-c", "3", "-W", "5", "-6", "h"]),
("linux", 4, ["ping", "-c", "3", "-W", "5", "-4", "h"]),
("win32", None, ["ping", "-n", "3", "-w", "5000", "h"]),
("win32", 6, ["ping", "-n", "3", "-w", "5000", "-6", "h"]),
("darwin", None, ["ping", "-c", "3", "-W", "5", "h"]),
("darwin", 4, ["ping", "-c", "3", "-W", "5000", "h"]),
("darwin", 6, ["ping6", "-c", "3", "h"]),
])
def test_ping_command(monkeypatch, platform, family, expected):
monkeypatch.setattr(sys, "platform", platform)
assert pingutil.ping_command("h", 3, 5, family) == expected
+250
View File
@@ -0,0 +1,250 @@
"""Host-level overdue alerting driven by client-reported protocol presence."""
import asyncio
import time
from hbd.common.proto import dicttos
from hbd.server import hbdclass, udp
from hbd.server.hbdclass import Connection as C
from hbd.server.udp import handle_datagram, parse_message
class _FakeTransport:
def sendto(self, data, addr):
pass
def _ctx():
return {"config": {}, "hbdclass": hbdclass, "msg_to_websockets": None,
"DEBUG": 0, "verbose": False}
def _fresh(name):
hbdclass.Host.hosts.pop(name, None)
def _send(name, addr, ctx, **fields):
msg = parse_message(dicttos("HTB", {"name": name, "id": 0, **fields}))
handle_datagram(msg, (addr, 50000), _FakeTransport(), ctx)
def _events(monkeypatch):
events = []
monkeypatch.setattr(udp, "eventlog", lambda u, level, msg, **kw: events.append((level, msg)))
return events
def _dual_stack(name, ctx):
_fresh(name)
_send(name, "127.0.0.1", ctx)
_send(name, "::1", ctx)
return hbdclass.Host.hosts[name]
def test_flags_are_stored_on_host():
ctx = _ctx()
_fresh("pres-store")
_send("pres-store", "127.0.0.1", ctx, p4=1, p6=0)
h = hbdclass.Host.hosts["pres-store"]
assert h.presence["IPv4"]["present"] is True
assert h.presence["IPv6"]["present"] is False
def test_no_flags_leaves_presence_unknown():
ctx = _ctx()
_fresh("pres-legacy")
_send("pres-legacy", "127.0.0.1", ctx)
assert hbdclass.Host.hosts["pres-legacy"].presence == {}
def test_absent_flag_moves_overdue_family_to_absent_and_clears_alert(monkeypatch):
_events(monkeypatch)
ctx = _ctx()
h = _dual_stack("pres-absent", ctx)
v6 = h.connections["IPv6"]
v6.state = C.OVERDUE
udp._set_connectivity_alert(h, "IPv6", "CRITICAL")
_send("pres-absent", "127.0.0.1", ctx, p4=1, p6=0)
assert v6.state == C.ABSENT
assert "connectivity.IPv6" not in h.alert_states
assert h.status() == "up"
def test_absent_flag_does_not_touch_family_that_is_up():
ctx = _ctx()
h = _dual_stack("pres-up-stays", ctx)
_send("pres-up-stays", "127.0.0.1", ctx, p4=1, p6=0)
assert h.connections["IPv6"].state == C.UP # traffic overrides the flag
def test_present_again_arms_grace_timer():
ctx = _ctx()
async def scenario():
_fresh("pres-back")
_send("pres-back", "127.0.0.1", ctx, interval=10)
_send("pres-back", "::1", ctx, interval=10)
h = hbdclass.Host.hosts["pres-back"]
v6 = h.connections["IPv6"]
v6.cancel_overdue_timer()
v6.state = C.ABSENT
_send("pres-back", "127.0.0.1", ctx, interval=10, p6=1)
assert v6.state == C.UNKNOWN and v6.overdue_timer is not None
for c in h.connections.values():
c.cancel_overdue_timer()
asyncio.run(scenario())
def test_one_family_overdue_is_simplex_all_overdue_is_critical(monkeypatch):
events = _events(monkeypatch)
ctx = _ctx()
h = _dual_stack("pres-simplex", ctx)
on_overdue, _ = udp._make_timer_callbacks("pres-simplex", h, ctx)
asyncio.run(on_overdue(h.connections["IPv6"]))
assert h.status() == "simplex"
assert ("INFO", "IPv6 overdue (simplex)") in events
assert not any(level == "CRITICAL" for level, _ in events)
assert "connectivity.IPv6" not in h.alert_states
asyncio.run(on_overdue(h.connections["IPv4"]))
assert h.status() == "overdue"
assert [e for e in events if e[0] == "CRITICAL"] == [("CRITICAL", "IPv4, IPv6 overdue")]
assert {"connectivity.IPv4", "connectivity.IPv6"} <= set(h.alert_states)
def test_timer_expiry_on_absent_family_is_silent(monkeypatch):
events = _events(monkeypatch)
ctx = _ctx()
h = _dual_stack("pres-quiet", ctx)
_send("pres-quiet", "127.0.0.1", ctx, p4=1, p6=0) # v6 still UP => left alone
events.clear()
on_overdue, _ = udp._make_timer_callbacks("pres-quiet", h, ctx)
asyncio.run(on_overdue(h.connections["IPv6"]))
assert h.connections["IPv6"].state == C.ABSENT
assert events == []
assert h.status() == "up"
def test_recovery_from_simplex_is_info_only(monkeypatch):
events = _events(monkeypatch)
ctx = _ctx()
h = _dual_stack("pres-rec-simplex", ctx)
v6 = h.connections["IPv6"]
v6.state, v6.statetime = C.OVERDUE, time.time() - 100
events.clear()
_send("pres-rec-simplex", "::1", ctx)
assert v6.state == C.UP
assert [e[0] for e in events] == ["INFO"] and "back after" in events[0][1]
def test_recovery_from_overdue_is_recover_and_clears_all_alerts(monkeypatch):
events = _events(monkeypatch)
ctx = _ctx()
h = _dual_stack("pres-rec-overdue", ctx)
for afam in ("IPv4", "IPv6"):
c = h.connections[afam]
c.state, c.statetime = C.OVERDUE, time.time() - 100
udp._set_connectivity_alert(h, afam, "CRITICAL")
events.clear()
_send("pres-rec-overdue", "127.0.0.1", ctx)
assert [e[0] for e in events] == ["RECOVER"]
assert h.status() == "simplex"
assert "connectivity.IPv6" not in h.alert_states # left overdue => all cleared
assert "connectivity.IPv4" not in h.alert_states
def test_restore_skips_absent_and_sets_no_alert_for_simplex():
ctx = _ctx()
async def scenario():
h = _dual_stack("pres-restore", ctx)
h.connections["IPv6"].state = C.OVERDUE # v4 UP + v6 OVERDUE => simplex
udp.restore_connection_timers(hbdclass, ctx)
assert "connectivity.IPv6" not in h.alert_states
h.connections["IPv6"].cancel_overdue_timer()
h.connections["IPv6"].state = C.ABSENT
h.set_presence("IPv6", False, time.time())
udp.restore_connection_timers(hbdclass, ctx)
assert h.connections["IPv6"].state == C.ABSENT
for c in h.connections.values():
c.cancel_overdue_timer()
asyncio.run(scenario())
def _past_drop(conn):
conn.state, conn.statetime = C.OVERDUE, time.time() - (udp.DROPOVERDUE + 10)
def test_restore_past_drop_window_keeps_alert_single_family():
ctx = _ctx()
async def scenario():
_fresh("pres-drop-single")
_send("pres-drop-single", "127.0.0.1", ctx)
h = hbdclass.Host.hosts["pres-drop-single"]
_past_drop(h.connections["IPv4"])
udp.restore_connection_timers(hbdclass, ctx)
assert h.connections["IPv4"].state == C.UNKNOWN
assert "connectivity.IPv4" in h.alert_states
asyncio.run(scenario())
def test_restore_past_drop_window_keeps_alerts_dual_family():
ctx = _ctx()
async def scenario():
h = _dual_stack("pres-drop-dual", ctx)
for c in h.connections.values():
_past_drop(c)
udp.restore_connection_timers(hbdclass, ctx)
assert {"connectivity.IPv4", "connectivity.IPv6"} <= set(h.alert_states)
asyncio.run(scenario())
def test_v4_only_host_overdue_is_single_critical(monkeypatch):
events = _events(monkeypatch)
ctx = _ctx()
_fresh("pres-v4only")
_send("pres-v4only", "127.0.0.1", ctx)
h = hbdclass.Host.hosts["pres-v4only"]
on_overdue, _ = udp._make_timer_callbacks("pres-v4only", h, ctx)
asyncio.run(on_overdue(h.connections["IPv4"]))
assert [e for e in events if e[0] == "CRITICAL"] == [("CRITICAL", "IPv4 overdue")]
assert "connectivity.IPv4" in h.alert_states
asyncio.run(on_overdue(h.connections["IPv4"]))
assert len([e for e in events if e[0] == "CRITICAL"]) == 1
def test_all_families_absent_v4_only_host_still_alerts_when_it_dies(monkeypatch):
events = _events(monkeypatch)
ctx = _ctx()
_fresh("pres-allabsent")
_send("pres-allabsent", "127.0.0.1", ctx, p4=0)
h = hbdclass.Host.hosts["pres-allabsent"]
on_overdue, _ = udp._make_timer_callbacks("pres-allabsent", h, ctx)
asyncio.run(on_overdue(h.connections["IPv4"]))
assert [e for e in events if e[0] == "CRITICAL"] == [("CRITICAL", "IPv4 overdue")]
assert h.connections["IPv4"].state == C.OVERDUE
assert h.status() == "overdue"
assert "connectivity.IPv4" in h.alert_states
def test_last_present_family_is_not_parked_when_other_is_absent(monkeypatch):
events = _events(monkeypatch)
ctx = _ctx()
h = _dual_stack("pres-lastfam", ctx)
_send("pres-lastfam", "127.0.0.1", ctx, p4=0, p6=0)
events.clear()
on_overdue, _ = udp._make_timer_callbacks("pres-lastfam", h, ctx)
asyncio.run(on_overdue(h.connections["IPv4"]))
assert h.connections["IPv4"].state == C.ABSENT
assert not any(level == "CRITICAL" for level, _ in events)
assert h.status() == "up"
asyncio.run(on_overdue(h.connections["IPv6"]))
assert [e for e in events if e[0] == "CRITICAL"] == [("CRITICAL", "IPv6 overdue")]
assert h.connections["IPv6"].state == C.OVERDUE
assert h.status() == "overdue"
+153
View File
@@ -0,0 +1,153 @@
"""Tests for RTT history capture in udp.py's handle_datagram."""
import time
from hbd.common.proto import dicttos
from hbd.server import hbdclass
from hbd.server.udp import handle_datagram, parse_message
class _FakeTransport:
def __init__(self):
self.sent = []
def sendto(self, data, addr):
self.sent.append((data, addr))
def _htb(name, rtt=None, interval=0, boot=0):
d = {"name": name, "interval": interval, "id": 0}
if rtt is not None:
d["rtt"] = rtt
if boot:
d["boot"] = boot
return parse_message(dicttos("HTB", d))
def _base_ctx():
return {
"config": {},
"hbdclass": hbdclass,
"msg_to_websockets": None,
"DEBUG": 0,
"verbose": False,
}
def test_handle_datagram_records_rtt_history_for_new_connection():
hbdclass.Host.hosts.pop("rtt-hist-host", None)
handle_datagram(_htb("rtt-hist-host", rtt=42.5), ("127.0.0.1", 50000),
_FakeTransport(), _base_ctx())
host = hbdclass.Host.hosts["rtt-hist-host"]
samples = host.plugin_data.get("rtt_ipv4")
assert samples is not None
assert len(samples) == 1
ts, data = samples[0]
assert data == {"rtt": 42.5}
assert isinstance(ts, float)
def test_handle_datagram_appends_rtt_history_across_heartbeats():
hbdclass.Host.hosts.pop("rtt-hist-host2", None)
transport = _FakeTransport()
ctx = _base_ctx()
handle_datagram(_htb("rtt-hist-host2", rtt=10.0), ("127.0.0.1", 50000), transport, ctx)
handle_datagram(_htb("rtt-hist-host2", rtt=20.0), ("127.0.0.1", 50000), transport, ctx)
host = hbdclass.Host.hosts["rtt-hist-host2"]
samples = host.plugin_data["rtt_ipv4"]
assert [d["rtt"] for _, d in samples] == [10.0, 20.0]
def test_handle_datagram_skips_rtt_history_when_rtt_missing():
hbdclass.Host.hosts.pop("rtt-hist-host3", None)
handle_datagram(_htb("rtt-hist-host3", rtt=None), ("127.0.0.1", 50000),
_FakeTransport(), _base_ctx())
host = hbdclass.Host.hosts["rtt-hist-host3"]
assert "rtt_ipv4" not in host.plugin_data
def test_request_update_fires_on_recovery_even_with_rtt_history():
"""Regression for Finding 1: rtt_* keys must not permanently disable the
request_update gate. A connection recovering from a non-UP state must
still be asked to resend real plugin data, even though rtt_ipv4 already
holds samples from before the drop.
"""
hbdclass.Host.hosts.pop("rtt-hist-host4", None)
transport = _FakeTransport()
ctx = _base_ctx()
# First heartbeat: brand-new host, no plugin data at all yet.
handle_datagram(_htb("rtt-hist-host4", rtt=15.0), ("127.0.0.1", 50000), transport, ctx)
host = hbdclass.Host.hosts["rtt-hist-host4"]
assert host.plugin_data.get("rtt_ipv4") # rtt history now non-empty
# Simulate a recovery: connection was dropped (e.g. OVERDUE->UP after a
# missed heartbeat) and is about to come back UP on the next heartbeat.
conn = host.connections["IPv4"]
conn.state = hbdclass.Connection.DOWN
transport.sent.clear()
handle_datagram(_htb("rtt-hist-host4", rtt=16.0), ("127.0.0.1", 50000), transport, ctx)
ack_data, _ = transport.sent[0]
ack = parse_message(ack_data)
assert ack.get("request_update")
def test_ordinary_recovery_preserves_real_plugin_data_and_rtt_history():
"""An ordinary reconnect (no boot flag) — e.g. OVERDUE->UP after a
transient network blip — must NOT wipe already-collected real plugin
data (e.g. cpu_monitor, os_info) or rtt_* history. Only an actual
client reboot invalidates that data (see the boot-flag test below).
"""
hbdclass.Host.hosts.pop("rtt-hist-host5", None)
transport = _FakeTransport()
ctx = _base_ctx()
for rtt in (10.0, 11.0, 12.0):
handle_datagram(_htb("rtt-hist-host5", rtt=rtt), ("127.0.0.1", 50000), transport, ctx)
host = hbdclass.Host.hosts["rtt-hist-host5"]
assert len(host.plugin_data["rtt_ipv4"]) == 3
# Simulate a drop, and pretend the client had previously sent real
# plugin data (collected before the connection went down).
conn = host.connections["IPv4"]
conn.state = hbdclass.Connection.DOWN
host.add_plugin_data("os_info", {"os": "linux"}, timestamp=time.time())
assert "os_info" in host.plugin_data
# Ordinary recovery heartbeat — no boot flag.
handle_datagram(_htb("rtt-hist-host5", rtt=13.0), ("127.0.0.1", 50000), transport, ctx)
assert "os_info" in host.plugin_data
assert len(host.plugin_data["rtt_ipv4"]) == 4
def test_boot_recovery_clears_real_plugin_data_but_preserves_rtt_history():
"""A recovery heartbeat carrying the boot flag (client process actually
restarted) must still wipe stale real plugin data, while rtt_* history
(still a valid measurement, unaffected by a client reboot) survives.
"""
hbdclass.Host.hosts.pop("rtt-hist-host6", None)
transport = _FakeTransport()
ctx = _base_ctx()
for rtt in (10.0, 11.0, 12.0):
handle_datagram(_htb("rtt-hist-host6", rtt=rtt), ("127.0.0.1", 50000), transport, ctx)
host = hbdclass.Host.hosts["rtt-hist-host6"]
assert len(host.plugin_data["rtt_ipv4"]) == 3
conn = host.connections["IPv4"]
conn.state = hbdclass.Connection.DOWN
host.add_plugin_data("os_info", {"os": "linux"}, timestamp=time.time())
assert "os_info" in host.plugin_data
# Recovery heartbeat with boot=1 — client process actually restarted.
handle_datagram(_htb("rtt-hist-host6", rtt=13.0, boot=1), ("127.0.0.1", 50000), transport, ctx)
assert "os_info" not in host.plugin_data
assert len(host.plugin_data["rtt_ipv4"]) == 4