Compare commits

..
20 Commits
Author SHA1 Message Date
Andreas Wrede 1f4dd7cb1f version 5.4.0
Release / release (push) Successful in 44s
2026-07-23 13:17:33 -07:00
Andreas WredeandClaude Opus 4.8 ae3f2fc70f feat: gate remote command execution behind allow_remote_command
CMD packets arrive as unauthenticated UDP datagrams, yet every hbc client
executed the shell command they carry without any opt-in. Add an
allow_remote_command config key, default false: when off, the command is
logged and refused with "Refused: allow_remote_command is false" (visible in
the server event log under the command service), and subprocess is never
reached. When on, the client warns at startup that it will execute CMD
packets.

Applied to all four clients that handle CMD — hbc, hbc_windows.py,
hbc_mini.py, and the C hbc_mini — since gating only one leaves the others
wide open. The C client reads the same key from ~/.hbc.json and needs a
rebuild to pick it up. UPD (self-update) is unchanged.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-23 13:10:41 -07:00
Andreas WredeandClaude Opus 4.8 4414967bdc feat: flapping detection suppresses notification storms
A (host, service) pair is flapping once it exceeds flap_count warning or
critical notifications within flap_interval minutes. The notification that
trips the state carries "Now flapping!! No more messages!" and every later
one is dropped, including RECOVER. The state ends silently flap_interval
minutes after a RECOVER, provided no further alert arrived meanwhile.

Hooked into notify.send_notification, the single choke point for channel
delivery, so only outbound notifications are suppressed — eventlog keeps
recording, leaving the journal and /log with the full history of the flap.

Threshold alerts key on their metric path, so a flapping disk check cannot
silence a CPU alert; connectivity, boot and shutdown events key on the host
itself. State lives at module level in flap.py and is never pickled.

Flapping pairs surface in Host.stateinfo() and render as an amber badge on
the live dashboard. Config: flap_count (5), flap_interval (10 minutes),
0 in either disables; both editable on the settings page.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-23 09:53:46 -07:00
Andreas Wrede e3b0e5041f add: info on deplyment 2026-07-23 09:51:32 -07:00
andreasandClaude Fable 5 92982c0ca7 fix: harden events log — non-dict ring entries, off-loop journal reads, no default-config singleton
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-10 15:51:34 -04:00
andreasandClaude Fable 5 00fdbcaa5b fix: skip journal scheduling when the event loop is not running
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-10 15:39:42 -04:00
andreasandClaude Fable 5 600e68b509 feat: remove log section from live dashboard (moved to /log)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-10 15:29:39 -04:00
andreasandClaude Fable 5 b26e853a1c fix: dedup log rows between API seed and websocket tail
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-10 15:25:01 -04:00
andreasandClaude Fable 5 36adad5b3b feat: dedicated /log page with journal-backed history and nav item
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-10 15:20:17 -04:00
andreasandClaude Fable 5 7591268a89 feat: GET /api/0/log — paged, filtered events journal API
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-10 15:15:46 -04:00
andreasandClaude Fable 5 7afc06a329 fix: journal startup/shutdown lifecycle events (fired outside journal lifetime)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-10 15:11:30 -04:00
andreasandClaude Fable 5 a3f303e6ba feat: eventlog writes to dedicated events journal; init/backfill/close wiring
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-10 15:03:48 -04:00
andreasandClaude Fable 5 16c2922bea feat: events journal read path with filters and backward pagination
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-10 14:57:32 -04:00
andreasandClaude Fable 5 0343eb4a24 fix: drop typing names not yet used from journal.py import
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-10 14:53:24 -04:00
andreasandClaude Fable 5 fb5570155b feat: events journal write primitives (log_event, backfill, get_events_journal)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-10 14:47:42 -04:00
andreasandClaude Fable 5 8d268adf70 feat: Live Dashboard redesign — 6 columns, hover details, last-alert column
Replaces the 11-column table with severity-sorted record rows: host,
combined status chip, latency, alert chips, time-in-state, and the most
recent alert message. Per-family address/state/latency/last-change move
to a hover card on the status chip and permanently to a new Connectivity
table in the Host Overview info section (host info API now includes
connections). Last-alert is seeded from /api/0/alerts and kept fresh
from the event stream; the event log below uses the same row idiom with
its filters intact. WS reconnect state shows in the toolbar instead of
a modal.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NfPpSpccTWBfZg1FTveyaU
2026-07-10 13:45:03 -04:00
andreasandClaude Fable 5 5d9f161706 feat: extend the record-row design system to Host Overview, Alerts, and About
Extract the settings redesign's tokens and components into shared
static/hbd-ui.css + hbd-ui.js and dedupe settings.html against them.
About becomes yaml-key sections with kv rows; Alerts gets stat tiles,
chip filters, and alert record rows (same fetch/ack logic); Host
Overview keeps its DOM and live-update JS but is re-skinned to the
token system with a page toolbar. Live Dashboard intentionally
untouched pending its own redesign.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NfPpSpccTWBfZg1FTveyaU
2026-07-10 09:37:58 -04:00
andreasandClaude Fable 5 067feacd59 fix: settings toolbar sticks below the site nav instead of scrolling away
Sticky offsets (toolbar, rail, mobile chips, anchor margins) derive from
the measured nav/toolbar heights via CSS variables, since the fixed nav's
height varies with viewport width.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NfPpSpccTWBfZg1FTveyaU
2026-07-10 09:19:21 -04:00
andreasandClaude Fable 5 c564fafe73 fix: settings page not scrollable — restore html/body overflow override
static/style.css pins html/body to height:100% with overflow:hidden for
the dashboard pages; the redesign dropped the counter-override the old
template carried.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NfPpSpccTWBfZg1FTveyaU
2026-07-10 08:20:21 -04:00
andreasandClaude Fable 5 8eeae48964 feat: unified record-row redesign of the settings page
One idiom for every section: mono identifier, badges, fact chips, inline
accordion editor. Scalar config sections merge into one grouped 'server'
section. Sidebar becomes YAML-key nav (mobile: horizontal chips); sticky
toolbar carries pending-changes state and Publish. Staging, publish,
channel CRUD, and permission gating are unchanged — same payload shapes
and endpoints.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NfPpSpccTWBfZg1FTveyaU
2026-07-10 08:08:58 -04:00
35 changed files with 3462 additions and 3120 deletions
+26
View File
@@ -2,6 +2,32 @@
All notable changes to this project are documented here, organized by release.
## [5.4.0]
### Added
- gate remote command execution behind allow_remote_command
- flapping detection suppresses notification storms
- remove log section from live dashboard (moved to /log)
- dedicated /log page with journal-backed history and nav item
- GET /api/0/log — paged, filtered events journal API
- eventlog writes to dedicated events journal; init/backfill/close wiring
- events journal read path with filters and backward pagination
- events journal write primitives (log_event, backfill, get_events_journal)
- Live Dashboard redesign — 6 columns, hover details, last-alert column
- extend the record-row design system to Host Overview, Alerts, and About
- unified record-row redesign of the settings page
### Fixed
- harden events log — non-dict ring entries, off-loop journal reads, no default-config singleton
- skip journal scheduling when the event loop is not running
- dedup log rows between API seed and websocket tail
- journal startup/shutdown lifecycle events (fired outside journal lifetime)
- drop typing names not yet used from journal.py import
- settings toolbar sticks below the site nav instead of scrolling away
- settings page not scrollable — restore html/body overflow override
---
## [5.3.12]
### Added
+4
View File
@@ -2,3 +2,7 @@
2. Minimum code that solves the problem. Nothing speculative.
3. Touch only what you must. Clean up only your own mess.
4. Define success criteria. Loop until verified.
This project's hbd daemon is deployed and in production on host w02
under user 'nagios', which you can reach via ssh nagios@w02.
The hbc client is used on 30 hosts in multiple networks.
+33 -2
View File
@@ -20,7 +20,7 @@ A lightweight UDP-based host monitoring system. Monitored hosts run a client (`h
└────────────────────┘ └────────────────────────────┘
```
**Package:** `hbd` v5.3.12
**Package:** `hbd` v5.4.0
**Python:** 3.11+
### Subpackages
@@ -116,6 +116,11 @@ dyndomains:
# Threshold alert re-notification interval (seconds)
threshold_renotify_interval: 3600
# Flap detection — silence a service/host after flap_count warning or critical
# notifications within flap_interval minutes (flap_count: 0 disables)
flap_count: 5
flap_interval: 10
# Notification channels
notification_channels:
pushover_ops:
@@ -207,6 +212,9 @@ hb_port: 50003 # Server UDP port
interval: 10 # Heartbeat interval (seconds)
owner: alice # Optional: claim ownership of this host
allow_remote_command: false # Execute shell commands sent by the server in CMD packets.
# Default false — see Remote command execution below.
plugins:
cpu_monitor:
interval: 300 # Override collection interval
@@ -237,6 +245,27 @@ plugins:
- If a connection fails to open at startup, IPv6 connections are dropped after 3 consecutive failures. IPv4 connections retry indefinitely.
- In daemon mode (`-d`), all log output goes to syslog (`LOG_DAEMON` facility).
### Remote command execution
The server can queue a shell command for a host, delivered in a `CMD` packet. Because
heartbeat packets are unauthenticated UDP, the client only runs those commands when the
host opts in:
```yaml
allow_remote_command: true
```
With the default `false`, the command is logged and refused, and the client replies
`Refused: allow_remote_command is false` — visible in the server's event log under the
`command` service, so a queued command never fails silently. When enabled, `hbc` logs a
warning at startup naming the risk. `SIGHUP` re-execs the client, so a config change takes
effect on reload.
All four clients enforce this: `hbc`, `hbc_windows.py`, `hbc_mini.py`, and the C
`hbc_mini` (which reads the same key from `~/.hbc.json` and needs a rebuild to pick the
change up). It does not gate `UPD` (self-update via `hb_install.sh`), which remains
ungated.
---
## UDP Protocol
@@ -254,7 +283,7 @@ Payload format: `key=value;key=value;...`
| `HTB` | client → server | Heartbeat (name, timestamp, RTT, acks, interval) |
| `PLG` | client → server | Plugin data (plugin name + metrics) |
| `ACK` | server → client | Acknowledgment |
| `CMD` | server → client | Execute a shell command on the client |
| `CMD` | server → client | Execute a shell command on the client (requires `allow_remote_command`) |
| `UPD` | server → client | Trigger self-update via `hb_install.sh` |
Value encoding:
@@ -400,6 +429,8 @@ hosts:
Notifications are sent on state transitions (OK → WARNING, WARNING → CRITICAL, CRITICAL → OK). De-escalations (CRITICAL → WARNING) do not trigger a notification. Ongoing alerts generate a re-notification every `threshold_renotify_interval` seconds (default: 3600). Alerts can be acknowledged via the web UI or API to suppress re-notifications.
A service or host that exceeds `flap_count` warning/critical notifications within `flap_interval` minutes is marked **flapping**: the tripping notification carries `Now flapping!! No more messages!` and further notifications are suppressed until it stays OK for `flap_interval` minutes. Flapping hosts are badged on the live dashboard; the event log keeps recording throughout. See [docs/NOTIFICATIONS.md](docs/NOTIFICATIONS.md#flap-detection).
### RTT thresholds
The server measures heartbeat round-trip time and supports RTT thresholds using the same format:
+15
View File
@@ -406,6 +406,21 @@ Potential improvements for future versions:
- Journal file encryption
- Signed journal entries
## Events journal
Alert/connectivity events shown on the `/log` page are written to a second,
dedicated journal (one JSON object per line, same rotation mechanism):
| Key | Default | Meaning |
|---|---|---|
| `events_journal_file` | `events.journal` | Filename inside `journal_dir` |
| `events_journal_max_size` | 10 MB | Rotation threshold |
| `events_journal_max_backups` | 10 | Rotated files kept |
`journal_dir` and `journal_enabled` are shared with the message journal.
On startup, if the events journal file is empty, it is seeded once from the
pickled in-memory message ring so history survives the upgrade.
## See Also
- [Configuration Guide](../hbd/config.py) - Full configuration options
+39
View File
@@ -9,6 +9,7 @@ Notifications are dispatched to the **owner and managers** of a host, each via t
```
Alert event (udp.py / threshold.py)
└─ notify.send_notification(host_name, Notification)
├─ flap.observe(host, service, level) → pass | trip | suppress
├─ look up host.owner + host.managers
├─ for each user → user.notification_channels
└─ for each channel → _dispatch_to_channel (filtered by min_level)
@@ -19,6 +20,7 @@ Every notification carries:
- **body** — detail message (metric value, threshold, duration)
- **url** — link to the plugin metrics page (`{base_url}/plugins#{hostname}`)
- **level** — `RECOVER | WARNING | CRITICAL | INFO`
- **service** — flap-detection key within the host (empty = the host itself)
## Configuration
@@ -268,6 +270,43 @@ min_level: WARNING
Reminder notifications (re-notify) are sent only for CRITICAL level alerts.
## Flap detection
A check that toggles between OK and alerting produces a notification per swing. Flap
detection silences it after the first few.
A **`(host, service)`** pair is flapping once it exceeds `flap_count` WARNING/CRITICAL
notifications within `flap_interval` minutes. Threshold alerts key on their metric path,
so a flapping disk check does not silence an unrelated CPU alert; connectivity, boot and
shutdown events key on the host itself.
```yaml
flap_count: 5 # notifications within the window that trip flapping (0 disables)
flap_interval: 10 # minutes — both the counting window and the quiet window
```
Lifecycle:
| Event | Effect |
|---|---|
| Alerts 1..`flap_count` within the window | Delivered normally |
| Alert `flap_count + 1` | Delivered with ` Now flapping!! No more messages!` appended to the body |
| Every notification after that | Dropped — including RECOVER and INFO |
| RECOVER while flapping | Dropped, and starts the `flap_interval` quiet window |
| WARNING/CRITICAL during the quiet window | Restarts the quiet window; still flapping |
| Quiet window elapses | Flapping ends **silently** — no notification |
Only outbound notifications are suppressed. `notify.eventlog` keeps recording every event,
so the journal and the `/log` page retain the full history of the flap.
Flapping pairs appear in each host's `stateinfo()` under `flapping` (a list of service
keys; `""` means the host itself) and render as an amber **flapping** badge next to the
host name on the live dashboard, with the affected services in its tooltip.
State lives at module level in `hbd/server/flap.py` and is never pickled — a server restart
starts every check with a clean slate. Hosts with `watch: false` never reach the
notification path, so they never flap.
## API reference
### `send_notification(host_name, notif) -> dict`
+1 -1
View File
@@ -14,4 +14,4 @@ Install options:
"""
__all__ = ["__version__"]
__version__ = "5.3.12"
__version__ = "5.4.0"
+3
View File
@@ -19,6 +19,9 @@ CLIENT_DEFAULTS = {
# Host identity
"owner": None, # Optional username to set as this host's owner on the server
# Security
"allow_remote_command": False, # Execute shell commands received in CMD packets from the server
# Runtime flags
"foreground": False,
"verbose": False,
+18 -2
View File
@@ -37,6 +37,10 @@ dorestart = False
shutdown_event: Optional[asyncio.Event] = None
active_tasks: List[asyncio.Task] = []
# Set from config in async_main. Off by default: a CMD packet is an unauthenticated
# UDP datagram, so executing one must be opted into per host.
allow_remote_command = False
class AsyncConnection:
"""Async UDP connection to a heartbeat server."""
@@ -183,7 +187,7 @@ class HeartbeatProtocol(asyncio.DatagramProtocol):
async def handle_command(conn: AsyncConnection, msg: dict):
"""Execute a command received from server."""
"""Execute a command received from server, if allow_remote_command is set."""
import subprocess
cmd = msg.get("cmd", "")
@@ -191,6 +195,15 @@ async def handle_command(conn: AsyncConnection, msg: dict):
return
logger = logging.getLogger("hbc.command")
if not allow_remote_command:
logger.warning(f"Refused command (allow_remote_command is false): {cmd}")
await conn.sendto({
"service": "command",
"msg": "Refused: allow_remote_command is false",
})
return
logger.info(f"Executing command: {cmd}")
try:
@@ -501,11 +514,12 @@ async def cleanup(connections: List[AsyncConnection]):
async def async_main(args, config):
"""Async main function."""
global running, shutdown_event, active_tasks, send_shutdown
global running, shutdown_event, active_tasks, send_shutdown, allow_remote_command
# Create shutdown event
shutdown_event = asyncio.Event()
active_tasks = []
allow_remote_command = bool(config.get("allow_remote_command", False))
logger = logging.getLogger("hbc.main")
@@ -519,6 +533,8 @@ async def async_main(args, config):
interval = config.get("interval", INTERVAL)
logger.info(f"hbc {__version__} on {iam} -> {hb_hosts} port={hb_port}, interval={interval}s")
if allow_remote_command:
logger.warning("allow_remote_command is true — CMD packets from the server will be executed")
af_filter = (socket.AF_INET if getattr(args, "ipv4_only", False)
else socket.AF_INET6 if getattr(args, "ipv6_only", False)
+4
View File
@@ -30,6 +30,10 @@ SERVER_DEFAULTS = {
"grace": 2, # Grace period (extra seconds before notifying after a missed heartbeat)
"threshold_renotify_interval": 3600, # Seconds between threshold re-notifications
# Flap detection (0 in either key disables it)
"flap_count": 5, # Warning/critical notifications within flap_interval that trip flapping
"flap_interval": 10, # Minutes: the counting window, and the quiet window after an OK
# User management
"users": {}, # username -> {full_name, avatar, password, admin, notification_channels}
"default_owner": None, # Username that owns hosts with no explicit owner
+1
View File
@@ -19,6 +19,7 @@ def _make_yaml() -> YAML:
_SERVER_KEYS = [
"hbd_port", "hbd_host", "ws_port", "wss_port", "hb_port",
"interval", "grace", "base_url", "threshold_renotify_interval",
"flap_count", "flap_interval",
"logfile", "pidfile", "pickfile", "journal_enabled", "journal_dir",
"journal_max_size", "journal_max_backups", "default_owner",
"default_threshold_config",
+119
View File
@@ -0,0 +1,119 @@
"""Flap detection: silence checks that toggle faster than they are useful.
A ``(host, service)`` pair is *flapping* once it exceeds ``flap_count``
WARNING/CRITICAL notifications within ``flap_interval`` minutes. The
notification that trips the threshold is delivered with ``FLAP_MARKER``
appended; every notification for that pair afterwards is dropped. The state
clears silently ``flap_interval`` minutes after a RECOVER, provided no further
WARNING/CRITICAL arrived in the meantime.
Only outbound notifications are affected — ``notify.eventlog`` keeps recording
every event, so the journal and ``/log`` retain the full history of the flap.
State lives here at module level rather than on ``Host`` so it is never
pickled: a restart starts every check with a clean slate.
"""
import logging
import time
logger = logging.getLogger(__name__)
FLAP_MARKER = "Now flapping!! No more messages!"
# Actions returned by observe()
PASS = "pass" # deliver unchanged
TRIP = "trip" # deliver with FLAP_MARKER appended
SUPPRESS = "suppress" # drop
_count = 0 # flap_count: alerts within the window needed to trip
_window = 0.0 # flap_interval in seconds
# {(host, service): {"events": [ts, ...], "flapping": bool, "ok_since": float|None}}
_state: dict = {}
def setup(cfg) -> None:
"""Read flap_count / flap_interval from *cfg* (also called on reload)."""
global _count, _window
_count = int(cfg.get("flap_count", 0) or 0)
_window = float(cfg.get("flap_interval", 0) or 0) * 60.0
def _enabled() -> bool:
return _count > 0 and _window > 0
def _label(key) -> str:
host, service = key
return f"{host}/{service}" if service else host
def _sweep(st: dict, now: float) -> bool:
"""Clear the flapping flag once the post-RECOVER quiet window has elapsed."""
if st["flapping"] and st["ok_since"] is not None and now - st["ok_since"] >= _window:
st["flapping"] = False
st["ok_since"] = None
st["events"].clear()
return True
return False
def observe(host: str, service: str, level: str) -> str:
"""Record a notification for *host*/*service* and return what to do with it.
Returns PASS, TRIP or SUPPRESS.
"""
if not _enabled():
return PASS
now = time.time()
key = (host or "", service or "")
level = (level or "").upper()
st = _state.get(key)
if st is not None and _sweep(st, now):
logger.info("flapping cleared for %s", _label(key))
if level in ("WARNING", "CRITICAL"):
if st is None:
st = _state[key] = {"events": [], "flapping": False, "ok_since": None}
# An alert inside the quiet window means it never settled.
st["ok_since"] = None
st["events"] = [t for t in st["events"] if now - t < _window]
st["events"].append(now)
if st["flapping"]:
return SUPPRESS
if len(st["events"]) > _count:
st["flapping"] = True
logger.info(
"flapping detected for %s (%d alerts in %.0f min)",
_label(key), len(st["events"]), _window / 60,
)
return TRIP
return PASS
if st is None or not st["flapping"]:
return PASS
if level == "RECOVER":
# Starts the quiet window; the state ends silently when it elapses.
st["ok_since"] = now
return SUPPRESS
def flapping_services(host: str) -> list:
"""Return the services of *host* that are currently flapping.
An empty string in the result means the host itself (connectivity, boot,
shutdown) rather than a named service.
"""
if not _enabled():
return []
now = time.time()
flapping = []
for (h, service), st in _state.items():
if h != host:
continue
_sweep(st, now)
if st["flapping"]:
flapping.append(service)
return sorted(flapping)
+4
View File
@@ -428,6 +428,10 @@ class Host:
ddict["alert_critical_unacked"] = critical_unacked
ddict["alert_critical_acked"] = critical_acked
# Flap detection state (module-level in flap.py, never pickled)
from . import flap
ddict["flapping"] = flap.flapping_services(self.name)
# User access
ddict["owner"] = getattr(self, "owner", None)
ddict["managers"] = list(getattr(self, "managers", []))
+81 -1
View File
@@ -20,6 +20,7 @@ from . import users as users_mod
from . import oauth as oauth_mod
from . import ws as ws_mod
from . import configio as configio_mod
from . import journal as journal_mod
from . import config_access
logger = logging.getLogger(__name__)
@@ -258,12 +259,25 @@ def _build_host_info(host, threshold_checker=None) -> dict:
key=lambda x: x["metric"],
)
connections = [
{
"family": getattr(conn, "afam", family),
"addr": getattr(conn, "addr", ""),
"state": getattr(conn, "state", ""),
"rtt": (getattr(conn, "rtts", None) or [None])[0],
"statetime": getattr(conn, "statetime", None),
"lastbeat": getattr(conn, "lastbeat", None),
}
for family, conn in sorted(host.connections.items())
]
return {
"owner": getattr(host, "owner", None),
"managers": list(getattr(host, "managers", [])),
"hbc_version": hbc_version,
"hbc_type": hbc_type,
"last_packet": last_packet,
"connections": connections,
"thresholds": thresholds,
}
@@ -349,6 +363,50 @@ async def start(
lst = data.msgs[-30:]
return web.json_response(lst)
async def api_log(request):
"""Paged, filtered read of the events journal (newest first)."""
user, err = _require_auth(request)
if err:
return err
qa = request.rel_url.query
try:
limit = max(1, min(int(qa.get("limit", "100")), 1000))
except ValueError:
limit = 100
before = None
if qa.get("before"):
try:
before = float(qa["before"])
except ValueError:
return web.json_response({"error": "invalid before"}, status=400)
host_f = qa.get("host") or None
level_f = qa.get("level") or None
q_f = qa.get("q") or None
def visible(ev):
h = ev.get("host")
return not h or ws_mod._user_can_see_host(user, h)
ej = journal_mod.get_events_journal(config)
if ej.enabled and ej.journal_path.is_file():
# File scan can be large; keep it off the loop that services UDP/WS
events, more = await asyncio.get_running_loop().run_in_executor(
None,
lambda: journal_mod.read_events(
ej.journal_dir, ej.journal_file,
limit=limit, before=before, host=host_f, level=level_f, q=q_f,
predicate=visible,
),
)
else:
# Journal disabled or not yet written: serve the in-memory ring
events, more = journal_mod.filter_events(
reversed(data.msgs),
limit=limit, before=before, host=host_f, level=level_f, q=q_f,
predicate=visible,
)
return web.json_response({"events": events, "more": more})
async def cmd(request):
user, err = _require_auth(request)
if err:
@@ -453,7 +511,6 @@ async def start(
for h in sorted(hbdclass.Host.hosts)
if _can_operate_host(current_user, hbdclass.Host.hosts[h])
],
messages=data.msgs[-30:],
current_user=current_user.to_dict() if current_user else None,
active_page="live",
)
@@ -739,6 +796,27 @@ async def start(
)
return web.Response(text=body, content_type="text/html")
async def log_page(request):
"""Render the event log page."""
current_user, _ = _require_auth_redirect(request)
pkg_dir = os.path.dirname(__file__)
templates_dir = config.get("templates_dir", os.path.join(pkg_dir, "templates"))
env = jinja2.Environment(loader=jinja2.FileSystemLoader(templates_dir), autoescape=True)
host = request.host # includes port if non-standard
forwarded_proto = request.headers.get("X-Forwarded-Proto", "")
is_secure = request.secure or forwarded_proto.lower() == "https"
scheme = "wss" if is_secure else "ws"
heartbeat_ws_url = f"{scheme}://{host}/ws"
tmpl = env.get_template("log.html")
body = tmpl.render(
title="Log - Heartbeat",
header="Log",
heartbeat_ws_url=heartbeat_ws_url,
current_user=current_user.to_dict() if current_user else None,
active_page="log",
)
return web.Response(text=body, content_type="text/html")
# -------------------------------------------------------------------------
# Auth endpoints
# -------------------------------------------------------------------------
@@ -1766,6 +1844,7 @@ async def start(
web.get("/api/0/hosts", api_hosts),
web.get("/api/0/alert_summary", api_alert_summary),
web.get("/api/0/messages", api_messages),
web.get("/api/0/log", api_log),
web.get("/api/0/hosts/{hostname}/plugins", api_host_plugins),
web.get("/api/0/hosts/{hostname}/plugins/{plugin_name}", api_host_plugin_detail),
web.get("/api/0/hosts/{hostname}/alerts", api_host_alerts),
@@ -1781,6 +1860,7 @@ async def start(
web.get("/live", live),
web.get("/plugins", plugins_page),
web.get("/alerts", alerts_page),
web.get("/log", log_page),
web.get("/about", about_page),
web.get("/profile", profile_page),
web.get("/settings", settings_page),
+133 -1
View File
@@ -11,7 +11,7 @@ import os
import asyncio
from datetime import datetime
from pathlib import Path
from typing import Dict, Any, Optional
from typing import Dict, Any, Optional, Callable, Iterable, List, Tuple, Union
logger = logging.getLogger(__name__)
@@ -153,6 +153,31 @@ class MessageJournal:
except Exception as e:
logger.error(f"Error writing to journal: {e}")
async def log_event(self, event: Dict[str, Any]):
"""Write a caller-provided dict verbatim as one JSONL line (with rotation)."""
if not self.enabled or not self._initialized:
return
async with self._lock:
try:
line = json.dumps(event, separators=(',', ':')) + '\n'
nbytes = len(line.encode('utf-8'))
if self._current_size + nbytes > self.max_size:
await self._rotate()
if self._file_handle:
self._file_handle.write(line)
self._file_handle.flush()
self._current_size += nbytes
except Exception as e:
logger.error(f"Error writing event to journal: {e}")
async def backfill(self, events: Iterable[Dict[str, Any]]):
"""One-time seed: write *events* only if the journal file is currently empty."""
if not self.enabled or not self._initialized or self._current_size > 0:
return
for ev in events:
if isinstance(ev, dict):
await self.log_event(ev)
async def _rotate(self):
"""
Rotate the journal file.
@@ -309,6 +334,88 @@ class MessageJournal:
}
def _iter_journal_events(journal_dir: Union[str, Path], journal_file: str) -> Iterable[Dict[str, Any]]:
"""Yield event dicts from the journal files, newest event first.
Reads the current file, then rotated backups newest-to-oldest (backup
names embed rotation timestamps, so filename sort is chronological).
"""
dirp = Path(journal_dir)
files = [dirp / journal_file]
files.extend(sorted(dirp.glob(journal_file + '.*'), reverse=True))
for f in files:
if not f.is_file():
continue
try:
lines = f.read_text(encoding='utf-8', errors='replace').splitlines()
except OSError as e:
logger.warning(f"Cannot read journal file {f}: {e}")
continue
for line in reversed(lines):
try:
ev = json.loads(line)
except ValueError:
continue
if isinstance(ev, dict):
yield ev
def filter_events(
events: Iterable[Dict[str, Any]],
limit: int = 100,
before: Optional[float] = None,
host: Optional[str] = None,
level: Optional[str] = None,
q: Optional[str] = None,
predicate: Optional[Callable[[Dict[str, Any]], bool]] = None,
) -> Tuple[List[Dict[str, Any]], bool]:
"""Filter an iterable of event dicts already ordered newest-first.
Returns (events, more): up to *limit* matching events, and whether at
least one further matching event exists beyond the limit.
"""
host_l = host.lower() if host else None
level_l = level.lower() if level else None
q_l = q.lower() if q else None
out: List[Dict[str, Any]] = []
for ev in events:
if not isinstance(ev, dict):
continue
ts = ev.get('ts')
if before is not None and (not isinstance(ts, (int, float)) or ts >= before):
continue
if host_l and host_l not in str(ev.get('host') or '').lower():
continue
if level_l and str(ev.get('level') or '').lower() != level_l:
continue
if q_l and q_l not in str(ev.get('message') or '').lower():
continue
if predicate is not None and not predicate(ev):
continue
if len(out) >= limit:
return out, True
out.append(ev)
return out, False
def read_events(
journal_dir: Union[str, Path],
journal_file: str = 'events.journal',
*,
limit: int = 100,
before: Optional[float] = None,
host: Optional[str] = None,
level: Optional[str] = None,
q: Optional[str] = None,
predicate: Optional[Callable[[Dict[str, Any]], bool]] = None,
) -> Tuple[List[Dict[str, Any]], bool]:
"""Read filtered events newest-first from the events journal files."""
return filter_events(
_iter_journal_events(journal_dir, journal_file),
limit=limit, before=before, host=host, level=level, q=q, predicate=predicate,
)
# Global journal instance
_journal_instance: Optional[MessageJournal] = None
@@ -340,3 +447,28 @@ async def log_message(msg: Dict[str, Any], addr: tuple, timestamp: Optional[floa
"""
journal = get_journal()
await journal.log_message(msg, addr, timestamp)
# Global events journal instance (human-readable event log, written by notify.eventlog)
_events_journal_instance: Optional[MessageJournal] = None
def get_events_journal(config: Optional[Dict[str, Any]] = None) -> MessageJournal:
"""Get or create the global events journal instance.
Uses the events_journal_* config keys; shares journal_dir and
journal_enabled with the raw-datagram journal.
"""
global _events_journal_instance
if _events_journal_instance is None:
cfg = config or {}
_events_journal_instance = MessageJournal(
{
'journal_dir': cfg.get('journal_dir', '/var/log/heartbeat'),
'journal_file': cfg.get('events_journal_file', 'events.journal'),
'journal_max_size': cfg.get('events_journal_max_size', 10 * 1024 * 1024),
'journal_max_backups': cfg.get('events_journal_max_backups', 10),
'journal_enabled': cfg.get('journal_enabled', True),
}
)
return _events_journal_instance
+29 -2
View File
@@ -165,6 +165,15 @@ async def _run_async(config, config_path=None):
msg_journal = journal_mod.get_journal(config)
await msg_journal.initialize()
# Initialize events journal (human-readable event log for the /log page)
events_journal = journal_mod.get_events_journal(config)
await events_journal.initialize()
if data.msgs:
# One-time seed on upgrade: only writes when the journal file is empty
await events_journal.backfill(data.msgs)
notify_mod.eventlog(None, "INFO", f"hbd version {__version__} starting up")
# Initialize threshold checker
threshold_checker = threshold_mod.ThresholdChecker(
config=config,
@@ -379,6 +388,26 @@ async def _run_async(config, config_path=None):
except Exception as e:
logger.warning("Error closing message journal: %s", e)
# Journal the shutdown event now; run()'s shutdown eventlog fires after this journal closes
try:
await events_journal.log_event(
{
"ts": time.time(),
"host": None,
"level": "INFO",
"service": None,
"message": f"hbd version {__version__} shutdown",
}
)
except Exception as e:
logger.debug("Could not journal shutdown event: %s", e)
# Close events journal
try:
await events_journal.close()
except Exception as e:
logger.warning("Error closing events journal: %s", e)
# Signal DNS worker to exit and await it
try:
if "dns_task" in locals() and dns_task:
@@ -487,8 +516,6 @@ def run(config, config_path=None):
except Exception as e:
logger.warning("Failed to write pidfile %s: %s", pidfile, e)
eventlog(None, "INFO", f"hbd version {__version__} starting up")
if config_path:
logger.info(f"Config file: {config_path} (reload with SIGHUP)")
else:
+34 -1
View File
@@ -25,6 +25,7 @@ from dataclasses import dataclass, field
from typing import Optional
from . import data
from . import flap as flap_mod
from . import ws as ws_mod
logger = logging.getLogger(__name__)
@@ -33,6 +34,7 @@ msg_to_websockets = ws_mod.broadcast
# Module-level state set via setup()
_config: dict = {}
_loop: Optional[asyncio.AbstractEventLoop] = None
# Tracks which channels fired a WARNING/CRITICAL per host.
# {host_name: set of channel_names} — used to route RECOVER to the same channels.
@@ -62,6 +64,7 @@ class Notification:
body: str # detail message
level: str # RECOVER | WARNING | CRITICAL | INFO
url: str = "" # link to plugin metrics page
service: str = "" # flap-detection key within the host ("" = the host itself)
# ---------------------------------------------------------------------------
@@ -70,14 +73,18 @@ class Notification:
def setup(cfg: dict, loop: Optional[asyncio.AbstractEventLoop] = None):
"""Initialize notifier from configuration dict."""
global _config
global _config, _loop
_config = dict(cfg)
flap_mod.setup(_config)
if loop is not None:
_loop = loop
def reload_config(cfg: dict):
"""Reload notification configuration on SIGHUP."""
global _config
_config = dict(cfg)
flap_mod.setup(_config)
logger.info("Notification configuration reloaded")
@@ -131,6 +138,21 @@ def eventlog(host, lvl, m, service=None):
except Exception as e:
logger.warning("failed to write to logfile: %s", e)
msg_to_websockets("message", msg)
_journal_event(msg)
def _journal_event(msg: dict):
"""Schedule an async write of *msg* to the events journal (no-op without a loop)."""
if _loop is None or not _loop.is_running():
return
from . import journal as journal_mod
ej = journal_mod._events_journal_instance
if ej is None or not ej.enabled:
return
try:
asyncio.run_coroutine_threadsafe(ej.log_event(msg), _loop)
except Exception as e:
logger.warning("failed to schedule events journal write: %s", e)
# ---------------------------------------------------------------------------
@@ -421,11 +443,22 @@ async def send_notification(host_name: str, notif: Notification) -> dict:
notification_channels, and dispatches. Silently does nothing if
no users are configured.
Flap detection runs first: once *host_name*/*notif.service* is flapping the
notification is dropped, and the one that trips the state carries
``flap.FLAP_MARKER``.
Returns a dict of {channel_name: bool} results.
"""
from . import users as users_mod
from . import hbdclass
action = flap_mod.observe(host_name, notif.service, notif.level)
if action == flap_mod.SUPPRESS:
logger.debug("flapping: suppressed %s notification for %s", notif.level, host_name)
return {}
if action == flap_mod.TRIP:
notif.body = f"{notif.body} {flap_mod.FLAP_MARKER}"
if not users_mod.users_enabled():
return {}
+6
View File
@@ -386,6 +386,12 @@ def get_settings_sections(config: dict, threshold_checker=None, user=None) -> li
"Extra seconds to wait after a missed heartbeat before sending notifications.", editable=True),
field("threshold_renotify_interval", "Re-notify interval", "duration",
"How often to re-send notifications for ongoing threshold alerts.", editable=True),
field("flap_count", "Flap count", "number",
"Warning/critical notifications within the flap interval that mark a "
"service as flapping and silence it. 0 disables flap detection.", editable=True),
field("flap_interval", "Flap interval", "number",
"Minutes: the window flap count is measured over, and how long a service "
"must stay OK before flapping ends.", editable=True),
field("autosave_interval", "Autosave interval", "duration",
"How often the server saves its state to disk."),
field("base_url", "Base URL", "text",
+155
View File
@@ -0,0 +1,155 @@
/* hbd-ui.css shared design system for hbd pages.
*
* Green-biased neutrals, teal accent, mono identifiers. Pages link this
* after static/style.css and build on the tokens + components below.
* Introduced with the settings-page redesign; see settings.html for the
* fullest use of the row/editor idiom.
*/
:root {
--st-paper: #f4f6f4;
--st-surface: #ffffff;
--st-surface-2: #eef1ee;
--st-ink: #1d2420;
--st-muted: #5f6a63;
--st-faint: #8a948d;
--st-line: #dde3de;
--st-line-soft: #e8ece8;
--st-accent: #0e7280;
--st-accent-ink: #ffffff;
--st-accent-soft: #e0eef0;
--st-ok: #3f9c5a;
--st-ok-soft: #e2f2e7;
--st-warn: #d9982a;
--st-warn-soft: #f9efdc;
--st-crit: #d64545;
--st-crit-soft: #f9e3e3;
--st-badge-owner: #ede7f6; --st-badge-owner-ink: #5e35b1;
--st-badge-mgr: #e3f2fd; --st-badge-mgr-ink: #1565c0;
--st-mono: ui-monospace, "SF Mono", "Cascadia Mono", Menlo, Consolas, monospace;
}
html[data-theme="dark"] {
--st-paper: #131816;
--st-surface: #1b211e;
--st-surface-2: #222925;
--st-ink: #e4eae6;
--st-muted: #9aa69e;
--st-faint: #6e7a72;
--st-line: #2c342f;
--st-line-soft: #252c28;
--st-accent: #4cc4d4;
--st-accent-ink: #0d2326;
--st-accent-soft: #143b40;
--st-ok: #5cba77;
--st-ok-soft: #1c2f22;
--st-warn: #e0ad4e;
--st-warn-soft: #33290f;
--st-crit: #e06c6c;
--st-crit-soft: #3a1a1a;
--st-badge-owner: #33294a; --st-badge-owner-ink: #b39ddb;
--st-badge-mgr: #1b3350; --st-badge-mgr-ink: #90caf9;
}
/* ── sticky toolbar under the site nav ── */
.st-toolbar {
position: sticky; top: var(--nav-h, 48px); z-index: 90;
display: flex; align-items: center; gap: 12px; flex-wrap: wrap;
padding: 9px 20px;
background: var(--st-surface); border-bottom: 1px solid var(--st-line);
}
.st-toolbar .brand { font-family: var(--st-mono); font-size: 15px; font-weight: 700; letter-spacing: -.02em; color: var(--st-ink); }
.st-toolbar .brand .tld { color: var(--st-accent); }
.st-toolbar .hintline { font-size: 12px; color: var(--st-faint); }
.st-toolbar .spacer { flex: 1; }
/* ── ECG pulse divider ── */
.pulse { display: block; width: 100%; height: 14px; }
.pulse polyline { fill: none; stroke: var(--st-accent); stroke-width: 1.5; opacity: .55; }
/* ── buttons ── */
.btn {
font: 600 13px/1 inherit; border: 1px solid var(--st-line); border-radius: 6px;
background: var(--st-surface); color: var(--st-ink); padding: 7px 14px; cursor: pointer;
}
.btn:focus-visible, a:focus-visible { outline: 2px solid var(--st-accent); outline-offset: 2px; }
.btn.primary { background: var(--st-accent); border-color: var(--st-accent); color: var(--st-accent-ink); }
.btn.quiet { border-color: transparent; background: transparent; color: var(--st-muted); }
.btn.small { padding: 4px 10px; font-size: 12px; }
.btn.danger { color: var(--st-crit); }
.btn:disabled { opacity: .6; cursor: default; }
/* ── section headers as config keys ── */
section.st { margin-top: 26px; scroll-margin-top: calc(var(--bars-h, 94px) + 12px); }
.sec-head { display: flex; align-items: baseline; gap: 10px; margin-bottom: 8px; flex-wrap: wrap; }
.sec-head h2 { margin: 0; font-family: var(--st-mono); font-size: 15px; font-weight: 700; color: var(--st-ink); }
.sec-head h2 .colon { color: var(--st-accent); }
.sec-head .count { font-family: var(--st-mono); font-size: 11.5px; color: var(--st-faint); }
.sec-head .sub { font-size: 12px; color: var(--st-muted); }
.sec-head .add { margin-left: auto; }
/* ── record list + row ── */
.list { background: var(--st-surface); border: 1px solid var(--st-line); border-radius: 8px; overflow: hidden; }
.row {
display: grid; grid-template-columns: 190px minmax(0,1fr) auto;
gap: 4px 14px; align-items: center;
padding: 8px 14px; border-bottom: 1px solid var(--st-line-soft);
transition: background .12s ease;
}
@media (prefers-reduced-motion: reduce) { .row { transition: none; } }
.row:last-child { border-bottom: none; }
.row:hover { background: var(--st-surface-2); }
.row .id { font-family: var(--st-mono); font-size: 13px; font-weight: 600; color: var(--st-ink); display: flex; align-items: center; gap: 8px; min-width: 0; }
.row .id .name { overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }
.row .act { display: flex; gap: 4px; justify-content: flex-end; }
.dot { width: 8px; height: 8px; border-radius: 50%; flex: none; }
.dot.ok { background: var(--st-ok); }
.dot.warn { background: var(--st-warn); }
.dot.crit { background: var(--st-crit); }
.dot.off { background: transparent; border: 1.5px solid var(--st-faint); }
.facts { display: flex; flex-wrap: wrap; gap: 4px 6px; align-items: center; min-width: 0; }
.chip {
display: inline-flex; align-items: center; gap: 4px;
font-family: var(--st-mono); font-size: 11px; line-height: 1.3;
padding: 3px 7px; border-radius: 4px;
background: var(--st-surface-2); color: var(--st-muted);
}
.chip.owner { background: var(--st-badge-owner); color: var(--st-badge-owner-ink); }
.chip.mgr { background: var(--st-badge-mgr); color: var(--st-badge-mgr-ink); }
.chip.level { background: transparent; border: 1px solid var(--st-line); }
.chip.private { background: transparent; border: 1px dashed var(--st-badge-owner-ink); color: var(--st-badge-owner-ink); }
.chip.ok { background: var(--st-ok-soft); color: var(--st-ok); }
.chip.warn { background: var(--st-warn-soft); color: var(--st-warn); }
.chip.crit { background: var(--st-crit-soft); color: var(--st-crit); }
.chip .k { color: var(--st-faint); }
.chip .w { color: var(--st-warn); font-weight: 700; }
.chip .c { color: var(--st-crit); font-weight: 700; }
/* ── key/value rows (server groups, about page) ── */
.kv .row { grid-template-columns: 230px minmax(0,1fr); padding: 6px 14px; }
.group {
display: flex; align-items: center; gap: 10px;
padding: 7px 14px 5px; font-family: var(--st-mono); font-size: 11px; color: var(--st-faint);
background: var(--st-surface); border-bottom: 1px solid var(--st-line-soft);
}
.group .gbtn { margin-left: auto; }
.kv .val { font-family: var(--st-mono); font-size: 12.5px; color: var(--st-muted); min-width: 0; }
.kv .val a { color: var(--st-accent); text-decoration: none; }
.kv .val a:hover { text-decoration: underline; }
.kv .desc { display: block; font-size: 11px; color: var(--st-faint); margin-top: 2px; }
.val-masked { color: var(--st-faint); letter-spacing: 2px; }
.note { margin: 8px 2px 0; font-size: 12px; color: var(--st-faint); }
/* ── handheld defaults for rows ── */
@media (max-width: 760px) {
.st-toolbar { padding: 8px 10px; }
.st-toolbar .hintline { display: none; }
.row { grid-template-columns: minmax(0,1fr) auto; padding: 9px 12px; }
.row .id { grid-row: 1; grid-column: 1; }
.row .act { grid-row: 1; grid-column: 2; }
.row .facts { grid-column: 1 / -1; }
.kv .row { grid-template-columns: 1fr; }
section.st { scroll-margin-top: calc(var(--bars-h, 94px) + 50px); }
}
+16
View File
@@ -0,0 +1,16 @@
/* hbd-ui.js shared helpers for pages using hbd-ui.css.
*
* Keeps sticky toolbars stuck just below the fixed site nav, whose height
* varies with viewport width and wrapping.
*/
function _setStickyOffsets() {
const nav = document.querySelector('.nav');
const tb = document.querySelector('.st-toolbar');
const navH = nav ? nav.offsetHeight : 48;
const barsH = navH + (tb ? tb.offsetHeight : 46);
document.documentElement.style.setProperty('--nav-h', navH + 'px');
document.documentElement.style.setProperty('--bars-h', barsH + 'px');
}
window.addEventListener('resize', _setStickyOffsets);
document.addEventListener('DOMContentLoaded', _setStickyOffsets);
+76 -155
View File
@@ -1,189 +1,110 @@
<!DOCTYPE html>
<html>
{% include 'head.html' %}
<link rel="stylesheet" href="/static/hbd-ui.css">
<script src="/static/hbd-ui.js"></script>
<style>
html, body { overflow: visible; }
html, body { height: auto; overflow: visible; }
body { background: var(--st-paper); padding: 60px 0 0; }
.container {
max-width: 700px;
margin: 0 auto;
.about-main { max-width: 700px; margin: 0 auto; padding: 0 20px 60px; }
.hero {
display: flex; align-items: baseline; gap: 14px; flex-wrap: wrap;
padding: 26px 2px 0;
}
h1 {
color: #333;
margin-bottom: 4px;
font-size: 1.5em;
.hero .hb-logo {
font-family: var(--st-mono); font-size: 30px; font-weight: 700;
letter-spacing: -.03em; color: var(--st-ink);
}
.subtitle {
color: #666;
margin-bottom: 24px;
font-size: 0.9em;
}
.section {
background: #fff;
border-radius: 8px;
box-shadow: 0 1px 6px rgba(0,0,0,0.1);
padding: 20px 24px;
margin-bottom: 20px;
}
.section h2 {
font-size: 1em;
font-weight: 700;
color: #333;
margin: 0 0 16px;
padding-bottom: 10px;
border-bottom: 1px solid #eee;
text-transform: uppercase;
letter-spacing: 0.5px;
}
.info-row {
display: flex;
align-items: baseline;
padding: 8px 0;
border-bottom: 1px solid #f5f5f5;
font-size: 0.9em;
}
.info-row:last-child { border-bottom: none; }
.info-label {
width: 160px;
flex-shrink: 0;
color: #666;
font-size: 0.88em;
}
.info-value {
color: #222;
word-break: break-all;
}
.info-value a {
color: #0066cc;
text-decoration: none;
}
.info-value a:hover { text-decoration: underline; }
.version-badge {
display: inline-block;
padding: 3px 12px;
background: #e8f0fe;
color: #1a73e8;
border-radius: 12px;
font-size: 1.00em;
font-weight: 600;
font-family: monospace;
}
.hb-logo {
font-size: 2.5em;
font-weight: 700;
color: #0066cc;
letter-spacing: -1px;
margin-bottom: 6px;
}
.hb-tagline {
color: #555;
font-size: 0.95em;
}
.logo-section {
display: flex;
align-items: center;
gap: 20px;
padding: 8px 0 4px;
}
.logo-text { flex: 1; }
/* ── Dark mode ── */
html[data-theme="dark"] h1 { color: var(--text); }
html[data-theme="dark"] .subtitle { color: var(--text-sec); }
html[data-theme="dark"] .section { background: var(--surface); box-shadow: 0 1px 6px var(--shadow); }
html[data-theme="dark"] .section h2 { color: var(--text); border-bottom-color: var(--border); }
html[data-theme="dark"] .info-row { border-bottom-color: var(--border-4); }
html[data-theme="dark"] .info-label { color: var(--text-sec); }
html[data-theme="dark"] .info-value { color: var(--text); }
html[data-theme="dark"] .info-value a { color: var(--link); }
html[data-theme="dark"] .hb-logo { color: var(--link); }
html[data-theme="dark"] .hb-tagline { color: var(--text-sec); }
html[data-theme="dark"] .version-badge { background: #1a3255; color: #60a5fa; }
.hero .hb-logo .tld { color: var(--st-accent); }
.hero .version { font-family: var(--st-mono); font-size: 13px; color: var(--st-accent);
background: var(--st-accent-soft); border-radius: 999px; padding: 3px 12px; }
.hero .tagline { flex-basis: 100%; font-size: 13px; color: var(--st-muted); }
</style>
<body>
{% include 'nav.html' %}
<div class="container">
<h1>{{ header }}</h1>
<p class="subtitle">Heartbeat monitoring system</p>
<div class="st-toolbar">
<span class="brand">hbd<span class="tld">·about</span></span>
<span class="hintline">heartbeat monitoring system</span>
</div>
<svg class="pulse" viewBox="0 0 1200 14" preserveAspectRatio="none" aria-hidden="true">
<polyline points="0,10 340,10 352,10 358,3 364,13 370,1 378,12 384,10 560,10 572,10 578,3 584,13 590,1 598,12 604,10 1200,10"/>
</svg>
<div class="section">
<div class="logo-section">
<div class="logo-text">
<div class="hb-logo">Heartbeat</div>
<div class="hb-tagline">Lightweight host monitoring over UDP</div>
</div>
<span class="version-badge">v{{ hbd_version }}</span>
</div>
<div class="about-main">
<div class="hero">
<span class="hb-logo">heart<span class="tld">beat</span></span>
<span class="version">v{{ hbd_version }}</span>
<span class="tagline">Lightweight host monitoring over UDP</span>
</div>
<div class="section">
<h2>Version</h2>
<div class="info-row">
<span class="info-label">Server version</span>
<span class="info-value">{{ hbd_version }}</span>
<section class="st">
<div class="sec-head">
<h2>version<span class="colon">:</span></h2>
</div>
<div class="info-row">
<span class="info-label">Python</span>
<span class="info-value">{{ python_version }}</span>
<div class="list kv">
<div class="row">
<span class="id"><span class="name">server</span></span>
<span class="val">{{ hbd_version }}</span>
</div>
<div class="info-row">
<span class="info-label">License</span>
<span class="info-value">MIT</span>
<div class="row">
<span class="id"><span class="name">python</span></span>
<span class="val">{{ python_version }}</span>
</div>
<div class="row">
<span class="id"><span class="name">license</span></span>
<span class="val">MIT</span>
</div>
</div>
</section>
<div class="section">
<h2>Runtime</h2>
<div class="info-row">
<span class="info-label">Host</span>
<span class="info-value">{{ server_hostname }}</span>
<section class="st">
<div class="sec-head">
<h2>runtime<span class="colon">:</span></h2>
</div>
<div class="info-row">
<span class="info-label">Started</span>
<span class="info-value">{{ start_time_str }}</span>
<div class="list kv">
<div class="row">
<span class="id"><span class="name">host</span></span>
<span class="val">{{ server_hostname }}</span>
</div>
<div class="info-row">
<span class="info-label">Uptime</span>
<span class="info-value" id="uptime-value">{{ uptime_str }}</span>
<div class="row">
<span class="id"><span class="name">started</span></span>
<span class="val">{{ start_time_str }}</span>
</div>
<div class="info-row">
<span class="info-label">Hosts monitored</span>
<span class="info-value">{{ host_count }}</span>
<div class="row">
<span class="id"><span class="name">uptime</span></span>
<span class="val" id="uptime-value">{{ uptime_str }}</span>
</div>
<div class="row">
<span class="id"><span class="name">hosts_monitored</span></span>
<span class="val">{{ host_count }}</span>
</div>
</div>
</section>
<div class="section">
<h2>Contact &amp; Source</h2>
<div class="info-row">
<span class="info-label">Author</span>
<span class="info-value">Andreas Wrede</span>
<section class="st">
<div class="sec-head">
<h2>contact<span class="colon">:</span></h2>
</div>
<div class="info-row">
<span class="info-label">Email</span>
<span class="info-value"><a href="mailto:aew.hbd@wrede.ca">aew.hbd@wrede.ca</a></span>
<div class="list kv">
<div class="row">
<span class="id"><span class="name">author</span></span>
<span class="val">Andreas Wrede</span>
</div>
<div class="info-row">
<span class="info-label">Repository</span>
<span class="info-value"><a href="https://git.wrede.ca/andreas/heartbeat" target="_blank" rel="noopener">git.wrede.ca/andreas/heartbeat</a></span>
<div class="row">
<span class="id"><span class="name">email</span></span>
<span class="val"><a href="mailto:aew.hbd@wrede.ca">aew.hbd@wrede.ca</a></span>
</div>
<div class="row">
<span class="id"><span class="name">repository</span></span>
<span class="val"><a href="https://git.wrede.ca/andreas/heartbeat" target="_blank" rel="noopener">git.wrede.ca/andreas/heartbeat</a></span>
</div>
</div>
</section>
</div>
<script>
+132 -483
View File
@@ -1,376 +1,97 @@
<!DOCTYPE html>
<html>
{% include 'head.html' %}
<link rel="stylesheet" href="/static/hbd-ui.css">
<script src="/static/hbd-ui.js"></script>
<style>
html, body { height: auto; overflow: visible; }
body { background: var(--st-paper); padding: 60px 0 0; }
html, body {
height: auto;
overflow-y: auto;
.alerts-main { max-width: 1100px; margin: 0 auto; padding: 0 20px 60px; }
/* summary tiles */
.stats { display: flex; gap: 10px; flex-wrap: wrap; padding-top: 18px; }
.stat {
flex: 1 1 140px; display: flex; align-items: baseline; gap: 10px;
background: var(--st-surface); border: 1px solid var(--st-line); border-radius: 8px;
padding: 10px 14px;
}
.stat .num { font-family: var(--st-mono); font-size: 24px; font-weight: 700; font-variant-numeric: tabular-nums; }
.stat .lbl { font-family: var(--st-mono); font-size: 11.5px; color: var(--st-faint); }
.stat.crit .num { color: var(--st-crit); }
.stat.warn .num { color: var(--st-warn); }
.stat.ok .num { color: var(--st-ok); }
.container {
max-width: 1400px;
margin: 0 auto;
/* filter bar */
.filterbar { display: flex; gap: 6px; align-items: center; flex-wrap: wrap; margin: 14px 0 0; }
.filterbar .fbtn {
font-family: var(--st-mono); font-size: 12px; cursor: pointer;
color: var(--st-muted); background: var(--st-surface); border: 1px solid var(--st-line);
border-radius: 999px; padding: 5px 12px;
}
h1 { color: #333; margin-bottom: 5px; margin-top: 15px; font-size: 1.5em; }
.subtitle {
color: #666;
margin-bottom: 30px;
.filterbar .fbtn.active { background: var(--st-accent-soft); border-color: var(--st-accent); color: var(--st-accent); font-weight: 600; }
.filterbar input {
font: 12px var(--st-mono); color: var(--st-ink);
background: var(--st-surface); border: 1px solid var(--st-line); border-radius: 999px;
padding: 5px 12px; min-width: 170px;
}
.filterbar input.invalid { border-color: var(--st-crit); }
.filterbar .upd { margin-left: auto; font-size: 11.5px; color: var(--st-faint); }
.summary-cards {
display: flex;
flex-wrap: wrap;
gap: 10px;
margin-bottom: 16px;
/* alert row specifics */
.row.alert-crit { border-left: 3px solid var(--st-crit); padding-left: 11px; }
.row.alert-warn { border-left: 3px solid var(--st-warn); padding-left: 11px; }
.row.acked { opacity: .55; border-left-style: dashed; }
.row .id a { color: inherit; text-decoration: none; }
.row .id a:hover { color: var(--st-accent); }
.acked-chip { color: var(--st-ok); font-size: 12px; white-space: nowrap; }
.empty {
padding: 36px 14px; text-align: center; color: var(--st-faint); font-size: 13px;
}
.summary-card {
background: white;
border-radius: 6px;
padding: 6px 14px;
box-shadow: 0 1px 4px rgba(0,0,0,0.1);
display: flex;
align-items: center;
gap: 8px;
border-left: 4px solid #ddd;
.empty .big { font-size: 26px; color: var(--st-ok); display: block; margin-bottom: 6px; }
.errorbox {
background: var(--st-crit-soft); color: var(--st-crit);
border-radius: 8px; padding: 14px 16px; font-size: 13px;
}
.summary-card.critical { border-left-color: #ea1e0f; }
.summary-card.warning { border-left-color: #ff9800; }
.summary-card.ok { border-left-color: #4caf50; }
.summary-number {
font-size: 1.4em;
font-weight: bold;
line-height: 1;
}
.summary-number.critical { color: #ea1e0f; }
.summary-number.warning { color: #ff9800; }
.summary-number.ok { color: #4caf50; }
.summary-label {
color: #666;
font-size: 1.00em;
}
.filters {
background: white;
border-radius: 8px;
padding: 15px;
margin-bottom: 20px;
box-shadow: 0 2px 8px rgba(0,0,0,0.1);
display: flex;
gap: 15px;
align-items: center;
}
.filter-label {
font-weight: bold;
color: #555;
}
.filter-button {
padding: 8px 16px;
border: 2px solid #ddd;
background: white;
border-radius: 20px;
cursor: pointer;
transition: all 0.2s;
font-size: 0.9em;
}
.filter-button:hover {
border-color: #2196f3;
}
.filter-button.active {
background: #2196f3;
color: white;
border-color: #2196f3;
}
.filter-input {
padding: 7px 12px;
border: 2px solid #ddd;
border-radius: 20px;
font-size: 0.9em;
outline: none;
width: 200px;
transition: border-color 0.2s;
}
.filter-input:focus {
border-color: #2196f3;
}
.filter-input.invalid {
border-color: #f44336;
}
.alerts-container {
background: white;
border-radius: 8px;
padding: 20px;
box-shadow: 0 2px 8px rgba(0,0,0,0.1);
}
.alert-item {
border-left: 5px solid #ddd;
padding: 15px;
margin-bottom: 15px;
background: #fafafa;
border-radius: 4px;
display: flex;
justify-content: space-between;
align-items: center;
transition: all 0.2s;
}
.alert-item.acknowledged {
opacity: 0.8;
background: #f0f0f0;
}
.alert-item:hover {
box-shadow: 0 2px 8px rgba(0,0,0,0.1);
transform: translateX(5px);
}
.alert-item.critical {
border-left-color: #f44336;
background: #ffebee;
}
.alert-item.warning {
border-left-color: #ff9800;
background: #fff3e0;
}
.alert-item.unknown {
border-left-color: #9e9e9e;
background: #f5f5f5;
}
.alert-main {
flex: 1;
}
.alert-header {
display: flex;
align-items: center;
gap: 15px;
margin-bottom: 8px;
}
.alert-level {
padding: 4px 12px;
border-radius: 12px;
font-size: 0.75em;
font-weight: bold;
text-transform: uppercase;
letter-spacing: 0.5px;
}
.alert-level.critical {
background: #f44336;
color: white;
}
.alert-level.warning {
background: #ff9800;
color: white;
}
.alert-level.unknown {
background: #9e9e9e;
color: white;
}
.alert-hostname {
font-weight: bold;
color: #0066cc;
font-size: 1.1em;
text-decoration: none;
}
.alert-hostname:hover {
text-decoration: underline;
}
.alert-metric {
color: #0066cc;
font-size: 1.1em;
font-weight: normal;
}
.alert-details {
display: flex;
gap: 20px;
color: #666;
font-size: 0.9em;
}
.alert-value {
font-weight: bold;
color: #333;
}
.alert-duration {
color: #999;
font-size: 1.00em;
}
.alert-actions {
display: flex;
flex-direction: column;
gap: 8px;
margin-left: 15px;
}
.acknowledge-btn {
padding: 8px 16px;
background: #2196f3;
color: white;
border: none;
border-radius: 4px;
cursor: pointer;
font-size: 1.00em;
transition: all 0.2s;
white-space: nowrap;
}
.acknowledge-btn:hover {
background: #1976d2;
transform: scale(1.05);
}
.acknowledge-btn:disabled {
background: #ccc;
cursor: not-allowed;
transform: none;
}
.acknowledged-badge {
padding: 4px 8px;
background: #4caf50;
color: white;
border-radius: 4px;
font-size: 0.75em;
text-align: center;
white-space: nowrap;
}
.no-alerts {
text-align: center;
padding: 60px 20px;
color: #999;
}
.no-alerts-icon {
font-size: 4em;
margin-bottom: 20px;
}
.loading {
text-align: center;
padding: 40px;
color: #666;
}
.error {
background: #ffebee;
border-left: 4px solid #f44336;
padding: 20px;
margin: 20px 0;
border-radius: 4px;
color: #c62828;
}
.refresh-info {
text-align: center;
color: #999;
font-size: 1.00em;
margin-top: 20px;
padding-top: 20px;
border-top: 1px solid #e0e0e0;
}
.last-update {
color: #666;
font-size: 0.9em;
text-align: right;
margin-bottom: 15px;
}
/* ── Dark mode ── */
html[data-theme="dark"] h1 { color: var(--text); }
html[data-theme="dark"] .subtitle { color: var(--text-sec); }
html[data-theme="dark"] .summary-card { background: var(--surface); }
html[data-theme="dark"] .summary-label { color: var(--text-sec); }
html[data-theme="dark"] .filters { background: var(--surface); }
html[data-theme="dark"] .filter-label { color: var(--text-sec); }
html[data-theme="dark"] .filter-button { background: var(--surface-2); border-color: var(--border); color: var(--text); }
html[data-theme="dark"] .filter-button.active { background: #2196f3; color: #fff; border-color: #2196f3; }
html[data-theme="dark"] .filter-input { background: var(--input-bg); border-color: var(--input-border); color: var(--text); }
html[data-theme="dark"] .alerts-container { background: var(--surface); }
html[data-theme="dark"] .alert-item { background: var(--surface-2); }
html[data-theme="dark"] .alert-item.acknowledged { background: var(--surface-3); }
html[data-theme="dark"] .alert-item.critical { background: #2e0a0a; border-left-color: #f44336; }
html[data-theme="dark"] .alert-item.warning { background: #2e1a00; border-left-color: #ff9800; }
html[data-theme="dark"] .alert-item.unknown { background: var(--surface-2); }
html[data-theme="dark"] .alert-hostname { color: var(--link); }
html[data-theme="dark"] .alert-details { color: var(--text-sec); }
html[data-theme="dark"] .alert-value { color: var(--text); }
html[data-theme="dark"] .alert-duration { color: var(--text-muted); }
html[data-theme="dark"] .last-update { color: var(--text-sec); }
html[data-theme="dark"] .refresh-info { color: var(--text-muted); border-top-color: var(--border); }
html[data-theme="dark"] .no-alerts,
html[data-theme="dark"] .loading { color: var(--text-muted); }
</style>
<body>
{% include 'nav.html' %}
<div class="container">
<h1>{{ header }}</h1>
<p class="subtitle">Real-time monitoring alerts and threshold violations</p>
<div class="st-toolbar">
<span class="brand">hbd<span class="tld">·alerts</span></span>
<span class="hintline">threshold violations and reachability — refreshes every 15 s</span>
</div>
<svg class="pulse" viewBox="0 0 1200 14" preserveAspectRatio="none" aria-hidden="true">
<polyline points="0,10 340,10 352,10 358,3 364,13 370,1 378,12 384,10 560,10 572,10 578,3 584,13 590,1 598,12 604,10 1200,10"/>
</svg>
<div class="summary-cards" id="summary-cards">
<div class="summary-card critical">
<div class="summary-label">Critical</div>
<div class="summary-number critical" id="critical-count">-</div>
</div>
<div class="summary-card warning">
<div class="summary-label">Warning</div>
<div class="summary-number warning" id="warning-count">-</div>
</div>
<div class="summary-card ok">
<div class="summary-label">Total Hosts</div>
<div class="summary-number ok" id="host-count">-</div>
</div>
<div class="alerts-main">
<div class="stats">
<div class="stat crit"><span class="num" id="critical-count"></span><span class="lbl">critical</span></div>
<div class="stat warn"><span class="num" id="warning-count"></span><span class="lbl">warning</span></div>
<div class="stat ok"><span class="num" id="host-count"></span><span class="lbl">hosts</span></div>
</div>
<div class="filters">
<span class="filter-label">Show:</span>
<button class="filter-button active" onclick="filterAlerts('all')">All</button>
<button class="filter-button" onclick="filterAlerts('critical')">Critical Only</button>
<button class="filter-button" onclick="filterAlerts('warning')">Warning Only</button>
<input id="host-filter" class="filter-input" type="text" placeholder="host filter (regex)" oninput="onHostFilterInput(this)">
<div class="filterbar">
<button class="fbtn active" onclick="filterAlerts('all', this)">all</button>
<button class="fbtn" onclick="filterAlerts('critical', this)">critical</button>
<button class="fbtn" onclick="filterAlerts('warning', this)">warning</button>
<input id="host-filter" type="text" placeholder="host filter (regex)" oninput="onHostFilterInput(this)">
<span class="upd">updated <span id="last-update-time">never</span></span>
</div>
<div class="alerts-container">
<div class="last-update">Last updated: <span id="last-update-time">Never</span></div>
<div id="alerts-list">
<div class="loading">Loading alerts...</div>
</div>
<div class="refresh-info">
Auto-refreshing every 15 seconds
<section class="st" id="alerts">
<div class="sec-head">
<h2>alerts<span class="colon">:</span></h2>
<span class="count" id="alert-count"></span>
</div>
<div class="list" id="alerts-list">
<div class="empty">Loading alerts…</div>
</div>
</section>
</div>
<script>
@@ -378,207 +99,138 @@
let allAlerts = [];
let hostFilterRe = null;
function escHtml(s) {
return String(s).replace(/&/g,'&amp;').replace(/</g,'&lt;').replace(/>/g,'&gt;').replace(/"/g,'&quot;').replace(/'/g,'&#39;');
}
async function loadAlerts() {
try {
const response = await fetch('/api/0/alerts');
if (!response.ok) {
throw new Error(`HTTP ${response.status}`);
}
if (!response.ok) throw new Error(`HTTP ${response.status}`);
const data = await response.json();
allAlerts = data.alerts;
// Update summary cards
document.getElementById('critical-count').textContent = data.summary.critical || 0;
document.getElementById('warning-count').textContent = data.summary.warning || 0;
document.getElementById('host-count').textContent = data.host_count || 0;
// Update last update time
document.getElementById('last-update-time').textContent = new Date().toLocaleTimeString();
// Render alerts
renderAlerts(allAlerts);
} catch (error) {
document.getElementById('alerts-list').innerHTML =
`<div class="error">Failed to load alerts: ${error.message}</div>`;
`<div class="errorbox">Failed to load alerts: ${escHtml(error.message)}. Retrying automatically.</div>`;
}
}
function renderAlerts(alerts) {
const container = document.getElementById('alerts-list');
// Filter alerts based on current filter
let filteredAlerts = alerts;
let filtered = alerts;
if (currentFilter !== 'all') {
filteredAlerts = filteredAlerts.filter(alert =>
alert.level.toLowerCase() === currentFilter
);
filtered = filtered.filter(a => a.level.toLowerCase() === currentFilter);
}
if (hostFilterRe) {
filteredAlerts = filteredAlerts.filter(alert => hostFilterRe.test(alert.hostname));
filtered = filtered.filter(a => hostFilterRe.test(a.hostname));
}
if (filteredAlerts.length === 0) {
if (currentFilter === 'all' && alerts.length === 0) {
container.innerHTML = `
<div class="no-alerts">
<div class="no-alerts-icon"></div>
<h2>All Systems Normal</h2>
<p>No active alerts at this time</p>
</div>
`;
} else {
container.innerHTML = `
<div class="no-alerts">
<p>No ${currentFilter} alerts</p>
</div>
`;
}
document.getElementById('alert-count').textContent =
filtered.length === alerts.length ? alerts.length : `${filtered.length} of ${alerts.length}`;
if (filtered.length === 0) {
container.innerHTML = (currentFilter === 'all' && !hostFilterRe && alerts.length === 0)
? `<div class="empty"><span class="big"></span>All systems normal — no active alerts.</div>`
: `<div class="empty">No matching alerts.</div>`;
return;
}
let html = '';
for (const alert of filteredAlerts) {
html += renderAlert(alert);
}
container.innerHTML = html;
container.innerHTML = filtered.map(renderAlert).join('');
}
function renderAlert(alert) {
const level = alert.level.toLowerCase();
const cls = level === 'critical' ? 'crit' : (level === 'warning' ? 'warn' : 'level');
const duration = getDuration(alert.since);
const acknowledged = alert.acknowledged || false;
const acked = alert.acknowledged || false;
// Use formatted message if available, otherwise build from individual fields
let valueText = `Value: <span class="alert-value">${formatValue(alert.last_value)}</span>`;
const metric = (alert.metric_path.includes('.')
? alert.metric_path.slice(alert.metric_path.indexOf('.') + 1)
: alert.metric_path).replace(/_status_code$/, '');
let chips = `<span class="chip ${cls}">${escHtml(alert.level)}</span>`;
chips += `<span class="chip"><span class="k">metric</span> ${escHtml(metric)}</span>`;
if (alert.formatted_message) {
valueText += ` <span class="threshold-info">${alert.formatted_message}</span>`;
} else if (alert.threshold_value !== undefined && alert.threshold_value !== null && alert.operator) {
valueText += ` <span class="threshold-info">(threshold: ${alert.operator} ${formatValue(alert.threshold_value)})</span>`;
chips += `<span class="chip">${escHtml(alert.formatted_message)}</span>`;
} else {
chips += `<span class="chip"><span class="k">value</span> ${escHtml(formatValue(alert.last_value))}</span>`;
if (alert.threshold_value !== undefined && alert.threshold_value !== null && alert.operator) {
chips += `<span class="chip level">${escHtml(alert.operator)} ${escHtml(formatValue(alert.threshold_value))}</span>`;
}
}
if (alert.recovery_threshold !== undefined && alert.recovery_threshold !== null) {
const recOp = (alert.operator === '>' || alert.operator === '>=') ? '<' : '>';
valueText += ` <span class="threshold-info" style="color:#888">(recovers ${recOp} ${formatValue(alert.recovery_threshold)})</span>`;
chips += `<span class="chip level">recovers ${recOp} ${escHtml(formatValue(alert.recovery_threshold))}</span>`;
}
chips += `<span class="chip"><span class="k">for</span> ${duration}</span>`;
// Build actions section
let actionsHtml = '';
if (acknowledged) {
actionsHtml = `
<div class="alert-actions">
<div class="acknowledged-badge">✓ Acknowledged</div>
</div>
`;
} else {
actionsHtml = `
<div class="alert-actions">
<button class="acknowledge-btn" onclick="acknowledgeAlert('${alert.hostname}', '${alert.metric_path}', event)">
Acknowledge
</button>
</div>
`;
}
const act = acked
? `<span class="acked-chip">✓ acknowledged</span>`
: `<button class="btn small" onclick="acknowledgeAlert('${escHtml(alert.hostname)}', '${escHtml(alert.metric_path)}', event)">Acknowledge</button>`;
return `
<div class="alert-item ${level} ${acknowledged ? 'acknowledged' : ''}">
<div class="alert-main">
<div class="alert-header">
<span class="alert-level ${level}">${alert.level}</span>
<a class="alert-hostname" href="/plugins#${alert.hostname}">${alert.hostname}</a>
<span class="alert-metric">${(alert.metric_path.includes('.') ? alert.metric_path.slice(alert.metric_path.indexOf('.') + 1) : alert.metric_path).replace(/_status_code$/, '')}</span>
</div>
<div class="alert-details">
<span>${valueText}</span>
<span class="alert-duration">Active for ${duration}</span>
</div>
</div>
${actionsHtml}
</div>
`;
<div class="row alert-${cls === 'level' ? 'warn' : cls}${acked ? ' acked' : ''}">
<span class="id">
<span class="dot ${cls === 'crit' ? 'crit' : 'warn'}"></span>
<a class="name" href="/plugins#${encodeURIComponent(alert.hostname)}">${escHtml(alert.hostname)}</a>
</span>
<span class="facts">${chips}</span>
<span class="act">${act}</span>
</div>`;
}
function formatValue(value) {
if (typeof value === 'number') {
if (value > 1000) {
return value.toLocaleString();
}
if (value > 1000) return value.toLocaleString();
return value.toFixed(2);
}
return value;
}
function getDuration(timestamp) {
const now = Date.now() / 1000;
const seconds = Math.floor(now - timestamp);
if (seconds < 60) {
return `${seconds}s`;
} else if (seconds < 3600) {
return `${Math.floor(seconds / 60)}m`;
} else if (seconds < 86400) {
const hours = Math.floor(seconds / 3600);
const minutes = Math.floor((seconds % 3600) / 60);
return `${hours}h ${minutes}m`;
} else {
const days = Math.floor(seconds / 86400);
const hours = Math.floor((seconds % 86400) / 3600);
return `${days}d ${hours}h`;
const seconds = Math.floor(Date.now() / 1000 - timestamp);
if (seconds < 60) return `${seconds}s`;
if (seconds < 3600) return `${Math.floor(seconds / 60)}m`;
if (seconds < 86400) {
return `${Math.floor(seconds / 3600)}h ${Math.floor((seconds % 3600) / 60)}m`;
}
return `${Math.floor(seconds / 86400)}d ${Math.floor((seconds % 86400) / 3600)}h`;
}
function filterAlerts(filter) {
function filterAlerts(filter, btn) {
currentFilter = filter;
// Update active button
document.querySelectorAll('.filter-button').forEach(btn => {
btn.classList.remove('active');
});
event.target.classList.add('active');
// Re-render with new filter
document.querySelectorAll('.filterbar .fbtn').forEach(b => b.classList.remove('active'));
btn.classList.add('active');
renderAlerts(allAlerts);
}
async function acknowledgeAlert(hostname, metricPath, event) {
// Prevent event bubbling
if (event) {
event.stopPropagation();
}
// Disable the button
if (event) event.stopPropagation();
const button = event.target;
button.disabled = true;
button.textContent = 'Acknowledging...';
button.textContent = 'Acknowledging';
try {
const response = await fetch('/api/0/alerts/acknowledge', {
method: 'POST',
headers: {
'Content-Type': 'application/json',
},
body: JSON.stringify({
hostname: hostname,
metric_path: metricPath,
}),
headers: {'Content-Type': 'application/json'},
body: JSON.stringify({hostname: hostname, metric_path: metricPath}),
});
if (!response.ok) {
throw new Error(`HTTP ${response.status}`);
}
if (!response.ok) throw new Error(`HTTP ${response.status}`);
const result = await response.json();
// Update the alert in our local data
const alert = allAlerts.find(a => a.hostname === hostname && a.metric_path === metricPath);
if (alert) {
alert.acknowledged = true;
alert.acknowledged_at = result.acknowledged_at;
const a = allAlerts.find(x => x.hostname === hostname && x.metric_path === metricPath);
if (a) {
a.acknowledged = true;
a.acknowledged_at = result.acknowledged_at;
}
// Re-render alerts
renderAlerts(allAlerts);
} catch (error) {
alert(`Failed to acknowledge alert: ${error.message}`);
button.disabled = false;
@@ -603,10 +255,8 @@
renderAlerts(allAlerts);
}
// Auto-refresh every 15 seconds
setInterval(loadAlerts, 15000);
// Initialise filter from URL query string (?filter=...)
(function () {
const param = new URLSearchParams(window.location.search).get('filter');
if (param) {
@@ -616,7 +266,6 @@
}
})();
// Initial load
loadAlerts();
</script>
</body>
File diff suppressed because it is too large Load Diff
+228
View File
@@ -0,0 +1,228 @@
<!DOCTYPE html>
<html>
{% include 'head.html' %}
<link rel="stylesheet" href="/static/hbd-ui.css">
<script src="/static/hbd-ui.js"></script>
<style>
html, body { height: auto; overflow: visible; }
body { background: var(--st-paper); padding: 60px 0 0; }
.log-main { max-width: 1100px; margin: 0 auto; padding: 0 20px 60px; }
.st-toolbar input, .st-toolbar select {
font: 12px var(--st-mono); color: var(--st-ink);
background: var(--st-surface); border: 1px solid var(--st-line); border-radius: 999px;
padding: 5px 12px;
}
.live-dot { display: inline-flex; align-items: center; gap: 6px; font-size: 12px; font-weight: 600; color: var(--st-ok); }
.live-dot::before { content: ""; width: 7px; height: 7px; border-radius: 50%; background: var(--st-ok); }
.live-dot.reconnecting { color: var(--st-crit); }
.live-dot.reconnecting::before { background: var(--st-crit); }
.logrow {
display: grid; grid-template-columns: 130px 74px 150px minmax(0,1fr);
gap: 4px 14px; align-items: baseline;
padding: 4px 14px; border-bottom: 1px solid var(--st-line-soft);
font-family: var(--st-mono); font-size: 11.5px;
}
.logrow:last-child { border-bottom: none; }
.logrow .t { color: var(--st-faint); font-variant-numeric: tabular-nums; white-space: nowrap; }
.logrow .h { color: var(--st-ink); font-weight: 600; overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }
.logrow .m { color: var(--st-muted); overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }
.empty { padding: 24px 14px; text-align: center; color: var(--st-faint); font-size: 12.5px; }
.loadmore {
display: block; margin: 14px auto 0; cursor: pointer;
font-family: var(--st-mono); font-size: 12px; color: var(--st-muted);
background: var(--st-surface); border: 1px solid var(--st-line); border-radius: 999px;
padding: 6px 16px;
}
.loadmore:hover { color: var(--st-accent); border-color: var(--st-accent); }
.loadmore:disabled { opacity: .5; cursor: default; }
@media (max-width: 760px) {
.logrow { grid-template-columns: 66px minmax(0,1fr); }
.logrow .lvlc, .logrow .h { display: none; }
}
</style>
<body>
{% include 'nav.html' %}
<div class="st-toolbar">
<span class="brand">hbd<span class="tld">·log</span></span>
<span class="hintline">connectivity and alert events — full journal history</span>
<span class="spacer"></span>
<input type="text" id="filter-host" placeholder="host">
<select id="filter-level">
<option value="">level</option>
<option value="info">INFO</option>
<option value="warning">WARNING</option>
<option value="critical">CRITICAL</option>
<option value="recover">RECOVER</option>
<option value="unknown">UNKNOWN</option>
</select>
<input type="text" id="filter-msg" placeholder="message">
<span class="live-dot reconnecting" id="ws-status">connecting…</span>
</div>
<svg class="pulse" viewBox="0 0 1200 14" preserveAspectRatio="none" aria-hidden="true">
<polyline points="0,10 340,10 352,10 358,3 364,13 370,1 378,12 384,10 560,10 572,10 578,3 584,13 590,1 598,12 604,10 1200,10"/>
</svg>
<div class="log-main">
<section class="st">
<div class="sec-head">
<h2>log<span class="colon">:</span></h2>
<span class="count" id="log-count"></span>
</div>
<div class="list" id="messages">
<div class="empty">Loading events…</div>
</div>
<button class="loadmore" id="load-older" style="display:none">load older</button>
</section>
</div>
<script>
var PAGE_SIZE = 200;
var oldestTs = null; // ts of the oldest loaded row (pagination cursor)
var seq = 0; // guards against out-of-order fetch responses
var newestTs = null; // ts of the newest loaded row (dedup guard for the WS tail)
function escHtml(s) {
return String(s).replace(/&/g,'&amp;').replace(/</g,'&lt;').replace(/>/g,'&gt;').replace(/"/g,'&quot;').replace(/'/g,'&#39;');
}
function fmtClock(ts) {
var d = new Date(ts * 1000);
function p(n) { return n < 10 ? '0' + n : '' + n; }
var t = p(d.getHours()) + ':' + p(d.getMinutes()) + ':' + p(d.getSeconds());
return (d.toDateString() === new Date().toDateString())
? t
: d.getFullYear() + '-' + p(d.getMonth() + 1) + '-' + p(d.getDate()) + ' ' + t;
}
function logRowHtml(msg) {
var lvl = (msg.level || 'INFO');
var lc = lvl.toLowerCase();
var chipCls = lc === 'critical' ? 'crit' : (lc === 'warning' ? 'warn' : (lc === 'recover' ? 'ok' : ''));
var short = lc === 'critical' ? 'CRIT' : (lc === 'warning' ? 'WARN' : escHtml(lvl));
return '<div class="logrow">'
+ '<span class="t">' + fmtClock(msg.ts) + '</span>'
+ '<span class="lvlc"><span class="chip ' + chipCls + '">' + short + '</span></span>'
+ '<span class="h">' + escHtml(msg.host || '') + (msg.service ? ' · ' + escHtml(msg.service) : '') + '</span>'
+ '<span class="m">' + escHtml(msg.message || '') + '</span>'
+ '</div>';
}
function currentFilters() {
return {
host: document.getElementById('filter-host').value.trim(),
level: document.getElementById('filter-level').value,
q: document.getElementById('filter-msg').value.trim()
};
}
function matchesFilters(msg) {
var f = currentFilters();
if (f.host && !(msg.host || '').toLowerCase().includes(f.host.toLowerCase())) return false;
if (f.level && (msg.level || '').toLowerCase() !== f.level) return false;
if (f.q && !(msg.message || '').toLowerCase().includes(f.q.toLowerCase())) return false;
return true;
}
function updateCount() {
var n = document.querySelectorAll('#messages .logrow').length;
document.getElementById('log-count').textContent = n ? n + ' events loaded' : '';
}
async function fetchLog(reset) {
var mySeq = ++seq;
var box = document.getElementById('messages');
var btn = document.getElementById('load-older');
btn.disabled = true;
var f = currentFilters();
var params = new URLSearchParams();
params.set('limit', String(PAGE_SIZE));
if (f.host) params.set('host', f.host);
if (f.level) params.set('level', f.level);
if (f.q) params.set('q', f.q);
if (!reset && oldestTs != null) params.set('before', String(oldestTs));
var data;
try {
var r = await fetch('/api/0/log?' + params.toString());
if (!r.ok) throw new Error('HTTP ' + r.status);
data = await r.json();
} catch (e) {
if (mySeq === seq && reset) {
box.innerHTML = '<div class="empty">Failed to load events: ' + escHtml(e.message) + '</div>';
}
btn.disabled = false;
return;
}
if (mySeq !== seq) return; // a newer request superseded this one
if (reset) { box.innerHTML = ''; oldestTs = null; }
var html = '';
data.events.forEach(function (ev) {
html += logRowHtml(ev);
if (ev.ts != null && (oldestTs == null || ev.ts < oldestTs)) oldestTs = ev.ts;
if (ev.ts != null && (newestTs == null || ev.ts > newestTs)) newestTs = ev.ts;
});
box.insertAdjacentHTML('beforeend', html);
if (!box.children.length) box.innerHTML = '<div class="empty">No events found.</div>';
btn.style.display = data.more ? '' : 'none';
btn.disabled = false;
updateCount();
}
function onFilterChange() { fetchLog(true); }
function debounce(fn, ms) {
var t;
return function () { clearTimeout(t); t = setTimeout(fn, ms); };
}
document.getElementById('filter-host').addEventListener('input', debounce(onFilterChange, 300));
document.getElementById('filter-msg').addEventListener('input', debounce(onFilterChange, 300));
document.getElementById('filter-level').addEventListener('change', onFilterChange);
document.getElementById('load-older').addEventListener('click', function () { fetchLog(false); });
// ---- live tail over the shared websocket --------------------------------
function setWsStatus(ok) {
var el = document.getElementById('ws-status');
el.classList.toggle('reconnecting', !ok);
el.textContent = ok ? 'live' : 'reconnecting…';
}
function WS_Connect() {
if (!("WebSocket" in window)) return;
var ws_hbd = new WebSocket("{{ heartbeat_ws_url }}");
ws_hbd.onopen = function () {
setWsStatus(true);
ws_hbd.send("heartbeat_web");
};
ws_hbd.onmessage = function (event) {
var state = JSON.parse(event.data);
// history replays are covered by the API seed; only tail new events
if (state.type !== "message" || state.history) return;
var msg = state.data;
if (!matchesFilters(msg)) return;
if (msg.ts != null && newestTs != null && msg.ts <= newestTs) return; // already loaded via the API
if (msg.ts != null && (newestTs == null || msg.ts > newestTs)) newestTs = msg.ts;
var box = document.getElementById('messages');
var placeholder = box.querySelector('.empty');
if (placeholder) placeholder.remove();
box.insertAdjacentHTML('afterbegin', logRowHtml(msg));
updateCount();
};
ws_hbd.onclose = function () {
setWsStatus(false);
setTimeout(WS_Connect, 3000);
};
}
// ---- boot ----------------------------------------------------------------
fetchLog(true);
WS_Connect();
</script>
</body>
</html>
+1
View File
@@ -6,6 +6,7 @@
<a href="/live"{% if active_page == "live" %} class="active"{% endif %}>Live Dashboard</a>
<a href="/plugins"{% if active_page == "plugins" %} class="active"{% endif %}>Host Overview</a>
<a href="/alerts"{% if active_page == "alerts" %} class="active"{% endif %}>Alerts</a>
<a href="/log"{% if active_page == "log" %} class="active"{% endif %}>Log</a>
{% if current_user %}
<a href="/settings"{% if active_page == "settings" %} class="active"{% endif %}>Settings</a>
{% endif %}
+175 -325
View File
@@ -2,38 +2,30 @@
<html>
{% include 'head.html' %}
<link rel="stylesheet" href="/static/hbd-ui.css">
<script src="/static/hbd-ui.js"></script>
<style>
body { overflow: hidden; }
body { overflow: hidden; background: var(--st-paper); }
/* This page scrolls inside .container, not the body — the toolbar is
always in view, and sticky misbehaves inside an overflow:hidden body. */
.st-toolbar { position: static; }
.container {
max-width: 1400px;
margin: 0 auto;
max-height: calc(100vh - 120px);
max-height: calc(100vh - var(--bars-h, 94px) - 30px);
overflow-y: auto;
padding-right: 10px;
padding: 0 10px 20px;
}
h1 {
color: #333;
margin-bottom: 5px;
margin-top: 15px;
font-size: 1.5em;
}
.subtitle {
color: #666;
margin-bottom: 15px;
font-size: 0.9em;
}
/* ── Host cards ─────────────────────────────────────────────── */
/* ── Host cards: one record list per host ───────────────────── */
.host-card {
background: white;
border-radius: 6px;
padding: 0;
margin-bottom: 10px;
box-shadow: 0 1px 4px rgba(0,0,0,0.1);
background: var(--st-surface);
border: 1px solid var(--st-line);
border-radius: 8px;
margin-bottom: 8px;
}
.host-header {
@@ -42,115 +34,74 @@
gap: 10px;
cursor: pointer;
user-select: none;
padding: 10px 15px;
border-radius: 6px;
padding: 8px 14px;
border-radius: 8px;
}
.host-header:hover { background: var(--st-surface-2); border-radius: 8px 8px 0 0; }
.host-card.collapsed .host-header:hover { border-radius: 8px; }
.host-header:hover { background: #f9f9f9; border-radius: 6px 6px 0 0; }
.host-card.collapsed .host-header:hover { border-radius: 6px; }
.host-left {
display: flex;
align-items: center;
gap: 8px;
flex-shrink: 0;
}
.host-left { display: flex; align-items: center; gap: 8px; flex-shrink: 0; }
.collapse-icon {
font-size: 1em;
color: #888;
font-size: .85em;
color: var(--st-faint);
transition: transform 0.2s;
min-width: 16px;
min-width: 14px;
}
@media (prefers-reduced-motion: reduce) { .collapse-icon, .acc-icon { transition: none; } }
.host-card.collapsed .collapse-icon { transform: rotate(-90deg); }
.host-name {
font-size: 1.05em;
font-weight: bold;
color: #333;
font-family: var(--st-mono);
font-size: 13.5px;
font-weight: 600;
color: var(--st-ink);
white-space: nowrap;
}
/* ── Glance strip ───────────────────────────────────────────── */
.glance-strip {
display: flex;
align-items: center;
gap: 8px;
flex: 1;
flex-wrap: wrap;
padding: 0 12px;
}
.glance-strip { display: flex; align-items: center; gap: 5px; flex: 1; flex-wrap: wrap; padding: 0 8px; }
.glance-chip {
font-size: 0.78em;
padding: 2px 9px;
border-radius: 10px;
font-weight: 500;
white-space: nowrap;
background: #e8f5e9;
color: #2e7d32;
display: inline-flex; align-items: center;
font-family: var(--st-mono); font-size: 11px; line-height: 1.3;
padding: 3px 7px; border-radius: 4px; white-space: nowrap;
background: var(--st-ok-soft); color: var(--st-ok);
}
.glance-chip.warn { background: #fff3e0; color: #e65100; }
.glance-chip.crit { background: #ffebee; color: #b71c1c; }
.glance-chip.neutral { background: #f5f5f5; color: #555; }
.glance-loading { font-size: 0.8em; color: #bbb; font-style: italic; }
.glance-chip.warn { background: var(--st-warn-soft); color: var(--st-warn); }
.glance-chip.crit { background: var(--st-crit-soft); color: var(--st-crit); }
.glance-chip.neutral { background: var(--st-surface-2); color: var(--st-muted); }
.glance-loading { font-size: 11.5px; color: var(--st-faint); font-style: italic; }
/* ── Host right zone ────────────────────────────────────────── */
.host-right {
display: flex;
align-items: center;
gap: 10px;
flex-shrink: 0;
}
.host-right { display: flex; align-items: center; gap: 8px; flex-shrink: 0; }
.nagios-badge {
font-size: 0.75em;
font-weight: bold;
padding: 2px 10px;
border-radius: 10px;
background: #9e9e9e;
color: white;
text-transform: uppercase;
white-space: nowrap;
font-family: var(--st-mono); font-size: 10.5px; font-weight: 700;
padding: 3px 9px; border-radius: 4px;
background: var(--st-surface-2); color: var(--st-muted);
text-transform: uppercase; white-space: nowrap;
}
.nagios-badge.ok { background: #4caf50; }
.nagios-badge.warning { background: #ff9800; }
.nagios-badge.critical { background: #f44336; }
.nagios-badge.ok { background: var(--st-ok-soft); color: var(--st-ok); }
.nagios-badge.warning { background: var(--st-warn-soft); color: var(--st-warn); }
.nagios-badge.critical { background: var(--st-crit-soft); color: var(--st-crit); }
.os-label {
font-size: 0.75em;
color: #999;
white-space: nowrap;
max-width: 200px;
overflow: hidden;
text-overflow: ellipsis;
font-size: 11px; color: var(--st-faint); white-space: nowrap;
max-width: 200px; overflow: hidden; text-overflow: ellipsis;
}
.host-action-btn {
font-size: 0.75em;
font-weight: bold;
padding: 3px 10px;
border-radius: 4px;
border: none;
cursor: pointer;
text-decoration: none;
white-space: nowrap;
font: 600 12px/1 inherit;
padding: 4px 10px; border-radius: 6px;
border: 1px solid var(--st-line);
background: var(--st-surface);
cursor: pointer; text-decoration: none; white-space: nowrap;
}
.host-action-btn.update-btn {
background: #e3f2fd;
color: #1565c0;
}
.host-action-btn.update-btn:hover { background: #bbdefb; }
.host-action-btn.delete-btn {
background: #ffebee;
color: #c62828;
}
.host-action-btn.delete-btn:hover { background: #ffcdd2; }
.host-action-btn.update-btn { color: var(--st-accent); }
.host-action-btn.update-btn:hover { background: var(--st-accent-soft); border-color: var(--st-accent); }
.host-action-btn.delete-btn { color: var(--st-crit); }
.host-action-btn.delete-btn:hover { background: var(--st-crit-soft); border-color: var(--st-crit); }
/* ── Action result toast ───────────────────────────────────── */
#action-toast {
@@ -158,8 +109,8 @@
bottom: 24px;
left: 50%;
transform: translateX(-50%) translateY(20px);
background: #323232;
color: #fff;
background: var(--st-ink);
color: var(--st-paper);
padding: 12px 22px;
border-radius: 6px;
font-size: 0.9em;
@@ -171,296 +122,171 @@
z-index: 9000;
white-space: pre-wrap;
}
#action-toast.show {
opacity: 1;
transform: translateX(-50%) translateY(0);
}
#action-toast.error { background: #c62828; }
#action-toast.show { opacity: 1; transform: translateX(-50%) translateY(0); }
#action-toast.error { background: var(--st-crit); color: #fff; }
/* ── Host body ──────────────────────────────────────────────── */
.host-body {
padding: 8px 15px 12px;
border-top: 1px solid #f0f0f0;
}
.host-body { padding: 8px 14px 12px; border-top: 1px solid var(--st-line-soft); }
.host-card.collapsed .host-body { display: none; }
/* ── Plugin accordions ──────────────────────────────────────── */
.plugin-accordion {
border: 1px solid #e8e8e8;
border-radius: 4px;
border: 1px solid var(--st-line-soft);
border-radius: 6px;
margin-bottom: 5px;
overflow: hidden;
}
.plugin-acc-header {
display: flex;
align-items: center;
gap: 10px;
padding: 7px 12px;
cursor: pointer;
background: #fafafa;
display: flex; align-items: center; gap: 10px;
padding: 6px 12px; cursor: pointer;
background: var(--st-surface-2);
user-select: none;
}
.plugin-acc-header:hover { background: var(--st-accent-soft); }
.plugin-acc-header:hover { background: #f0f4ff; }
.acc-icon {
font-size: 0.7em;
color: #999;
transition: transform 0.15s;
min-width: 12px;
}
.acc-icon { font-size: 0.7em; color: var(--st-faint); transition: transform 0.15s; min-width: 12px; }
.plugin-accordion:not(.collapsed) .acc-icon { transform: rotate(90deg); }
.plugin-label {
font-weight: 600;
font-size: 1.00em;
color: #444;
font-family: var(--st-mono);
font-weight: 600; font-size: 12.5px;
color: var(--st-ink);
min-width: 140px;
}
.plugin-summary {
font-size: 0.82em;
color: #888;
flex: 1;
}
.plugin-summary { font-family: var(--st-mono); font-size: 11.5px; color: var(--st-faint); flex: 1; }
.plugin-accordion.collapsed .plugin-acc-body { display: none; }
.plugin-acc-body { padding: 10px 12px; }
/* ── Tables ─────────────────────────────────────────────────── */
.data-table {
width: 100%;
border-collapse: collapse;
font-size: 1.00em;
background: #fff;
box-shadow: 0 1px 3px rgba(0,0,0,0.08);
border-radius: 4px;
font-size: 12.5px;
background: var(--st-surface);
border: 1px solid var(--st-line-soft);
border-radius: 6px;
overflow: hidden;
margin-bottom: 10px;
font-variant-numeric: tabular-nums;
}
.data-table thead { background: #2196f3; color: white; }
.data-table thead { background: var(--st-surface-2); }
.data-table th {
padding: 7px 10px;
text-align: left;
font-weight: 600;
text-transform: uppercase;
font-size: 0.75em;
letter-spacing: 0.4px;
font-family: var(--st-mono);
font-size: 10.5px; text-transform: uppercase; letter-spacing: .05em;
color: var(--st-faint); font-weight: 600;
text-align: left; padding: 5px 8px;
border-bottom: 1px solid var(--st-line);
}
.data-table th.num { text-align: right; }
.data-table th.center { text-align: center; }
.data-table td {
/* padding: 6px 10px; */
border-top: 1px solid #e8e8e8;
color: #333;
padding: 4px 8px;
border-top: 1px solid var(--st-line-soft);
color: var(--st-ink);
}
.data-table td.num {
text-align: right;
font-family: 'Courier New', monospace;
font-size: 0.95em;
}
.data-table td.num { text-align: right; font-family: var(--st-mono); }
.data-table td.center { text-align: center; }
.data-table td.key { color: #666; font-weight: 500; width: 38%; }
.data-table td.key { color: var(--st-muted); font-weight: 500; width: 38%; }
.data-table tbody tr:hover { background: var(--st-surface-2); }
.data-table tbody tr:nth-child(even) { background: #fafafa; }
.data-table tbody tr:hover { background: #f0f4ff; }
.iface-name { font-weight: bold; color: #2196f3; }
.iface-name { font-family: var(--st-mono); font-weight: 600; color: var(--st-accent); }
/* ── Percent bars ───────────────────────────────────────────── */
.bar-wrap {
display: flex;
align-items: center;
gap: 8px;
}
.bar-wrap { display: flex; align-items: center; gap: 8px; min-width: 120px; }
.bar-track {
display: inline-block;
width: 70px;
height: 6px;
background: #e0e0e0;
border-radius: 3px;
vertical-align: middle;
flex-shrink: 0;
flex: 1; height: 7px; border-radius: 4px;
background: var(--st-line-soft); overflow: hidden; min-width: 60px;
}
.bar-fill {
height: 6px;
border-radius: 3px;
background: #4caf50;
max-width: 100%;
}
.bar-fill.warn { background: #ff9800; }
.bar-fill.crit { background: #f44336; }
.bar-fill { height: 100%; border-radius: 4px; background: var(--st-ok); }
.bar-fill.warn { background: var(--st-warn); }
.bar-fill.crit { background: var(--st-crit); }
/* ── Disk two-table layout ──────────────────────────────────── */
.flex-tables {
display: flex;
gap: 14px;
flex-wrap: wrap;
}
.flex-tables { display: flex; gap: 14px; flex-wrap: wrap; }
.flex-tables > div { flex: 1 1 380px; }
.table-section-label {
font-size: 0.78em;
font-weight: 600;
text-transform: uppercase;
color: #888;
letter-spacing: 0.4px;
margin-bottom: 4px;
font-family: var(--st-mono);
font-size: 11px; font-weight: 600; color: var(--st-faint);
text-transform: uppercase; letter-spacing: .05em;
margin: 4px 0 6px;
}
/* ── Status / misc ──────────────────────────────────────────── */
.status-up { color: var(--st-ok); font-weight: bold; }
.status-down { color: var(--st-crit); font-weight: bold; }
.status-up { color: #4caf50; font-weight: bold; }
.status-down { color: #f44336; font-weight: bold; }
.pct-ok { color: var(--st-ok); font-weight: bold; }
.pct-warn { color: var(--st-warn); font-weight: bold; }
.pct-crit { color: var(--st-crit); font-weight: bold; }
.pct-ok { color: #2e7d32; font-weight: bold; }
.pct-warn { color: #e65100; font-weight: bold; }
.pct-crit { color: #b71c1c; font-weight: bold; }
.check-ok { background: var(--st-ok-soft); }
.check-warning { background: var(--st-warn-soft); }
.check-critical { background: var(--st-crit-soft); }
.check-unknown { background: var(--st-surface-2); }
.check-ok { background: #e8f5e9; }
.check-warning { background: #fff3e0; }
.check-critical { background: #ffebee; }
.check-unknown { background: #f5f5f5; }
.check-status-ok { color: var(--st-ok); font-weight: bold; }
.check-status-warning { color: var(--st-warn); font-weight: bold; }
.check-status-critical { color: var(--st-crit); font-weight: bold; }
.check-status-unknown { color: var(--st-faint); font-weight: bold; }
.check-status-ok { color: #2e7d32; font-weight: bold; }
.check-status-warning { color: #e65100; font-weight: bold; }
.check-status-critical { color: #b71c1c; font-weight: bold; }
.check-status-unknown { color: #777; font-weight: bold; }
.check-output { font-size: 0.9em; color: #555; }
.check-output { font-size: 0.9em; color: var(--st-muted); }
.timestamp {
color: #bbb;
font-size: 0.75em;
margin-top: 8px;
padding-top: 6px;
border-top: 1px solid #f0f0f0;
text-align: right;
}
.no-data {
text-align: center;
padding: 20px;
color: #aaa;
font-style: italic;
font-size: 0.9em;
}
.loading {
text-align: center;
padding: 12px;
color: #aaa;
font-size: 1.00em;
font-size: 11px; color: var(--st-faint);
margin-top: 6px; padding-top: 6px;
border-top: 1px solid var(--st-line-soft);
}
.no-data { text-align: center; padding: 40px; color: var(--st-faint); }
.loading { text-align: center; padding: 12px; color: var(--st-faint); font-style: italic; }
.error {
background: #ffebee;
border-left: 3px solid #f44336;
padding: 8px 12px;
margin: 8px 0;
border-radius: 3px;
color: #c62828;
font-size: 1.00em;
background: var(--st-crit-soft);
border-left: 3px solid var(--st-crit);
padding: 12px 16px; margin: 10px 0;
border-radius: 6px; color: var(--st-crit);
}
/* ── Scrollbar ──────────────────────────────────────────────── */
.container::-webkit-scrollbar { width: 8px; }
.container::-webkit-scrollbar-track { background: #f1f1f1; border-radius: 4px; }
.container::-webkit-scrollbar-thumb { background: #ccc; border-radius: 4px; }
.container::-webkit-scrollbar-thumb:hover { background: #999; }
.container::-webkit-scrollbar-track { background: var(--st-surface-2); border-radius: 4px; }
.container::-webkit-scrollbar-thumb { background: var(--st-line); border-radius: 4px; }
.container::-webkit-scrollbar-thumb:hover { background: var(--st-faint); }
/* ── Host info section ──────────────────────────────────────────────────── */
/* ── Host info section ──────────────────────────────────────── */
.host-info-section {
padding: 12px 16px;
background: #fafafa;
border-bottom: 1px solid #e0e0e0;
font-size: 1.00em;
background: var(--st-surface-2);
border-radius: 6px;
padding: 8px 12px;
margin-bottom: 8px;
font-size: 12.5px;
}
.info-meta {
display: grid;
grid-template-columns: max-content 1fr;
gap: 3px 14px;
margin-bottom: 10px;
}
.info-label { font-weight: 600; color: #555; white-space: nowrap; }
.info-value { color: #222; }
.info-meta { display: flex; flex-wrap: wrap; gap: 4px 18px; }
.info-label { font-family: var(--st-mono); font-size: 11px; font-weight: 600; color: var(--st-faint); white-space: nowrap; }
.info-value { color: var(--st-ink); }
.info-thresholds-title {
font-weight: 600;
color: #555;
margin-bottom: 6px;
font-family: var(--st-mono);
font-size: 11px; font-weight: 600; color: var(--st-faint);
text-transform: uppercase; letter-spacing: .05em;
margin: 8px 0 4px;
}
.info-note { color: #888; font-style: italic; }
.info-loading { color: #bbb; font-style: italic; }
.threshold-covers { font-size: 1.00em; color: #777; font-style: italic; }
/* ── Dark mode ── */
html[data-theme="dark"] h1 { color: var(--text); }
html[data-theme="dark"] .subtitle { color: var(--text-sec); }
html[data-theme="dark"] .host-card { background: var(--surface); }
html[data-theme="dark"] .host-header:hover { background: var(--surface-2); }
html[data-theme="dark"] .host-name { color: var(--text); }
html[data-theme="dark"] .collapse-icon,
html[data-theme="dark"] .acc-icon { color: var(--text-muted); }
html[data-theme="dark"] .host-body { border-top-color: var(--border-3); }
html[data-theme="dark"] .plugin-accordion { border-color: var(--border); }
html[data-theme="dark"] .plugin-acc-header { background: var(--surface-2); }
html[data-theme="dark"] .plugin-acc-header:hover { background: var(--surface-3); }
html[data-theme="dark"] .plugin-label { color: var(--text-2); }
html[data-theme="dark"] .plugin-summary { color: var(--text-muted); }
html[data-theme="dark"] .data-table { background: var(--surface); }
html[data-theme="dark"] .data-table td { border-top-color: var(--border); color: var(--text); }
html[data-theme="dark"] .data-table td.key { color: var(--text-sec); }
html[data-theme="dark"] .data-table tbody tr:nth-child(even) { background: var(--surface-2); }
html[data-theme="dark"] .data-table tbody tr:hover { background: #1e3a5f; }
html[data-theme="dark"] .bar-track { background: var(--border); }
html[data-theme="dark"] .table-section-label { color: var(--text-muted); }
html[data-theme="dark"] .no-data,
html[data-theme="dark"] .loading { color: var(--text-dim); }
html[data-theme="dark"] .timestamp { color: var(--text-dim); border-top-color: var(--border-3); }
html[data-theme="dark"] .glance-chip.neutral { background: var(--surface-3); color: var(--text-sec); }
html[data-theme="dark"] .os-label { color: var(--text-muted); }
html[data-theme="dark"] .host-info-section { background: var(--surface-2); border-bottom-color: var(--border); }
html[data-theme="dark"] .info-label { color: var(--text-3); }
html[data-theme="dark"] .info-value { color: var(--text); }
html[data-theme="dark"] .info-thresholds-title { color: var(--text-3); }
html[data-theme="dark"] .info-note,
html[data-theme="dark"] .info-loading,
html[data-theme="dark"] .threshold-covers { color: var(--text-muted); }
html[data-theme="dark"] .check-ok { background: #0d2e17; }
html[data-theme="dark"] .check-warning { background: #2e1a00; }
html[data-theme="dark"] .check-critical { background: #2e0a0a; }
html[data-theme="dark"] .check-unknown { background: var(--surface-2); }
html[data-theme="dark"] .check-output { color: var(--text-sec); }
html[data-theme="dark"] .container::-webkit-scrollbar-track { background: var(--surface-2); }
html[data-theme="dark"] .container::-webkit-scrollbar-thumb { background: var(--border); }
.info-note { color: var(--st-faint); font-style: italic; }
.info-loading { color: var(--st-faint); font-style: italic; }
.threshold-covers { font-size: 11.5px; color: var(--st-faint); font-style: italic; }
</style>
<body>
{% include 'nav.html' %}
<div class="st-toolbar">
<span class="brand">hbd<span class="tld">·overview</span></span>
<span class="hintline">per-host metrics — expand a host for plugin details</span>
</div>
<svg class="pulse" viewBox="0 0 1200 14" preserveAspectRatio="none" aria-hidden="true">
<polyline points="0,10 340,10 352,10 358,3 364,13 370,1 378,12 384,10 560,10 572,10 578,3 584,13 590,1 598,12 604,10 1200,10"/>
</svg>
<div class="container">
<h1>{{ header }}</h1>
<p class="subtitle">Per-host system metrics — expand a host to see plugin details</p>
{% if not hosts %}
<div class="no-data">
@@ -618,6 +444,30 @@
<span class="info-label">Last Packet</span><span class="info-value">${lastPkt}</span>
</div>`;
if (data.connections && data.connections.length) {
html += `<div class="info-thresholds-title">Connectivity</div>
<table class="data-table"><thead><tr>
<th>Family</th><th>Address</th><th>State</th>
<th class="num">RTT</th><th>Last Change</th><th>Last Packet</th>
</tr></thead><tbody>`;
for (const c of data.connections) {
const st = escHtml(c.state || '—');
const stCls = c.state === 'up' ? 'status-up' : (c.state ? 'status-down' : '');
const rtt = (c.state === 'up' && c.rtt) ? Math.round(c.rtt) + ' ms' : '—';
const chg = c.statetime ? new Date(c.statetime * 1000).toLocaleString() : '—';
const seen = c.lastbeat ? new Date(c.lastbeat * 1000).toLocaleString() : '—';
html += `<tr>
<td class="key">${escHtml(c.family)}</td>
<td>${escHtml(c.addr || '—')}</td>
<td><span class="${stCls}">${st}</span></td>
<td class="num">${rtt}</td>
<td>${chg}</td>
<td>${seen}</td>
</tr>`;
}
html += `</tbody></table>`;
}
if (data.thresholds === null) {
html += `<div class="info-note">Threshold alerting not configured.</div>`;
} else if (data.thresholds.length === 0) {
File diff suppressed because it is too large Load Diff
+2
View File
@@ -1251,6 +1251,7 @@ class ThresholdChecker:
title=title,
body=body,
level=lvl,
service=short_path,
),
))
@@ -1534,6 +1535,7 @@ class ThresholdChecker:
title=f"[REMINDER/{alert_state.level.name}] {host_name} {short_path}",
body=body,
level=alert_state.level.name,
service=short_path,
),
))
logger.info("Re-notification sent: %s", message)
+1 -1
View File
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
[project]
name = "hbd"
version = "5.3.12"
version = "5.4.0"
description = "Heartbeat monitoring system — client (hbc) and server (hbd)"
readme = "README.md"
requires-python = ">=3.11"
+14 -1
View File
@@ -375,6 +375,10 @@ static const char *jstr(const jval_t *v, const char *def) {
* Config
* ============================================================ */
/* Set from config in cfg_load. Off by default: a CMD packet is an unauthenticated
* UDP datagram, so executing one must be opted into per host. */
static bool g_allow_remote_command = false;
typedef struct {
int hb_port, interval;
char owner[256];
@@ -429,6 +433,9 @@ static void config_load(config_t *cfg, const char *path) {
if ((v = jget(root, "hb_port"))) cfg->hb_port = jint(v, cfg->hb_port);
if ((v = jget(root, "interval"))) cfg->interval = jint(v, cfg->interval);
if ((v = jget(root, "owner"))) snprintf(cfg->owner, sizeof(cfg->owner), "%s", jstr(v, ""));
if ((v = jget(root, "allow_remote_command"))) g_allow_remote_command = jint(v, 0) != 0;
if (g_allow_remote_command)
LOGI("allow_remote_command is true - CMD packets from the server will be executed");
jval_t *plugins = jget(root, "plugins");
@@ -591,7 +598,13 @@ static void conn_recv(conn_t *c) {
LOGD("ACK rtt=%.1fms", c->rtt);
} else if (strcmp(id, "CMD") == 0) {
const char *cmd = kv_get(&msg, "cmd");
if (cmd) {
if (cmd && !g_allow_remote_command) {
LOGI("refused command (allow_remote_command is false): %s", cmd);
kvdict_t rep; kv_clear(&rep);
kv_set(&rep, "service", "command");
kv_set(&rep, "msg", "Refused: allow_remote_command is false");
conn_send(c, "HTB", &rep);
} else if (cmd) {
LOGI("CMD: %s", cmd);
char out[4096] = "";
FILE *p = popen(cmd, "r");
+15 -1
View File
@@ -41,7 +41,7 @@ from pathlib import Path
from typing import Any, Dict, List, Optional, Tuple
# updated by scripts/bumpminor.sh
__version__ = "5.3.12"
__version__ = "5.4.0"
# ---------------------------------------------------------------------------
# Protocol (mirrors hbd/common/proto.py)
@@ -115,9 +115,14 @@ _DEFAULTS: Dict[str, Any] = {
"hb_port": 50003,
"interval": 10,
"owner": None,
"allow_remote_command": False, # Execute shell commands received in CMD packets
"plugins": {},
}
# Set from config in main(). Off by default: a CMD packet is an unauthenticated
# UDP datagram, so executing one must be opted into per host.
_allow_remote_command = False
def _load_config(path: Optional[str] = None) -> Dict[str, Any]:
cfg = dict(_DEFAULTS)
@@ -870,6 +875,10 @@ async def _handle_command(conn: AsyncConnection, msg: Dict[str, Any]):
if not cmd:
return
log = logging.getLogger("hbc.cmd")
if not _allow_remote_command:
log.warning("refused command (allow_remote_command is false): %s", cmd)
await conn.sendto({"service": "command", "msg": "Refused: allow_remote_command is false"})
return
log.info("exec: %s", cmd)
try:
out = subprocess.check_output(
@@ -1180,6 +1189,11 @@ def main(argv=None):
cfg = _load_config(args.configfile)
global _allow_remote_command
_allow_remote_command = bool(cfg.get("allow_remote_command", False))
if _allow_remote_command:
logging.warning("allow_remote_command is true — CMD packets from the server will be executed")
if args.daemon:
_daemonize()
_reconfigure_syslog(level)
+14
View File
@@ -112,9 +112,14 @@ _DEFAULTS: Dict[str, Any] = {
"hb_port": 50003,
"interval": 10,
"owner": None,
"allow_remote_command": False, # Execute shell commands received in CMD packets
"plugins": {},
}
# Set from config in main(). Off by default: a CMD packet is an unauthenticated
# UDP datagram, so executing one must be opted into per host.
_allow_remote_command = False
def _load_config(path: Optional[str] = None) -> Dict[str, Any]:
cfg = dict(_DEFAULTS)
@@ -886,6 +891,10 @@ async def _handle_command(conn: AsyncConnection, msg: Dict[str, Any]):
if not cmd:
return
log = logging.getLogger("hbc.cmd")
if not _allow_remote_command:
log.warning("refused command (allow_remote_command is false): %s", cmd)
await conn.sendto({"service": "command", "msg": "Refused: allow_remote_command is false"})
return
log.info("exec: %s", cmd)
try:
out = subprocess.check_output(
@@ -1180,6 +1189,11 @@ def main(argv=None):
cfg = _load_config(args.configfile)
global _allow_remote_command
_allow_remote_command = bool(cfg.get("allow_remote_command", False))
if _allow_remote_command:
logging.warning("allow_remote_command is true — CMD packets from the server will be executed")
try:
rc = asyncio.run(_async_main(args, cfg))
except KeyboardInterrupt:
+103
View File
@@ -0,0 +1,103 @@
"""Tests for the allow_remote_command gate on CMD packets in hbc."""
import asyncio
import subprocess
import pytest
from hbd.client import main as client_main
from hbd.client.config import CLIENT_DEFAULTS, load_config
class FakeConn:
"""Stand-in for AsyncConnection that records what the client sends back."""
def __init__(self):
self.sent = []
async def sendto(self, msg, msg_id="HTB"):
self.sent.append(msg)
@pytest.fixture
def conn(monkeypatch):
"""A fake connection with remote commands disabled and subprocess booby-trapped."""
monkeypatch.setattr(client_main, "allow_remote_command", False)
def explode(*a, **kw):
raise AssertionError("subprocess must not run when the command is refused")
monkeypatch.setattr(subprocess, "check_output", explode)
return FakeConn()
def test_default_is_false():
assert CLIENT_DEFAULTS["allow_remote_command"] is False
assert load_config("/nonexistent/hbc.yaml")["allow_remote_command"] is False
assert client_main.allow_remote_command is False
def test_refused_when_disabled(conn):
asyncio.run(client_main.handle_command(conn, {"cmd": "id"}))
assert conn.sent == [{
"service": "command",
"msg": "Refused: allow_remote_command is false",
}]
def test_refusal_reported_for_every_attempt(conn):
for _ in range(3):
asyncio.run(client_main.handle_command(conn, {"cmd": "rm -rf /"}))
assert len(conn.sent) == 3
def test_empty_command_is_ignored_either_way(conn, monkeypatch):
asyncio.run(client_main.handle_command(conn, {}))
monkeypatch.setattr(client_main, "allow_remote_command", True)
asyncio.run(client_main.handle_command(conn, {"cmd": ""}))
assert conn.sent == []
def test_executed_when_enabled(conn, monkeypatch):
monkeypatch.setattr(client_main, "allow_remote_command", True)
monkeypatch.setattr(subprocess, "check_output", lambda *a, **kw: b"uid=0(root)")
asyncio.run(client_main.handle_command(conn, {"cmd": "id"}))
assert conn.sent == [{"service": "command", "msg": "OK uid=0(root)"}]
def test_failure_reported_when_enabled(conn, monkeypatch):
monkeypatch.setattr(client_main, "allow_remote_command", True)
def fail(*a, **kw):
raise subprocess.CalledProcessError(1, "false")
monkeypatch.setattr(subprocess, "check_output", fail)
asyncio.run(client_main.handle_command(conn, {"cmd": "false"}))
assert len(conn.sent) == 1
assert conn.sent[0]["msg"].startswith("CalledProcessError ")
def test_config_file_can_opt_in(tmp_path):
cfg_file = tmp_path / "hbc.yaml"
cfg_file.write_text("allow_remote_command: true\n", encoding="utf-8")
assert load_config(str(cfg_file))["allow_remote_command"] is True
@pytest.mark.parametrize("configured, expected", [(True, True), (False, False), (None, False)])
def test_async_main_sets_the_flag_from_config(monkeypatch, configured, expected):
"""async_main must apply the config value before any CMD packet can arrive."""
monkeypatch.setattr(client_main, "allow_remote_command", not expected)
class Bail(Exception):
pass
# gethostname() is the first call after the flag is assigned — bail there so the
# test never opens a socket.
def bail():
raise Bail
monkeypatch.setattr(client_main.socket, "gethostname", bail)
cfg = {} if configured is None else {"allow_remote_command": configured}
with pytest.raises(Bail):
asyncio.run(client_main.async_main(object(), cfg))
assert client_main.allow_remote_command is expected
+265
View File
@@ -0,0 +1,265 @@
"""Tests for the dedicated events journal (write path and read path)."""
import asyncio
import json
from hbd.server import journal
def _make_journal(tmp_path, **overrides):
cfg = {"journal_dir": str(tmp_path), "journal_file": "events.journal"}
cfg.update(overrides)
j = journal.MessageJournal(cfg)
assert asyncio.run(j.initialize())
return j
def _read_lines(tmp_path, name="events.journal"):
return (tmp_path / name).read_text(encoding="utf-8").splitlines()
EV1 = {"ts": 1000.0, "host": "h1", "level": "INFO", "service": None, "message": "host up"}
EV2 = {"ts": 2000.0, "host": "h2", "level": "CRITICAL", "service": "cpu", "message": "cpu high"}
def test_log_event_writes_one_json_line(tmp_path):
j = _make_journal(tmp_path)
asyncio.run(j.log_event(EV1))
asyncio.run(j.close())
lines = _read_lines(tmp_path)
assert len(lines) == 1
assert json.loads(lines[0]) == EV1
def test_log_event_appends_in_order(tmp_path):
j = _make_journal(tmp_path)
asyncio.run(j.log_event(EV1))
asyncio.run(j.log_event(EV2))
asyncio.run(j.close())
lines = _read_lines(tmp_path)
assert [json.loads(ln)["ts"] for ln in lines] == [1000.0, 2000.0]
def test_log_event_rotates_at_max_size(tmp_path):
# max_size fits one serialized event line (~77 bytes) but not two, so the
# second write triggers exactly one rotation
j = _make_journal(tmp_path, journal_max_size=120)
asyncio.run(j.log_event(EV1))
asyncio.run(j.log_event(EV2))
asyncio.run(j.close())
backups = list(tmp_path.glob("events.journal.*"))
assert len(backups) == 1
assert json.loads(backups[0].read_text().splitlines()[0]) == EV1
assert json.loads(_read_lines(tmp_path)[0]) == EV2
def test_log_event_noop_when_disabled(tmp_path):
j = journal.MessageJournal(
{"journal_dir": str(tmp_path), "journal_file": "events.journal", "journal_enabled": False}
)
asyncio.run(j.initialize())
asyncio.run(j.log_event(EV1))
assert not (tmp_path / "events.journal").exists()
def test_backfill_seeds_empty_journal(tmp_path):
j = _make_journal(tmp_path)
asyncio.run(j.backfill([EV1, EV2]))
asyncio.run(j.close())
lines = _read_lines(tmp_path)
assert len(lines) == 2
assert json.loads(lines[0]) == EV1
def test_backfill_skipped_when_journal_nonempty(tmp_path):
(tmp_path / "events.journal").write_text(json.dumps(EV1) + "\n")
j = _make_journal(tmp_path) # initialize() picks up the existing size
asyncio.run(j.backfill([EV2]))
asyncio.run(j.close())
assert len(_read_lines(tmp_path)) == 1
def test_get_events_journal_uses_events_config_keys(tmp_path):
journal._events_journal_instance = None
try:
ej = journal.get_events_journal(
{
"journal_dir": str(tmp_path),
"events_journal_file": "ev.jsonl",
"events_journal_max_size": 12345,
"events_journal_max_backups": 3,
}
)
assert ej.journal_file == "ev.jsonl"
assert ej.max_size == 12345
assert ej.max_backups == 3
assert ej.journal_dir == tmp_path
# singleton: second call returns the same instance
assert journal.get_events_journal() is ej
finally:
journal._events_journal_instance = None
def test_get_events_journal_defaults(tmp_path):
journal._events_journal_instance = None
try:
ej = journal.get_events_journal({"journal_dir": str(tmp_path)})
assert ej.journal_file == "events.journal"
assert ej.max_size == 10 * 1024 * 1024
assert ej.max_backups == 10
finally:
journal._events_journal_instance = None
# ---- read path -------------------------------------------------------------
def _write_journal(path, events):
path.write_text("".join(json.dumps(e) + "\n" for e in events), encoding="utf-8")
def _evts(*ts_list):
return [
{"ts": float(t), "host": f"host{i}", "level": "INFO", "service": None, "message": f"msg {t}"}
for i, t in enumerate(ts_list)
]
def test_read_events_newest_first(tmp_path):
_write_journal(tmp_path / "events.journal", _evts(1, 2, 3))
events, more = journal.read_events(tmp_path)
assert [e["ts"] for e in events] == [3.0, 2.0, 1.0]
assert more is False
def test_read_events_limit_and_more(tmp_path):
_write_journal(tmp_path / "events.journal", _evts(1, 2, 3))
events, more = journal.read_events(tmp_path, limit=2)
assert [e["ts"] for e in events] == [3.0, 2.0]
assert more is True
def test_read_events_before_cursor(tmp_path):
_write_journal(tmp_path / "events.journal", _evts(1, 2, 3))
events, _ = journal.read_events(tmp_path, before=3.0)
assert [e["ts"] for e in events] == [2.0, 1.0]
def test_read_events_spans_rotated_files(tmp_path):
# rotated backup holds the oldest events; current file the newest
_write_journal(tmp_path / "events.journal.20260101-000000", _evts(1, 2))
_write_journal(tmp_path / "events.journal.20260201-000000", _evts(3, 4))
_write_journal(tmp_path / "events.journal", _evts(5, 6))
events, more = journal.read_events(tmp_path, limit=10)
assert [e["ts"] for e in events] == [6.0, 5.0, 4.0, 3.0, 2.0, 1.0]
assert more is False
def test_read_events_pagination_across_files(tmp_path):
_write_journal(tmp_path / "events.journal.20260101-000000", _evts(1, 2))
_write_journal(tmp_path / "events.journal", _evts(3, 4))
page1, more1 = journal.read_events(tmp_path, limit=3)
assert [e["ts"] for e in page1] == [4.0, 3.0, 2.0]
assert more1 is True
page2, more2 = journal.read_events(tmp_path, limit=3, before=page1[-1]["ts"])
assert [e["ts"] for e in page2] == [1.0]
assert more2 is False
def test_read_events_host_filter_substring_case_insensitive(tmp_path):
evs = [
{"ts": 1.0, "host": "Wentworth", "level": "INFO", "service": None, "message": "a"},
{"ts": 2.0, "host": "winter", "level": "INFO", "service": None, "message": "b"},
]
_write_journal(tmp_path / "events.journal", evs)
events, _ = journal.read_events(tmp_path, host="went")
assert [e["host"] for e in events] == ["Wentworth"]
def test_read_events_level_filter_exact_case_insensitive(tmp_path):
evs = [
{"ts": 1.0, "host": "h", "level": "CRITICAL", "service": None, "message": "a"},
{"ts": 2.0, "host": "h", "level": "INFO", "service": None, "message": "b"},
]
_write_journal(tmp_path / "events.journal", evs)
events, _ = journal.read_events(tmp_path, level="critical")
assert [e["level"] for e in events] == ["CRITICAL"]
def test_read_events_message_filter(tmp_path):
evs = [
{"ts": 1.0, "host": "h", "level": "INFO", "service": None, "message": "disk almost full"},
{"ts": 2.0, "host": "h", "level": "INFO", "service": None, "message": "all quiet"},
]
_write_journal(tmp_path / "events.journal", evs)
events, _ = journal.read_events(tmp_path, q="Disk")
assert [e["ts"] for e in events] == [1.0]
def test_read_events_skips_malformed_lines(tmp_path):
p = tmp_path / "events.journal"
p.write_text('{"ts": 1.0, "host": "h", "level": "INFO", "message": "ok"}\nnot json\n[1,2]\n')
events, _ = journal.read_events(tmp_path)
assert [e["ts"] for e in events] == [1.0]
def test_read_events_predicate(tmp_path):
_write_journal(tmp_path / "events.journal", _evts(1, 2))
events, _ = journal.read_events(tmp_path, predicate=lambda e: e["host"] == "host0")
assert [e["host"] for e in events] == ["host0"]
def test_read_events_missing_dir(tmp_path):
events, more = journal.read_events(tmp_path / "nope")
assert events == [] and more is False
def test_filter_events_over_in_memory_ring():
ring = _evts(1, 2, 3) # oldest-first, like data.msgs
events, more = journal.filter_events(reversed(ring), limit=2)
assert [e["ts"] for e in events] == [3.0, 2.0]
assert more is True
def test_filter_events_skips_non_dict_entries():
ring = [
{"ts": 1.0, "host": "h", "level": "INFO", "service": None, "message": "a"},
"legacy string entry",
None,
{"ts": 2.0, "host": "h", "level": "INFO", "service": None, "message": "b"},
]
events, more = journal.filter_events(reversed(ring), limit=10)
assert [e["ts"] for e in events] == [2.0, 1.0]
assert more is False
# ---- eventlog wiring --------------------------------------------------------
def test_eventlog_writes_to_events_journal(tmp_path):
from hbd.server import data, notify
journal._events_journal_instance = None
saved_msgs = data.msgs
data.msgs = []
try:
ej = journal.get_events_journal({"journal_dir": str(tmp_path)})
async def scenario():
await ej.initialize()
notify.setup({}, loop=asyncio.get_running_loop())
notify.eventlog("h1", "INFO", "hello world")
await asyncio.sleep(0.05) # let the scheduled journal write run
await ej.close()
asyncio.run(scenario())
lines = _read_lines(tmp_path)
assert len(lines) == 1
ev = json.loads(lines[0])
assert ev["host"] == "h1"
assert ev["level"] == "INFO"
assert ev["message"] == "hello world"
assert isinstance(ev["ts"], float)
finally:
journal._events_journal_instance = None
data.msgs = saved_msgs
notify._loop = None
+197
View File
@@ -0,0 +1,197 @@
"""Tests for flap detection (hbd.server.flap) and its hook in notify.send_notification."""
import asyncio
import json
import pytest
from hbd.server import flap, hbdclass, notify, users as users_mod
@pytest.fixture(autouse=True)
def clean_flap_state():
"""Every test starts with an empty state and a 3-in-10-minutes config."""
flap._state.clear()
flap.setup({"flap_count": 3, "flap_interval": 10})
yield
flap._state.clear()
flap.setup({})
def alerts(n, host="h1", service="cpu", level="CRITICAL"):
return [flap.observe(host, service, level) for _ in range(n)]
def advance(seconds, host="h1", service="cpu"):
"""Backdate a key's recorded timestamps to simulate the clock moving on."""
st = flap._state[(host, service)]
st["events"] = [t - seconds for t in st["events"]]
if st["ok_since"] is not None:
st["ok_since"] -= seconds
# --- tripping ---------------------------------------------------------------
def test_alerts_up_to_flap_count_pass():
assert alerts(3) == [flap.PASS] * 3
assert flap.flapping_services("h1") == []
def test_exceeding_flap_count_trips_then_suppresses():
assert alerts(5) == [flap.PASS, flap.PASS, flap.PASS, flap.TRIP, flap.SUPPRESS]
assert flap.flapping_services("h1") == ["cpu"]
def test_warning_and_critical_both_count():
flap.observe("h1", "cpu", "WARNING")
flap.observe("h1", "cpu", "CRITICAL")
flap.observe("h1", "cpu", "WARNING")
assert flap.observe("h1", "cpu", "CRITICAL") == flap.TRIP
def test_alerts_outside_the_window_do_not_count():
alerts(3)
advance(11 * 60) # older than flap_interval
assert alerts(3) == [flap.PASS] * 3
assert flap.flapping_services("h1") == []
def test_recover_alone_never_trips():
assert [flap.observe("h1", "cpu", "RECOVER") for _ in range(5)] == [flap.PASS] * 5
# --- suppression while flapping --------------------------------------------
def test_recover_and_info_are_suppressed_while_flapping():
alerts(4)
assert flap.observe("h1", "cpu", "RECOVER") == flap.SUPPRESS
assert flap.observe("h1", "cpu", "INFO") == flap.SUPPRESS
def test_info_passes_when_not_flapping():
assert flap.observe("h1", "cpu", "INFO") == flap.PASS
# --- clearing ---------------------------------------------------------------
def test_clears_silently_one_interval_after_recover():
alerts(4)
flap.observe("h1", "cpu", "RECOVER")
advance(10 * 60)
assert flap.flapping_services("h1") == []
assert flap.observe("h1", "cpu", "CRITICAL") == flap.PASS
def test_still_flapping_before_the_interval_elapses():
alerts(4)
flap.observe("h1", "cpu", "RECOVER")
advance(9 * 60)
assert flap.flapping_services("h1") == ["cpu"]
assert flap.observe("h1", "cpu", "CRITICAL") == flap.SUPPRESS
def test_alert_during_the_quiet_window_keeps_it_flapping():
alerts(4)
flap.observe("h1", "cpu", "RECOVER")
advance(9 * 60)
flap.observe("h1", "cpu", "CRITICAL") # resets the quiet window
advance(2 * 60)
assert flap.flapping_services("h1") == ["cpu"]
# --- keying -----------------------------------------------------------------
def test_services_and_hosts_are_tracked_independently():
alerts(4, service="cpu")
assert flap.observe("h1", "disk", "CRITICAL") == flap.PASS
assert flap.observe("h2", "cpu", "CRITICAL") == flap.PASS
assert flap.flapping_services("h1") == ["cpu"]
assert flap.flapping_services("h2") == []
def test_host_level_events_use_the_empty_service_key():
alerts(4, service="")
assert flap.flapping_services("h1") == [""]
# --- disabled ---------------------------------------------------------------
@pytest.mark.parametrize("cfg", [
{"flap_count": 0, "flap_interval": 10},
{"flap_count": 3, "flap_interval": 0},
{},
])
def test_disabled_never_suppresses(cfg):
flap.setup(cfg)
assert alerts(20) == [flap.PASS] * 20
assert flap.flapping_services("h1") == []
# --- integration through the real dispatch path -----------------------------
NOTIFY_CFG = {
"flap_count": 3,
"flap_interval": 10,
"notification_channels": {"ch1": {"type": "pushover", "token": "t", "user": "u"}},
"users": {"alice": {"notification_channels": ["ch1"]}},
}
@pytest.fixture
def delivered(monkeypatch):
"""Wire up a host, a user and a stub channel driver; collect what gets delivered."""
sent = []
monkeypatch.setattr(notify, "_config", dict(notify._config)) # restored on teardown
notify.setup(NOTIFY_CFG)
users_mod.load_users(NOTIFY_CFG)
monkeypatch.setitem(notify._DRIVERS, "pushover", lambda cfg, n: sent.append(n.body) or True)
host = hbdclass.Host("flaphost")
host.watched = True
host.owner = "alice"
yield sent, host
hbdclass.Host.hosts.pop("flaphost", None)
users_mod.load_users({})
def _notify(level, service, body):
n = notify.Notification(title=f"[{level}] flaphost", body=body, level=level, service=service)
asyncio.run(notify.send_notification("flaphost", n))
def test_marker_on_the_tripping_notification_then_nothing(delivered):
sent, host = delivered
for i in range(5):
_notify("CRITICAL", "cpu", f"cpu = 9{i}")
_notify("RECOVER", "cpu", "cpu = 12")
assert sent == ["cpu = 90", "cpu = 91", "cpu = 92", f"cpu = 93 {flap.FLAP_MARKER}"]
assert host.stateinfo()["flapping"] == ["cpu"]
def test_a_flapping_service_does_not_silence_another(delivered):
sent, host = delivered
for i in range(5):
_notify("CRITICAL", "cpu", "cpu = 99")
sent.clear()
_notify("CRITICAL", "disk", "disk = 91")
assert sent == ["disk = 91"]
def test_notifications_resume_after_the_state_clears(delivered):
sent, host = delivered
for i in range(5):
_notify("CRITICAL", "cpu", "cpu = 99")
_notify("RECOVER", "cpu", "cpu = 12")
advance(10 * 60, host="flaphost")
assert host.stateinfo()["flapping"] == []
sent.clear()
_notify("CRITICAL", "cpu", "cpu = 99")
assert sent == ["cpu = 99"]
def test_stateinfo_flapping_survives_json_encoding(delivered):
sent, host = delivered
for i in range(5):
_notify("CRITICAL", "", "IPv4 overdue") # host-level, empty service key
info = host.stateinfo()
assert info["flapping"] == [""]
assert json.loads(json.dumps(info))["flapping"] == [""]
+28
View File
@@ -172,3 +172,31 @@ def test_build_host_info_covers_empty_when_exact_matches_only():
result = _build_host_info(host, threshold_checker=checker)
t = result["thresholds"][0]
assert t["covers"] == []
class _FakeConnFull:
def __init__(self, afam, addr, state, rtt, statetime, lastbeat):
self.afam = afam
self.addr = addr
self.state = state
self.rtts = [rtt]
self.statetime = statetime
self.lastbeat = lastbeat
def test_build_host_info_includes_connections():
host = _FakeHost(connections={
"IPv4": _FakeConnFull("IPv4", "10.0.0.5", "up", 14.2, 1000.0, 2000.0),
"IPv6": _FakeConnFull("IPv6", "fd00::5", "overdue", 0, 1500.0, 1900.0),
})
result = _build_host_info(host)
conns = {c["family"]: c for c in result["connections"]}
assert conns["IPv4"] == {"family": "IPv4", "addr": "10.0.0.5", "state": "up",
"rtt": 14.2, "statetime": 1000.0, "lastbeat": 2000.0}
assert conns["IPv6"]["state"] == "overdue"
def test_build_host_info_connections_empty():
host = _FakeHost()
result = _build_host_info(host)
assert result["connections"] == []