Compare commits

..
26 Commits
Author SHA1 Message Date
andreasandClaude Sonnet 5 e8d9c4a958 docs: complete the 5.4.5 changelog entry
Release / release (push) Successful in 44s
The release script dropped every scoped feat:/fix: commit from the
generated entry (fixed in the prior commit); this backfills the mini
client presence feature and both C-client fixes.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uc9V1v7iXruubpoLiSE24M
2026-09-22 05:48:03 -04:00
andreasandClaude Sonnet 5 328f248c43 fix: bumpminor.sh changelog grep drops scoped feat/fix commits
The grep/sed only matched unscoped conventional-commit subjects
("feat: ...", "fix: ..."), so every scoped commit ("feat(server): ...")
was silently dropped from the generated changelog entry. Both 5.4.4 and
5.4.5 needed a manual follow-up fix for this. Broaden the pattern to
accept an optional (scope) and drop it from the changelog line.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uc9V1v7iXruubpoLiSE24M
2026-09-22 05:47:52 -04:00
andreas 747a8c27c9 version 5.4.5
Release / release (push) Successful in 45s
2026-09-22 05:26:12 -04:00
andreasandClaude Sonnet 5 5243f0a834 fix: shutdown ordering, silent probe failures, and unused import in the mini clients
- hbc_mini.c: move the probe-thread pthread_join to after the shutdown HTB send,
  so the clean-shutdown notification is no longer delayed behind (and possibly
  lost to a SIGKILL racing) a probe thread blocked in popen()/ping.
- hbc_mini.c: log a LOGW when pthread_create() fails to start the probe thread,
  and when probe_v4/probe_v6 are not IP literals (probe_init), so both previously
  silent no-op failure modes are now diagnosable. Diagnostic only — no behavior
  change to the PROBE_UNKNOWN fallback path.
- hbc_mini.py: remove the unused 'import struct' (F401).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uc9V1v7iXruubpoLiSE24M
2026-09-22 05:16:36 -04:00
andreasandClaude Sonnet 5 3c6f7fc868 docs: document the connectivity probe config for hbc_mini.py and hbc_mini.c
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uc9V1v7iXruubpoLiSE24M
2026-09-22 05:01:42 -04:00
andreasandClaude Sonnet 5 adbd2f9b46 feat(mini-c): send p4/p6 presence flags and connectivity metric from hbc_mini.c
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uc9V1v7iXruubpoLiSE24M
2026-09-22 04:50:17 -04:00
andreasandClaude Sonnet 5 248b17edca feat(mini-c): connectivity probe core with unit tests
Adds the pure/core probe logic for the standalone C client (hbc_mini.c):
gateway text parsers for /proc/net/route, /proc/net/ipv6_route and BSD
`route -n get`, a ping-output RTT parser, and the probe_step state machine
(probe_t/probe_fam_t/probe_ops_t) plus probe_flags and probe_metric_json.
config_t gains probe_enabled/probe_interval/probe_v4/probe_v6.

Temporary stub functions (sys_local_address, sys_ping, sys_gateway,
mono_ts) stand in for real syscalls until Task 4 wires them into main().

New scripts/c/test_probe.c test harness (#include's hbc_mini.c with main
renamed out of the way) plus a `make test` target; scripts/c/.gitignore
now ignores the test_probe binary.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uc9V1v7iXruubpoLiSE24M
2026-09-22 04:38:42 -04:00
andreasandClaude Sonnet 5 3639464ba6 feat(mini): send p4/p6 presence flags and connectivity metric from hbc_mini.py
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uc9V1v7iXruubpoLiSE24M
2026-09-22 04:31:22 -04:00
andreasandClaude Sonnet 5 997279dce8 feat(mini): per-protocol connectivity probe in hbc_mini.py
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uc9V1v7iXruubpoLiSE24M
2026-09-22 04:24:08 -04:00
andreasandClaude Sonnet 5 00d6ba607e docs: complete the 5.4.4 changelog entry
Release / release (push) Successful in 45s
The release script only picks up unscoped feat:/fix: subjects, so the
presence/simplex feature was missing from the generated entry.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uc9V1v7iXruubpoLiSE24M
2026-09-21 10:01:52 -04:00
andreas cd5de7f205 version 5.4.4
Release / release (push) Successful in 48s
2026-09-21 10:00:43 -04:00
andreasandClaude Sonnet 5 95355374f8 fix: never park the last present family as absent
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uc9V1v7iXruubpoLiSE24M
2026-09-21 09:50:43 -04:00
andreasandClaude Sonnet 5 8122c8d7dc feat(ui): show host status, simplex and absent protocols
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uc9V1v7iXruubpoLiSE24M
2026-09-21 09:36:25 -04:00
andreasandClaude Sonnet 5 3970f4cdde fix(server): restore keeps connectivity alert for connections dropped to unknown
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uc9V1v7iXruubpoLiSE24M
2026-09-21 09:34:38 -04:00
andreasandClaude Sonnet 5 c3736f47ee feat(server): monitor only present protocols; host-level overdue and simplex alerts
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uc9V1v7iXruubpoLiSE24M
2026-09-21 09:28:26 -04:00
andreasandClaude Sonnet 5 2f1cad9156 feat(server): host presence, absent state and host-level status
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uc9V1v7iXruubpoLiSE24M
2026-09-21 09:25:33 -04:00
andreasandClaude Sonnet 5 1898bc52e5 feat(client): report per-protocol presence in HTB and network_monitor
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uc9V1v7iXruubpoLiSE24M
2026-09-21 09:23:33 -04:00
andreasandClaude Sonnet 5 c61a571995 fix(client): keep ping-less family unknown after route returns
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uc9V1v7iXruubpoLiSE24M
2026-09-21 09:22:03 -04:00
andreasandClaude Sonnet 5 9e8d68e712 feat(client): per-protocol connectivity probe
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uc9V1v7iXruubpoLiSE24M
2026-09-21 09:19:48 -04:00
andreasandClaude Sonnet 5 ad19d2596a fix(client): default_gateway never raises
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uc9V1v7iXruubpoLiSE24M
2026-09-21 09:17:48 -04:00
andreasandClaude Sonnet 5 82ba69d827 feat(client): best-effort default gateway lookup
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uc9V1v7iXruubpoLiSE24M
2026-09-21 09:15:17 -04:00
andreasandClaude Sonnet 5 9519169b47 refactor: extract ping helpers into pingutil
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uc9V1v7iXruubpoLiSE24M
2026-09-21 09:12:53 -04:00
andreas f838920f7c version 5.4.3
Release / release (push) Successful in 1m16s
2026-08-17 13:54:02 -04:00
andreasandClaude Sonnet 5 45d11a66b5 fix: fix RTT y-axis label clipping, add x-axis tickmarks, enlarge charts
Y-axis labels on the RTT chart were truncated above 99ms because the
left margin was a fixed guess; it's now sized to the widest tick
label. The x-axis only showed start/end timestamps; added evenly
spaced intermediate tickmarks and labels. Plot area enlarged 15%.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-08-17 13:53:57 -04:00
andreas a42d783c0c version 5.4.2
Release / release (push) Successful in 1m10s
2026-08-17 09:24:23 -04:00
andreasandClaude Sonnet 5 18e33656c0 fix: preserve chart history across reconnects, show gaps for missing data
Only wipe real (non-RTT) plugin data on an actual client reboot (boot
flag), not on every ordinary OVERDUE/DOWN -> UP recovery. A transient
network blip no longer erases CPU/memory/etc. history.

Also split the shared time-series chart into separate line/area segments
wherever the gap between samples is much larger than the typical spacing,
so missing data (host overdue, or history that simply hasn't accumulated
across a drop) renders as a visual gap instead of an interpolated line.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DEimzMv4Q5EjFg3hoiZ69T
2026-08-17 09:13:49 -04:00
29 changed files with 2913 additions and 159 deletions
+46
View File
@@ -2,6 +2,52 @@
All notable changes to this project are documented here, organized by release.
## [5.4.5]
### Added
- `hbc_mini.py` and `hbc_mini.c` (the dependency-free clients) get the same per-protocol connectivity probe as `hbc`: `p4`/`p6` flags on every heartbeat and a `connectivity` metric (presence, own address, gateway, RTT) via `network_monitor`, with gateway lookup on Linux, FreeBSD/NetBSD/DragonFly (both clients) and macOS/Windows (`hbc_mini.py` only)
- `scripts/c/test_probe.c` and a `make test` target — the C client's first automated test suite
### Fixed
- `hbc_mini.c`: the clean-shutdown heartbeat could be delayed or lost behind an in-flight connectivity probe on SIGTERM; it is now sent before the probe thread is joined
- `hbc_mini.c`: a connectivity probe that fails to start, or is configured with a non-IP-literal target, now logs a warning instead of failing silently
- `hbc_mini.py`: removed an unused import
---
## [5.4.4]
### Added
- client reports per-protocol (IPv4/IPv6) internet presence: a connectivity probe (route check every ~5s, ICMP echo every ~60s, debounced) sends `p4`/`p6` flags on every heartbeat and a `connectivity` metric (presence, own address, gateway, RTT) via `network_monitor`
- server monitors only the protocols a client reports as present; a protocol reported absent is shown as `absent` and raises no alert
- host status (`up`, `simplex`, `overdue`, `down`, `unknown`) computed on the server: a host is overdue only when every present protocol is overdue; `simplex` (both present, one silent) has the same severity as `up` and is logged without notification
- live and plugins pages show the host status, `simplex`/`absent` protocols and a Connectivity table
- `hbc` config keys under `network_monitor`: `probe_enabled`, `probe_v4`, `probe_v6`, `probe_interval`
### Changed
- overdue and recovery alerts are raised per host, not per protocol: a single CRITICAL when the last present protocol goes overdue; a protocol coming back while another stayed up is logged as INFO instead of a RECOVER notification (also applies to clients that do not report presence)
### Fixed
- never park the last present family as absent, so a host that reports no protocol as present still alerts when it dies
- keep the connectivity alert for a host overdue past the drop window across an `hbd` restart
- `default_gateway` never raises; a protocol whose probe cannot run stays unknown rather than absent
---
## [5.4.3]
### Fixed
- fix RTT y-axis label clipping, add x-axis tickmarks, enlarge charts
---
## [5.4.2]
### Fixed
- preserve chart history across reconnects, show gaps for missing data
---
## [5.4.1]
### Added
+45 -2
View File
@@ -20,7 +20,7 @@ A lightweight UDP-based host monitoring system. Monitored hosts run a client (`h
└────────────────────┘ └────────────────────────────┘
```
**Package:** `hbd` v5.4.1
**Package:** `hbd` v5.4.5
**Python:** 3.11+
### Subpackages
@@ -659,7 +659,28 @@ Config: `~/.hbc.json` (JSON format, same keys as `~/.hbc.yaml`).
| `cpu_monitor` | Linux (`/proc/stat`; no per-core, no frequency) |
| `memory_monitor` | Linux (`/proc/meminfo`) |
| `disk_monitor` | Linux, macOS, BSD (`df -P`) |
| `network_monitor` | Linux (`/proc/net/dev`) |
| `network_monitor` | Linux (`/proc/net/dev`); per-protocol connectivity probe (all platforms) |
`network_monitor` also reports per-protocol (IPv4/IPv6) internet presence — whether the host
has a working route and reply to a well-known address, plus its own address, the local
gateway, and RTT — as `p4`/`p6` flags on every heartbeat and a `connectivity` object in the
plugin data. Configurable, defaults shown:
```json
{
"plugins": {
"network_monitor": {
"probe_enabled": true,
"probe_v4": "1.1.1.1",
"probe_v6": "2606:4700:4700::1111",
"probe_interval": 60
}
}
}
```
`probe_v4`/`probe_v6` must be IP literals (no hostnames). Gateway lookup works on Linux,
FreeBSD/NetBSD/DragonFly/macOS (`route -n get`), and Windows (`route print`).
Not available vs full `hbc`: no YAML config, no `filesystem_info`, no `zfs_monitor`, no IPv6 early-fail protection.
@@ -712,6 +733,7 @@ SIGHUP causes the process to restart itself via `execv()`. SIGTERM/SIGINT trigge
| `disk_monitor` | All | `df -P` subprocess |
| `network_monitor` | Linux | `/proc/net/dev` |
| `network_monitor` | FreeBSD, NetBSD, DragonFly | `getifaddrs()` + `AF_LINK` |
| `network_monitor` (connectivity probe) | Linux, FreeBSD, NetBSD, DragonFly | UDP `connect()` route check + one ICMP echo per protocol |
| `ping_monitor` | All | `ping` subprocess |
| `nagios_runner` | All | `popen()` subprocess |
@@ -743,12 +765,33 @@ SIGHUP causes the process to restart itself via `execv()`. SIGTERM/SIGINT trigge
}
```
`network_monitor` also runs the per-protocol connectivity probe described above, in a
dedicated thread (its ICMP echo can block for a few seconds). Same config keys and defaults
as `hbc_mini.py`:
```json
{
"plugins": {
"network_monitor": {
"probe_enabled": true,
"probe_v4": "1.1.1.1",
"probe_v6": "2606:4700:4700::1111",
"probe_interval": 60
}
}
}
```
Gateway lookup on the C client covers Linux and FreeBSD/NetBSD/DragonFly only (no macOS or
Windows build of `hbc_mini.c` exists).
### Differences from `hbc_mini.py`
- No `filesystem_info` or `zfs_monitor` plugins
- `UPD` (self-update) messages are logged but not acted on
- No IPv6 early-fail protection
- Config is JSON only (`~/.hbc.json`), no YAML
- No macOS or Windows build (Linux and the BSDs only)
---
+1 -1
View File
@@ -14,4 +14,4 @@ Install options:
"""
__all__ = ["__version__"]
__version__ = "5.4.1"
__version__ = "5.4.5"
+204
View File
@@ -0,0 +1,204 @@
"""Per-protocol internet presence probe.
Tier 1 (every tier1_interval seconds, sends no packets): a UDP connect() to the probe
address reveals whether a route exists and which source address would be used.
Tier 2 (every probe_interval seconds, sooner after a failure or a change of address or
gateway): one ICMP echo to the probe address. Absence needs fail_limit consecutive
failures (or no route at all); presence needs one success.
The result is read by the heartbeat sender (p4/p6 flags on every HTB) and by the
network_monitor plugin (connectivity metric) via the shared instance.
"""
import asyncio
import logging
import socket
import sys
import time
from dataclasses import asdict, dataclass, replace
from typing import Any, Awaitable, Callable, Dict, Optional, Set, Tuple
from hbd.client import gateway as gateway_mod
from hbd.client.pingutil import PingUnavailable, run_ping
logger = logging.getLogger("hbc.connectivity")
FAMILIES = ("ipv4", "ipv6")
FLAG_KEYS = {"ipv4": "p4", "ipv6": "p6"}
DEFAULT_PROBES = {"ipv4": "1.1.1.1", "ipv6": "2606:4700:4700::1111"}
class ProbeUnavailable(Exception):
"""The address family cannot be probed on this host (e.g. no IPv6 support)."""
@dataclass
class FamilyState:
present: Optional[bool] = None # None = unknown (probe cannot run yet/at all)
addr: Optional[str] = None
gateway: Optional[str] = None
rtt: Optional[float] = None
def local_address(family: str, target: str) -> Optional[str]:
"""Source address the kernel would use to reach target, or None if there is no route.
A UDP connect() sends nothing. Raises ProbeUnavailable if the family is unsupported.
"""
af = socket.AF_INET if family == "ipv4" else socket.AF_INET6
try:
sock = socket.socket(af, socket.SOCK_DGRAM)
except OSError as e:
raise ProbeUnavailable(str(e)) from e
try:
sock.connect((target, 9))
return str(sock.getsockname()[0])
except OSError:
return None
finally:
sock.close()
async def ping_rtt(target: str, family: str) -> Optional[float]:
"""One ICMP echo; returns RTT in ms or None if no reply."""
result = await run_ping(target, 1, 2, 4 if family == "ipv4" else 6)
rtt = result["rtt_avg"]
return None if rtt == float("inf") else rtt
class Connectivity:
def __init__(
self,
probes: Optional[Dict[str, str]] = None,
probe_interval: float = 60.0,
retry_interval: float = 10.0,
tier1_interval: float = 5.0,
fail_limit: int = 2,
local_address_fn: Callable[[str, str], Optional[str]] = local_address,
ping_fn: Callable[[str, str], Awaitable[Optional[float]]] = ping_rtt,
gateway_fn: Callable[[str], Awaitable[Optional[str]]] = gateway_mod.default_gateway,
clock: Callable[[], float] = time.monotonic,
):
self.probes = {**DEFAULT_PROBES, **(probes or {})}
self.probe_interval = probe_interval
self.retry_interval = retry_interval
self.tier1_interval = tier1_interval
self.fail_limit = fail_limit
self._local_address = local_address_fn
self._ping = ping_fn
self._gateway_fn = gateway_fn
self._clock = clock
# Reading the gateway is a cheap /proc read on Linux but a subprocess elsewhere,
# so elsewhere it is refreshed only when a tier-2 probe is due.
self._cheap_gateway = sys.platform.startswith("linux")
self.states: Dict[str, FamilyState] = {f: FamilyState() for f in FAMILIES}
self._fails: Dict[str, int] = {f: 0 for f in FAMILIES}
self._next_ping: Dict[str, float] = {f: 0.0 for f in FAMILIES}
self._last_key: Dict[str, Optional[Tuple[Optional[str], Optional[str]]]] = {
f: None for f in FAMILIES}
self._ping_disabled: Set[str] = set()
@classmethod
def from_config(cls, cfg: Optional[Dict[str, Any]]) -> "Connectivity":
cfg = cfg or {}
probes = {}
if cfg.get("probe_v4"):
probes["ipv4"] = str(cfg["probe_v4"])
if cfg.get("probe_v6"):
probes["ipv6"] = str(cfg["probe_v6"])
return cls(probes=probes, probe_interval=float(cfg.get("probe_interval", 60)))
# ---- readers (cheap; safe to call on every heartbeat) --------------------------
def snapshot(self) -> Dict[str, FamilyState]:
return {f: replace(s) for f, s in self.states.items()}
def flags(self) -> Dict[str, bool]:
"""HTB flags for families whose presence is known."""
return {FLAG_KEYS[f]: s.present for f, s in self.states.items() if s.present is not None}
def as_metric(self) -> Dict[str, Dict[str, Any]]:
return {f: asdict(s) for f, s in self.states.items()}
# ---- probing -------------------------------------------------------------------
async def refresh(self) -> None:
for fam in FAMILIES:
try:
await self._refresh_family(fam)
except Exception:
logger.exception("connectivity probe failed for %s", fam)
async def run(self, stop: asyncio.Event) -> None:
while not stop.is_set():
await self.refresh()
try:
await asyncio.wait_for(stop.wait(), timeout=self.tier1_interval)
except asyncio.TimeoutError:
pass
async def _gateway(self, fam: str) -> Optional[str]:
try:
return await self._gateway_fn(fam)
except Exception:
return None
async def _refresh_family(self, fam: str) -> None:
st = self.states[fam]
try:
addr = self._local_address(fam, self.probes[fam])
except ProbeUnavailable:
st.present = None
return
if addr is None: # tier 1: no route => absent immediately
self._fails[fam] = 0
self._next_ping[fam] = 0.0
self._last_key[fam] = None
st.present, st.addr, st.gateway, st.rtt = False, None, None, None
return
now = self._clock()
gw = await self._gateway(fam) if self._cheap_gateway else st.gateway
key = (addr, gw)
due = key != self._last_key[fam] or now >= self._next_ping[fam]
if due and not self._cheap_gateway:
gw = await self._gateway(fam)
key = (addr, gw)
st.addr, st.gateway = addr, gw
self._last_key[fam] = key
if fam in self._ping_disabled:
st.present = None # cannot probe: unknown, never absent, while a route exists
elif due:
await self._tier2(fam, st, now)
async def _tier2(self, fam: str, st: FamilyState, now: float) -> None:
try:
rtt = await self._ping(self.probes[fam], fam)
except PingUnavailable:
logger.warning("ping unavailable; %s presence stays unknown", fam)
self._ping_disabled.add(fam)
st.present = None
return
except (asyncio.TimeoutError, OSError):
rtt = None
if rtt is None:
self._fails[fam] += 1
self._next_ping[fam] = now + self.retry_interval
if self._fails[fam] >= self.fail_limit:
st.present, st.rtt = False, None
else:
self._fails[fam] = 0
self._next_ping[fam] = now + self.probe_interval
st.present, st.rtt = True, rtt
# ---- shared instance (heartbeat sender and network_monitor plugin) -----------------
_shared: Optional[Connectivity] = None
def set_shared(conn: Optional[Connectivity]) -> None:
global _shared
_shared = conn
def get_shared() -> Optional[Connectivity]:
return _shared
+103
View File
@@ -0,0 +1,103 @@
"""Best-effort default-gateway lookup (informational only; never affects presence)."""
import asyncio
import re
import socket
import struct
import sys
from typing import List, Optional, Tuple
_RTF_GATEWAY = 0x2
_RTF_REJECT = 0x200
def parse_proc_route_v4(text: str) -> Optional[str]:
"""Default gateway from /proc/net/route (lowest metric wins)."""
best: Optional[Tuple[int, str]] = None
for line in text.splitlines()[1:]:
f = line.split()
if len(f) < 8:
continue
_iface, dest, gw, flags = f[0], f[1], f[2], f[3] # noqa: F841
if dest != "00000000" or not int(flags, 16) & _RTF_GATEWAY:
continue
metric = int(f[6])
if best is None or metric < best[0]:
best = (metric, socket.inet_ntoa(struct.pack("<L", int(gw, 16))))
return best[1] if best else None
def parse_proc_route_v6(text: str) -> Optional[str]:
"""Default gateway from /proc/net/ipv6_route (lowest metric wins)."""
best: Optional[Tuple[int, str]] = None
for line in text.splitlines():
f = line.split()
if len(f) < 10:
continue
dest, plen, nexthop, metric, flags, iface = f[0], f[1], f[4], f[5], f[8], f[9]
if dest != "0" * 32 or plen != "00" or nexthop == "0" * 32 or iface == "lo":
continue
fl = int(flags, 16)
if not fl & _RTF_GATEWAY or fl & _RTF_REJECT:
continue
m = int(metric, 16)
if best is None or m < best[0]:
best = (m, socket.inet_ntop(socket.AF_INET6, bytes.fromhex(nexthop)))
return best[1] if best else None
def parse_macos_route(text: str) -> Optional[str]:
"""Gateway from `route -n get [-inet6] default`."""
m = re.search(r"^\s*gateway:\s*(\S+)", text, re.MULTILINE)
return m.group(1) if m else None
def parse_windows_route(text: str, family: str) -> Optional[str]:
"""Default gateway from `route print -4` / `route print -6` (lowest metric wins)."""
best: Optional[Tuple[int, str]] = None
for line in text.splitlines():
if family == "ipv4":
f = line.split()
if len(f) >= 5 and f[0] == "0.0.0.0" and f[1] == "0.0.0.0" and f[2] != "On-link":
cand = (int(f[4]), f[2])
else:
continue
else:
m = re.match(r"^\s*\d+\s+(\d+)\s+::/0\s+(\S+)\s*$", line)
if not m or m.group(2) == "On-link":
continue
cand = (int(m.group(1)), m.group(2))
if best is None or cand[0] < best[0]:
best = cand
return best[1] if best else None
async def _run(cmd: List[str]) -> str:
proc = await asyncio.create_subprocess_exec(
*cmd, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.DEVNULL,
)
try:
out, _ = await asyncio.wait_for(proc.communicate(), timeout=5)
except asyncio.TimeoutError:
proc.kill()
await proc.wait()
raise
return out.decode(errors="replace")
async def default_gateway(family: str) -> Optional[str]:
"""Return the default gateway address for "ipv4"/"ipv6", or None on any failure."""
v4 = family == "ipv4"
try:
if sys.platform.startswith("linux"):
with open("/proc/net/route" if v4 else "/proc/net/ipv6_route") as fh:
text = fh.read()
return parse_proc_route_v4(text) if v4 else parse_proc_route_v6(text)
if sys.platform == "darwin":
cmd = ["route", "-n", "get"] + ([] if v4 else ["-inet6"]) + ["default"]
return parse_macos_route(await _run(cmd))
if sys.platform == "win32":
return parse_windows_route(await _run(["route", "print", "-4" if v4 else "-6"]), family)
except Exception:
return None
return None
+23 -7
View File
@@ -25,6 +25,7 @@ from .. import __version__
# Import plugin system
from .plugin import PluginRegistry, PluginLoader, InfoPlugin, MonitorPlugin
from .connectivity import Connectivity, set_shared
# Constants
PORT = 50003
@@ -281,7 +282,16 @@ async def handle_update(conn: AsyncConnection, _msg: dict): # pyright: ignore[r
stop()
async def heartbeat_sender(conn: AsyncConnection, interval: int):
def build_heartbeat_msg(conn, interval: int, connectivity: Optional[Connectivity] = None) -> dict:
"""HTB payload; carries p4/p6 presence flags when the probe knows them."""
msg = {"acks": conn.ackcount, "rtt": conn.rtts[-1], "interval": interval}
if connectivity is not None:
msg.update(connectivity.flags())
return msg
async def heartbeat_sender(conn: AsyncConnection, interval: int,
connectivity: Optional[Connectivity] = None):
"""Send periodic heartbeats, retrying the connection if it is not open.
IPv6 connections that fail to open before their first successful send are
@@ -291,6 +301,7 @@ async def heartbeat_sender(conn: AsyncConnection, interval: int):
Args:
conn: Connection to send on
interval: Heartbeat interval in seconds
connectivity: Optional probe whose presence flags are attached to every heartbeat
"""
logger = logging.getLogger("hbc.heartbeat")
IPV6_EARLY_FAIL_LIMIT = 3
@@ -328,11 +339,7 @@ async def heartbeat_sender(conn: AsyncConnection, interval: int):
continue
try:
msg = {
"acks": conn.ackcount,
"rtt": conn.rtts[-1],
"interval": interval
}
msg = build_heartbeat_msg(conn, interval, connectivity)
await conn.sendto(msg, "HTB")
except asyncio.CancelledError:
@@ -619,9 +626,18 @@ async def async_main(args, config):
loop.add_signal_handler(signal.SIGHUP, _sighup)
# Start async tasks
# Per-protocol internet presence probe (shared with the network_monitor plugin)
net_cfg = dict((config.get("plugins") or {}).get("network_monitor")
or config.get("network_monitor") or {})
connectivity = None
if net_cfg.get("probe_enabled", True):
connectivity = Connectivity.from_config(net_cfg)
set_shared(connectivity)
active_tasks.append(asyncio.create_task(connectivity.run(shutdown_event)))
# Heartbeat senders (one per connection)
for conn in connections:
task = asyncio.create_task(heartbeat_sender(conn, interval))
task = asyncio.create_task(heartbeat_sender(conn, interval, connectivity))
active_tasks.append(task)
# Plugin collector (uses all connections, but we'll use first one)
+73
View File
@@ -0,0 +1,73 @@
"""System ping helpers shared by ping_monitor and the connectivity probe."""
import asyncio
import re
import sys
from typing import Dict, List, Optional
_INF = float("inf")
class PingUnavailable(Exception):
"""The system ping binary could not be executed."""
def ping_command(host: str, count: int, timeout: int, family: Optional[int] = None) -> List[str]:
"""Build the platform ping command. family is 4, 6 or None (let ping decide)."""
if sys.platform == "win32":
cmd = ["ping", "-n", str(count), "-w", str(timeout * 1000)]
if family:
cmd.append(f"-{family}")
return cmd + [host]
if sys.platform == "darwin" and family:
if family == 6:
return ["ping6", "-c", str(count), host]
return ["ping", "-c", str(count), "-W", str(timeout * 1000), host]
cmd = ["ping", "-c", str(count), "-W", str(timeout)]
if family:
cmd.append(f"-{family}")
return cmd + [host]
def parse_ping_output(output: str) -> Dict[str, float]:
"""Parse rtt_min/avg/max/loss from ping output (Linux, macOS, BSD)."""
loss = 100.0
loss_match = re.search(r"(\d+(?:\.\d+)?)\s*%\s*packet\s*loss", output)
if loss_match:
loss = float(loss_match.group(1))
# Linux: "rtt min/avg/max/mdev = x/x/x/x ms"; macOS: "round-trip min/avg/max/stddev = ..."
rtt_match = re.search(
r"(?:rtt|round-trip)\s+min/avg/max/\S+\s*=\s*([\d.]+)/([\d.]+)/([\d.]+)",
output,
)
if rtt_match:
return {
"rtt_min": float(rtt_match.group(1)),
"rtt_avg": float(rtt_match.group(2)),
"rtt_max": float(rtt_match.group(3)),
"loss": loss,
}
return {"rtt_min": _INF, "rtt_avg": _INF, "rtt_max": _INF, "loss": loss}
async def run_ping(host: str, count: int, timeout: int, family: Optional[int] = None) -> Dict[str, float]:
"""Run ping and return parsed metrics.
Raises PingUnavailable if the ping binary is missing and asyncio.TimeoutError
(after killing the process) if it hangs.
"""
cmd = ping_command(host, count, timeout, family)
try:
proc = await asyncio.create_subprocess_exec(
*cmd, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE,
)
except FileNotFoundError as e:
raise PingUnavailable(str(e)) from e
try:
stdout, _ = await asyncio.wait_for(proc.communicate(), timeout=timeout * count + 2)
except asyncio.TimeoutError:
proc.kill()
await proc.wait()
raise
return parse_ping_output(stdout.decode(errors="replace"))
+11
View File
@@ -13,6 +13,7 @@ except ImportError:
psutil = None
from hbd.client.plugin import MonitorPlugin
from hbd.client.connectivity import get_shared
logger = logging.getLogger(__name__)
@@ -32,6 +33,10 @@ class NetworkMonitorPlugin(MonitorPlugin):
interfaces: List of interfaces to monitor (default: all)
include_connections: Include connection statistics (default: True)
include_addresses: Include interface addresses (default: False)
probe_enabled: Run the per-protocol internet presence probe (default: True)
probe_v4: IPv4 address to ping (default: 1.1.1.1)
probe_v6: IPv6 address to ping (default: 2606:4700:4700::1111)
probe_interval: Seconds between ICMP probes while healthy (default: 60)
"""
name = "network_monitor"
@@ -103,6 +108,7 @@ class NetworkMonitorPlugin(MonitorPlugin):
- etc.
- addresses: Interface address information (if include_addresses)
- Dict keyed by interface name with address details
- connectivity: per-protocol presence {ipv4, ipv6} -> {present, addr, gateway, rtt}
"""
if psutil is None:
logger.error("psutil not available")
@@ -228,6 +234,11 @@ class NetworkMonitorPlugin(MonitorPlugin):
except Exception as e:
logger.warning(f"Could not collect interface stats: {e}")
# Per-protocol internet presence, address and gateway (from the shared probe)
shared = get_shared()
if shared is not None:
metrics['connectivity'] = shared.as_metric()
return metrics
+3 -41
View File
@@ -40,9 +40,9 @@ threshold_configs:
import asyncio
import re
import sys
from typing import Any, Dict, Optional
from hbd.client.pingutil import PingUnavailable, run_ping
from hbd.client.plugin import MonitorPlugin
@@ -84,51 +84,13 @@ class PingMonitorPlugin(MonitorPlugin):
async def _ping(self, host: str) -> Dict[str, float]:
"""Run a system ping command and return rtt_min/avg/max/loss."""
if sys.platform == "win32":
cmd = ["ping", "-n", str(self.count), "-w", str(self.timeout * 1000), host]
else:
cmd = ["ping", "-c", str(self.count), "-W", str(self.timeout), host]
try:
proc = await asyncio.create_subprocess_exec(
*cmd,
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.PIPE,
)
stdout, _ = await asyncio.wait_for(
proc.communicate(),
timeout=self.timeout * self.count + 2,
)
output = stdout.decode(errors="replace")
except (asyncio.TimeoutError, FileNotFoundError, OSError) as e:
return await run_ping(host, self.count, self.timeout)
except (asyncio.TimeoutError, PingUnavailable, OSError) as e:
self.logger.warning("ping_monitor: ping failed for %s: %s", host, e)
return {"rtt_min": float("inf"), "rtt_avg": float("inf"),
"rtt_max": float("inf"), "loss": 100.0}
# Parse packet loss
loss = 100.0
loss_match = re.search(r"(\d+(?:\.\d+)?)\s*%\s*packet\s*loss", output)
if loss_match:
loss = float(loss_match.group(1))
# Parse rtt min/avg/max — Linux: "rtt min/avg/max/mdev = x/x/x/x ms"
# macOS: "round-trip min/avg/max/stddev = x/x/x/x ms"
rtt_match = re.search(
r"(?:rtt|round-trip)\s+min/avg/max/\S+\s*=\s*([\d.]+)/([\d.]+)/([\d.]+)",
output,
)
if rtt_match:
return {
"rtt_min": float(rtt_match.group(1)),
"rtt_avg": float(rtt_match.group(2)),
"rtt_max": float(rtt_match.group(3)),
"loss": loss,
}
# Host unreachable or all packets lost
return {"rtt_min": float("inf"), "rtt_avg": float("inf"),
"rtt_max": float("inf"), "loss": loss}
async def _collect_metrics(self) -> Dict[str, Any]:
data: Dict[str, Any] = {}
tasks = {host: asyncio.create_task(self._ping(host)) for host in self.hosts}
+51 -1
View File
@@ -29,6 +29,7 @@ class Connection:
UP = "up"
DOWN = "down"
OVERDUE = "overdue"
ABSENT = "absent"
def __init__(self, host, cid, addr, afam):
self.host = host
@@ -94,7 +95,9 @@ class Connection:
now = time.time()
if not Null:
d["addr"] = self.addr
if self.rtts[-1]:
if self.state == Connection.ABSENT:
d["rtt"] = ""
elif self.rtts[-1]:
d["rtt"] = "%d" % round(self.rtts[-1])
elif self.state == Connection.UNKNOWN:
d["rtt"] = ""
@@ -299,6 +302,8 @@ class Host:
self.alert_states = {}
# Stale-data timers: {plugin_name: asyncio.TimerHandle}
self.plugin_timers = {}
# Client-reported protocol presence: {afam: {"present": bool, "ts": float}}
self.presence: dict = {}
# User access control
self.owner: str | None = None # username of owner
self.managers: list = [] # usernames with manager role
@@ -322,6 +327,7 @@ class Host:
d["name"] = "<b>%s</b>" % d["name"]
d["dyn"] = str(self.dyn)
d["num"] = self.num
d["status"] = self.status()
# Add alert counts (split by acknowledged status)
warning_unacked = 0
@@ -444,6 +450,7 @@ class Host:
_, os_data = latest_os
hbc_version = os_data.get("hbc_version")
ddict["hbc_version"] = hbc_version
ddict["status"] = self.status()
return ddict
@@ -459,6 +466,47 @@ class Host:
else:
return addr.find(".") > 0
def set_presence(self, afam, present, now):
"""Record the client's report of whether it has internet connectivity on afam."""
self.presence[afam] = {"present": bool(present), "ts": now}
def flag_present(self, afam):
"""Client-reported presence; True when the client never reported (legacy clients)."""
entry = getattr(self, "presence", {}).get(afam)
return True if entry is None else entry["present"]
def family_present(self, afam):
"""True if afam should be monitored: the client says it is present, or heartbeats
are demonstrably arriving on it (real traffic overrides a false "absent")."""
conn = self.connections.get(afam)
return self.flag_present(afam) or (conn is not None and conn.state == Connection.UP)
def status(self):
"""Host-level status: down, overdue, simplex, up or unknown.
overdue: every present family is overdue. simplex: both present, one up, one overdue
(same severity as up). Families the client reports absent are not considered,
unless none is present: then every connection not parked as ABSENT is considered.
"""
conns = self.connections
if any(c.state == Connection.DOWN for c in conns.values()):
return "down"
present = [c for afam, c in conns.items() if self.family_present(afam)]
if not present:
# The client reports no protocol present (e.g. its internet probe fails on every
# family) yet may still be heard: monitor whatever is not parked as absent, so a
# host that then dies is "overdue", not silently "unknown".
present = [c for c in conns.values() if c.state != Connection.ABSENT]
up = sum(c.state == Connection.UP for c in present)
overdue = sum(c.state == Connection.OVERDUE for c in present)
if up and overdue:
return "simplex"
if up:
return "up"
if overdue:
return "overdue"
return "unknown"
def conndata(self, cid, addr, rtt, now):
if addr[0:7] == "::ffff:":
addr = addr[7:]
@@ -499,6 +547,8 @@ class Host:
self.monitors = []
if not hasattr(self, "plugin_timers"):
self.plugin_timers = {}
if not hasattr(self, "presence"):
self.presence = {}
pass
+18 -15
View File
@@ -73,6 +73,7 @@
.tipbox .up { color: var(--st-ok); font-weight: 700; }
.tipbox .dn { color: var(--st-crit); font-weight: 700; }
.tipbox .ov { color: var(--st-warn); font-weight: 700; }
.tipbox .ab { color: var(--st-faint); font-weight: 700; }
.tipbox .meta { margin-top: 6px; padding-top: 6px; border-top: 1px solid var(--st-line-soft); font-size: 11px; color: var(--st-faint); }
@media (max-width: 760px) {
@@ -134,18 +135,18 @@
function worstConnection(data) {
var worst = null;
(data.connections || []).forEach(function (c) {
if (!c.state || c.state === 'unknown') return;
if (!c.state || c.state === 'unknown' || c.state === 'absent') return;
if (!worst || (STATE_RANK[c.state] || 0) > (STATE_RANK[worst.state] || 0)) worst = c;
});
return worst;
}
function severity(data) {
var w = worstConnection(data);
var st = data.status;
if ((data.alert_critical_unacked || 0) > 0 || (data.alert_critical_acked || 0) > 0) return 'crit';
if (w && w.state === 'down') return 'crit';
if (st === 'down') return 'crit';
if ((data.alert_warning_unacked || 0) > 0 || (data.alert_warning_acked || 0) > 0) return 'warn';
if (w && w.state === 'overdue') return 'warn';
if (st === 'overdue') return 'warn';
return 'ok';
}
var SEV_ORDER = { crit: 0, warn: 1, ok: 2 };
@@ -171,17 +172,17 @@
// ---- host row rendering ---------------------------------------------
function statusChip(data) {
var w = worstConnection(data);
if (!w) return '<span class="chip outline">—</span>';
var cls = w.state === 'up' ? 'ok' : (w.state === 'down' ? 'crit' : 'warn');
return '<span class="chip ' + cls + '">' + escHtml(w.state) + '</span>';
var st = data.status;
if (!st || st === 'unknown') return '<span class="chip outline">—</span>';
var cls = (st === 'up' || st === 'simplex') ? 'ok' : (st === 'down' ? 'crit' : 'warn');
return '<span class="chip ' + cls + '">' + escHtml(st) + '</span>';
}
function tipCard(data) {
var rows = '';
(data.connections || []).forEach(function (c) {
if (!c.state || c.state === 'unknown') return;
var cls = c.state === 'up' ? 'up' : (c.state === 'down' ? 'dn' : 'ov');
var cls = c.state === 'up' ? 'up' : (c.state === 'down' ? 'dn' : (c.state === 'absent' ? 'ab' : 'ov'));
var fam = (c.afam || '').replace('IPv', 'v');
var rtt = (c.state === 'up' && c.rtts && c.rtts[0]) ? Math.round(c.rtts[0]) + ' ms' : '—';
rows += '<tr><td>' + escHtml(fam) + '</td><td>' + escHtml(c.addr || '—') + '</td>'
@@ -247,7 +248,9 @@
else if (data.hbc_version !== HBD_VERSION) {
staleTag = '<span class="stale" title="agent v' + escHtml(String(data.hbc_version)) + ' ≠ server v' + HBD_VERSION + '">v' + escHtml(String(data.hbc_version)) + '</span>';
}
var dotCls = w ? (w.state === 'up' ? sev : (w.state === 'down' ? 'down' : 'overdue')) : 'off';
var st = data.status;
var dotCls = (!st || st === 'unknown') ? 'off'
: ((st === 'up' || st === 'simplex') ? sev : (st === 'down' ? 'down' : 'overdue'));
var flapTag = '';
var flapping = data.flapping || [];
if (flapping.length) {
@@ -286,11 +289,11 @@
var total = 0, up = 0, overdue = 0, down = 0;
Object.values(hostData).forEach(function (d) {
total++;
var w = worstConnection(d);
if (!w) return;
if (w.state === 'up') up++;
else if (w.state === 'overdue') overdue++;
else if (w.state === 'down') down++;
var st = d.status;
if (!st || st === 'unknown') return;
if (st === 'up' || st === 'simplex') up++;
else if (st === 'overdue') overdue++;
else if (st === 'down') down++;
});
var parts = [total + ' hosts', up + ' up'];
if (overdue) parts.push(overdue + ' overdue');
+85 -28
View File
@@ -456,7 +456,7 @@
</tr></thead><tbody>`;
for (const c of data.connections) {
const st = escHtml(c.state || '—');
const stCls = c.state === 'up' ? 'status-up' : (c.state ? 'status-down' : '');
const stCls = c.state === 'up' ? 'status-up' : (c.state === 'absent' ? '' : (c.state ? 'status-down' : ''));
const rtt = (c.state === 'up' && c.rtt) ? Math.round(c.rtt) + ' ms' : '—';
const chg = c.statetime ? new Date(c.statetime * 1000).toLocaleString() : '—';
const seen = c.lastbeat ? new Date(c.lastbeat * 1000).toLocaleString() : '—';
@@ -827,13 +827,10 @@
if (!el || pts.length < 2) { if (el) el.style.display = 'none'; return; }
const unitSuffix = opts.unitSuffix || '';
const W = 600, H = 80, PAD = { top: 6, right: 8, bottom: 18, left: 28 };
const cW = W - PAD.left - PAD.right;
const cH = H - PAD.top - PAD.bottom;
const W = 690, H = 92, PAD = { top: 6, right: 8, bottom: 18, left: 28 };
const tMin = pts[0].t, tMax = pts[pts.length - 1].t;
const tRange = tMax - tMin || 1;
const x = t => PAD.left + ((t - tMin) / tRange) * cW;
// Auto-scale Y axis with 10% padding, optionally clamped to opts.yDomain
const vMin = Math.min(...pts.map(p => p.v));
@@ -845,39 +842,80 @@
const yLow = Math.max(domainLow, vMin - vPad);
const yHigh = Math.min(domainHigh, vMax + vPad);
const yRange = yHigh - yLow || 1;
const y = v => PAD.top + cH - ((v - yLow) / yRange) * cH;
// Build polyline points and filled area path
const linePoints = pts.map(p => `${x(p.t).toFixed(1)},${y(p.v).toFixed(1)}`).join(' ');
const areaPath = `M${x(pts[0].t).toFixed(1)},${(PAD.top + cH).toFixed(1)} ` +
pts.map(p => `L${x(p.t).toFixed(1)},${y(p.v).toFixed(1)}`).join(' ') +
` L${x(pts[pts.length-1].t).toFixed(1)},${(PAD.top + cH).toFixed(1)} Z`;
// Compute nice tick step for ~3-5 grid lines, then size the left
// margin to fit the widest label (values/units can run to 3+ digits,
// e.g. RTT samples above 99ms) instead of a fixed guess that clips them.
const rawStep = yRange / 4;
const mag = Math.pow(10, Math.floor(Math.log10(rawStep || 1)));
const niceStep = [1, 2, 5, 10].map(f => f * mag).find(s => yRange / s <= 5) || mag * 10;
const tickStart = Math.ceil(yLow / niceStep) * niceStep;
const yTicks = [];
for (let v = tickStart; v <= yHigh + 0.001; v += niceStep) yTicks.push(v);
const yTickLabels = yTicks.map(v => (Number.isInteger(v) ? v : v.toFixed(1)) + unitSuffix);
const maxLabelLen = yTickLabels.reduce((m, s) => Math.max(m, s.length), 0);
PAD.left = Math.max(28, Math.ceil(maxLabelLen * 5.5) + 10);
const cW = W - PAD.left - PAD.right;
const cH = H - PAD.top - PAD.bottom;
const x = t => PAD.left + ((t - tMin) / tRange) * cW;
const y = v => PAD.top + cH - ((v - yLow) / yRange) * cH;
// Color based on latest value
const latest = pts[pts.length - 1].v;
const { stroke: strokeColor, fill: fillColor } = opts.colorFor(latest);
// Compute nice tick step for ~3-5 grid lines
const rawStep = yRange / 4;
const mag = Math.pow(10, Math.floor(Math.log10(rawStep || 1)));
const niceStep = [1, 2, 5, 10].map(f => f * mag).find(s => yRange / s <= 5) || mag * 10;
const tickStart = Math.ceil(yLow / niceStep) * niceStep;
let gridLines = '';
for (let v = tickStart; v <= yHigh + 0.001; v += niceStep) {
const yy = y(v).toFixed(1);
const label = (Number.isInteger(v) ? v : v.toFixed(1)) + unitSuffix;
gridLines += `<line x1="${PAD.left}" y1="${yy}" x2="${PAD.left + cW}" y2="${yy}" stroke="#e0e0e0" stroke-width="1"/>`;
gridLines += `<text x="${(PAD.left - 3).toFixed(1)}" y="${yy}" text-anchor="end" dominant-baseline="middle" font-size="8" fill="#999">${label}</text>`;
// Split into segments wherever the gap between consecutive samples is
// much larger than the typical spacing (e.g. the host was overdue/down
// for a while) — draw each segment separately so missing data reads as
// a visual gap instead of an interpolated line across dead time.
const deltas = [];
for (let i = 1; i < pts.length; i++) deltas.push(pts[i].t - pts[i - 1].t);
deltas.sort((a, b) => a - b);
const medianDelta = deltas[Math.floor(deltas.length / 2)];
const gapThreshold = medianDelta * 2.5;
const segments = [[pts[0]]];
for (let i = 1; i < pts.length; i++) {
if (pts[i].t - pts[i - 1].t > gapThreshold) segments.push([]);
segments[segments.length - 1].push(pts[i]);
}
// X-axis time labels
let linePolylines = '';
let areaPaths = '';
for (const seg of segments) {
if (seg.length < 2) continue;
const segPoints = seg.map(p => `${x(p.t).toFixed(1)},${y(p.v).toFixed(1)}`).join(' ');
linePolylines += `<polyline points="${segPoints}" fill="none" stroke="${strokeColor}" stroke-width="1.5" stroke-linejoin="round"/>`;
const segArea = `M${x(seg[0].t).toFixed(1)},${(PAD.top + cH).toFixed(1)} ` +
seg.map(p => `L${x(p.t).toFixed(1)},${y(p.v).toFixed(1)}`).join(' ') +
` L${x(seg[seg.length-1].t).toFixed(1)},${(PAD.top + cH).toFixed(1)} Z`;
areaPaths += `<path d="${segArea}" fill="${fillColor}" opacity="0.6"/>`;
}
let gridLines = '';
yTicks.forEach((v, i) => {
const yy = y(v).toFixed(1);
gridLines += `<line x1="${PAD.left}" y1="${yy}" x2="${PAD.left + cW}" y2="${yy}" stroke="#e0e0e0" stroke-width="1"/>`;
gridLines += `<text x="${(PAD.left - 3).toFixed(1)}" y="${yy}" text-anchor="end" dominant-baseline="middle" font-size="8" fill="#999">${yTickLabels[i]}</text>`;
});
// X-axis: start/end plus evenly spaced intermediate tickmarks + labels
const fmt = ts => {
const d = new Date(ts * 1000);
return d.toLocaleTimeString([], { hour: '2-digit', minute: '2-digit' });
};
const xLabels = `
<text x="${PAD.left}" y="${H - 2}" text-anchor="start" font-size="8" fill="#999">${fmt(pts[0].t)}</text>
<text x="${PAD.left + cW}" y="${H - 2}" text-anchor="end" font-size="8" fill="#999">${fmt(pts[pts.length-1].t)}</text>`;
const xTickCount = 5;
const xAxisY = (PAD.top + cH).toFixed(1);
let xAxisMarks = '';
let xLabels = '';
for (let i = 0; i < xTickCount; i++) {
const t = tMin + (tRange * i) / (xTickCount - 1);
const xx = x(t).toFixed(1);
const anchor = i === 0 ? 'start' : (i === xTickCount - 1 ? 'end' : 'middle');
xAxisMarks += `<line x1="${xx}" y1="${xAxisY}" x2="${xx}" y2="${(PAD.top + cH + 3).toFixed(1)}" stroke="#ccc" stroke-width="1"/>`;
xLabels += `<text x="${xx}" y="${H - 2}" text-anchor="${anchor}" font-size="8" fill="#999">${fmt(t)}</text>`;
}
el.innerHTML = `<svg viewBox="0 0 ${W} ${H}" preserveAspectRatio="none"
style="width:100%;height:${H}px;display:block;">
@@ -890,9 +928,10 @@
<line x1="${PAD.left}" y1="${PAD.top}" x2="${PAD.left}" y2="${PAD.top + cH}" stroke="#ccc" stroke-width="1"/>
<line x1="${PAD.left}" y1="${PAD.top + cH}" x2="${PAD.left + cW}" y2="${PAD.top + cH}" stroke="#ccc" stroke-width="1"/>
<g clip-path="url(#${opts.clipId})">
<path d="${areaPath}" fill="${fillColor}" opacity="0.6"/>
<polyline points="${linePoints}" fill="none" stroke="${strokeColor}" stroke-width="1.5" stroke-linejoin="round"/>
${areaPaths}
${linePolylines}
</g>
${xAxisMarks}
${xLabels}
</svg>`;
}
@@ -1104,6 +1143,24 @@
function renderNetworkTables(d) {
let html = '';
if (d.connectivity) {
let t = '<table class="data-table"><thead><tr><th>Protocol</th><th>Internet</th>'
+ '<th>Address</th><th>Gateway</th><th class="num">RTT</th></tr></thead><tbody>';
for (const fam of ['ipv4', 'ipv6']) {
const c = d.connectivity[fam];
if (!c) continue;
const known = c.present === true || c.present === false;
const cls = c.present === true ? 'status-up' : '';
const txt = c.present === true ? '✓ present' : (c.present === false ? 'absent' : '?');
t += `<tr><td>${fam === 'ipv4' ? 'IPv4' : 'IPv6'}</td>
<td class="${cls}">${txt}</td>
<td>${escHtml(c.addr || '—')}</td><td>${escHtml(c.gateway || '—')}</td>
<td class="num">${known && c.rtt != null ? Math.round(c.rtt) + ' ms' : '—'}</td></tr>`;
}
t += '</tbody></table>';
html += `<div class="table-section-label">Connectivity</div>${t}`;
}
if (d.interfaces && Object.keys(d.interfaces).length) {
const sample = Object.values(d.interfaces)[0];
const hasDelta = 'bytes_sent_delta' in sample;
+110 -29
View File
@@ -194,6 +194,40 @@ def _set_connectivity_alert(host, afam, level_name):
state.update(level, level_name)
_PRESENCE_KEYS = {"p4": "IPv4", "p6": "IPv6"}
def _apply_presence(host, msg, now, uname, ctx, skip_conn):
"""Record client-reported protocol presence and settle the affected connections.
A family reported absent whose connection is already overdue/unknown is parked as
ABSENT (timer cancelled, connectivity alert cleared). One still UP is left alone; its
timer expiry parks it. A family reported present again while ABSENT restarts a grace
timer. The connection that carried this very heartbeat (skip_conn) is never touched.
"""
cfg = ctx.get("config", {})
for key, afam in _PRESENCE_KEYS.items():
if key not in msg:
continue
present = bool(msg[key])
host.set_presence(afam, present, now)
conn = host.connections.get(afam)
if conn is None or conn is skip_conn:
continue
cls = conn.__class__
if not present:
if conn.getstate() in (cls.OVERDUE, cls.UNKNOWN):
conn.cancel_overdue_timer()
conn.newstate(cls.ABSENT, now)
_set_connectivity_alert(host, afam, "OK")
elif conn.getstate() == cls.ABSENT:
conn.newstate(cls.UNKNOWN, now)
interval = int(msg.get("interval", 0) or 0) or host.interval
if interval > 0:
on_overdue, _ = _make_timer_callbacks(uname, host, ctx)
conn.reset_overdue_timer(interval + cfg.get("grace", 2), on_overdue)
def _make_timer_callbacks(uname, host, ctx):
"""Return (on_overdue, on_unknown) async callbacks for connection timer logic.
@@ -210,26 +244,52 @@ def _make_timer_callbacks(uname, host, ctx):
msg_to_websockets("host", host.stateinfo())
async def on_overdue(connection):
if connection.getstate() != connection.__class__.UP:
cls = connection.__class__
if connection.getstate() not in (cls.UP, cls.UNKNOWN):
return
now = time.time()
connection.newstate(connection.__class__.OVERDUE, now, cfg.get("grace", 2))
msg = f"{connection.afam} overdue"
eventlog(uname, "CRITICAL", msg)
if host.watched:
asyncio.create_task(notify_mod.send_notification(
uname,
notify_mod.Notification(title=f"[CRITICAL] {uname}", body=msg, level="CRITICAL"),
))
# Track in alert_states so the Alerts Dashboard shows this
_set_connectivity_alert(host, connection.afam, "CRITICAL")
if threshold_checker:
threshold_checker.check_value(
host_name=uname,
metric_path="rtt",
value=float("inf"),
alert_states=host.alert_states,
)
afam = connection.afam
# Park only while some other family is still present: the last monitored family
# must never be parked silently, or a host whose client reports no internet on any
# protocol would die without an alert.
if not host.flag_present(afam) and any(
af != afam and host.family_present(af) for af in host.connections):
# The client says this protocol has no connectivity: not an outage.
connection.newstate(cls.ABSENT, now)
connection.cancel_overdue_timer()
_set_connectivity_alert(host, afam, "OK")
if msg_to_websockets:
msg_to_websockets("host", host.stateinfo())
return
# If any connection is DOWN (clean shutdown reported) Host.status() is "down", so a
# later overdue of the other family raises no separate event/alert; the shutdown did.
prev = host.status()
connection.newstate(cls.OVERDUE, now, cfg.get("grace", 2))
new = host.status()
if new == "overdue" and prev != "overdue":
names = ", ".join(sorted(af for af, c in host.connections.items()
if c.getstate() == cls.OVERDUE))
msg = f"{names} overdue"
eventlog(uname, "CRITICAL", msg)
if host.watched:
asyncio.create_task(notify_mod.send_notification(
uname,
notify_mod.Notification(title=f"[CRITICAL] {uname}", body=msg, level="CRITICAL"),
))
# Track in alert_states so the Alerts Dashboard shows this
for af, c in host.connections.items():
if c.getstate() == cls.OVERDUE:
_set_connectivity_alert(host, af, "CRITICAL")
if threshold_checker:
threshold_checker.check_value(
host_name=uname,
metric_path="rtt",
value=float("inf"),
alert_states=host.alert_states,
)
elif new == "simplex":
eventlog(uname, "INFO", f"{afam} overdue (simplex)")
if msg_to_websockets:
msg_to_websockets("host", host.stateinfo())
connection.reset_overdue_timer(DROPOVERDUE, on_unknown)
@@ -270,6 +330,8 @@ def restore_connection_timers(hbdclass, ctx):
interval = host.interval
for afam, conn in list(host.connections.items()):
state = conn.getstate()
if state == hbdclass.Connection.ABSENT:
continue
if state == hbdclass.Connection.DOWN:
_set_connectivity_alert(host, afam, "CRITICAL")
continue
@@ -295,6 +357,7 @@ def restore_connection_timers(hbdclass, ctx):
restored += 1
elif state == hbdclass.Connection.OVERDUE:
host_overdue = host.status() == "overdue"
elapsed_overdue = now - conn.statetime
remaining = DROPOVERDUE - elapsed_overdue
if remaining <= 1:
@@ -304,6 +367,9 @@ def restore_connection_timers(hbdclass, ctx):
"Marking %s/%s UNKNOWN (overdue %.1f days)",
uname, afam, elapsed_overdue / 86400,
)
# A connection dropped to UNKNOWN keeps its alert, like the
# UNKNOWN branch above and on_unknown do.
host_overdue = True
else:
conn.reset_overdue_timer(remaining, on_unknown)
logger.debug(
@@ -313,7 +379,8 @@ def restore_connection_timers(hbdclass, ctx):
# Ensure the connectivity alert is set — it may be missing if
# hbd was shut down before the on_overdue callback had a chance
# to record it.
_set_connectivity_alert(host, afam, "CRITICAL")
if host_overdue:
_set_connectivity_alert(host, afam, "CRITICAL")
restored += 1
logger.info("Restored timers for %d connection(s)", restored)
@@ -517,25 +584,36 @@ def handle_datagram(msg: dict, addr, transport, ctx: dict):
if message:
eventlog(uname, "INFO", message, service=service)
_apply_presence(host, msg, now, uname, ctx, conn)
prev_status = host.status()
if conn.getstate() != hbdcls.Connection.UP:
# Transition to UP and log/notify if appropriate
lasts = conn.state
d = conn.newstate(hbdcls.Connection.UP, now)
# On reboot, pre-boot plugin data and derived alerts are stale.
# Cancel all plugin timers and wipe plugin state so timers restart
# cleanly from the first two post-boot samples.
for pname in list(host.plugin_timers):
host.cancel_plugin_timer(pname)
for pname in [k for k in host.plugin_data if not _is_rtt_key(k)]:
del host.plugin_data[pname]
if boot:
# On reboot, pre-boot plugin data and derived alerts are stale.
# Cancel all plugin timers and wipe plugin state so timers restart
# cleanly from the first two post-boot samples. An ordinary
# reconnect (no boot flag) doesn't invalidate the client's
# already-collected data, so it's left alone — this keeps chart
# history intact across a transient network blip.
for pname in list(host.plugin_timers):
host.cancel_plugin_timer(pname)
for pname in [k for k in host.plugin_data if not _is_rtt_key(k)]:
del host.plugin_data[pname]
stale_plugin_keys = [
k for k in host.alert_states
if k not in ("rtt",) and not k.startswith("connectivity.")
]
for k in stale_plugin_keys:
del host.alert_states[k]
# Clear connectivity alert now that the host is back up
# Clear connectivity alert now that the host is back up; once the host is no longer
# overdue no family may keep a connectivity alert (e.g. the simplex case).
_set_connectivity_alert(host, conn.afam, "OK")
if host.status() in ("up", "simplex"):
for af in host.connections:
_set_connectivity_alert(host, af, "OK")
# Don't log/notify RECOVER for a brand-new host seen for the first time —
# it was never down, it just hasn't been seen before.
if not newh:
@@ -547,8 +625,11 @@ def handle_datagram(msg: dict, addr, transport, ctx: dict):
else:
m = "%s back after being %s for %s" % (conn.afam, lasts, dur(d))
if m:
eventlog(uname, "RECOVER", m)
if host.watched:
# RECOVER (and a notification) only if the host itself was overdue/down;
# a single family coming back while another stayed up is informational.
level = "RECOVER" if prev_status in ("overdue", "down", "unknown") else "INFO"
eventlog(uname, level, m)
if host.watched and level == "RECOVER":
asyncio.create_task(notify_mod.send_notification(
uname,
notify_mod.Notification(title=f"[RECOVER] {uname}", body=m, level="RECOVER"),
+1 -1
View File
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
[project]
name = "hbd"
version = "5.4.1"
version = "5.4.5"
description = "Heartbeat monitoring system — client (hbc) and server (hbd)"
readme = "README.md"
requires-python = ">=3.11"
+4 -2
View File
@@ -9,8 +9,10 @@ sed -i".bak" "s/\*\*Package:\*\* \`hbd\` v[0-9.]*/\*\*Package:\*\* \`hbd\` v$VE
# Update CHANGELOG.md with commits since last tag
LASTTAG=$(git describe --tags --abbrev=0 2>/dev/null || true)
ADDED=$(git log "${LASTTAG:+$LASTTAG..}HEAD" --pretty="%s" | grep "^feat:" | sed 's/^feat: /- /')
FIXED=$(git log "${LASTTAG:+$LASTTAG..}HEAD" --pretty="%s" | grep "^fix:" | sed 's/^fix: /- /')
# Matches both unscoped ("feat: ...") and scoped ("feat(server): ...") conventional-commit
# subjects; the scope is dropped from the changelog line.
ADDED=$(git log "${LASTTAG:+$LASTTAG..}HEAD" --pretty="%s" | grep -E "^feat(\([^)]*\))?:" | sed -E 's/^feat(\([^)]*\))?: /- /')
FIXED=$(git log "${LASTTAG:+$LASTTAG..}HEAD" --pretty="%s" | grep -E "^fix(\([^)]*\))?:" | sed -E 's/^fix(\([^)]*\))?: /- /')
{
printf "## [%s]\n" "$VER"
[ -n "$ADDED" ] && printf "\n### Added\n%s\n" "$ADDED"
+1
View File
@@ -1,2 +1,3 @@
hbc_mini
hbc_mini_dbg
test_probe
+8 -2
View File
@@ -7,7 +7,7 @@ SRC = hbc_mini.c
# FreeBSD/NetBSD keep zlib in base; no extra flags needed.
# On some NetBSD installs pthreads may need -lpthread from pkgsrc.
.PHONY: all clean debug
.PHONY: all clean debug test
all: $(TARGET)
@@ -17,5 +17,11 @@ $(TARGET): $(SRC)
debug: $(SRC)
$(CC) -g -fsanitize=address,undefined -o $(TARGET)_dbg $< $(LDFLAGS)
test: test_probe
./test_probe
test_probe: test_probe.c $(SRC)
$(CC) $(CFLAGS) -o $@ test_probe.c $(LDFLAGS)
clean:
rm -f $(TARGET) $(TARGET)_dbg
rm -f $(TARGET) $(TARGET)_dbg test_probe
+381 -5
View File
@@ -6,6 +6,12 @@
*
* Drop on any host and run: ./hbc_mini <hbd-host>
* Config: ~/.hbc.json (same keys as Python version)
*
* Presence probe (built in): reports per protocol (IPv4/IPv6) whether the host has internet
* connectivity, as p4/p6 flags on every heartbeat, so hbd stops treating a missing IPv6 (or
* IPv4) path as an outage. Config, in the network_monitor section of ~/.hbc.json:
* probe_enabled (default true), probe_v4 (default 1.1.1.1), probe_v6 (default
* 2606:4700:4700::1111; IP literals only), probe_interval (default 60 s)
*/
#ifdef __linux__
@@ -396,6 +402,10 @@ typedef struct {
char net_skip[MAX_NET_SKIP][64];
int net_skip_count;
bool probe_enabled;
int probe_interval;
char probe_v4[64], probe_v6[64];
} config_t;
static void config_defaults(config_t *c) {
@@ -410,6 +420,10 @@ static void config_defaults(config_t *c) {
c->net_interval = 300;
snprintf(c->net_skip[0], 64, "lo");
c->net_skip_count = 1;
c->probe_enabled = true;
c->probe_interval = 60;
snprintf(c->probe_v4, sizeof(c->probe_v4), "1.1.1.1");
snprintf(c->probe_v6, sizeof(c->probe_v6), "2606:4700:4700::1111");
}
static void config_load(config_t *cfg, const char *path) {
@@ -498,6 +512,10 @@ static void config_load(config_t *cfg, const char *path) {
jval_t *net = plugins ? jget(plugins,"network_monitor") : jget(root,"network_monitor");
if (net && net->type == JT_OBJECT) {
if ((v=jget(net,"interval"))) cfg->net_interval = jint(v, cfg->net_interval);
if ((v=jget(net,"probe_enabled"))) cfg->probe_enabled = jint(v, 1) != 0;
if ((v=jget(net,"probe_interval"))) cfg->probe_interval = jint(v, cfg->probe_interval);
if ((v=jget(net,"probe_v4"))) snprintf(cfg->probe_v4, sizeof(cfg->probe_v4), "%s", jstr(v, cfg->probe_v4));
if ((v=jget(net,"probe_v6"))) snprintf(cfg->probe_v6, sizeof(cfg->probe_v6), "%s", jstr(v, cfg->probe_v6));
jval_t *sk = jget(net,"skip_interfaces");
if (sk && sk->type == JT_ARRAY) {
cfg->net_skip_count = 0;
@@ -984,6 +1002,341 @@ static void plugin_disk_monitor(conn_t *c, const config_t *cfg) {
LOGD("sent disk_monitor");
}
/* ============================================================
* Connectivity probe: per-protocol internet presence (IPv4 / IPv6)
*
* Tier 1 (every PROBE_TIER1 s, sends no packets): a UDP connect() to the probe address shows
* whether a route exists and which source address would be used. Tier 2 (every probe_interval
* s, sooner after a failure or a change of address or gateway): one ICMP echo, only when tier 1
* found a route. Absent = no route, or PROBE_FAIL_LIMIT consecutive failed echoes; present =
* one success; unknown (flag omitted) when the family cannot be probed. The result rides on
* every HTB as p4/p6 and inside the network_monitor metric. Mirrors hbd/client/connectivity.py.
* ============================================================ */
#define PROBE_UNKNOWN (-1)
#define PROBE_FAIL_LIMIT 2
#define PROBE_TIER1 5 /* seconds between route checks */
typedef struct {
int present; /* PROBE_UNKNOWN, 0 = absent, 1 = present */
char addr[INET6_ADDRSTRLEN];
char gateway[INET6_ADDRSTRLEN];
double rtt; /* ms, < 0 when unknown */
int fails; /* consecutive failed echoes */
double next_ping; /* monotonic seconds */
char last_key[2 * INET6_ADDRSTRLEN + 2];
bool ping_disabled; /* no usable ping binary */
} probe_fam_t;
typedef struct {
/* 1 = route exists (source address in out), 0 = no route, -1 = family unsupported/unusable */
int (*local_address)(int af, const char *target, char *out, size_t n);
/* RTT in ms; -1 = no reply; -2 = ping unavailable */
double (*ping)(int af, const char *target);
/* fills out with the default gateway, "" when unknown */
void (*gateway)(int af, char *out, size_t n);
} probe_ops_t;
typedef struct {
probe_fam_t fam[2]; /* [0] = IPv4, [1] = IPv6 */
char target[2][64];
double probe_interval, retry_interval;
bool cheap_gateway; /* gateway read is cheap (/proc), so read every tier 1 */
pthread_mutex_t mu; /* guards fam[] between the probe thread and readers */
} probe_t;
/* Lowest-metric default gateway from /proc/net/route. The kernel prints the raw network-order
* u32 in host order, so it can be stored straight into s_addr on any endianness. */
static void gw_parse_proc_v4(const char *text, char *out, size_t n) {
long best = -1; out[0] = '\0';
const char *p = strchr(text, '\n'); /* skip the header line */
while (p && *++p) {
char iface[64], dest[16], gw[16]; unsigned flags; int refcnt, use, metric;
if (sscanf(p, "%63s %15s %15s %x %d %d %d", iface, dest, gw, &flags, &refcnt, &use, &metric) == 7
&& !strcmp(dest, "00000000") && (flags & 0x2)) {
if (best < 0 || metric < best) {
struct in_addr a; a.s_addr = (uint32_t)strtoul(gw, NULL, 16);
if (inet_ntop(AF_INET, &a, out, n)) best = metric;
}
}
p = strchr(p, '\n');
}
}
/* Lowest-metric default gateway from /proc/net/ipv6_route (skips RTF_REJECT and "lo"). */
static void gw_parse_proc_v6(const char *text, char *out, size_t n) {
long best = -1; out[0] = '\0';
const char *zero = "00000000000000000000000000000000";
for (const char *p = text; p && *p; ) {
char dest[40], src[40], nh[40], iface[64];
unsigned plen, splen, metric, refcnt, use, flags;
if (sscanf(p, "%32s %x %32s %x %32s %x %x %x %x %63s",
dest, &plen, src, &splen, nh, &metric, &refcnt, &use, &flags, iface) == 10
&& !strcmp(dest, zero) && plen == 0 && strcmp(nh, zero) && strcmp(iface, "lo")
&& (flags & 0x2) && !(flags & 0x200)) {
if (best < 0 || (long)metric < best) {
struct in6_addr a; memset(&a, 0, sizeof(a));
bool ok = strlen(nh) == 32;
for (int i = 0; ok && i < 16; i++) {
unsigned b;
if (sscanf(nh + 2 * i, "%2x", &b) != 1) ok = false; else a.s6_addr[i] = (uint8_t)b;
}
if (ok && inet_ntop(AF_INET6, &a, out, n)) best = (long)metric;
}
}
p = strchr(p, '\n'); if (p) p++;
}
}
/* Gateway from `route -n get [-inet6] default` (the BSDs). */
static void gw_parse_route_get(const char *text, char *out, size_t n) {
out[0] = '\0';
const char *p = strstr(text, "gateway:");
if (!p) return;
p += 8;
while (*p == ' ' || *p == '\t') p++;
size_t i = 0;
while (*p && !isspace((unsigned char)*p) && i + 1 < n) out[i++] = *p++;
out[i] = '\0';
}
/* Average RTT from ping output: -1.0 no reply, -2.0 ping could not be run. Works for Linux
* ("rtt min/avg/max/mdev") and BSD ("round-trip min/avg/max/stddev"). */
static double ping_parse_rtt(const char *out) {
if (strstr(out, "not found") || strstr(out, "No such file")) return -2.0;
const char *p = strstr(out, "min/avg/max");
if (!p) return -1.0;
p = strchr(p, '=');
double mn, avg, mx;
if (p && sscanf(p + 1, " %lf/%lf/%lf", &mn, &avg, &mx) == 3) return avg;
return -1.0;
}
/* One probe step for one family (f: 0 = IPv4, 1 = IPv6). Works on *s only, so the probe thread
* can run it on a private copy without holding the mutex during the (blocking) echo. */
static void probe_step(const probe_t *p, probe_fam_t *s, int f, const probe_ops_t *ops, double now) {
int af = f ? AF_INET6 : AF_INET;
char addr[INET6_ADDRSTRLEN] = "", gw[INET6_ADDRSTRLEN] = "";
int r = ops->local_address(af, p->target[f], addr, sizeof(addr));
if (r < 0) { s->present = PROBE_UNKNOWN; return; }
if (r == 0) { /* tier 1: no route => absent immediately */
s->fails = 0; s->next_ping = 0; s->last_key[0] = '\0';
s->present = 0; s->addr[0] = '\0'; s->gateway[0] = '\0'; s->rtt = -1;
return;
}
if (p->cheap_gateway) ops->gateway(af, gw, sizeof(gw));
else snprintf(gw, sizeof(gw), "%s", s->gateway);
char key[sizeof(s->last_key)];
snprintf(key, sizeof(key), "%s|%s", addr, gw);
bool due = strcmp(key, s->last_key) != 0 || now >= s->next_ping;
if (due && !p->cheap_gateway) {
ops->gateway(af, gw, sizeof(gw));
snprintf(key, sizeof(key), "%s|%s", addr, gw);
}
snprintf(s->addr, sizeof(s->addr), "%s", addr);
snprintf(s->gateway, sizeof(s->gateway), "%s", gw);
snprintf(s->last_key, sizeof(s->last_key), "%s", key);
if (s->ping_disabled) { s->present = PROBE_UNKNOWN; return; } /* cannot probe: unknown, never absent, while a route exists */
if (!due) return;
double rtt = ops->ping(af, p->target[f]);
if (rtt < -1.5) { s->ping_disabled = true; s->present = PROBE_UNKNOWN; return; }
if (rtt < 0) {
s->fails++;
s->next_ping = now + p->retry_interval;
if (s->fails >= PROBE_FAIL_LIMIT) { s->present = 0; s->rtt = -1; }
} else {
s->fails = 0;
s->next_ping = now + p->probe_interval;
s->present = 1; s->rtt = rtt;
}
}
/* HTB flags for the families whose presence is known. */
static void probe_flags(probe_t *p, kvdict_t *d) {
pthread_mutex_lock(&p->mu);
if (p->fam[0].present >= 0) kv_set_int(d, "p4", p->fam[0].present);
if (p->fam[1].present >= 0) kv_set_int(d, "p6", p->fam[1].present);
pthread_mutex_unlock(&p->mu);
}
/* The `connectivity` metric as a JSON object; returns its length or -1 on overflow. */
static int probe_metric_json(probe_t *p, char *out, size_t n) {
static const char *names[2] = { "ipv4", "ipv6" };
size_t len = 0;
len += (size_t)snprintf(out + len, n - len, "{");
pthread_mutex_lock(&p->mu);
for (int f = 0; f < 2 && len < n; f++) {
const probe_fam_t *s = &p->fam[f];
char pres[8], addr[64], gw[64], rtt[32];
snprintf(pres, sizeof(pres), "%s", s->present < 0 ? "null" : s->present ? "true" : "false");
if (s->addr[0]) snprintf(addr, sizeof(addr), "\"%s\"", s->addr); else snprintf(addr, sizeof(addr), "null");
if (s->gateway[0]) snprintf(gw, sizeof(gw), "\"%s\"", s->gateway); else snprintf(gw, sizeof(gw), "null");
if (s->rtt >= 0) snprintf(rtt, sizeof(rtt), "%.3f", s->rtt); else snprintf(rtt, sizeof(rtt), "null");
len += (size_t)snprintf(out + len, n - len,
"%s\"%s\":{\"present\":%s,\"addr\":%s,\"gateway\":%s,\"rtt\":%s}",
f ? "," : "", names[f], pres, addr, gw, rtt);
}
pthread_mutex_unlock(&p->mu);
if (len + 2 > n) { out[0] = '\0'; return -1; }
out[len++] = '}'; out[len] = '\0';
return (int)len;
}
static double mono_ts(void) {
struct timespec ts;
clock_gettime(CLOCK_MONOTONIC, &ts);
return ts.tv_sec + ts.tv_nsec * 1e-9;
}
/* Source address the kernel would use to reach target (an IP literal). A UDP connect() sends
* nothing. Returns 1 = route (address in out), 0 = no route, -1 = family unsupported or the
* target is not an IP literal (unknown, never absent). */
static int sys_local_address(int af, const char *target, char *out, size_t n) {
int fd = socket(af, SOCK_DGRAM, 0);
if (fd < 0) return -1;
struct sockaddr_storage sa; socklen_t sl;
memset(&sa, 0, sizeof(sa));
if (af == AF_INET) {
struct sockaddr_in *s4 = (struct sockaddr_in *)&sa;
s4->sin_family = AF_INET; s4->sin_port = htons(9);
if (inet_pton(AF_INET, target, &s4->sin_addr) != 1) { close(fd); return -1; }
sl = sizeof(*s4);
} else {
struct sockaddr_in6 *s6 = (struct sockaddr_in6 *)&sa;
s6->sin6_family = AF_INET6; s6->sin6_port = htons(9);
if (inet_pton(AF_INET6, target, &s6->sin6_addr) != 1) { close(fd); return -1; }
sl = sizeof(*s6);
}
int rc = 0;
if (connect(fd, (struct sockaddr *)&sa, sl) == 0) {
struct sockaddr_storage me; socklen_t ml = sizeof(me);
if (getsockname(fd, (struct sockaddr *)&me, &ml) == 0) {
const void *src = (af == AF_INET)
? (const void *)&((struct sockaddr_in *)&me)->sin_addr
: (const void *)&((struct sockaddr_in6 *)&me)->sin6_addr;
if (inet_ntop(af, src, out, (socklen_t)n)) rc = 1;
}
}
close(fd);
return rc;
}
static char *slurp(FILE *f) {
size_t cap = 4096, len = 0;
char *buf = malloc(cap);
if (!buf) return NULL;
size_t r;
while ((r = fread(buf + len, 1, cap - len - 1, f)) > 0) {
len += r;
if (len + 1 >= cap) {
if (cap >= 65536) break;
cap *= 2;
char *nb = realloc(buf, cap);
if (!nb) { free(buf); return NULL; }
buf = nb;
}
}
buf[len] = '\0';
return buf;
}
static bool g_have_timeout = false; /* timeout(1) exists: bounds ping6 on the BSDs */
/* One echo request: RTT in ms, -1.0 no reply, -2.0 ping could not be run. The target is an IP
* literal from the config; anything else is refused before it can reach the shell. */
static double sys_ping(int af, const char *target) {
for (const char *c = target; *c; c++)
if (!(isxdigit((unsigned char)*c) || *c == '.' || *c == ':')) return -2.0;
char cmd[256];
#ifdef __linux__
snprintf(cmd, sizeof(cmd), "ping -c 1 -W 2 -w 3 %s %s 2>&1", af == AF_INET ? "-4" : "-6", target);
#else
/* BSD ping: -W is in milliseconds, and IPv6 is reached through ping6 */
if (af == AF_INET)
snprintf(cmd, sizeof(cmd), "%sping -c 1 -W 2000 %s 2>&1", g_have_timeout ? "timeout 4 " : "", target);
else
snprintf(cmd, sizeof(cmd), "%sping6 -c 1 %s 2>&1", g_have_timeout ? "timeout 4 " : "", target);
#endif
FILE *f = popen(cmd, "r");
if (!f) return -2.0;
char *out = slurp(f);
pclose(f);
if (!out) return -1.0;
double rtt = ping_parse_rtt(out);
free(out);
return rtt;
}
/* Default gateway for the family, "" when unknown. Best effort; never affects presence. */
static void sys_gateway(int af, char *out, size_t n) {
out[0] = '\0';
#ifdef __linux__
FILE *f = fopen(af == AF_INET ? "/proc/net/route" : "/proc/net/ipv6_route", "r");
if (!f) return;
char *text = slurp(f);
fclose(f);
if (!text) return;
if (af == AF_INET) gw_parse_proc_v4(text, out, n); else gw_parse_proc_v6(text, out, n);
free(text);
#else
FILE *f = popen(af == AF_INET ? "route -n get default 2>/dev/null"
: "route -n get -inet6 default 2>/dev/null", "r");
if (!f) return;
char *text = slurp(f);
pclose(f);
if (!text) return;
gw_parse_route_get(text, out, n);
free(text);
#endif
}
static probe_t g_probe;
static bool g_probe_enabled = false;
static void probe_init(probe_t *p, const config_t *cfg) {
memset(p, 0, sizeof(*p));
snprintf(p->target[0], sizeof(p->target[0]), "%s", cfg->probe_v4);
snprintf(p->target[1], sizeof(p->target[1]), "%s", cfg->probe_v6);
p->probe_interval = cfg->probe_interval > 0 ? cfg->probe_interval : 60;
p->retry_interval = 10;
#ifdef __linux__
p->cheap_gateway = true; /* /proc read; elsewhere a subprocess only when due */
#endif
p->fam[0].present = p->fam[1].present = PROBE_UNKNOWN;
p->fam[0].rtt = p->fam[1].rtt = -1;
pthread_mutex_init(&p->mu, NULL);
g_have_timeout = system("command -v timeout >/dev/null 2>&1") == 0;
g_probe_enabled = cfg->probe_enabled;
/* Diagnostic only: sys_local_address already turns a non-literal target into PROBE_UNKNOWN
* (via inet_pton failing there), so this does not change behavior — it just stops that from
* being silent. */
struct in_addr a4;
struct in6_addr a6;
if (inet_pton(AF_INET, p->target[0], &a4) != 1)
LOGW("probe_v4 is not an IPv4 literal: %s", p->target[0]);
if (inet_pton(AF_INET6, p->target[1], &a6) != 1)
LOGW("probe_v6 is not an IPv6 literal: %s", p->target[1]);
}
static void *probe_thread(void *arg) {
probe_t *p = arg;
static const probe_ops_t ops = { sys_local_address, sys_ping, sys_gateway };
while (g_running) {
double now = mono_ts();
for (int f = 0; f < 2 && g_running; f++) {
pthread_mutex_lock(&p->mu); probe_fam_t tmp = p->fam[f]; pthread_mutex_unlock(&p->mu);
probe_step(p, &tmp, f, &ops, now); /* may block in ping: no lock held */
pthread_mutex_lock(&p->mu); p->fam[f] = tmp; pthread_mutex_unlock(&p->mu);
}
for (int i = 0; i < PROBE_TIER1 && g_running; i++) sleep(1);
}
return NULL;
}
/* ============================================================
* Plugin: network_monitor
* Linux: /proc/net/dev
@@ -1077,16 +1430,26 @@ static void plugin_network_monitor(conn_t *c, const config_t *cfg) {
json[jlen++] = '}'; json[jlen] = '\0';
memcpy(g_net_prev, curr, cn * sizeof(net_stat_t));
g_net_prev_n = cn; g_net_prev_ts = now;
if (first) { free(json); return; }
kvdict_t d; kv_clear(&d);
kv_set(&d, "plugin", "network_monitor");
char *jval = malloc(MAX_VAL + 1);
if (jval) { snprintf(jval, MAX_VAL, "@%s", json); kv_set(&d, "interfaces", jval); free(jval); }
if (!first) {
char *jval = malloc(MAX_VAL + 1);
if (jval) { snprintf(jval, MAX_VAL, "@%s", json); kv_set(&d, "interfaces", jval); free(jval); }
}
bool have_conn = false;
if (g_probe_enabled) {
char cj[512];
cj[0] = '@';
if (probe_metric_json(&g_probe, cj + 1, sizeof(cj) - 1) > 0) {
kv_set(&d, "connectivity", cj);
have_conn = true;
}
}
free(json);
if (first && !have_conn) return;
kv_set_dbl(&d, "_timestamp", now_ts());
kv_set_int(&d, "_interval", cfg->net_interval);
conn_send(c, "PLG", &d);
free(json);
LOGD("sent network_monitor");
}
@@ -1323,6 +1686,7 @@ int main(int argc, char **argv) {
config_t cfg;
config_load(&cfg, cfgpath);
probe_init(&g_probe, &cfg);
if (do_daemon) {
daemonize();
@@ -1405,6 +1769,13 @@ int main(int argc, char **argv) {
pthread_t tid;
pthread_create(&tid, NULL, monitor_thread, &tctx);
pthread_t ptid; bool probe_running = false;
if (g_probe_enabled) {
int prc = pthread_create(&ptid, NULL, probe_thread, &g_probe);
if (prc == 0) probe_running = true;
else LOGW("pthread_create(probe): %s", strerror(prc));
}
/* Main heartbeat + receive loop */
time_t last_hb = 0;
while (g_running) {
@@ -1415,6 +1786,7 @@ int main(int argc, char **argv) {
kv_set_int(&hb, "acks", g_conns[i].ack_count);
kv_set_dbl(&hb, "rtt", g_conns[i].rtt);
kv_set_int(&hb, "interval", cfg.interval);
if (g_probe_enabled) probe_flags(&g_probe, &hb);
conn_send(&g_conns[i], "HTB", &hb);
}
last_hb = now;
@@ -1449,6 +1821,10 @@ int main(int argc, char **argv) {
conn_send(primary, "HTB", &sh);
usleep(300000);
}
/* Joined after the shutdown HTB goes out: the probe thread is never cancelled (it can be
* inside popen), so joining it can block for a few seconds — long enough to lose the
* clean-shutdown notification to an impatient supervisor's SIGKILL. */
if (probe_running) pthread_join(ptid, NULL);
for (int i = 0; i < g_nconns; i++) conn_close(&g_conns[i]);
if (g_use_syslog) closelog();
if (g_restart) { LOGI("restarting..."); execv(argv[0], argv); }
+243
View File
@@ -0,0 +1,243 @@
/* Unit tests for the connectivity-probe helpers in hbc_mini.c.
* Build and run: make test (live check of this host: ./test_probe --live)
* The source file is included directly so its static functions can be exercised; its main()
* is renamed out of the way.
*/
#define main hbc_mini_main
#include "hbc_mini.c"
#undef main
static int g_run = 0, g_fail = 0;
#define CHECK(cond) do { g_run++; if (!(cond)) { g_fail++; \
fprintf(stderr, "FAIL %s:%d: %s\n", __FILE__, __LINE__, #cond); } } while (0)
#define CHECK_STR(a, b) do { g_run++; if (strcmp((a), (b)) != 0) { g_fail++; \
fprintf(stderr, "FAIL %s:%d: \"%s\" != \"%s\"\n", __FILE__, __LINE__, (a), (b)); } } while (0)
static const char PROC_V4[] =
"Iface\tDestination\tGateway \tFlags\tRefCnt\tUse\tMetric\tMask\t\tMTU\tWindow\tIRTT\n"
"wlan0\t00000000\t0100A8C0\t0003\t0\t0\t600\t00000000\t0\t0\t0\n"
"eth0\t00000000\t0101A8C0\t0003\t0\t0\t100\t00000000\t0\t0\t0\n"
"wlan0\t0000A8C0\t00000000\t0001\t0\t0\t600\t00FFFFFF\t0\t0\t0\n";
static const char PROC_V6[] =
"00000000000000000000000000000000 00 00000000000000000000000000000000 00 "
"fe800000000000000000000000000001 00000400 00000001 00000000 00000003 wlan0\n"
"00000000000000000000000000000000 00 00000000000000000000000000000000 00 "
"00000000000000000000000000000000 ffffffff 00000001 00000000 00200200 lo\n"
"fe800000000000000000000000000000 40 00000000000000000000000000000000 00 "
"00000000000000000000000000000000 00000100 00000001 00000000 00000001 wlan0\n";
static const char ROUTE_GET[] =
" route to: 0.0.0.0\ndestination: 0.0.0.0\n mask: 0.0.0.0\n"
" gateway: 192.0.2.1\n fib: 0\n interface: vtnet0\n";
static const char LINUX_PING[] =
"PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.\n"
"64 bytes from 1.1.1.1: icmp_seq=1 ttl=57 time=11.2 ms\n\n"
"--- 1.1.1.1 ping statistics ---\n"
"1 packets transmitted, 1 received, 0% packet loss, time 0ms\n"
"rtt min/avg/max/mdev = 11.234/11.234/11.234/0.000 ms\n";
static const char FREEBSD_PING[] =
"PING 1.1.1.1 (1.1.1.1): 56 data bytes\n"
"64 bytes from 1.1.1.1: icmp_seq=0 ttl=59 time=4.006 ms\n\n"
"--- 1.1.1.1 ping statistics ---\n"
"1 packets transmitted, 1 packets received, 0.0% packet loss, 1 packets out of wait time\n"
"round-trip min/avg/max/stddev = 4.006/4.006/4.006/0.000 ms\n";
static void test_gateway_parsers(void) {
char gw[64];
gw_parse_proc_v4(PROC_V4, gw, sizeof(gw));
CHECK_STR(gw, "192.168.1.1"); /* eth0, metric 100 beats wlan0, 600 */
gw_parse_proc_v4("Iface\tDestination\n", gw, sizeof(gw));
CHECK_STR(gw, "");
gw_parse_proc_v6(PROC_V6, gw, sizeof(gw));
CHECK_STR(gw, "fe80::1"); /* reject and non-default lines skipped */
gw_parse_route_get(ROUTE_GET, gw, sizeof(gw));
CHECK_STR(gw, "192.0.2.1");
gw_parse_route_get("route: writing to routing socket: not in table\n", gw, sizeof(gw));
CHECK_STR(gw, "");
}
static void test_ping_parser(void) {
CHECK(ping_parse_rtt(LINUX_PING) > 11.23 && ping_parse_rtt(LINUX_PING) < 11.24);
CHECK(ping_parse_rtt(FREEBSD_PING) > 4.0 && ping_parse_rtt(FREEBSD_PING) < 4.01);
CHECK(ping_parse_rtt("1 packets transmitted, 0 received, 100% packet loss\n") == -1.0);
CHECK(ping_parse_rtt("sh: 1: ping: not found\n") == -2.0);
CHECK(ping_parse_rtt("ping: command not found\n") == -2.0);
}
/* ---- scriptable network for probe_step ------------------------------------------------ */
static struct {
char addr[2][INET6_ADDRSTRLEN]; /* "" = no route */
char gw[2][INET6_ADDRSTRLEN];
double rtt[2]; /* -1 no reply, -2 ping unavailable */
int unsupported[2];
int npings[2];
} net;
static int fake_local(int af, const char *t, char *out, size_t n) {
int f = af == AF_INET6; (void)t;
if (net.unsupported[f]) return -1;
if (!net.addr[f][0]) return 0;
snprintf(out, n, "%s", net.addr[f]);
return 1;
}
static double fake_ping(int af, const char *t) { int f = af == AF_INET6; (void)t; net.npings[f]++; return net.rtt[f]; }
static void fake_gw(int af, char *out, size_t n) { snprintf(out, n, "%s", net.gw[af == AF_INET6]); }
static const probe_ops_t FAKE = { fake_local, fake_ping, fake_gw };
static void net_reset(void) {
memset(&net, 0, sizeof(net));
snprintf(net.addr[0], INET6_ADDRSTRLEN, "192.0.2.5");
snprintf(net.addr[1], INET6_ADDRSTRLEN, "2001:db8::5");
snprintf(net.gw[0], INET6_ADDRSTRLEN, "192.0.2.1");
snprintf(net.gw[1], INET6_ADDRSTRLEN, "fe80::1");
net.rtt[0] = 10.0; net.rtt[1] = 20.0;
}
static void probe_reset(probe_t *p) {
memset(p, 0, sizeof(*p));
snprintf(p->target[0], sizeof(p->target[0]), "1.1.1.1");
snprintf(p->target[1], sizeof(p->target[1]), "2606:4700:4700::1111");
p->probe_interval = 60; p->retry_interval = 10; p->cheap_gateway = true;
p->fam[0].present = p->fam[1].present = PROBE_UNKNOWN;
p->fam[0].rtt = p->fam[1].rtt = -1;
pthread_mutex_init(&p->mu, NULL);
}
static void step_both(probe_t *p, double now) {
probe_step(p, &p->fam[0], 0, &FAKE, now);
probe_step(p, &p->fam[1], 1, &FAKE, now);
}
static void test_present_with_details(void) {
net_reset(); probe_t p; probe_reset(&p);
step_both(&p, 1000);
CHECK(p.fam[0].present == 1 && p.fam[1].present == 1);
CHECK_STR(p.fam[0].addr, "192.0.2.5");
CHECK_STR(p.fam[0].gateway, "192.0.2.1");
CHECK(p.fam[0].rtt == 10.0);
kvdict_t d; kv_clear(&d);
probe_flags(&p, &d);
CHECK_STR(kv_get(&d, "p4") ? kv_get(&d, "p4") : "(null)", "1");
CHECK_STR(kv_get(&d, "p6") ? kv_get(&d, "p6") : "(null)", "1");
char js[512];
CHECK(probe_metric_json(&p, js, sizeof(js)) > 0);
CHECK(strstr(js, "\"ipv6\":{\"present\":true,\"addr\":\"2001:db8::5\",\"gateway\":\"fe80::1\"") != NULL);
}
static void test_no_route_is_absent_and_wastes_no_ping(void) {
net_reset(); net.addr[1][0] = '\0'; probe_t p; probe_reset(&p);
step_both(&p, 1000);
CHECK(p.fam[1].present == 0 && net.npings[1] == 0 && net.npings[0] == 1);
kvdict_t d; kv_clear(&d);
probe_flags(&p, &d);
CHECK_STR(kv_get(&d, "p6") ? kv_get(&d, "p6") : "(null)", "0");
char js[512];
probe_metric_json(&p, js, sizeof(js));
CHECK(strstr(js, "\"ipv6\":{\"present\":false,\"addr\":null,\"gateway\":null,\"rtt\":null}") != NULL);
}
static void test_debounce_and_rate_limit(void) {
net_reset(); probe_t p; probe_reset(&p);
step_both(&p, 1000);
step_both(&p, 1000); /* cached: no second ping */
CHECK(net.npings[0] == 1 && net.npings[1] == 1);
net.rtt[1] = -1;
step_both(&p, 1060); /* first failure: still present */
CHECK(p.fam[1].present == 1);
step_both(&p, 1070); /* retry interval elapsed: flips */
CHECK(p.fam[1].present == 0);
net.rtt[1] = 25;
step_both(&p, 1080); /* first success recovers */
CHECK(p.fam[1].present == 1);
}
static void test_first_failure_is_unknown_and_omitted(void) {
net_reset(); net.rtt[0] = -1; probe_t p; probe_reset(&p);
step_both(&p, 1000);
CHECK(p.fam[0].present == PROBE_UNKNOWN);
kvdict_t d; kv_clear(&d);
probe_flags(&p, &d);
CHECK(kv_get(&d, "p4") == NULL && kv_get(&d, "p6") != NULL);
char js[512];
probe_metric_json(&p, js, sizeof(js));
CHECK(strstr(js, "\"ipv4\":{\"present\":null") != NULL);
}
static void test_address_or_gateway_change_repings(void) {
net_reset(); probe_t p; probe_reset(&p);
step_both(&p, 1000);
snprintf(net.addr[0], INET6_ADDRSTRLEN, "198.51.100.7");
step_both(&p, 1000);
CHECK(net.npings[0] == 2);
snprintf(net.gw[0], INET6_ADDRSTRLEN, "192.0.2.254");
step_both(&p, 1000);
CHECK(net.npings[0] == 3);
}
static void test_route_loss_then_return_pings_immediately(void) {
net_reset(); probe_t p; probe_reset(&p);
step_both(&p, 1000);
net.addr[1][0] = '\0';
step_both(&p, 1000);
CHECK(p.fam[1].present == 0);
snprintf(net.addr[1], INET6_ADDRSTRLEN, "2001:db8::5");
step_both(&p, 1000);
CHECK(net.npings[1] == 2 && p.fam[1].present == 1);
}
static void test_missing_ping_stays_unknown_even_after_route_loss(void) {
net_reset(); net.rtt[0] = -2; probe_t p; probe_reset(&p);
step_both(&p, 1000);
step_both(&p, 1600);
CHECK(p.fam[0].present == PROBE_UNKNOWN && net.npings[0] == 1);
net.addr[0][0] = '\0';
step_both(&p, 1700);
CHECK(p.fam[0].present == 0); /* no route is definitive */
snprintf(net.addr[0], INET6_ADDRSTRLEN, "192.0.2.5");
step_both(&p, 1800);
CHECK(p.fam[0].present == PROBE_UNKNOWN && net.npings[0] == 1);
}
static void test_unsupported_family_is_unknown(void) {
net_reset(); net.unsupported[1] = 1; probe_t p; probe_reset(&p);
step_both(&p, 1000);
CHECK(p.fam[1].present == PROBE_UNKNOWN && p.fam[0].present == 1);
}
static int live_mode(void) {
static const probe_ops_t sys_ops = { sys_local_address, sys_ping, sys_gateway };
probe_t p; probe_reset(&p);
p.cheap_gateway = false;
snprintf(p.target[0], sizeof(p.target[0]), "1.1.1.1");
for (int round = 0; round < 3; round++) {
double now = mono_ts();
for (int f = 0; f < 2; f++) probe_step(&p, &p.fam[f], f, &sys_ops, now);
sleep(round < 2 ? 11 : 0);
}
char js[512];
probe_metric_json(&p, js, sizeof(js));
kvdict_t d; kv_clear(&d);
probe_flags(&p, &d);
printf("%s\np4=%s p6=%s\n", js, kv_get(&d, "p4") ? kv_get(&d, "p4") : "-",
kv_get(&d, "p6") ? kv_get(&d, "p6") : "-");
return 0;
}
int main(int argc, char **argv) {
if (argc > 1 && !strcmp(argv[1], "--live")) return live_mode();
test_gateway_parsers();
test_ping_parser();
test_present_with_details();
test_no_route_is_absent_and_wastes_no_ping();
test_debounce_and_rate_limit();
test_first_failure_is_unknown_and_omitted();
test_address_or_gateway_change_repings();
test_route_loss_then_return_pings_immediately();
test_missing_ping_stays_unknown_even_after_route_loss();
test_unsupported_family_is_unknown();
printf("%d checks, %d failed\n", g_run, g_fail);
return g_fail ? 1 : 0;
}
+353 -19
View File
@@ -10,14 +10,20 @@ Built-in plugins (always available):
os_info — OS name, kernel, distro, Python version
ping_monitor — ICMP ping RTT to configured hosts
nagios_runner — run Nagios-compatible check scripts
network_monitor — per-interface TX/RX bytes/s (Linux); per-protocol connectivity (all platforms)
Linux-only plugins (via /proc, no extra tools needed):
cpu_monitor — CPU %, load average, core count
memory_monitor — RAM and swap usage
network_monitor — per-interface TX/RX bytes/s
Cross-platform (via df subprocess):
disk_monitor — per-mount usage and capacity
Presence probe (built in): reports per protocol (IPv4/IPv6) whether the host has internet
connectivity, as p4/p6 flags on every heartbeat, so hbd stops treating a missing IPv6 (or
IPv4) path as an outage. Config, in the network_monitor section of ~/.hbc.json:
probe_enabled (default true), probe_v4 (default 1.1.1.1), probe_v6 (default
2606:4700:4700::1111; IP literals only), probe_interval (default 60 s)
"""
import argparse
@@ -36,12 +42,13 @@ import time
import zlib
from abc import ABC, abstractmethod
from collections import defaultdict
from dataclasses import asdict, dataclass
from logging.handlers import SysLogHandler
from pathlib import Path
from typing import Any, Dict, List, Optional, Tuple
from typing import Any, Awaitable, Callable, Dict, List, Optional, Set, Tuple
# updated by scripts/bumpminor.sh
__version__ = "5.4.1"
__version__ = "5.4.5"
# ---------------------------------------------------------------------------
# Protocol (mirrors hbd/common/proto.py)
@@ -637,6 +644,315 @@ class DiskMonitorPlugin(MonitorPlugin):
return {"partitions": partitions} if partitions else {}
# ---------------------------------------------------------------------------
# Connectivity probe — per-protocol internet presence (IPv4 / IPv6)
#
# Tier 1 (every tier1_interval s, sends no packets): a UDP connect() to the probe address
# shows whether a route exists and which source address would be used. Tier 2 (every
# probe_interval s, sooner after a failure or a change of address or gateway): one ICMP echo,
# only when tier 1 found a route. A family is absent after no route at all or fail_limit
# consecutive failed echoes, present after one success, and unknown (flag omitted) when it
# cannot be probed. The result rides on every HTB as p4/p6 and, when network_monitor runs, as
# its `connectivity` metric. Mirrors hbd/client/connectivity.py.
# ---------------------------------------------------------------------------
_FAMILIES = ("ipv4", "ipv6")
_FLAG_KEYS = {"ipv4": "p4", "ipv6": "p6"}
_DEFAULT_PROBES = {"ipv4": "1.1.1.1", "ipv6": "2606:4700:4700::1111"}
_BSD_PLATFORMS = ("darwin", "freebsd", "netbsd", "openbsd", "dragonfly")
class _ProbeUnavailable(Exception):
"""The address family cannot be probed on this host (e.g. no IPv6 support)."""
class _PingUnavailable(Exception):
"""The system ping binary could not be executed."""
def _parse_proc_route_v4(text: str) -> Optional[str]:
"""Default gateway from /proc/net/route (lowest metric wins)."""
best: Optional[Tuple[int, str]] = None
for line in text.splitlines()[1:]:
f = line.split()
if len(f) < 8 or f[1] != "00000000" or not int(f[3], 16) & 0x2:
continue
metric = int(f[6])
if best is None or metric < best[0]:
# the kernel prints the raw network-order u32 in host order: use the host byte order
best = (metric, socket.inet_ntoa(int(f[2], 16).to_bytes(4, sys.byteorder)))
return best[1] if best else None
def _parse_proc_route_v6(text: str) -> Optional[str]:
"""Default gateway from /proc/net/ipv6_route (lowest metric wins)."""
best: Optional[Tuple[int, str]] = None
for line in text.splitlines():
f = line.split()
if len(f) < 10:
continue
dest, plen, nexthop, metric, flags, iface = f[0], f[1], f[4], f[5], f[8], f[9]
if dest != "0" * 32 or plen != "00" or nexthop == "0" * 32 or iface == "lo":
continue
fl = int(flags, 16)
if not fl & 0x2 or fl & 0x200: # need RTF_GATEWAY, skip RTF_REJECT
continue
m = int(metric, 16)
if best is None or m < best[0]:
best = (m, socket.inet_ntop(socket.AF_INET6, bytes.fromhex(nexthop)))
return best[1] if best else None
def _parse_route_get(text: str) -> Optional[str]:
"""Gateway from `route -n get [-inet6] default` (macOS and the BSDs)."""
m = re.search(r"^\s*gateway:\s*(\S+)", text, re.MULTILINE)
return m.group(1) if m else None
def _parse_windows_route(text: str, family: str) -> Optional[str]:
"""Default gateway from `route print -4` / `route print -6` (lowest metric wins)."""
best: Optional[Tuple[int, str]] = None
for line in text.splitlines():
if family == "ipv4":
f = line.split()
if not (len(f) >= 5 and f[0] == "0.0.0.0" and f[1] == "0.0.0.0" and f[2] != "On-link"):
continue
cand = (int(f[4]), f[2])
else:
m = re.match(r"^\s*\d+\s+(\d+)\s+::/0\s+(\S+)\s*$", line)
if not m or m.group(2) == "On-link":
continue
cand = (int(m.group(1)), m.group(2))
if best is None or cand[0] < best[0]:
best = cand
return best[1] if best else None
async def _run_cmd(cmd: List[str], timeout: float = 5.0) -> str:
proc = await asyncio.create_subprocess_exec(
*cmd, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.DEVNULL)
try:
out, _ = await asyncio.wait_for(proc.communicate(), timeout=timeout)
except asyncio.TimeoutError:
proc.kill()
await proc.wait()
raise
return out.decode(errors="replace")
async def _default_gateway(family: str) -> Optional[str]:
"""Default gateway for "ipv4"/"ipv6", or None on any failure (informational only)."""
v4 = family == "ipv4"
try:
if sys.platform.startswith("linux"):
with open("/proc/net/route" if v4 else "/proc/net/ipv6_route") as fh:
text = fh.read()
return _parse_proc_route_v4(text) if v4 else _parse_proc_route_v6(text)
if sys.platform.startswith(_BSD_PLATFORMS):
cmd = ["route", "-n", "get"] + ([] if v4 else ["-inet6"]) + ["default"]
return _parse_route_get(await _run_cmd(cmd))
if sys.platform == "win32":
text = await _run_cmd(["route", "print", "-4" if v4 else "-6"])
return _parse_windows_route(text, family)
except Exception:
return None
return None
def _probe_ping_command(host: str, family: str) -> List[str]:
"""Command for ONE echo request with a ~2 s reply wait."""
v4 = family == "ipv4"
if sys.platform == "win32":
return ["ping", "-n", "1", "-w", "2000", "-4" if v4 else "-6", host]
if sys.platform.startswith(_BSD_PLATFORMS):
# BSD/macOS -W is in milliseconds, and IPv6 is reached through ping6.
return ["ping", "-c", "1", "-W", "2000", host] if v4 else ["ping6", "-c", "1", host]
return ["ping", "-c", "1", "-W", "2", "-4" if v4 else "-6", host]
def _parse_ping_rtt(output: str) -> Optional[float]:
"""Average RTT in ms from ping output, or None when there was no reply."""
m = re.search(r"(?:rtt|round-trip)\s+min/avg/max/\S+\s*=\s*([\d.]+)/([\d.]+)/([\d.]+)", output)
if m:
return float(m.group(2))
if "TTL=" in output: # Windows reply line (also when localized)
m = re.search(r"[=<]\s*(\d+)\s*ms", output)
return float(m.group(1)) if m else 0.0
return None
async def _ping_rtt(target: str, family: str) -> Optional[float]:
"""One ICMP echo; RTT in ms, or None if no reply. Raises _PingUnavailable without ping."""
try:
proc = await asyncio.create_subprocess_exec(
*_probe_ping_command(target, family),
stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE)
except FileNotFoundError as e:
raise _PingUnavailable(str(e))
try:
out, _ = await asyncio.wait_for(proc.communicate(), timeout=6)
except asyncio.TimeoutError:
proc.kill()
await proc.wait()
return None
return _parse_ping_rtt(out.decode(errors="replace"))
def _local_address(family: str, target: str) -> Optional[str]:
"""Source address the kernel would use to reach target, or None if there is no route.
A UDP connect() sends nothing. Raises _ProbeUnavailable if the family is unsupported.
"""
af = socket.AF_INET if family == "ipv4" else socket.AF_INET6
try:
sock = socket.socket(af, socket.SOCK_DGRAM)
except OSError as e:
raise _ProbeUnavailable(str(e))
try:
sock.connect((target, 9))
return str(sock.getsockname()[0])
except OSError:
return None
finally:
sock.close()
@dataclass
class _FamilyState:
present: Optional[bool] = None # None = unknown (probe cannot run yet/at all)
addr: Optional[str] = None
gateway: Optional[str] = None
rtt: Optional[float] = None
class _Connectivity:
def __init__(
self,
probes: Optional[Dict[str, str]] = None,
probe_interval: float = 60.0,
retry_interval: float = 10.0,
tier1_interval: float = 5.0,
fail_limit: int = 2,
local_address_fn: Callable[[str, str], Optional[str]] = _local_address,
ping_fn: Callable[[str, str], Awaitable[Optional[float]]] = _ping_rtt,
gateway_fn: Callable[[str], Awaitable[Optional[str]]] = _default_gateway,
clock: Callable[[], float] = time.monotonic,
):
self.probes = dict(_DEFAULT_PROBES)
self.probes.update(probes or {})
self.probe_interval = probe_interval
self.retry_interval = retry_interval
self.tier1_interval = tier1_interval
self.fail_limit = fail_limit
self._local_address = local_address_fn
self._ping = ping_fn
self._gateway_fn = gateway_fn
self._clock = clock
# Reading the gateway is a cheap /proc read on Linux but a subprocess elsewhere, so
# elsewhere it is refreshed only when a tier-2 probe is due.
self._cheap_gateway = sys.platform.startswith("linux")
self.states: Dict[str, _FamilyState] = {f: _FamilyState() for f in _FAMILIES}
self._fails: Dict[str, int] = {f: 0 for f in _FAMILIES}
self._next_ping: Dict[str, float] = {f: 0.0 for f in _FAMILIES}
self._last_key: Dict[str, Optional[Tuple[Optional[str], Optional[str]]]] = {
f: None for f in _FAMILIES}
self._ping_disabled: Set[str] = set()
@classmethod
def from_config(cls, cfg: Optional[Dict[str, Any]]) -> "_Connectivity":
cfg = cfg or {}
probes = {}
if cfg.get("probe_v4"):
probes["ipv4"] = str(cfg["probe_v4"])
if cfg.get("probe_v6"):
probes["ipv6"] = str(cfg["probe_v6"])
return cls(probes=probes, probe_interval=float(cfg.get("probe_interval", 60)))
# ---- readers (cheap; safe to call on every heartbeat) ---------------------------
def snapshot(self) -> Dict[str, _FamilyState]:
return {f: _FamilyState(**asdict(s)) for f, s in self.states.items()}
def flags(self) -> Dict[str, bool]:
"""HTB flags for families whose presence is known."""
return {_FLAG_KEYS[f]: s.present for f, s in self.states.items() if s.present is not None}
def as_metric(self) -> Dict[str, Dict[str, Any]]:
return {f: asdict(s) for f, s in self.states.items()}
# ---- probing ----------------------------------------------------------------------
async def refresh(self) -> None:
for fam in _FAMILIES:
try:
await self._refresh_family(fam)
except Exception:
logging.getLogger("hbc.probe").exception("connectivity probe failed for %s", fam)
async def run(self) -> None:
while _running:
await self.refresh()
await _sleep(self.tier1_interval)
async def _gateway(self, fam: str) -> Optional[str]:
try:
return await self._gateway_fn(fam)
except Exception:
return None
async def _refresh_family(self, fam: str) -> None:
st = self.states[fam]
try:
addr = self._local_address(fam, self.probes[fam])
except _ProbeUnavailable:
st.present = None
return
if addr is None: # tier 1: no route => absent immediately
self._fails[fam] = 0
self._next_ping[fam] = 0.0
self._last_key[fam] = None
st.present, st.addr, st.gateway, st.rtt = False, None, None, None
return
now = self._clock()
gw = await self._gateway(fam) if self._cheap_gateway else st.gateway
key = (addr, gw)
due = key != self._last_key[fam] or now >= self._next_ping[fam]
if due and not self._cheap_gateway:
gw = await self._gateway(fam)
key = (addr, gw)
st.addr, st.gateway = addr, gw
self._last_key[fam] = key
if fam in self._ping_disabled:
st.present = None # cannot probe: unknown, never absent, while a route exists
elif due:
await self._tier2(fam, st, now)
async def _tier2(self, fam: str, st: _FamilyState, now: float) -> None:
try:
rtt = await self._ping(self.probes[fam], fam)
except _PingUnavailable:
logging.getLogger("hbc.probe").warning("ping unavailable; %s presence stays unknown", fam)
self._ping_disabled.add(fam)
st.present = None
return
except (asyncio.TimeoutError, OSError):
rtt = None
if rtt is None:
self._fails[fam] += 1
self._next_ping[fam] = now + self.retry_interval
if self._fails[fam] >= self.fail_limit:
st.present, st.rtt = False, None
else:
self._fails[fam] = 0
self._next_ping[fam] = now + self.probe_interval
st.present, st.rtt = True, rtt
# The running probe (None when probe_enabled is false); shared by the heartbeat sender and
# the network_monitor plugin. Set in _async_main.
_probe: Optional[_Connectivity] = None
# ---------------------------------------------------------------------------
# Plugin: network_monitor (Linux /proc/net/dev)
# ---------------------------------------------------------------------------
@@ -658,7 +974,7 @@ def _read_net_dev() -> Dict[str, Tuple[int, int]]:
class NetworkMonitorPlugin(MonitorPlugin):
name = "network_monitor"
description = "Network I/O rates via /proc/net/dev (Linux)"
description = "Network I/O rates via /proc/net/dev (Linux) and per-protocol internet connectivity"
interval = 300
def __init__(self, config: Optional[Dict[str, Any]] = None):
@@ -667,19 +983,25 @@ class NetworkMonitorPlugin(MonitorPlugin):
self.interval = cfg.get("interval", 300)
self.skip_ifaces: set = set(cfg.get("skip_interfaces", ["lo"]))
self._prev: Optional[Tuple[float, Dict[str, Tuple[int, int]]]] = None
self._have_dev = False
async def initialize(self) -> bool:
if platform.system() != "Linux":
self.skip_reason = "Linux only (/proc/net/dev not available)"
dev = _read_net_dev() if platform.system() == "Linux" else {}
self._have_dev = bool(dev)
if not self._have_dev and _probe is None:
self.skip_reason = "no /proc/net/dev (Linux only) and the connectivity probe is disabled"
return False
dev = _read_net_dev()
if not dev:
self.skip_reason = "/proc/net/dev not readable"
return False
self._prev = (time.time(), dev) # prime first delta
if self._have_dev:
self._prev = (time.time(), dev) # prime first delta
return True
async def _collect_metrics(self) -> Dict[str, Any]:
data: Dict[str, Any] = await self._collect_interfaces() if self._have_dev else {}
if _probe is not None:
data["connectivity"] = _probe.as_metric()
return data
async def _collect_interfaces(self) -> Dict[str, Any]:
now = time.time()
curr = _read_net_dev()
if not curr or self._prev is None:
@@ -940,15 +1262,21 @@ async def _handle_update(conn: AsyncConnection):
# Heartbeat sender
# ---------------------------------------------------------------------------
async def _heartbeat_sender(conn: AsyncConnection, interval: int):
def _build_heartbeat_msg(conn: AsyncConnection, interval: int,
probe: Optional[_Connectivity] = None) -> Dict[str, Any]:
"""HTB payload; carries p4/p6 presence flags when the probe knows them."""
msg: Dict[str, Any] = {"acks": conn.ackcount, "rtt": conn.rtts[-1], "interval": interval}
if probe is not None:
msg.update(probe.flags())
return msg
async def _heartbeat_sender(conn: AsyncConnection, interval: int,
probe: Optional[_Connectivity] = None):
log = logging.getLogger("hbc.hb")
while _running:
try:
await conn.sendto({
"acks": conn.ackcount,
"rtt": conn.rtts[-1],
"interval": interval,
})
await conn.sendto(_build_heartbeat_msg(conn, interval, probe))
except Exception as e:
log.error("send: %s", e)
await _sleep(interval)
@@ -1056,7 +1384,7 @@ def _reconfigure_syslog(level: int):
# ---------------------------------------------------------------------------
async def _async_main(args, cfg: Dict[str, Any]) -> int:
global _running, _shutdown_event, _active_tasks, send_shutdown
global _running, _shutdown_event, _active_tasks, send_shutdown, _probe
_running = True
_shutdown_event = asyncio.Event()
_active_tasks = []
@@ -1113,6 +1441,10 @@ async def _async_main(args, cfg: Dict[str, Any]) -> int:
c.close()
return 0
# Per-protocol internet presence probe (shared with the network_monitor plugin)
net_cfg = dict((cfg.get("plugins") or {}).get("network_monitor") or cfg.get("network_monitor") or {})
_probe = _Connectivity.from_config(net_cfg) if net_cfg.get("probe_enabled", True) else None
plugins = await _load_plugins(cfg)
loop = asyncio.get_event_loop()
@@ -1126,8 +1458,10 @@ async def _async_main(args, cfg: Dict[str, Any]) -> int:
loop.add_signal_handler(signal.SIGHUP, _sighup)
if _probe is not None:
_active_tasks.append(asyncio.create_task(_probe.run()))
for conn in connections:
_active_tasks.append(asyncio.create_task(_heartbeat_sender(conn, interval)))
_active_tasks.append(asyncio.create_task(_heartbeat_sender(conn, interval, _probe)))
if plugins and connections:
_active_tasks.append(asyncio.create_task(_plugin_collector(connections[0], plugins)))
+28
View File
@@ -0,0 +1,28 @@
from hbd.client.main import build_heartbeat_msg
class _Conn:
ackcount = 3
rtts = [0, 12.5]
class _Conn2:
def __init__(self, flags):
self._flags = flags
def flags(self):
return self._flags
def test_heartbeat_without_connectivity_is_unchanged():
assert build_heartbeat_msg(_Conn(), 60) == {"acks": 3, "rtt": 12.5, "interval": 60}
def test_heartbeat_carries_presence_flags():
msg = build_heartbeat_msg(_Conn(), 60, _Conn2({"p4": True, "p6": False}))
assert msg == {"acks": 3, "rtt": 12.5, "interval": 60, "p4": True, "p6": False}
def test_heartbeat_omits_unknown_flags():
msg = build_heartbeat_msg(_Conn(), 60, _Conn2({}))
assert "p4" not in msg and "p6" not in msg
+173
View File
@@ -0,0 +1,173 @@
import asyncio
from hbd.client.connectivity import Connectivity, ProbeUnavailable
from hbd.client.pingutil import PingUnavailable
class Net:
"""Scriptable network: addr None = no route, 'unavailable' = family unsupported."""
def __init__(self):
self.addr = {"ipv4": "192.0.2.5", "ipv6": "2001:db8::5"}
self.gw = {"ipv4": "192.0.2.1", "ipv6": "fe80::1"}
self.rtt = {"ipv4": 10.0, "ipv6": 20.0} # None = ping fails, 'unavailable' = no ping binary
self.pings = []
self.now = 1000.0
def local(self, fam, target):
a = self.addr[fam]
if a == "unavailable":
raise ProbeUnavailable("no such family")
return a
async def ping(self, target, fam):
self.pings.append(fam)
r = self.rtt[fam]
if r == "unavailable":
raise PingUnavailable("no ping")
return r
async def gateway(self, fam):
return self.gw[fam]
def clock(self):
return self.now
def make(net, **kw):
c = Connectivity(local_address_fn=net.local, ping_fn=net.ping, gateway_fn=net.gateway,
clock=net.clock, **kw)
c._cheap_gateway = True
return c
def refresh(c):
asyncio.run(c.refresh())
def test_present_with_addr_gateway_rtt():
net = Net()
c = make(net)
refresh(c)
s = c.snapshot()["ipv4"]
assert (s.present, s.addr, s.gateway, s.rtt) == (True, "192.0.2.5", "192.0.2.1", 10.0)
assert c.flags() == {"p4": True, "p6": True}
assert c.as_metric()["ipv6"]["gateway"] == "fe80::1"
def test_no_route_is_immediately_absent():
net = Net()
net.addr["ipv6"] = None
c = make(net)
refresh(c)
assert c.flags() == {"p4": True, "p6": False}
assert net.pings == ["ipv4"] # no ping wasted on a family without a route
assert c.snapshot()["ipv6"].addr is None
def test_ping_is_debounced_and_rate_limited():
net = Net()
c = make(net, probe_interval=60.0, retry_interval=10.0)
refresh(c)
refresh(c) # same instant: cached, no second ping
assert net.pings == ["ipv4", "ipv6"]
net.now += 60
net.rtt["ipv6"] = None # first failure: still present (debounce)
refresh(c)
assert c.flags()["p6"] is True
net.now += 10 # retry_interval elapsed: second failure flips it
refresh(c)
assert c.flags()["p6"] is False
def test_first_probe_failing_once_is_unknown():
net = Net()
net.rtt["ipv4"] = None
c = make(net)
refresh(c)
assert "p4" not in c.flags() and c.flags()["p6"] is True
def test_recovers_on_first_success():
net = Net()
net.rtt["ipv6"] = None
c = make(net, retry_interval=10.0)
refresh(c)
net.now += 10
refresh(c)
assert c.flags()["p6"] is False
net.rtt["ipv6"] = 25.0
net.now += 10
refresh(c)
assert c.flags()["p6"] is True
def test_address_change_invalidates_cache():
net = Net()
c = make(net)
refresh(c)
net.addr["ipv4"] = "198.51.100.7" # moved networks, no time has passed
refresh(c)
assert net.pings.count("ipv4") == 2
assert c.snapshot()["ipv4"].addr == "198.51.100.7"
def test_gateway_change_invalidates_cache():
net = Net()
c = make(net)
refresh(c)
net.gw["ipv4"] = "192.0.2.254"
refresh(c)
assert net.pings.count("ipv4") == 2
def test_route_loss_then_return_pings_immediately():
net = Net()
c = make(net)
refresh(c)
net.addr["ipv6"] = None
refresh(c)
net.addr["ipv6"] = "2001:db8::5"
refresh(c)
assert net.pings.count("ipv6") == 2 and c.flags()["p6"] is True
def test_missing_ping_binary_leaves_presence_unknown_and_stops_retrying():
net = Net()
net.rtt["ipv4"] = "unavailable"
c = make(net)
refresh(c)
net.now += 600
refresh(c)
assert "p4" not in c.flags()
assert net.pings.count("ipv4") == 1
def test_unsupported_family_is_unknown():
net = Net()
net.addr["ipv6"] = "unavailable"
c = make(net)
refresh(c)
assert "p6" not in c.flags() and c.flags()["p4"] is True
def test_from_config():
c = Connectivity.from_config({"probe_v4": "9.9.9.9", "probe_interval": 30})
assert c.probes["ipv4"] == "9.9.9.9"
assert c.probes["ipv6"] == "2606:4700:4700::1111"
assert c.probe_interval == 30.0
def test_ping_less_family_stays_unknown_after_route_returns():
net = Net()
net.rtt["ipv4"] = "unavailable"
c = make(net)
refresh(c)
assert "p4" not in c.flags()
net.addr["ipv4"] = None
refresh(c)
assert c.flags()["p4"] is False # no route is definitive
net.addr["ipv4"] = "192.0.2.5"
refresh(c)
assert "p4" not in c.flags()
assert net.pings.count("ipv4") == 1 # missing binary is not retried
+75
View File
@@ -0,0 +1,75 @@
import asyncio
import struct
from hbd.client import gateway
PROC_V4 = (
"Iface\tDestination\tGateway \tFlags\tRefCnt\tUse\tMetric\tMask\t\tMTU\tWindow\tIRTT\n"
"wlan0\t00000000\t0100A8C0\t0003\t0\t0\t600\t00000000\t0\t0\t0\n"
"eth0\t00000000\t0101A8C0\t0003\t0\t0\t100\t00000000\t0\t0\t0\n"
"wlan0\t0000A8C0\t00000000\t0001\t0\t0\t600\t00FFFFFF\t0\t0\t0\n"
)
PROC_V6 = (
"00000000000000000000000000000000 00 00000000000000000000000000000000 00 "
"fe800000000000000000000000000001 00000400 00000001 00000000 00000003 wlan0\n"
"00000000000000000000000000000000 00 00000000000000000000000000000000 00 "
"00000000000000000000000000000000 ffffffff 00000001 00000000 00200200 lo\n"
"fe800000000000000000000000000000 40 00000000000000000000000000000000 00 "
"00000000000000000000000000000000 00000100 00000001 00000000 00000001 wlan0\n"
)
MACOS = """ route to: default
destination: default
mask: default
gateway: 192.168.1.1
interface: en0
"""
WIN_V4 = """IPv4 Route Table
Active Routes:
Network Destination Netmask Gateway Interface Metric
0.0.0.0 0.0.0.0 192.168.1.1 192.168.1.50 25
0.0.0.0 0.0.0.0 10.0.0.1 10.0.0.9 50
127.0.0.0 255.0.0.0 On-link 127.0.0.1 331
"""
WIN_V6 = """IPv6 Route Table
Active Routes:
If Metric Network Destination Gateway
12 281 ::/0 fe80::1
1 331 ::1/128 On-link
"""
def test_proc_v4_picks_lowest_metric_gateway():
assert gateway.parse_proc_route_v4(PROC_V4) == "192.168.1.1" # eth0, metric 100
def test_proc_v4_none_without_default_route():
assert gateway.parse_proc_route_v4(PROC_V4.splitlines()[0] + "\n") is None
def test_proc_v6_skips_reject_and_non_default():
assert gateway.parse_proc_route_v6(PROC_V6) == "fe80::1"
def test_macos():
assert gateway.parse_macos_route(MACOS) == "192.168.1.1"
assert gateway.parse_macos_route("route: writing to routing socket: not in table") is None
def test_windows_v4_lowest_metric():
assert gateway.parse_windows_route(WIN_V4, "ipv4") == "192.168.1.1"
def test_windows_v6():
assert gateway.parse_windows_route(WIN_V6, "ipv6") == "fe80::1"
def test_default_gateway_never_raises_on_parser_error(monkeypatch):
"""default_gateway catches all exceptions; never raises even if parser fails."""
monkeypatch.setattr("sys.platform", "linux")
monkeypatch.setattr(gateway, "parse_proc_route_v4", lambda _: (_ for _ in ()).throw(struct.error))
result = asyncio.run(gateway.default_gateway("ipv4"))
assert result is None
+381
View File
@@ -0,0 +1,381 @@
"""Connectivity probe in the standalone scripts/hbc_mini.py (stdlib-only single-file client)."""
import asyncio
import importlib.util
import pathlib
import sys
import pytest
_PATH = pathlib.Path(__file__).resolve().parents[1] / "scripts" / "hbc_mini.py"
_SPEC = importlib.util.spec_from_file_location("hbc_mini", _PATH)
mini = importlib.util.module_from_spec(_SPEC)
sys.modules["hbc_mini"] = mini # dataclasses resolve their module by name
_SPEC.loader.exec_module(mini)
PROC_V4 = (
"Iface\tDestination\tGateway \tFlags\tRefCnt\tUse\tMetric\tMask\t\tMTU\tWindow\tIRTT\n"
"wlan0\t00000000\t0100A8C0\t0003\t0\t0\t600\t00000000\t0\t0\t0\n"
"eth0\t00000000\t0101A8C0\t0003\t0\t0\t100\t00000000\t0\t0\t0\n"
"wlan0\t0000A8C0\t00000000\t0001\t0\t0\t600\t00FFFFFF\t0\t0\t0\n"
)
PROC_V6 = (
"00000000000000000000000000000000 00 00000000000000000000000000000000 00 "
"fe800000000000000000000000000001 00000400 00000001 00000000 00000003 wlan0\n"
"00000000000000000000000000000000 00 00000000000000000000000000000000 00 "
"00000000000000000000000000000000 ffffffff 00000001 00000000 00200200 lo\n"
"fe800000000000000000000000000000 40 00000000000000000000000000000000 00 "
"00000000000000000000000000000000 00000100 00000001 00000000 00000001 wlan0\n"
)
ROUTE_GET = """ route to: 0.0.0.0
destination: 0.0.0.0
mask: 0.0.0.0
gateway: 192.0.2.1
fib: 0
interface: vtnet0
"""
WIN_V4 = """IPv4 Route Table
Active Routes:
Network Destination Netmask Gateway Interface Metric
0.0.0.0 0.0.0.0 192.168.1.1 192.168.1.50 25
0.0.0.0 0.0.0.0 10.0.0.1 10.0.0.9 50
127.0.0.0 255.0.0.0 On-link 127.0.0.1 331
"""
WIN_V6 = """IPv6 Route Table
Active Routes:
If Metric Network Destination Gateway
12 281 ::/0 fe80::1
1 331 ::1/128 On-link
"""
LINUX_PING = """PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
64 bytes from 1.1.1.1: icmp_seq=1 ttl=57 time=11.2 ms
--- 1.1.1.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 11.234/11.234/11.234/0.000 ms
"""
FREEBSD_PING = """PING 1.1.1.1 (1.1.1.1): 56 data bytes
64 bytes from 1.1.1.1: icmp_seq=0 ttl=59 time=4.006 ms
--- 1.1.1.1 ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss, 1 packets out of wait time
round-trip min/avg/max/stddev = 4.006/4.006/4.006/0.000 ms
"""
WIN_PING = """Pinging 1.1.1.1 with 32 bytes of data:
Reply from 1.1.1.1: bytes=32 time=12ms TTL=57
Ping statistics for 1.1.1.1:
Packets: Sent = 1, Received = 1, Lost = 0 (0% loss),
"""
LOST_PING = """--- 1.1.1.1 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms
"""
# ---- gateway parsers ---------------------------------------------------------------
def test_proc_v4_lowest_metric_wins():
assert mini._parse_proc_route_v4(PROC_V4) == "192.168.1.1" # eth0, metric 100
def test_proc_v4_none_without_default_route():
assert mini._parse_proc_route_v4(PROC_V4.splitlines()[0] + "\n") is None
def test_proc_v6_skips_reject_and_non_default():
assert mini._parse_proc_route_v6(PROC_V6) == "fe80::1"
def test_route_get_bsd_and_macos():
assert mini._parse_route_get(ROUTE_GET) == "192.0.2.1"
assert mini._parse_route_get("route: writing to routing socket: not in table") is None
def test_windows_routes():
assert mini._parse_windows_route(WIN_V4, "ipv4") == "192.168.1.1"
assert mini._parse_windows_route(WIN_V6, "ipv6") == "fe80::1"
def test_default_gateway_never_raises(monkeypatch):
monkeypatch.setattr(sys, "platform", "linux")
monkeypatch.setattr(mini, "_parse_proc_route_v4", lambda t: 1 / 0)
assert asyncio.run(mini._default_gateway("ipv4")) is None
# ---- ping ---------------------------------------------------------------------------
@pytest.mark.parametrize("platform,family,expected", [
("linux", "ipv4", ["ping", "-c", "1", "-W", "2", "-4", "h"]),
("linux", "ipv6", ["ping", "-c", "1", "-W", "2", "-6", "h"]),
("freebsd15", "ipv4", ["ping", "-c", "1", "-W", "2000", "h"]),
("freebsd15", "ipv6", ["ping6", "-c", "1", "h"]),
("darwin", "ipv4", ["ping", "-c", "1", "-W", "2000", "h"]),
("darwin", "ipv6", ["ping6", "-c", "1", "h"]),
("win32", "ipv4", ["ping", "-n", "1", "-w", "2000", "-4", "h"]),
("win32", "ipv6", ["ping", "-n", "1", "-w", "2000", "-6", "h"]),
])
def test_probe_ping_command(monkeypatch, platform, family, expected):
monkeypatch.setattr(sys, "platform", platform)
assert mini._probe_ping_command("h", family) == expected
def test_parse_ping_rtt():
assert mini._parse_ping_rtt(LINUX_PING) == 11.234
assert mini._parse_ping_rtt(FREEBSD_PING) == 4.006
assert mini._parse_ping_rtt(WIN_PING) == 12.0
assert mini._parse_ping_rtt(LOST_PING) is None
assert mini._parse_ping_rtt("ping: connect: Network is unreachable") is None
# ---- probe state machine ------------------------------------------------------------
class Net:
"""Scriptable network: addr None = no route, 'unavailable' = family unsupported."""
def __init__(self):
self.addr = {"ipv4": "192.0.2.5", "ipv6": "2001:db8::5"}
self.gw = {"ipv4": "192.0.2.1", "ipv6": "fe80::1"}
self.rtt = {"ipv4": 10.0, "ipv6": 20.0} # None = ping fails, 'unavailable' = no ping binary
self.pings = []
self.now = 1000.0
def local(self, fam, target):
a = self.addr[fam]
if a == "unavailable":
raise mini._ProbeUnavailable("no such family")
return a
async def ping(self, target, fam):
self.pings.append(fam)
r = self.rtt[fam]
if r == "unavailable":
raise mini._PingUnavailable("no ping")
return r
async def gateway(self, fam):
return self.gw[fam]
def clock(self):
return self.now
def make(net, **kw):
c = mini._Connectivity(local_address_fn=net.local, ping_fn=net.ping, gateway_fn=net.gateway,
clock=net.clock, **kw)
c._cheap_gateway = True
return c
def refresh(c):
asyncio.run(c.refresh())
def test_present_with_addr_gateway_rtt():
net = Net()
c = make(net)
refresh(c)
s = c.snapshot()["ipv4"]
assert (s.present, s.addr, s.gateway, s.rtt) == (True, "192.0.2.5", "192.0.2.1", 10.0)
assert c.flags() == {"p4": True, "p6": True}
assert c.as_metric()["ipv6"]["gateway"] == "fe80::1"
def test_no_route_is_immediately_absent():
net = Net()
net.addr["ipv6"] = None
c = make(net)
refresh(c)
assert c.flags() == {"p4": True, "p6": False}
assert net.pings == ["ipv4"]
assert c.snapshot()["ipv6"].addr is None
def test_ping_is_debounced_and_rate_limited():
net = Net()
c = make(net, probe_interval=60.0, retry_interval=10.0)
refresh(c)
refresh(c)
assert net.pings == ["ipv4", "ipv6"]
net.now += 60
net.rtt["ipv6"] = None
refresh(c)
assert c.flags()["p6"] is True
net.now += 10
refresh(c)
assert c.flags()["p6"] is False
def test_first_probe_failing_once_is_unknown():
net = Net()
net.rtt["ipv4"] = None
c = make(net)
refresh(c)
assert "p4" not in c.flags() and c.flags()["p6"] is True
def test_recovers_on_first_success():
net = Net()
net.rtt["ipv6"] = None
c = make(net, retry_interval=10.0)
refresh(c)
net.now += 10
refresh(c)
assert c.flags()["p6"] is False
net.rtt["ipv6"] = 25.0
net.now += 10
refresh(c)
assert c.flags()["p6"] is True
def test_address_change_invalidates_cache():
net = Net()
c = make(net)
refresh(c)
net.addr["ipv4"] = "198.51.100.7"
refresh(c)
assert net.pings.count("ipv4") == 2
assert c.snapshot()["ipv4"].addr == "198.51.100.7"
def test_gateway_change_invalidates_cache():
net = Net()
c = make(net)
refresh(c)
net.gw["ipv4"] = "192.0.2.254"
refresh(c)
assert net.pings.count("ipv4") == 2
def test_route_loss_then_return_pings_immediately():
net = Net()
c = make(net)
refresh(c)
net.addr["ipv6"] = None
refresh(c)
net.addr["ipv6"] = "2001:db8::5"
refresh(c)
assert net.pings.count("ipv6") == 2 and c.flags()["p6"] is True
def test_missing_ping_binary_leaves_presence_unknown_and_stops_retrying():
net = Net()
net.rtt["ipv4"] = "unavailable"
c = make(net)
refresh(c)
net.now += 600
refresh(c)
assert "p4" not in c.flags()
assert net.pings.count("ipv4") == 1
def test_ping_less_family_stays_unknown_after_route_returns():
net = Net()
net.rtt["ipv4"] = "unavailable"
c = make(net)
refresh(c)
assert "p4" not in c.flags()
net.addr["ipv4"] = None
refresh(c)
assert c.flags()["p4"] is False # no route is definitive
net.addr["ipv4"] = "192.0.2.5"
refresh(c)
assert "p4" not in c.flags() and net.pings.count("ipv4") == 1
def test_unsupported_family_is_unknown():
net = Net()
net.addr["ipv6"] = "unavailable"
c = make(net)
refresh(c)
assert "p6" not in c.flags() and c.flags()["p4"] is True
def test_from_config():
c = mini._Connectivity.from_config({"probe_v4": "9.9.9.9", "probe_interval": 30})
assert c.probes["ipv4"] == "9.9.9.9"
assert c.probes["ipv6"] == "2606:4700:4700::1111"
assert c.probe_interval == 30.0
def test_local_address_on_this_host_never_raises():
# Real UDP connect(): either an address string, None (no route) or _ProbeUnavailable.
for fam, target in (("ipv4", "1.1.1.1"), ("ipv6", "2606:4700:4700::1111")):
try:
r = mini._local_address(fam, target)
except mini._ProbeUnavailable:
continue
assert r is None or isinstance(r, str)
# ---- heartbeat message and plugin wiring ---------------------------------------------
class _Conn:
ackcount = 3
rtts = [0.0, 12.5]
class _FlagsProbe:
def __init__(self, flags):
self._flags = flags
def flags(self):
return self._flags
def test_heartbeat_without_probe_is_unchanged():
assert mini._build_heartbeat_msg(_Conn(), 60) == {"acks": 3, "rtt": 12.5, "interval": 60}
def test_heartbeat_carries_presence_flags():
msg = mini._build_heartbeat_msg(_Conn(), 60, _FlagsProbe({"p4": True, "p6": False}))
assert msg == {"acks": 3, "rtt": 12.5, "interval": 60, "p4": True, "p6": False}
def test_heartbeat_omits_unknown_flags():
msg = mini._build_heartbeat_msg(_Conn(), 60, _FlagsProbe({}))
assert "p4" not in msg and "p6" not in msg
def test_flags_survive_the_wire_encoding():
msg = mini._build_heartbeat_msg(_Conn(), 60, _FlagsProbe({"p4": True, "p6": False}))
out = mini._stodict(mini._dicttos("HTB", dict(msg, name="h", id=1)))
assert out["p4"] == 1 and out["p6"] == 0
class _FakeProbe:
def as_metric(self):
return {"ipv4": {"present": True, "addr": "192.0.2.5", "gateway": "192.0.2.1", "rtt": 9.0},
"ipv6": {"present": False, "addr": None, "gateway": None, "rtt": None}}
def test_plugin_reports_connectivity_alongside_interfaces(monkeypatch):
monkeypatch.setattr(mini, "_probe", _FakeProbe())
monkeypatch.setattr(mini.platform, "system", lambda: "Linux")
plugin = mini.NetworkMonitorPlugin({})
assert asyncio.run(plugin.initialize()) is True
data = asyncio.run(plugin._collect_metrics())
assert data["connectivity"]["ipv4"]["present"] is True
assert data["connectivity"]["ipv6"]["present"] is False
def test_plugin_probe_only_mode_on_non_linux(monkeypatch):
monkeypatch.setattr(mini, "_probe", _FakeProbe())
monkeypatch.setattr(mini.platform, "system", lambda: "FreeBSD")
plugin = mini.NetworkMonitorPlugin({})
assert asyncio.run(plugin.initialize()) is True
assert list(asyncio.run(plugin._collect_metrics())) == ["connectivity"]
def test_plugin_still_skips_on_non_linux_without_probe(monkeypatch):
monkeypatch.setattr(mini, "_probe", None)
monkeypatch.setattr(mini.platform, "system", lambda: "FreeBSD")
plugin = mini.NetworkMonitorPlugin({})
assert asyncio.run(plugin.initialize()) is False
assert plugin.skip_reason
def test_plugin_omits_connectivity_without_probe(monkeypatch):
monkeypatch.setattr(mini, "_probe", None)
monkeypatch.setattr(mini.platform, "system", lambda: "Linux")
plugin = mini.NetworkMonitorPlugin({})
assert asyncio.run(plugin.initialize()) is True
assert "connectivity" not in asyncio.run(plugin._collect_metrics())
+120
View File
@@ -0,0 +1,120 @@
import time
import pytest
from hbd.server import hbdclass
from hbd.server.hbdclass import Connection as C
def _host(name):
hbdclass.Host.hosts.pop(name, None)
return hbdclass.Host(name)
def _conn(host, afam, state):
addr = "192.0.2.1" if afam == "IPv4" else "2001:db8::1"
conn, _ = host.conndata(0, addr, None, 1000.0)
conn.state = state
return conn
CASES = [
# v4 state, v6 state, presence flags, expected status
(C.UP, C.UP, {}, "up"),
(C.UP, C.OVERDUE, {}, "simplex"), # both present (legacy) => simplex
(C.UP, C.OVERDUE, {"IPv6": True}, "simplex"),
(C.UP, C.OVERDUE, {"IPv6": False}, "up"), # v6 not present => not monitored
(C.OVERDUE, C.OVERDUE, {}, "overdue"),
(C.OVERDUE, C.ABSENT, {"IPv6": False}, "overdue"), # only present family is overdue
(C.UP, C.ABSENT, {"IPv6": False}, "up"),
(C.DOWN, C.UP, {}, "down"),
(C.UNKNOWN, C.UNKNOWN, {}, "unknown"),
(C.OVERDUE, C.UNKNOWN, {}, "overdue"),
(C.UP, C.UNKNOWN, {}, "up"),
(C.OVERDUE, C.UP, {"IPv6": False}, "simplex"), # HTB arriving => present despite flag
]
@pytest.mark.parametrize("s4,s6,flags,expected", CASES)
def test_status_matrix(s4, s6, flags, expected):
h = _host("status-matrix")
_conn(h, "IPv4", s4)
_conn(h, "IPv6", s6)
for afam, present in flags.items():
h.set_presence(afam, present, time.time())
assert h.status() == expected
def test_status_single_family_host():
h = _host("status-v4only")
_conn(h, "IPv4", C.UP)
assert h.status() == "up"
h.connections["IPv4"].state = C.OVERDUE
assert h.status() == "overdue"
def test_status_no_connections_is_unknown():
assert _host("status-empty").status() == "unknown"
def test_family_present_rules():
h = _host("fp")
v6 = _conn(h, "IPv6", C.OVERDUE)
assert h.family_present("IPv6") is True # never reported => present
h.set_presence("IPv6", False, 1.0)
assert h.family_present("IPv6") is False
v6.state = C.UP # real traffic overrides the flag
assert h.family_present("IPv6") is True
assert h.flag_present("IPv6") is False
def test_stateinfo_and_statedict_expose_status():
h = _host("status-exposed")
_conn(h, "IPv4", C.UP)
assert h.stateinfo()["status"] == "up"
assert h.statedict()["status"] == "up"
def test_absent_connection_statedict_is_neutral():
h = _host("absent-dict")
conn = _conn(h, "IPv6", C.ABSENT)
conn.rtts.append(42.0)
d = conn.statedict()
assert d["state"] == "absent" and d["rtt"] == "" and d["rttstate"] == "absent"
def test_presence_survives_pickle_and_old_pickles_get_default():
import pickle
h = _host("pickled")
h.set_presence("IPv6", False, 5.0)
h2 = pickle.loads(pickle.dumps(h))
assert h2.presence == {"IPv6": {"present": False, "ts": 5.0}}
del h2.__dict__["presence"] # simulate a pre-feature pickle
h2.fixup()
assert h2.presence == {}
def _all_absent(name, s4, s6=None):
h = _host(name)
_conn(h, "IPv4", s4)
h.set_presence("IPv4", False, time.time())
if s6 is not None:
_conn(h, "IPv6", s6)
h.set_presence("IPv6", False, time.time())
return h
def test_all_absent_single_family_overdue_is_overdue():
assert _all_absent("abs-single", C.OVERDUE).status() == "overdue"
def test_all_absent_dual_stack_one_absent_one_overdue_is_overdue():
assert _all_absent("abs-dual-od", C.ABSENT, C.OVERDUE).status() == "overdue"
def test_all_absent_dual_stack_v6_up_is_up():
assert _all_absent("abs-dual-up", C.ABSENT, C.UP).status() == "up"
def test_all_absent_all_unknown_is_unknown():
assert _all_absent("abs-unknown", C.UNKNOWN, C.UNKNOWN).status() == "unknown"
@@ -0,0 +1,26 @@
import asyncio
from hbd.client import connectivity
from hbd.client.plugins.network_monitor import NetworkMonitorPlugin
class _FakeShared:
def as_metric(self):
return {"ipv4": {"present": True, "addr": "192.0.2.5", "gateway": "192.0.2.1", "rtt": 9.0},
"ipv6": {"present": False, "addr": None, "gateway": None, "rtt": None}}
def test_collect_includes_connectivity_when_probe_running():
connectivity.set_shared(_FakeShared())
try:
data = asyncio.run(NetworkMonitorPlugin()._collect_metrics())
finally:
connectivity.set_shared(None)
assert data["connectivity"]["ipv4"]["present"] is True
assert data["connectivity"]["ipv6"]["present"] is False
def test_collect_omits_connectivity_without_probe():
connectivity.set_shared(None)
data = asyncio.run(NetworkMonitorPlugin()._collect_metrics())
assert "connectivity" not in data
+60
View File
@@ -0,0 +1,60 @@
import sys
import pytest
from hbd.client import pingutil
LINUX_OK = """PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
64 bytes from 1.1.1.1: icmp_seq=1 ttl=57 time=11.2 ms
--- 1.1.1.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 11.234/11.234/11.234/0.000 ms
"""
MACOS_OK = """PING 1.1.1.1 (1.1.1.1): 56 data bytes
64 bytes from 1.1.1.1: icmp_seq=0 ttl=57 time=12.3 ms
--- 1.1.1.1 ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 12.3/12.3/12.3/0.000 ms
"""
LOST = """--- 1.1.1.1 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms
"""
def test_parse_linux():
r = pingutil.parse_ping_output(LINUX_OK)
assert r == {"rtt_min": 11.234, "rtt_avg": 11.234, "rtt_max": 11.234, "loss": 0.0}
def test_parse_macos():
r = pingutil.parse_ping_output(MACOS_OK)
assert r["rtt_avg"] == 12.3 and r["loss"] == 0.0
def test_parse_lost():
r = pingutil.parse_ping_output(LOST)
assert r["rtt_avg"] == float("inf") and r["loss"] == 100.0
def test_parse_garbage_is_unreachable():
r = pingutil.parse_ping_output("ping: connect: Network is unreachable")
assert r["rtt_avg"] == float("inf") and r["loss"] == 100.0
@pytest.mark.parametrize("platform,family,expected", [
("linux", None, ["ping", "-c", "3", "-W", "5", "h"]),
("linux", 6, ["ping", "-c", "3", "-W", "5", "-6", "h"]),
("linux", 4, ["ping", "-c", "3", "-W", "5", "-4", "h"]),
("win32", None, ["ping", "-n", "3", "-w", "5000", "h"]),
("win32", 6, ["ping", "-n", "3", "-w", "5000", "-6", "h"]),
("darwin", None, ["ping", "-c", "3", "-W", "5", "h"]),
("darwin", 4, ["ping", "-c", "3", "-W", "5000", "h"]),
("darwin", 6, ["ping6", "-c", "3", "h"]),
])
def test_ping_command(monkeypatch, platform, family, expected):
monkeypatch.setattr(sys, "platform", platform)
assert pingutil.ping_command("h", 3, 5, family) == expected
+250
View File
@@ -0,0 +1,250 @@
"""Host-level overdue alerting driven by client-reported protocol presence."""
import asyncio
import time
from hbd.common.proto import dicttos
from hbd.server import hbdclass, udp
from hbd.server.hbdclass import Connection as C
from hbd.server.udp import handle_datagram, parse_message
class _FakeTransport:
def sendto(self, data, addr):
pass
def _ctx():
return {"config": {}, "hbdclass": hbdclass, "msg_to_websockets": None,
"DEBUG": 0, "verbose": False}
def _fresh(name):
hbdclass.Host.hosts.pop(name, None)
def _send(name, addr, ctx, **fields):
msg = parse_message(dicttos("HTB", {"name": name, "id": 0, **fields}))
handle_datagram(msg, (addr, 50000), _FakeTransport(), ctx)
def _events(monkeypatch):
events = []
monkeypatch.setattr(udp, "eventlog", lambda u, level, msg, **kw: events.append((level, msg)))
return events
def _dual_stack(name, ctx):
_fresh(name)
_send(name, "127.0.0.1", ctx)
_send(name, "::1", ctx)
return hbdclass.Host.hosts[name]
def test_flags_are_stored_on_host():
ctx = _ctx()
_fresh("pres-store")
_send("pres-store", "127.0.0.1", ctx, p4=1, p6=0)
h = hbdclass.Host.hosts["pres-store"]
assert h.presence["IPv4"]["present"] is True
assert h.presence["IPv6"]["present"] is False
def test_no_flags_leaves_presence_unknown():
ctx = _ctx()
_fresh("pres-legacy")
_send("pres-legacy", "127.0.0.1", ctx)
assert hbdclass.Host.hosts["pres-legacy"].presence == {}
def test_absent_flag_moves_overdue_family_to_absent_and_clears_alert(monkeypatch):
_events(monkeypatch)
ctx = _ctx()
h = _dual_stack("pres-absent", ctx)
v6 = h.connections["IPv6"]
v6.state = C.OVERDUE
udp._set_connectivity_alert(h, "IPv6", "CRITICAL")
_send("pres-absent", "127.0.0.1", ctx, p4=1, p6=0)
assert v6.state == C.ABSENT
assert "connectivity.IPv6" not in h.alert_states
assert h.status() == "up"
def test_absent_flag_does_not_touch_family_that_is_up():
ctx = _ctx()
h = _dual_stack("pres-up-stays", ctx)
_send("pres-up-stays", "127.0.0.1", ctx, p4=1, p6=0)
assert h.connections["IPv6"].state == C.UP # traffic overrides the flag
def test_present_again_arms_grace_timer():
ctx = _ctx()
async def scenario():
_fresh("pres-back")
_send("pres-back", "127.0.0.1", ctx, interval=10)
_send("pres-back", "::1", ctx, interval=10)
h = hbdclass.Host.hosts["pres-back"]
v6 = h.connections["IPv6"]
v6.cancel_overdue_timer()
v6.state = C.ABSENT
_send("pres-back", "127.0.0.1", ctx, interval=10, p6=1)
assert v6.state == C.UNKNOWN and v6.overdue_timer is not None
for c in h.connections.values():
c.cancel_overdue_timer()
asyncio.run(scenario())
def test_one_family_overdue_is_simplex_all_overdue_is_critical(monkeypatch):
events = _events(monkeypatch)
ctx = _ctx()
h = _dual_stack("pres-simplex", ctx)
on_overdue, _ = udp._make_timer_callbacks("pres-simplex", h, ctx)
asyncio.run(on_overdue(h.connections["IPv6"]))
assert h.status() == "simplex"
assert ("INFO", "IPv6 overdue (simplex)") in events
assert not any(level == "CRITICAL" for level, _ in events)
assert "connectivity.IPv6" not in h.alert_states
asyncio.run(on_overdue(h.connections["IPv4"]))
assert h.status() == "overdue"
assert [e for e in events if e[0] == "CRITICAL"] == [("CRITICAL", "IPv4, IPv6 overdue")]
assert {"connectivity.IPv4", "connectivity.IPv6"} <= set(h.alert_states)
def test_timer_expiry_on_absent_family_is_silent(monkeypatch):
events = _events(monkeypatch)
ctx = _ctx()
h = _dual_stack("pres-quiet", ctx)
_send("pres-quiet", "127.0.0.1", ctx, p4=1, p6=0) # v6 still UP => left alone
events.clear()
on_overdue, _ = udp._make_timer_callbacks("pres-quiet", h, ctx)
asyncio.run(on_overdue(h.connections["IPv6"]))
assert h.connections["IPv6"].state == C.ABSENT
assert events == []
assert h.status() == "up"
def test_recovery_from_simplex_is_info_only(monkeypatch):
events = _events(monkeypatch)
ctx = _ctx()
h = _dual_stack("pres-rec-simplex", ctx)
v6 = h.connections["IPv6"]
v6.state, v6.statetime = C.OVERDUE, time.time() - 100
events.clear()
_send("pres-rec-simplex", "::1", ctx)
assert v6.state == C.UP
assert [e[0] for e in events] == ["INFO"] and "back after" in events[0][1]
def test_recovery_from_overdue_is_recover_and_clears_all_alerts(monkeypatch):
events = _events(monkeypatch)
ctx = _ctx()
h = _dual_stack("pres-rec-overdue", ctx)
for afam in ("IPv4", "IPv6"):
c = h.connections[afam]
c.state, c.statetime = C.OVERDUE, time.time() - 100
udp._set_connectivity_alert(h, afam, "CRITICAL")
events.clear()
_send("pres-rec-overdue", "127.0.0.1", ctx)
assert [e[0] for e in events] == ["RECOVER"]
assert h.status() == "simplex"
assert "connectivity.IPv6" not in h.alert_states # left overdue => all cleared
assert "connectivity.IPv4" not in h.alert_states
def test_restore_skips_absent_and_sets_no_alert_for_simplex():
ctx = _ctx()
async def scenario():
h = _dual_stack("pres-restore", ctx)
h.connections["IPv6"].state = C.OVERDUE # v4 UP + v6 OVERDUE => simplex
udp.restore_connection_timers(hbdclass, ctx)
assert "connectivity.IPv6" not in h.alert_states
h.connections["IPv6"].cancel_overdue_timer()
h.connections["IPv6"].state = C.ABSENT
h.set_presence("IPv6", False, time.time())
udp.restore_connection_timers(hbdclass, ctx)
assert h.connections["IPv6"].state == C.ABSENT
for c in h.connections.values():
c.cancel_overdue_timer()
asyncio.run(scenario())
def _past_drop(conn):
conn.state, conn.statetime = C.OVERDUE, time.time() - (udp.DROPOVERDUE + 10)
def test_restore_past_drop_window_keeps_alert_single_family():
ctx = _ctx()
async def scenario():
_fresh("pres-drop-single")
_send("pres-drop-single", "127.0.0.1", ctx)
h = hbdclass.Host.hosts["pres-drop-single"]
_past_drop(h.connections["IPv4"])
udp.restore_connection_timers(hbdclass, ctx)
assert h.connections["IPv4"].state == C.UNKNOWN
assert "connectivity.IPv4" in h.alert_states
asyncio.run(scenario())
def test_restore_past_drop_window_keeps_alerts_dual_family():
ctx = _ctx()
async def scenario():
h = _dual_stack("pres-drop-dual", ctx)
for c in h.connections.values():
_past_drop(c)
udp.restore_connection_timers(hbdclass, ctx)
assert {"connectivity.IPv4", "connectivity.IPv6"} <= set(h.alert_states)
asyncio.run(scenario())
def test_v4_only_host_overdue_is_single_critical(monkeypatch):
events = _events(monkeypatch)
ctx = _ctx()
_fresh("pres-v4only")
_send("pres-v4only", "127.0.0.1", ctx)
h = hbdclass.Host.hosts["pres-v4only"]
on_overdue, _ = udp._make_timer_callbacks("pres-v4only", h, ctx)
asyncio.run(on_overdue(h.connections["IPv4"]))
assert [e for e in events if e[0] == "CRITICAL"] == [("CRITICAL", "IPv4 overdue")]
assert "connectivity.IPv4" in h.alert_states
asyncio.run(on_overdue(h.connections["IPv4"]))
assert len([e for e in events if e[0] == "CRITICAL"]) == 1
def test_all_families_absent_v4_only_host_still_alerts_when_it_dies(monkeypatch):
events = _events(monkeypatch)
ctx = _ctx()
_fresh("pres-allabsent")
_send("pres-allabsent", "127.0.0.1", ctx, p4=0)
h = hbdclass.Host.hosts["pres-allabsent"]
on_overdue, _ = udp._make_timer_callbacks("pres-allabsent", h, ctx)
asyncio.run(on_overdue(h.connections["IPv4"]))
assert [e for e in events if e[0] == "CRITICAL"] == [("CRITICAL", "IPv4 overdue")]
assert h.connections["IPv4"].state == C.OVERDUE
assert h.status() == "overdue"
assert "connectivity.IPv4" in h.alert_states
def test_last_present_family_is_not_parked_when_other_is_absent(monkeypatch):
events = _events(monkeypatch)
ctx = _ctx()
h = _dual_stack("pres-lastfam", ctx)
_send("pres-lastfam", "127.0.0.1", ctx, p4=0, p6=0)
events.clear()
on_overdue, _ = udp._make_timer_callbacks("pres-lastfam", h, ctx)
asyncio.run(on_overdue(h.connections["IPv4"]))
assert h.connections["IPv4"].state == C.ABSENT
assert not any(level == "CRITICAL" for level, _ in events)
assert h.status() == "up"
asyncio.run(on_overdue(h.connections["IPv6"]))
assert [e for e in events if e[0] == "CRITICAL"] == [("CRITICAL", "IPv6 overdue")]
assert h.connections["IPv6"].state == C.OVERDUE
assert h.status() == "overdue"
+36 -6
View File
@@ -14,10 +14,12 @@ class _FakeTransport:
self.sent.append((data, addr))
def _htb(name, rtt=None, interval=0):
def _htb(name, rtt=None, interval=0, boot=0):
d = {"name": name, "interval": interval, "id": 0}
if rtt is not None:
d["rtt"] = rtt
if boot:
d["boot"] = boot
return parse_message(dicttos("HTB", d))
@@ -94,10 +96,11 @@ def test_request_update_fires_on_recovery_even_with_rtt_history():
assert ack.get("request_update")
def test_recovery_clears_real_plugin_data_but_preserves_rtt_history():
"""Regression for Finding 2: host.plugin_data.clear() on recovery must
wipe real client-collected plugin data while leaving rtt_* history
samples intact.
def test_ordinary_recovery_preserves_real_plugin_data_and_rtt_history():
"""An ordinary reconnect (no boot flag) — e.g. OVERDUE->UP after a
transient network blip — must NOT wipe already-collected real plugin
data (e.g. cpu_monitor, os_info) or rtt_* history. Only an actual
client reboot invalidates that data (see the boot-flag test below).
"""
hbdclass.Host.hosts.pop("rtt-hist-host5", None)
transport = _FakeTransport()
@@ -116,8 +119,35 @@ def test_recovery_clears_real_plugin_data_but_preserves_rtt_history():
host.add_plugin_data("os_info", {"os": "linux"}, timestamp=time.time())
assert "os_info" in host.plugin_data
# Recovery heartbeat.
# Ordinary recovery heartbeat — no boot flag.
handle_datagram(_htb("rtt-hist-host5", rtt=13.0), ("127.0.0.1", 50000), transport, ctx)
assert "os_info" in host.plugin_data
assert len(host.plugin_data["rtt_ipv4"]) == 4
def test_boot_recovery_clears_real_plugin_data_but_preserves_rtt_history():
"""A recovery heartbeat carrying the boot flag (client process actually
restarted) must still wipe stale real plugin data, while rtt_* history
(still a valid measurement, unaffected by a client reboot) survives.
"""
hbdclass.Host.hosts.pop("rtt-hist-host6", None)
transport = _FakeTransport()
ctx = _base_ctx()
for rtt in (10.0, 11.0, 12.0):
handle_datagram(_htb("rtt-hist-host6", rtt=rtt), ("127.0.0.1", 50000), transport, ctx)
host = hbdclass.Host.hosts["rtt-hist-host6"]
assert len(host.plugin_data["rtt_ipv4"]) == 3
conn = host.connections["IPv4"]
conn.state = hbdclass.Connection.DOWN
host.add_plugin_data("os_info", {"os": "linux"}, timestamp=time.time())
assert "os_info" in host.plugin_data
# Recovery heartbeat with boot=1 — client process actually restarted.
handle_datagram(_htb("rtt-hist-host6", rtt=13.0, boot=1), ("127.0.0.1", 50000), transport, ctx)
assert "os_info" not in host.plugin_data
assert len(host.plugin_data["rtt_ipv4"]) == 4