Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
e8d9c4a958 | ||
|
|
328f248c43 | ||
|
|
747a8c27c9 | ||
|
|
5243f0a834 | ||
|
|
3c6f7fc868 | ||
|
|
adbd2f9b46 | ||
|
|
248b17edca | ||
|
|
3639464ba6 | ||
|
|
997279dce8 | ||
|
|
00d6ba607e | ||
|
|
cd5de7f205 | ||
|
|
95355374f8 | ||
|
|
8122c8d7dc | ||
|
|
3970f4cdde | ||
|
|
c3736f47ee | ||
|
|
2f1cad9156 | ||
|
|
1898bc52e5 | ||
|
|
c61a571995 | ||
|
|
9e8d68e712 | ||
|
|
ad19d2596a | ||
|
|
82ba69d827 | ||
|
|
9519169b47 | ||
|
|
f838920f7c | ||
|
|
45d11a66b5 | ||
|
|
a42d783c0c | ||
|
|
18e33656c0 | ||
|
|
4ac5a34a43 | ||
|
|
6d39c5be47 | ||
|
|
cdb2183989 | ||
|
|
41c78e8e34 | ||
|
|
acf1eb8804 | ||
|
|
49b631f02d | ||
|
|
8204737ea6 | ||
|
|
5b2fc8fd46 | ||
|
|
70c3e93684 | ||
|
|
52968201fa | ||
|
|
a7b698e6b9 | ||
|
|
136b10239f | ||
|
|
1e46f33648 | ||
|
|
4c3a63e645 |
@@ -16,3 +16,4 @@ uv.lock
|
||||
.superpowers/
|
||||
rndc-key
|
||||
docs/superpowers/
|
||||
graphify-out/
|
||||
|
||||
@@ -2,6 +2,70 @@
|
||||
|
||||
All notable changes to this project are documented here, organized by release.
|
||||
|
||||
## [5.4.5]
|
||||
|
||||
### Added
|
||||
- `hbc_mini.py` and `hbc_mini.c` (the dependency-free clients) get the same per-protocol connectivity probe as `hbc`: `p4`/`p6` flags on every heartbeat and a `connectivity` metric (presence, own address, gateway, RTT) via `network_monitor`, with gateway lookup on Linux, FreeBSD/NetBSD/DragonFly (both clients) and macOS/Windows (`hbc_mini.py` only)
|
||||
- `scripts/c/test_probe.c` and a `make test` target — the C client's first automated test suite
|
||||
|
||||
### Fixed
|
||||
- `hbc_mini.c`: the clean-shutdown heartbeat could be delayed or lost behind an in-flight connectivity probe on SIGTERM; it is now sent before the probe thread is joined
|
||||
- `hbc_mini.c`: a connectivity probe that fails to start, or is configured with a non-IP-literal target, now logs a warning instead of failing silently
|
||||
- `hbc_mini.py`: removed an unused import
|
||||
|
||||
---
|
||||
|
||||
## [5.4.4]
|
||||
|
||||
### Added
|
||||
- client reports per-protocol (IPv4/IPv6) internet presence: a connectivity probe (route check every ~5s, ICMP echo every ~60s, debounced) sends `p4`/`p6` flags on every heartbeat and a `connectivity` metric (presence, own address, gateway, RTT) via `network_monitor`
|
||||
- server monitors only the protocols a client reports as present; a protocol reported absent is shown as `absent` and raises no alert
|
||||
- host status (`up`, `simplex`, `overdue`, `down`, `unknown`) computed on the server: a host is overdue only when every present protocol is overdue; `simplex` (both present, one silent) has the same severity as `up` and is logged without notification
|
||||
- live and plugins pages show the host status, `simplex`/`absent` protocols and a Connectivity table
|
||||
- `hbc` config keys under `network_monitor`: `probe_enabled`, `probe_v4`, `probe_v6`, `probe_interval`
|
||||
|
||||
### Changed
|
||||
- overdue and recovery alerts are raised per host, not per protocol: a single CRITICAL when the last present protocol goes overdue; a protocol coming back while another stayed up is logged as INFO instead of a RECOVER notification (also applies to clients that do not report presence)
|
||||
|
||||
### Fixed
|
||||
- never park the last present family as absent, so a host that reports no protocol as present still alerts when it dies
|
||||
- keep the connectivity alert for a host overdue past the drop window across an `hbd` restart
|
||||
- `default_gateway` never raises; a protocol whose probe cannot run stays unknown rather than absent
|
||||
|
||||
---
|
||||
|
||||
## [5.4.3]
|
||||
|
||||
### Fixed
|
||||
- fix RTT y-axis label clipping, add x-axis tickmarks, enlarge charts
|
||||
|
||||
---
|
||||
|
||||
## [5.4.2]
|
||||
|
||||
### Fixed
|
||||
- preserve chart history across reconnects, show gaps for missing data
|
||||
|
||||
---
|
||||
|
||||
## [5.4.1]
|
||||
|
||||
### Added
|
||||
- live-refresh Connectivity section and RTT charts every 30s
|
||||
- render RTT history chart per connection in Connectivity section
|
||||
- record RTT history per heartbeat for charting
|
||||
|
||||
### Fixed
|
||||
- correct empty-state copy on Host Overview page
|
||||
- stop synthetic rtt_* keys from masking real plugin-data checks
|
||||
- use the latest RTT sample, not the oldest, in host info API
|
||||
- hide synthetic rtt history keys from plugin accordion list
|
||||
- remove extra blank line to pass flake8 E303 check
|
||||
- clear flap state when a host is dropped
|
||||
- correct plugin import path in docs, wire up orphaned footer template
|
||||
|
||||
---
|
||||
|
||||
## [5.4.0]
|
||||
|
||||
### Added
|
||||
|
||||
@@ -1,8 +1,97 @@
|
||||
# CLAUDE.md
|
||||
|
||||
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
||||
|
||||
## Working principles
|
||||
|
||||
1. Don't assume. Don't hide confusion. Surface tradeoffs.
|
||||
2. Minimum code that solves the problem. Nothing speculative.
|
||||
3. Touch only what you must. Clean up only your own mess.
|
||||
4. Define success criteria. Loop until verified.
|
||||
|
||||
This project's hbd daemon is deployed and in production on host w02
|
||||
under user 'nagios', which you can reach via ssh nagios@w02.
|
||||
The hbc client is used on 30 hosts in multiple networks.
|
||||
## Commands
|
||||
|
||||
```bash
|
||||
# Run tests
|
||||
pytest -q # all tests
|
||||
pytest tests/test_threshold.py # single file
|
||||
pytest -k test_name # single test
|
||||
|
||||
# Lint and type check
|
||||
tox -e lint # flake8 over hbd/ and tests/
|
||||
tox -e mypy # mypy over hbd/
|
||||
|
||||
# Install for development
|
||||
pip install -e ".[all,dev]"
|
||||
|
||||
# Run server
|
||||
python -m hbd.server.cli serve -c .hb.yaml -f -v
|
||||
|
||||
# Generate a password hash
|
||||
hbd passwd <username>
|
||||
```
|
||||
|
||||
Tests live in `tests/` (pytest, imported as a package). A second directory `test/` contains only TLS fixtures (`.pem` files), not test code.
|
||||
|
||||
Line length: 111 characters (`black`, `flake8`, both configured).
|
||||
|
||||
## Architecture
|
||||
|
||||
The system has two components: `hbc` (client) and `hbd` (server), both in the `hbd` Python package.
|
||||
|
||||
```
|
||||
hbd/
|
||||
common/ # proto.py (encode/decode), utils.py
|
||||
client/ # hbc: plugins/, main.py, config.py, plugin.py
|
||||
server/ # hbd: all server modules
|
||||
```
|
||||
|
||||
### Server module map
|
||||
|
||||
| Module | Role |
|
||||
|---|---|
|
||||
| `cli.py` | Argument parsing, entry point, daemon startup |
|
||||
| `main.py` | Asyncio runtime: starts UDP, HTTP, WebSocket; pickle save/load |
|
||||
| `hbdclass.py` | `Host` and `Connection` domain objects; DNS queue |
|
||||
| `udp.py` | UDP datagram listener; processes `HTB` and `PLG` messages; sets/resets overdue timers |
|
||||
| `http.py` | aiohttp web server; all REST API routes (`/api/0/`) and page routes |
|
||||
| `ws.py` | WebSocket broadcast to connected dashboard clients |
|
||||
| `notify.py` | Notification dispatch (Pushover, email, Mattermost, Matrix, Signal, SMS); `eventlog()` is the single choke-point for all alert events |
|
||||
| `threshold.py` | Threshold evaluation against plugin metrics; state transitions (OK/WARNING/CRITICAL/UNKNOWN) |
|
||||
| `monitor.py` | Timer cleanup on shutdown (reachability is event-driven via timers in `udp.py`, not polled) |
|
||||
| `data.py` | Shared in-memory message ring buffer |
|
||||
| `journal.py` | JSONL message journal with size-based rotation |
|
||||
| `users.py` | Session management, password hashing (PBKDF2), role checks |
|
||||
| `config.py` | Config loading and defaults |
|
||||
| `configio.py` | Config file read/write via `ruamel.yaml` (preserves comments) |
|
||||
| `settings.py` | Settings sections for the web UI settings page |
|
||||
| `dns.py` | `nsupdate` integration for dynamic DNS |
|
||||
| `oauth.py` | OAuth2 login (Gitea) |
|
||||
|
||||
### Key data flows
|
||||
|
||||
**Heartbeat received:** `udp.py` decodes the `HTB` datagram → updates `Connection` state in `hbdclass.py` → resets overdue asyncio timer → broadcasts via `ws.py` → `notify.py` fires connectivity alerts on state change.
|
||||
|
||||
**Plugin data received:** `udp.py` decodes `PLG` datagram → stores on `Host` → `threshold.py` evaluates against configured thresholds → `notify.py` fires threshold alerts on state transitions.
|
||||
|
||||
**State persistence:** `Host.hosts` dict + `data.msgs` ring + active sessions are pickled every 5 minutes and on clean shutdown. Asyncio timers are stripped before pickling (`Connection.__getstate__`).
|
||||
|
||||
**Config reload:** SIGHUP → `configio.py` re-reads YAML → live-updates hosts, thresholds, users, notification channels. Port/cert/pickle/journal changes require a full restart.
|
||||
|
||||
### Client plugin system
|
||||
|
||||
Plugins in `hbd/client/plugins/` subclass `InfoPlugin` (collected once, on demand) or `MonitorPlugin` (periodic). `initialize()` returns `False` to self-disable. Data is sent as `PLG` UDP messages.
|
||||
|
||||
`hbc_mini.py` (scripts/) and `hbc_mini.c` (scripts/c/) are standalone single-file clients with no external dependencies.
|
||||
|
||||
### Protocol
|
||||
|
||||
All UDP messages: `!<ID>: <zlib-compressed key=value payload>`. Encoding in `hbd/common/proto.py`. Lists/dicts encoded as JSON with `@` prefix; booleans as `1`/`0`.
|
||||
|
||||
### Web UI
|
||||
|
||||
Jinja2 templates in `hbd/server/templates/`. Static assets in `hbd/server/static/`. Live pages (`/live`, `/plugins`) use WebSocket connections for real-time push.
|
||||
|
||||
### CI
|
||||
|
||||
Gitea Actions workflow at `.gitea/workflows/release.yml`.
|
||||
|
||||
@@ -20,7 +20,7 @@ A lightweight UDP-based host monitoring system. Monitored hosts run a client (`h
|
||||
└────────────────────┘ └────────────────────────────┘
|
||||
```
|
||||
|
||||
**Package:** `hbd` v5.4.0
|
||||
**Package:** `hbd` v5.4.5
|
||||
**Python:** 3.11+
|
||||
|
||||
### Subpackages
|
||||
@@ -659,7 +659,28 @@ Config: `~/.hbc.json` (JSON format, same keys as `~/.hbc.yaml`).
|
||||
| `cpu_monitor` | Linux (`/proc/stat`; no per-core, no frequency) |
|
||||
| `memory_monitor` | Linux (`/proc/meminfo`) |
|
||||
| `disk_monitor` | Linux, macOS, BSD (`df -P`) |
|
||||
| `network_monitor` | Linux (`/proc/net/dev`) |
|
||||
| `network_monitor` | Linux (`/proc/net/dev`); per-protocol connectivity probe (all platforms) |
|
||||
|
||||
`network_monitor` also reports per-protocol (IPv4/IPv6) internet presence — whether the host
|
||||
has a working route and reply to a well-known address, plus its own address, the local
|
||||
gateway, and RTT — as `p4`/`p6` flags on every heartbeat and a `connectivity` object in the
|
||||
plugin data. Configurable, defaults shown:
|
||||
|
||||
```json
|
||||
{
|
||||
"plugins": {
|
||||
"network_monitor": {
|
||||
"probe_enabled": true,
|
||||
"probe_v4": "1.1.1.1",
|
||||
"probe_v6": "2606:4700:4700::1111",
|
||||
"probe_interval": 60
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
`probe_v4`/`probe_v6` must be IP literals (no hostnames). Gateway lookup works on Linux,
|
||||
FreeBSD/NetBSD/DragonFly/macOS (`route -n get`), and Windows (`route print`).
|
||||
|
||||
Not available vs full `hbc`: no YAML config, no `filesystem_info`, no `zfs_monitor`, no IPv6 early-fail protection.
|
||||
|
||||
@@ -712,6 +733,7 @@ SIGHUP causes the process to restart itself via `execv()`. SIGTERM/SIGINT trigge
|
||||
| `disk_monitor` | All | `df -P` subprocess |
|
||||
| `network_monitor` | Linux | `/proc/net/dev` |
|
||||
| `network_monitor` | FreeBSD, NetBSD, DragonFly | `getifaddrs()` + `AF_LINK` |
|
||||
| `network_monitor` (connectivity probe) | Linux, FreeBSD, NetBSD, DragonFly | UDP `connect()` route check + one ICMP echo per protocol |
|
||||
| `ping_monitor` | All | `ping` subprocess |
|
||||
| `nagios_runner` | All | `popen()` subprocess |
|
||||
|
||||
@@ -743,12 +765,33 @@ SIGHUP causes the process to restart itself via `execv()`. SIGTERM/SIGINT trigge
|
||||
}
|
||||
```
|
||||
|
||||
`network_monitor` also runs the per-protocol connectivity probe described above, in a
|
||||
dedicated thread (its ICMP echo can block for a few seconds). Same config keys and defaults
|
||||
as `hbc_mini.py`:
|
||||
|
||||
```json
|
||||
{
|
||||
"plugins": {
|
||||
"network_monitor": {
|
||||
"probe_enabled": true,
|
||||
"probe_v4": "1.1.1.1",
|
||||
"probe_v6": "2606:4700:4700::1111",
|
||||
"probe_interval": 60
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Gateway lookup on the C client covers Linux and FreeBSD/NetBSD/DragonFly only (no macOS or
|
||||
Windows build of `hbc_mini.c` exists).
|
||||
|
||||
### Differences from `hbc_mini.py`
|
||||
|
||||
- No `filesystem_info` or `zfs_monitor` plugins
|
||||
- `UPD` (self-update) messages are logged but not acted on
|
||||
- No IPv6 early-fail protection
|
||||
- Config is JSON only (`~/.hbc.json`), no YAML
|
||||
- No macOS or Windows build (Linux and the BSDs only)
|
||||
|
||||
---
|
||||
|
||||
|
||||
+10
-10
@@ -21,7 +21,7 @@ Heartbeat's plugin system is designed to be simple yet powerful. Plugins are Pyt
|
||||
### Key Concepts
|
||||
|
||||
- **Plugin Registry**: Central registry that manages all loaded plugins
|
||||
- **Plugin Loader**: Automatically discovers and loads plugins from the `hbd/plugins/` directory
|
||||
- **Plugin Loader**: Automatically discovers and loads plugins from the `hbd/client/plugins/` directory
|
||||
- **Plugin Types**: InfoPlugin (static data) and MonitorPlugin (periodic metrics)
|
||||
- **Async/Await**: All plugin methods are async for non-blocking operation
|
||||
|
||||
@@ -64,7 +64,7 @@ Decide whether your plugin collects static information (InfoPlugin) or dynamic m
|
||||
|
||||
### Step 2: Create Plugin File
|
||||
|
||||
Create a new Python file in `hbd/plugins/` directory:
|
||||
Create a new Python file in `hbd/client/plugins/` directory:
|
||||
|
||||
```python
|
||||
"""
|
||||
@@ -82,7 +82,7 @@ try:
|
||||
except ImportError:
|
||||
psutil = None
|
||||
|
||||
from hbd.plugin import MonitorPlugin # or InfoPlugin
|
||||
from hbd.client.plugin import MonitorPlugin # or InfoPlugin
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -193,7 +193,7 @@ from pathlib import Path
|
||||
# Add parent directory to path
|
||||
sys.path.insert(0, str(Path(__file__).parent))
|
||||
|
||||
from hbd.plugins.my_awesome_plugin import MyAwesomePlugin
|
||||
from hbd.client.plugins.my_awesome_plugin import MyAwesomePlugin
|
||||
|
||||
async def test():
|
||||
# Create plugin instance
|
||||
@@ -224,7 +224,7 @@ Understanding the plugin lifecycle helps you implement plugins correctly:
|
||||
|
||||
```
|
||||
1. Plugin Discovery
|
||||
└─> Loader scans hbd/plugins/ directory
|
||||
└─> Loader scans hbd/client/plugins/ directory
|
||||
└─> Finds Python files (except those starting with _)
|
||||
└─> Imports modules
|
||||
|
||||
@@ -378,7 +378,7 @@ Document your plugin thoroughly:
|
||||
### Example 1: Simple InfoPlugin
|
||||
|
||||
```python
|
||||
from hbd.plugin import InfoPlugin
|
||||
from hbd.client.plugin import InfoPlugin
|
||||
import platform
|
||||
|
||||
class SimpleInfoPlugin(InfoPlugin):
|
||||
@@ -406,7 +406,7 @@ plugin = SimpleInfoPlugin
|
||||
### Example 2: MonitorPlugin with State
|
||||
|
||||
```python
|
||||
from hbd.plugin import MonitorPlugin
|
||||
from hbd.client.plugin import MonitorPlugin
|
||||
import time
|
||||
|
||||
class CounterPlugin(MonitorPlugin):
|
||||
@@ -442,7 +442,7 @@ plugin = CounterPlugin
|
||||
### Example 3: Plugin with External Command
|
||||
|
||||
```python
|
||||
from hbd.plugin import MonitorPlugin
|
||||
from hbd.client.plugin import MonitorPlugin
|
||||
import asyncio
|
||||
|
||||
class CommandPlugin(MonitorPlugin):
|
||||
@@ -561,7 +561,7 @@ python -m hbd.hbc -c test_config.yaml --verbose
|
||||
|
||||
## Further Reading
|
||||
|
||||
- [Plugin Framework Source](../hbd/plugin.py) - Core plugin implementation
|
||||
- [Built-in Plugins](../hbd/plugins/) - Examples of working plugins
|
||||
- [Plugin Framework Source](../hbd/client/plugin.py) - Core plugin implementation
|
||||
- [Built-in Plugins](../hbd/client/plugins/) - Examples of working plugins
|
||||
- [Nagios Integration](NAGIOS_INTEGRATION.md) - Running external plugins
|
||||
- [Configuration Guide](../hbd/config_example.yaml) - Full configuration reference
|
||||
|
||||
+1
-1
@@ -14,4 +14,4 @@ Install options:
|
||||
"""
|
||||
|
||||
__all__ = ["__version__"]
|
||||
__version__ = "5.4.0"
|
||||
__version__ = "5.4.5"
|
||||
|
||||
@@ -0,0 +1,204 @@
|
||||
"""Per-protocol internet presence probe.
|
||||
|
||||
Tier 1 (every tier1_interval seconds, sends no packets): a UDP connect() to the probe
|
||||
address reveals whether a route exists and which source address would be used.
|
||||
Tier 2 (every probe_interval seconds, sooner after a failure or a change of address or
|
||||
gateway): one ICMP echo to the probe address. Absence needs fail_limit consecutive
|
||||
failures (or no route at all); presence needs one success.
|
||||
|
||||
The result is read by the heartbeat sender (p4/p6 flags on every HTB) and by the
|
||||
network_monitor plugin (connectivity metric) via the shared instance.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import socket
|
||||
import sys
|
||||
import time
|
||||
from dataclasses import asdict, dataclass, replace
|
||||
from typing import Any, Awaitable, Callable, Dict, Optional, Set, Tuple
|
||||
|
||||
from hbd.client import gateway as gateway_mod
|
||||
from hbd.client.pingutil import PingUnavailable, run_ping
|
||||
|
||||
logger = logging.getLogger("hbc.connectivity")
|
||||
|
||||
FAMILIES = ("ipv4", "ipv6")
|
||||
FLAG_KEYS = {"ipv4": "p4", "ipv6": "p6"}
|
||||
DEFAULT_PROBES = {"ipv4": "1.1.1.1", "ipv6": "2606:4700:4700::1111"}
|
||||
|
||||
|
||||
class ProbeUnavailable(Exception):
|
||||
"""The address family cannot be probed on this host (e.g. no IPv6 support)."""
|
||||
|
||||
|
||||
@dataclass
|
||||
class FamilyState:
|
||||
present: Optional[bool] = None # None = unknown (probe cannot run yet/at all)
|
||||
addr: Optional[str] = None
|
||||
gateway: Optional[str] = None
|
||||
rtt: Optional[float] = None
|
||||
|
||||
|
||||
def local_address(family: str, target: str) -> Optional[str]:
|
||||
"""Source address the kernel would use to reach target, or None if there is no route.
|
||||
|
||||
A UDP connect() sends nothing. Raises ProbeUnavailable if the family is unsupported.
|
||||
"""
|
||||
af = socket.AF_INET if family == "ipv4" else socket.AF_INET6
|
||||
try:
|
||||
sock = socket.socket(af, socket.SOCK_DGRAM)
|
||||
except OSError as e:
|
||||
raise ProbeUnavailable(str(e)) from e
|
||||
try:
|
||||
sock.connect((target, 9))
|
||||
return str(sock.getsockname()[0])
|
||||
except OSError:
|
||||
return None
|
||||
finally:
|
||||
sock.close()
|
||||
|
||||
|
||||
async def ping_rtt(target: str, family: str) -> Optional[float]:
|
||||
"""One ICMP echo; returns RTT in ms or None if no reply."""
|
||||
result = await run_ping(target, 1, 2, 4 if family == "ipv4" else 6)
|
||||
rtt = result["rtt_avg"]
|
||||
return None if rtt == float("inf") else rtt
|
||||
|
||||
|
||||
class Connectivity:
|
||||
def __init__(
|
||||
self,
|
||||
probes: Optional[Dict[str, str]] = None,
|
||||
probe_interval: float = 60.0,
|
||||
retry_interval: float = 10.0,
|
||||
tier1_interval: float = 5.0,
|
||||
fail_limit: int = 2,
|
||||
local_address_fn: Callable[[str, str], Optional[str]] = local_address,
|
||||
ping_fn: Callable[[str, str], Awaitable[Optional[float]]] = ping_rtt,
|
||||
gateway_fn: Callable[[str], Awaitable[Optional[str]]] = gateway_mod.default_gateway,
|
||||
clock: Callable[[], float] = time.monotonic,
|
||||
):
|
||||
self.probes = {**DEFAULT_PROBES, **(probes or {})}
|
||||
self.probe_interval = probe_interval
|
||||
self.retry_interval = retry_interval
|
||||
self.tier1_interval = tier1_interval
|
||||
self.fail_limit = fail_limit
|
||||
self._local_address = local_address_fn
|
||||
self._ping = ping_fn
|
||||
self._gateway_fn = gateway_fn
|
||||
self._clock = clock
|
||||
# Reading the gateway is a cheap /proc read on Linux but a subprocess elsewhere,
|
||||
# so elsewhere it is refreshed only when a tier-2 probe is due.
|
||||
self._cheap_gateway = sys.platform.startswith("linux")
|
||||
self.states: Dict[str, FamilyState] = {f: FamilyState() for f in FAMILIES}
|
||||
self._fails: Dict[str, int] = {f: 0 for f in FAMILIES}
|
||||
self._next_ping: Dict[str, float] = {f: 0.0 for f in FAMILIES}
|
||||
self._last_key: Dict[str, Optional[Tuple[Optional[str], Optional[str]]]] = {
|
||||
f: None for f in FAMILIES}
|
||||
self._ping_disabled: Set[str] = set()
|
||||
|
||||
@classmethod
|
||||
def from_config(cls, cfg: Optional[Dict[str, Any]]) -> "Connectivity":
|
||||
cfg = cfg or {}
|
||||
probes = {}
|
||||
if cfg.get("probe_v4"):
|
||||
probes["ipv4"] = str(cfg["probe_v4"])
|
||||
if cfg.get("probe_v6"):
|
||||
probes["ipv6"] = str(cfg["probe_v6"])
|
||||
return cls(probes=probes, probe_interval=float(cfg.get("probe_interval", 60)))
|
||||
|
||||
# ---- readers (cheap; safe to call on every heartbeat) --------------------------
|
||||
def snapshot(self) -> Dict[str, FamilyState]:
|
||||
return {f: replace(s) for f, s in self.states.items()}
|
||||
|
||||
def flags(self) -> Dict[str, bool]:
|
||||
"""HTB flags for families whose presence is known."""
|
||||
return {FLAG_KEYS[f]: s.present for f, s in self.states.items() if s.present is not None}
|
||||
|
||||
def as_metric(self) -> Dict[str, Dict[str, Any]]:
|
||||
return {f: asdict(s) for f, s in self.states.items()}
|
||||
|
||||
# ---- probing -------------------------------------------------------------------
|
||||
async def refresh(self) -> None:
|
||||
for fam in FAMILIES:
|
||||
try:
|
||||
await self._refresh_family(fam)
|
||||
except Exception:
|
||||
logger.exception("connectivity probe failed for %s", fam)
|
||||
|
||||
async def run(self, stop: asyncio.Event) -> None:
|
||||
while not stop.is_set():
|
||||
await self.refresh()
|
||||
try:
|
||||
await asyncio.wait_for(stop.wait(), timeout=self.tier1_interval)
|
||||
except asyncio.TimeoutError:
|
||||
pass
|
||||
|
||||
async def _gateway(self, fam: str) -> Optional[str]:
|
||||
try:
|
||||
return await self._gateway_fn(fam)
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
async def _refresh_family(self, fam: str) -> None:
|
||||
st = self.states[fam]
|
||||
try:
|
||||
addr = self._local_address(fam, self.probes[fam])
|
||||
except ProbeUnavailable:
|
||||
st.present = None
|
||||
return
|
||||
|
||||
if addr is None: # tier 1: no route => absent immediately
|
||||
self._fails[fam] = 0
|
||||
self._next_ping[fam] = 0.0
|
||||
self._last_key[fam] = None
|
||||
st.present, st.addr, st.gateway, st.rtt = False, None, None, None
|
||||
return
|
||||
|
||||
now = self._clock()
|
||||
gw = await self._gateway(fam) if self._cheap_gateway else st.gateway
|
||||
key = (addr, gw)
|
||||
due = key != self._last_key[fam] or now >= self._next_ping[fam]
|
||||
if due and not self._cheap_gateway:
|
||||
gw = await self._gateway(fam)
|
||||
key = (addr, gw)
|
||||
st.addr, st.gateway = addr, gw
|
||||
self._last_key[fam] = key
|
||||
if fam in self._ping_disabled:
|
||||
st.present = None # cannot probe: unknown, never absent, while a route exists
|
||||
elif due:
|
||||
await self._tier2(fam, st, now)
|
||||
|
||||
async def _tier2(self, fam: str, st: FamilyState, now: float) -> None:
|
||||
try:
|
||||
rtt = await self._ping(self.probes[fam], fam)
|
||||
except PingUnavailable:
|
||||
logger.warning("ping unavailable; %s presence stays unknown", fam)
|
||||
self._ping_disabled.add(fam)
|
||||
st.present = None
|
||||
return
|
||||
except (asyncio.TimeoutError, OSError):
|
||||
rtt = None
|
||||
if rtt is None:
|
||||
self._fails[fam] += 1
|
||||
self._next_ping[fam] = now + self.retry_interval
|
||||
if self._fails[fam] >= self.fail_limit:
|
||||
st.present, st.rtt = False, None
|
||||
else:
|
||||
self._fails[fam] = 0
|
||||
self._next_ping[fam] = now + self.probe_interval
|
||||
st.present, st.rtt = True, rtt
|
||||
|
||||
|
||||
# ---- shared instance (heartbeat sender and network_monitor plugin) -----------------
|
||||
_shared: Optional[Connectivity] = None
|
||||
|
||||
|
||||
def set_shared(conn: Optional[Connectivity]) -> None:
|
||||
global _shared
|
||||
_shared = conn
|
||||
|
||||
|
||||
def get_shared() -> Optional[Connectivity]:
|
||||
return _shared
|
||||
@@ -0,0 +1,103 @@
|
||||
"""Best-effort default-gateway lookup (informational only; never affects presence)."""
|
||||
|
||||
import asyncio
|
||||
import re
|
||||
import socket
|
||||
import struct
|
||||
import sys
|
||||
from typing import List, Optional, Tuple
|
||||
|
||||
_RTF_GATEWAY = 0x2
|
||||
_RTF_REJECT = 0x200
|
||||
|
||||
|
||||
def parse_proc_route_v4(text: str) -> Optional[str]:
|
||||
"""Default gateway from /proc/net/route (lowest metric wins)."""
|
||||
best: Optional[Tuple[int, str]] = None
|
||||
for line in text.splitlines()[1:]:
|
||||
f = line.split()
|
||||
if len(f) < 8:
|
||||
continue
|
||||
_iface, dest, gw, flags = f[0], f[1], f[2], f[3] # noqa: F841
|
||||
if dest != "00000000" or not int(flags, 16) & _RTF_GATEWAY:
|
||||
continue
|
||||
metric = int(f[6])
|
||||
if best is None or metric < best[0]:
|
||||
best = (metric, socket.inet_ntoa(struct.pack("<L", int(gw, 16))))
|
||||
return best[1] if best else None
|
||||
|
||||
|
||||
def parse_proc_route_v6(text: str) -> Optional[str]:
|
||||
"""Default gateway from /proc/net/ipv6_route (lowest metric wins)."""
|
||||
best: Optional[Tuple[int, str]] = None
|
||||
for line in text.splitlines():
|
||||
f = line.split()
|
||||
if len(f) < 10:
|
||||
continue
|
||||
dest, plen, nexthop, metric, flags, iface = f[0], f[1], f[4], f[5], f[8], f[9]
|
||||
if dest != "0" * 32 or plen != "00" or nexthop == "0" * 32 or iface == "lo":
|
||||
continue
|
||||
fl = int(flags, 16)
|
||||
if not fl & _RTF_GATEWAY or fl & _RTF_REJECT:
|
||||
continue
|
||||
m = int(metric, 16)
|
||||
if best is None or m < best[0]:
|
||||
best = (m, socket.inet_ntop(socket.AF_INET6, bytes.fromhex(nexthop)))
|
||||
return best[1] if best else None
|
||||
|
||||
|
||||
def parse_macos_route(text: str) -> Optional[str]:
|
||||
"""Gateway from `route -n get [-inet6] default`."""
|
||||
m = re.search(r"^\s*gateway:\s*(\S+)", text, re.MULTILINE)
|
||||
return m.group(1) if m else None
|
||||
|
||||
|
||||
def parse_windows_route(text: str, family: str) -> Optional[str]:
|
||||
"""Default gateway from `route print -4` / `route print -6` (lowest metric wins)."""
|
||||
best: Optional[Tuple[int, str]] = None
|
||||
for line in text.splitlines():
|
||||
if family == "ipv4":
|
||||
f = line.split()
|
||||
if len(f) >= 5 and f[0] == "0.0.0.0" and f[1] == "0.0.0.0" and f[2] != "On-link":
|
||||
cand = (int(f[4]), f[2])
|
||||
else:
|
||||
continue
|
||||
else:
|
||||
m = re.match(r"^\s*\d+\s+(\d+)\s+::/0\s+(\S+)\s*$", line)
|
||||
if not m or m.group(2) == "On-link":
|
||||
continue
|
||||
cand = (int(m.group(1)), m.group(2))
|
||||
if best is None or cand[0] < best[0]:
|
||||
best = cand
|
||||
return best[1] if best else None
|
||||
|
||||
|
||||
async def _run(cmd: List[str]) -> str:
|
||||
proc = await asyncio.create_subprocess_exec(
|
||||
*cmd, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.DEVNULL,
|
||||
)
|
||||
try:
|
||||
out, _ = await asyncio.wait_for(proc.communicate(), timeout=5)
|
||||
except asyncio.TimeoutError:
|
||||
proc.kill()
|
||||
await proc.wait()
|
||||
raise
|
||||
return out.decode(errors="replace")
|
||||
|
||||
|
||||
async def default_gateway(family: str) -> Optional[str]:
|
||||
"""Return the default gateway address for "ipv4"/"ipv6", or None on any failure."""
|
||||
v4 = family == "ipv4"
|
||||
try:
|
||||
if sys.platform.startswith("linux"):
|
||||
with open("/proc/net/route" if v4 else "/proc/net/ipv6_route") as fh:
|
||||
text = fh.read()
|
||||
return parse_proc_route_v4(text) if v4 else parse_proc_route_v6(text)
|
||||
if sys.platform == "darwin":
|
||||
cmd = ["route", "-n", "get"] + ([] if v4 else ["-inet6"]) + ["default"]
|
||||
return parse_macos_route(await _run(cmd))
|
||||
if sys.platform == "win32":
|
||||
return parse_windows_route(await _run(["route", "print", "-4" if v4 else "-6"]), family)
|
||||
except Exception:
|
||||
return None
|
||||
return None
|
||||
+23
-7
@@ -25,6 +25,7 @@ from .. import __version__
|
||||
|
||||
# Import plugin system
|
||||
from .plugin import PluginRegistry, PluginLoader, InfoPlugin, MonitorPlugin
|
||||
from .connectivity import Connectivity, set_shared
|
||||
|
||||
# Constants
|
||||
PORT = 50003
|
||||
@@ -281,7 +282,16 @@ async def handle_update(conn: AsyncConnection, _msg: dict): # pyright: ignore[r
|
||||
stop()
|
||||
|
||||
|
||||
async def heartbeat_sender(conn: AsyncConnection, interval: int):
|
||||
def build_heartbeat_msg(conn, interval: int, connectivity: Optional[Connectivity] = None) -> dict:
|
||||
"""HTB payload; carries p4/p6 presence flags when the probe knows them."""
|
||||
msg = {"acks": conn.ackcount, "rtt": conn.rtts[-1], "interval": interval}
|
||||
if connectivity is not None:
|
||||
msg.update(connectivity.flags())
|
||||
return msg
|
||||
|
||||
|
||||
async def heartbeat_sender(conn: AsyncConnection, interval: int,
|
||||
connectivity: Optional[Connectivity] = None):
|
||||
"""Send periodic heartbeats, retrying the connection if it is not open.
|
||||
|
||||
IPv6 connections that fail to open before their first successful send are
|
||||
@@ -291,6 +301,7 @@ async def heartbeat_sender(conn: AsyncConnection, interval: int):
|
||||
Args:
|
||||
conn: Connection to send on
|
||||
interval: Heartbeat interval in seconds
|
||||
connectivity: Optional probe whose presence flags are attached to every heartbeat
|
||||
"""
|
||||
logger = logging.getLogger("hbc.heartbeat")
|
||||
IPV6_EARLY_FAIL_LIMIT = 3
|
||||
@@ -328,11 +339,7 @@ async def heartbeat_sender(conn: AsyncConnection, interval: int):
|
||||
continue
|
||||
|
||||
try:
|
||||
msg = {
|
||||
"acks": conn.ackcount,
|
||||
"rtt": conn.rtts[-1],
|
||||
"interval": interval
|
||||
}
|
||||
msg = build_heartbeat_msg(conn, interval, connectivity)
|
||||
await conn.sendto(msg, "HTB")
|
||||
|
||||
except asyncio.CancelledError:
|
||||
@@ -619,9 +626,18 @@ async def async_main(args, config):
|
||||
loop.add_signal_handler(signal.SIGHUP, _sighup)
|
||||
|
||||
# Start async tasks
|
||||
# Per-protocol internet presence probe (shared with the network_monitor plugin)
|
||||
net_cfg = dict((config.get("plugins") or {}).get("network_monitor")
|
||||
or config.get("network_monitor") or {})
|
||||
connectivity = None
|
||||
if net_cfg.get("probe_enabled", True):
|
||||
connectivity = Connectivity.from_config(net_cfg)
|
||||
set_shared(connectivity)
|
||||
active_tasks.append(asyncio.create_task(connectivity.run(shutdown_event)))
|
||||
|
||||
# Heartbeat senders (one per connection)
|
||||
for conn in connections:
|
||||
task = asyncio.create_task(heartbeat_sender(conn, interval))
|
||||
task = asyncio.create_task(heartbeat_sender(conn, interval, connectivity))
|
||||
active_tasks.append(task)
|
||||
|
||||
# Plugin collector (uses all connections, but we'll use first one)
|
||||
|
||||
@@ -0,0 +1,73 @@
|
||||
"""System ping helpers shared by ping_monitor and the connectivity probe."""
|
||||
|
||||
import asyncio
|
||||
import re
|
||||
import sys
|
||||
from typing import Dict, List, Optional
|
||||
|
||||
_INF = float("inf")
|
||||
|
||||
|
||||
class PingUnavailable(Exception):
|
||||
"""The system ping binary could not be executed."""
|
||||
|
||||
|
||||
def ping_command(host: str, count: int, timeout: int, family: Optional[int] = None) -> List[str]:
|
||||
"""Build the platform ping command. family is 4, 6 or None (let ping decide)."""
|
||||
if sys.platform == "win32":
|
||||
cmd = ["ping", "-n", str(count), "-w", str(timeout * 1000)]
|
||||
if family:
|
||||
cmd.append(f"-{family}")
|
||||
return cmd + [host]
|
||||
if sys.platform == "darwin" and family:
|
||||
if family == 6:
|
||||
return ["ping6", "-c", str(count), host]
|
||||
return ["ping", "-c", str(count), "-W", str(timeout * 1000), host]
|
||||
cmd = ["ping", "-c", str(count), "-W", str(timeout)]
|
||||
if family:
|
||||
cmd.append(f"-{family}")
|
||||
return cmd + [host]
|
||||
|
||||
|
||||
def parse_ping_output(output: str) -> Dict[str, float]:
|
||||
"""Parse rtt_min/avg/max/loss from ping output (Linux, macOS, BSD)."""
|
||||
loss = 100.0
|
||||
loss_match = re.search(r"(\d+(?:\.\d+)?)\s*%\s*packet\s*loss", output)
|
||||
if loss_match:
|
||||
loss = float(loss_match.group(1))
|
||||
|
||||
# Linux: "rtt min/avg/max/mdev = x/x/x/x ms"; macOS: "round-trip min/avg/max/stddev = ..."
|
||||
rtt_match = re.search(
|
||||
r"(?:rtt|round-trip)\s+min/avg/max/\S+\s*=\s*([\d.]+)/([\d.]+)/([\d.]+)",
|
||||
output,
|
||||
)
|
||||
if rtt_match:
|
||||
return {
|
||||
"rtt_min": float(rtt_match.group(1)),
|
||||
"rtt_avg": float(rtt_match.group(2)),
|
||||
"rtt_max": float(rtt_match.group(3)),
|
||||
"loss": loss,
|
||||
}
|
||||
return {"rtt_min": _INF, "rtt_avg": _INF, "rtt_max": _INF, "loss": loss}
|
||||
|
||||
|
||||
async def run_ping(host: str, count: int, timeout: int, family: Optional[int] = None) -> Dict[str, float]:
|
||||
"""Run ping and return parsed metrics.
|
||||
|
||||
Raises PingUnavailable if the ping binary is missing and asyncio.TimeoutError
|
||||
(after killing the process) if it hangs.
|
||||
"""
|
||||
cmd = ping_command(host, count, timeout, family)
|
||||
try:
|
||||
proc = await asyncio.create_subprocess_exec(
|
||||
*cmd, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE,
|
||||
)
|
||||
except FileNotFoundError as e:
|
||||
raise PingUnavailable(str(e)) from e
|
||||
try:
|
||||
stdout, _ = await asyncio.wait_for(proc.communicate(), timeout=timeout * count + 2)
|
||||
except asyncio.TimeoutError:
|
||||
proc.kill()
|
||||
await proc.wait()
|
||||
raise
|
||||
return parse_ping_output(stdout.decode(errors="replace"))
|
||||
@@ -13,6 +13,7 @@ except ImportError:
|
||||
psutil = None
|
||||
|
||||
from hbd.client.plugin import MonitorPlugin
|
||||
from hbd.client.connectivity import get_shared
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -32,6 +33,10 @@ class NetworkMonitorPlugin(MonitorPlugin):
|
||||
interfaces: List of interfaces to monitor (default: all)
|
||||
include_connections: Include connection statistics (default: True)
|
||||
include_addresses: Include interface addresses (default: False)
|
||||
probe_enabled: Run the per-protocol internet presence probe (default: True)
|
||||
probe_v4: IPv4 address to ping (default: 1.1.1.1)
|
||||
probe_v6: IPv6 address to ping (default: 2606:4700:4700::1111)
|
||||
probe_interval: Seconds between ICMP probes while healthy (default: 60)
|
||||
"""
|
||||
|
||||
name = "network_monitor"
|
||||
@@ -103,6 +108,7 @@ class NetworkMonitorPlugin(MonitorPlugin):
|
||||
- etc.
|
||||
- addresses: Interface address information (if include_addresses)
|
||||
- Dict keyed by interface name with address details
|
||||
- connectivity: per-protocol presence {ipv4, ipv6} -> {present, addr, gateway, rtt}
|
||||
"""
|
||||
if psutil is None:
|
||||
logger.error("psutil not available")
|
||||
@@ -228,6 +234,11 @@ class NetworkMonitorPlugin(MonitorPlugin):
|
||||
|
||||
except Exception as e:
|
||||
logger.warning(f"Could not collect interface stats: {e}")
|
||||
|
||||
# Per-protocol internet presence, address and gateway (from the shared probe)
|
||||
shared = get_shared()
|
||||
if shared is not None:
|
||||
metrics['connectivity'] = shared.as_metric()
|
||||
|
||||
return metrics
|
||||
|
||||
|
||||
@@ -40,9 +40,9 @@ threshold_configs:
|
||||
|
||||
import asyncio
|
||||
import re
|
||||
import sys
|
||||
from typing import Any, Dict, Optional
|
||||
|
||||
from hbd.client.pingutil import PingUnavailable, run_ping
|
||||
from hbd.client.plugin import MonitorPlugin
|
||||
|
||||
|
||||
@@ -84,51 +84,13 @@ class PingMonitorPlugin(MonitorPlugin):
|
||||
|
||||
async def _ping(self, host: str) -> Dict[str, float]:
|
||||
"""Run a system ping command and return rtt_min/avg/max/loss."""
|
||||
if sys.platform == "win32":
|
||||
cmd = ["ping", "-n", str(self.count), "-w", str(self.timeout * 1000), host]
|
||||
else:
|
||||
cmd = ["ping", "-c", str(self.count), "-W", str(self.timeout), host]
|
||||
|
||||
try:
|
||||
proc = await asyncio.create_subprocess_exec(
|
||||
*cmd,
|
||||
stdout=asyncio.subprocess.PIPE,
|
||||
stderr=asyncio.subprocess.PIPE,
|
||||
)
|
||||
stdout, _ = await asyncio.wait_for(
|
||||
proc.communicate(),
|
||||
timeout=self.timeout * self.count + 2,
|
||||
)
|
||||
output = stdout.decode(errors="replace")
|
||||
except (asyncio.TimeoutError, FileNotFoundError, OSError) as e:
|
||||
return await run_ping(host, self.count, self.timeout)
|
||||
except (asyncio.TimeoutError, PingUnavailable, OSError) as e:
|
||||
self.logger.warning("ping_monitor: ping failed for %s: %s", host, e)
|
||||
return {"rtt_min": float("inf"), "rtt_avg": float("inf"),
|
||||
"rtt_max": float("inf"), "loss": 100.0}
|
||||
|
||||
# Parse packet loss
|
||||
loss = 100.0
|
||||
loss_match = re.search(r"(\d+(?:\.\d+)?)\s*%\s*packet\s*loss", output)
|
||||
if loss_match:
|
||||
loss = float(loss_match.group(1))
|
||||
|
||||
# Parse rtt min/avg/max — Linux: "rtt min/avg/max/mdev = x/x/x/x ms"
|
||||
# macOS: "round-trip min/avg/max/stddev = x/x/x/x ms"
|
||||
rtt_match = re.search(
|
||||
r"(?:rtt|round-trip)\s+min/avg/max/\S+\s*=\s*([\d.]+)/([\d.]+)/([\d.]+)",
|
||||
output,
|
||||
)
|
||||
if rtt_match:
|
||||
return {
|
||||
"rtt_min": float(rtt_match.group(1)),
|
||||
"rtt_avg": float(rtt_match.group(2)),
|
||||
"rtt_max": float(rtt_match.group(3)),
|
||||
"loss": loss,
|
||||
}
|
||||
|
||||
# Host unreachable or all packets lost
|
||||
return {"rtt_min": float("inf"), "rtt_avg": float("inf"),
|
||||
"rtt_max": float("inf"), "loss": loss}
|
||||
|
||||
async def _collect_metrics(self) -> Dict[str, Any]:
|
||||
data: Dict[str, Any] = {}
|
||||
tasks = {host: asyncio.create_task(self._ping(host)) for host in self.hosts}
|
||||
|
||||
@@ -100,6 +100,17 @@ def observe(host: str, service: str, level: str) -> str:
|
||||
return SUPPRESS
|
||||
|
||||
|
||||
def clear_host(host: str) -> None:
|
||||
"""Discard all flap state for *host* (called when a host is dropped).
|
||||
|
||||
A dropped host may be mid-flap with no RECOVER ever received, in which
|
||||
case ``ok_since`` stays ``None`` and ``_sweep`` can never clear it on its
|
||||
own — the state would otherwise persist forever.
|
||||
"""
|
||||
for key in [k for k in _state if k[0] == host]:
|
||||
del _state[key]
|
||||
|
||||
|
||||
def flapping_services(host: str) -> list:
|
||||
"""Return the services of *host* that are currently flapping.
|
||||
|
||||
|
||||
+51
-1
@@ -29,6 +29,7 @@ class Connection:
|
||||
UP = "up"
|
||||
DOWN = "down"
|
||||
OVERDUE = "overdue"
|
||||
ABSENT = "absent"
|
||||
|
||||
def __init__(self, host, cid, addr, afam):
|
||||
self.host = host
|
||||
@@ -94,7 +95,9 @@ class Connection:
|
||||
now = time.time()
|
||||
if not Null:
|
||||
d["addr"] = self.addr
|
||||
if self.rtts[-1]:
|
||||
if self.state == Connection.ABSENT:
|
||||
d["rtt"] = ""
|
||||
elif self.rtts[-1]:
|
||||
d["rtt"] = "%d" % round(self.rtts[-1])
|
||||
elif self.state == Connection.UNKNOWN:
|
||||
d["rtt"] = ""
|
||||
@@ -299,6 +302,8 @@ class Host:
|
||||
self.alert_states = {}
|
||||
# Stale-data timers: {plugin_name: asyncio.TimerHandle}
|
||||
self.plugin_timers = {}
|
||||
# Client-reported protocol presence: {afam: {"present": bool, "ts": float}}
|
||||
self.presence: dict = {}
|
||||
# User access control
|
||||
self.owner: str | None = None # username of owner
|
||||
self.managers: list = [] # usernames with manager role
|
||||
@@ -322,6 +327,7 @@ class Host:
|
||||
d["name"] = "<b>%s</b>" % d["name"]
|
||||
d["dyn"] = str(self.dyn)
|
||||
d["num"] = self.num
|
||||
d["status"] = self.status()
|
||||
|
||||
# Add alert counts (split by acknowledged status)
|
||||
warning_unacked = 0
|
||||
@@ -444,6 +450,7 @@ class Host:
|
||||
_, os_data = latest_os
|
||||
hbc_version = os_data.get("hbc_version")
|
||||
ddict["hbc_version"] = hbc_version
|
||||
ddict["status"] = self.status()
|
||||
|
||||
return ddict
|
||||
|
||||
@@ -459,6 +466,47 @@ class Host:
|
||||
else:
|
||||
return addr.find(".") > 0
|
||||
|
||||
def set_presence(self, afam, present, now):
|
||||
"""Record the client's report of whether it has internet connectivity on afam."""
|
||||
self.presence[afam] = {"present": bool(present), "ts": now}
|
||||
|
||||
def flag_present(self, afam):
|
||||
"""Client-reported presence; True when the client never reported (legacy clients)."""
|
||||
entry = getattr(self, "presence", {}).get(afam)
|
||||
return True if entry is None else entry["present"]
|
||||
|
||||
def family_present(self, afam):
|
||||
"""True if afam should be monitored: the client says it is present, or heartbeats
|
||||
are demonstrably arriving on it (real traffic overrides a false "absent")."""
|
||||
conn = self.connections.get(afam)
|
||||
return self.flag_present(afam) or (conn is not None and conn.state == Connection.UP)
|
||||
|
||||
def status(self):
|
||||
"""Host-level status: down, overdue, simplex, up or unknown.
|
||||
|
||||
overdue: every present family is overdue. simplex: both present, one up, one overdue
|
||||
(same severity as up). Families the client reports absent are not considered,
|
||||
unless none is present: then every connection not parked as ABSENT is considered.
|
||||
"""
|
||||
conns = self.connections
|
||||
if any(c.state == Connection.DOWN for c in conns.values()):
|
||||
return "down"
|
||||
present = [c for afam, c in conns.items() if self.family_present(afam)]
|
||||
if not present:
|
||||
# The client reports no protocol present (e.g. its internet probe fails on every
|
||||
# family) yet may still be heard: monitor whatever is not parked as absent, so a
|
||||
# host that then dies is "overdue", not silently "unknown".
|
||||
present = [c for c in conns.values() if c.state != Connection.ABSENT]
|
||||
up = sum(c.state == Connection.UP for c in present)
|
||||
overdue = sum(c.state == Connection.OVERDUE for c in present)
|
||||
if up and overdue:
|
||||
return "simplex"
|
||||
if up:
|
||||
return "up"
|
||||
if overdue:
|
||||
return "overdue"
|
||||
return "unknown"
|
||||
|
||||
def conndata(self, cid, addr, rtt, now):
|
||||
if addr[0:7] == "::ffff:":
|
||||
addr = addr[7:]
|
||||
@@ -499,6 +547,8 @@ class Host:
|
||||
self.monitors = []
|
||||
if not hasattr(self, "plugin_timers"):
|
||||
self.plugin_timers = {}
|
||||
if not hasattr(self, "presence"):
|
||||
self.presence = {}
|
||||
|
||||
pass
|
||||
|
||||
|
||||
+14
-2
@@ -14,6 +14,7 @@ import logging
|
||||
from aiohttp import web
|
||||
import jinja2
|
||||
from . import data
|
||||
from . import flap as flap_mod
|
||||
from . import notify as notify_mod
|
||||
from . import settings as settings_mod
|
||||
from . import users as users_mod
|
||||
@@ -209,6 +210,16 @@ def _mask_config_for_api(config) -> dict:
|
||||
return result
|
||||
|
||||
|
||||
def _visible_plugin_names(plugin_data: dict) -> list:
|
||||
"""Plugin names to render as accordion sections.
|
||||
|
||||
Excludes synthetic rtt_ipv4 / rtt_ipv6 history streams (written by
|
||||
udp.py for charting) which aren't real client plugins and have no
|
||||
accordion renderer.
|
||||
"""
|
||||
return [p for p in plugin_data.keys() if not p.startswith("rtt_")]
|
||||
|
||||
|
||||
def _build_host_info(host, threshold_checker=None) -> dict:
|
||||
"""Assemble the info payload for GET /api/0/hosts/{hostname}/info."""
|
||||
hbc_version = None
|
||||
@@ -264,7 +275,7 @@ def _build_host_info(host, threshold_checker=None) -> dict:
|
||||
"family": getattr(conn, "afam", family),
|
||||
"addr": getattr(conn, "addr", ""),
|
||||
"state": getattr(conn, "state", ""),
|
||||
"rtt": (getattr(conn, "rtts", None) or [None])[0],
|
||||
"rtt": (getattr(conn, "rtts", None) or [None])[-1],
|
||||
"statetime": getattr(conn, "statetime", None),
|
||||
"lastbeat": getattr(conn, "lastbeat", None),
|
||||
}
|
||||
@@ -439,6 +450,7 @@ async def start(
|
||||
return web.json_response({"error": "Forbidden"}, status=403)
|
||||
eventlog(uname, "INFO", "dropped")
|
||||
del hbdclass.Host.hosts[uname]
|
||||
flap_mod.clear_host(uname)
|
||||
return web.Response(text="Done")
|
||||
|
||||
async def register(request):
|
||||
@@ -765,7 +777,7 @@ async def start(
|
||||
if host.plugin_data:
|
||||
hosts_with_plugins.append({
|
||||
"name": hostname,
|
||||
"plugins": list(host.plugin_data.keys()),
|
||||
"plugins": _visible_plugin_names(host.plugin_data),
|
||||
"is_owner": _can_own_host(current_user, host),
|
||||
"owner": host.owner,
|
||||
})
|
||||
|
||||
@@ -107,6 +107,8 @@
|
||||
</section>
|
||||
</div>
|
||||
|
||||
{% include 'foot.html' %}
|
||||
|
||||
<script>
|
||||
(function() {
|
||||
var startEpoch = {{ start_epoch }};
|
||||
|
||||
@@ -73,6 +73,7 @@
|
||||
.tipbox .up { color: var(--st-ok); font-weight: 700; }
|
||||
.tipbox .dn { color: var(--st-crit); font-weight: 700; }
|
||||
.tipbox .ov { color: var(--st-warn); font-weight: 700; }
|
||||
.tipbox .ab { color: var(--st-faint); font-weight: 700; }
|
||||
.tipbox .meta { margin-top: 6px; padding-top: 6px; border-top: 1px solid var(--st-line-soft); font-size: 11px; color: var(--st-faint); }
|
||||
|
||||
@media (max-width: 760px) {
|
||||
@@ -134,18 +135,18 @@
|
||||
function worstConnection(data) {
|
||||
var worst = null;
|
||||
(data.connections || []).forEach(function (c) {
|
||||
if (!c.state || c.state === 'unknown') return;
|
||||
if (!c.state || c.state === 'unknown' || c.state === 'absent') return;
|
||||
if (!worst || (STATE_RANK[c.state] || 0) > (STATE_RANK[worst.state] || 0)) worst = c;
|
||||
});
|
||||
return worst;
|
||||
}
|
||||
|
||||
function severity(data) {
|
||||
var w = worstConnection(data);
|
||||
var st = data.status;
|
||||
if ((data.alert_critical_unacked || 0) > 0 || (data.alert_critical_acked || 0) > 0) return 'crit';
|
||||
if (w && w.state === 'down') return 'crit';
|
||||
if (st === 'down') return 'crit';
|
||||
if ((data.alert_warning_unacked || 0) > 0 || (data.alert_warning_acked || 0) > 0) return 'warn';
|
||||
if (w && w.state === 'overdue') return 'warn';
|
||||
if (st === 'overdue') return 'warn';
|
||||
return 'ok';
|
||||
}
|
||||
var SEV_ORDER = { crit: 0, warn: 1, ok: 2 };
|
||||
@@ -171,17 +172,17 @@
|
||||
|
||||
// ---- host row rendering ---------------------------------------------
|
||||
function statusChip(data) {
|
||||
var w = worstConnection(data);
|
||||
if (!w) return '<span class="chip outline">—</span>';
|
||||
var cls = w.state === 'up' ? 'ok' : (w.state === 'down' ? 'crit' : 'warn');
|
||||
return '<span class="chip ' + cls + '">' + escHtml(w.state) + '</span>';
|
||||
var st = data.status;
|
||||
if (!st || st === 'unknown') return '<span class="chip outline">—</span>';
|
||||
var cls = (st === 'up' || st === 'simplex') ? 'ok' : (st === 'down' ? 'crit' : 'warn');
|
||||
return '<span class="chip ' + cls + '">' + escHtml(st) + '</span>';
|
||||
}
|
||||
|
||||
function tipCard(data) {
|
||||
var rows = '';
|
||||
(data.connections || []).forEach(function (c) {
|
||||
if (!c.state || c.state === 'unknown') return;
|
||||
var cls = c.state === 'up' ? 'up' : (c.state === 'down' ? 'dn' : 'ov');
|
||||
var cls = c.state === 'up' ? 'up' : (c.state === 'down' ? 'dn' : (c.state === 'absent' ? 'ab' : 'ov'));
|
||||
var fam = (c.afam || '').replace('IPv', 'v');
|
||||
var rtt = (c.state === 'up' && c.rtts && c.rtts[0]) ? Math.round(c.rtts[0]) + ' ms' : '—';
|
||||
rows += '<tr><td>' + escHtml(fam) + '</td><td>' + escHtml(c.addr || '—') + '</td>'
|
||||
@@ -247,7 +248,9 @@
|
||||
else if (data.hbc_version !== HBD_VERSION) {
|
||||
staleTag = '<span class="stale" title="agent v' + escHtml(String(data.hbc_version)) + ' ≠ server v' + HBD_VERSION + '">v' + escHtml(String(data.hbc_version)) + '</span>';
|
||||
}
|
||||
var dotCls = w ? (w.state === 'up' ? sev : (w.state === 'down' ? 'down' : 'overdue')) : 'off';
|
||||
var st = data.status;
|
||||
var dotCls = (!st || st === 'unknown') ? 'off'
|
||||
: ((st === 'up' || st === 'simplex') ? sev : (st === 'down' ? 'down' : 'overdue'));
|
||||
var flapTag = '';
|
||||
var flapping = data.flapping || [];
|
||||
if (flapping.length) {
|
||||
@@ -286,11 +289,11 @@
|
||||
var total = 0, up = 0, overdue = 0, down = 0;
|
||||
Object.values(hostData).forEach(function (d) {
|
||||
total++;
|
||||
var w = worstConnection(d);
|
||||
if (!w) return;
|
||||
if (w.state === 'up') up++;
|
||||
else if (w.state === 'overdue') overdue++;
|
||||
else if (w.state === 'down') down++;
|
||||
var st = d.status;
|
||||
if (!st || st === 'unknown') return;
|
||||
if (st === 'up' || st === 'simplex') up++;
|
||||
else if (st === 'overdue') overdue++;
|
||||
else if (st === 'down') down++;
|
||||
});
|
||||
var parts = [total + ' hosts', up + ' up'];
|
||||
if (overdue) parts.push(overdue + ' overdue');
|
||||
|
||||
@@ -1,2 +0,0 @@
|
||||
<!-- <label for="drawer-toggle" id="drawer-toggle-label"></label>
|
||||
s<header>{{ header }}</header> -->
|
||||
@@ -290,8 +290,8 @@
|
||||
|
||||
{% if not hosts %}
|
||||
<div class="no-data">
|
||||
<p>No hosts with plugin data available</p>
|
||||
<p style="font-size:0.9em;margin-top:10px;">Hosts will appear here once they start sending plugin metrics</p>
|
||||
<p>No hosts have connected yet</p>
|
||||
<p style="font-size:0.9em;margin-top:10px;">Hosts will appear here once they start sending heartbeats</p>
|
||||
</div>
|
||||
{% else %}
|
||||
<div id="hosts-container">
|
||||
@@ -445,14 +445,18 @@
|
||||
</div>`;
|
||||
|
||||
if (data.connections && data.connections.length) {
|
||||
html += `<div class="info-thresholds-title">Connectivity</div>
|
||||
<table class="data-table"><thead><tr>
|
||||
html += `<div class="info-thresholds-title">Connectivity</div>`;
|
||||
for (const c of data.connections) {
|
||||
html += `<div class="info-note">${escHtml(c.family)} · ${escHtml(c.addr || '—')} RTT</div>
|
||||
<div id="rtt-chart-${hostname}-${escHtml(c.family)}" style="margin-bottom:8px;"></div>`;
|
||||
}
|
||||
html += `<table class="data-table"><thead><tr>
|
||||
<th>Family</th><th>Address</th><th>State</th>
|
||||
<th class="num">RTT</th><th>Last Change</th><th>Last Packet</th>
|
||||
</tr></thead><tbody>`;
|
||||
for (const c of data.connections) {
|
||||
const st = escHtml(c.state || '—');
|
||||
const stCls = c.state === 'up' ? 'status-up' : (c.state ? 'status-down' : '');
|
||||
const stCls = c.state === 'up' ? 'status-up' : (c.state === 'absent' ? '' : (c.state ? 'status-down' : ''));
|
||||
const rtt = (c.state === 'up' && c.rtt) ? Math.round(c.rtt) + ' ms' : '—';
|
||||
const chg = c.statetime ? new Date(c.statetime * 1000).toLocaleString() : '—';
|
||||
const seen = c.lastbeat ? new Date(c.lastbeat * 1000).toLocaleString() : '—';
|
||||
@@ -495,6 +499,15 @@
|
||||
}
|
||||
|
||||
el.innerHTML = html;
|
||||
|
||||
if (data.connections && data.connections.length) {
|
||||
const rttThresholds = (data.thresholds || []).find(t => t.metric === 'rtt') || null;
|
||||
for (const c of data.connections) {
|
||||
fetchRttHistory(hostname, c.family)
|
||||
.then(samples => renderRttChart(hostname, c.family, samples, rttThresholds))
|
||||
.catch(() => {});
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async function fetchHostGlance(hostname) {
|
||||
@@ -809,84 +822,164 @@
|
||||
return json.samples || [];
|
||||
}
|
||||
|
||||
function renderCpuChart(hostname, samples) {
|
||||
const el = document.getElementById(`cpu-chart-${hostname}`);
|
||||
if (!el || !samples.length) return;
|
||||
function renderTimeSeriesChart(elId, pts, opts) {
|
||||
const el = document.getElementById(elId);
|
||||
if (!el || pts.length < 2) { if (el) el.style.display = 'none'; return; }
|
||||
|
||||
const pts = samples
|
||||
.filter(s => s.data.cpu_percent != null)
|
||||
.map(s => ({ t: s.timestamp, v: s.data.cpu_percent }));
|
||||
if (pts.length < 2) { el.style.display = 'none'; return; }
|
||||
|
||||
const W = 600, H = 80, PAD = { top: 6, right: 8, bottom: 18, left: 28 };
|
||||
const cW = W - PAD.left - PAD.right;
|
||||
const cH = H - PAD.top - PAD.bottom;
|
||||
const unitSuffix = opts.unitSuffix || '';
|
||||
const W = 690, H = 92, PAD = { top: 6, right: 8, bottom: 18, left: 28 };
|
||||
|
||||
const tMin = pts[0].t, tMax = pts[pts.length - 1].t;
|
||||
const tRange = tMax - tMin || 1;
|
||||
const x = t => PAD.left + ((t - tMin) / tRange) * cW;
|
||||
|
||||
// Auto-scale Y axis with 10% padding, clamped to [0, 100]
|
||||
// Auto-scale Y axis with 10% padding, optionally clamped to opts.yDomain
|
||||
const vMin = Math.min(...pts.map(p => p.v));
|
||||
const vMax = Math.max(...pts.map(p => p.v));
|
||||
const vRange = vMax - vMin || 1;
|
||||
const vPad = Math.max(vRange * 0.1, 1);
|
||||
const yLow = Math.max(0, vMin - vPad);
|
||||
const yHigh = Math.min(100, vMax + vPad);
|
||||
const domainLow = Array.isArray(opts.yDomain) ? opts.yDomain[0] : 0;
|
||||
const domainHigh = Array.isArray(opts.yDomain) ? opts.yDomain[1] : Infinity;
|
||||
const yLow = Math.max(domainLow, vMin - vPad);
|
||||
const yHigh = Math.min(domainHigh, vMax + vPad);
|
||||
const yRange = yHigh - yLow || 1;
|
||||
const y = v => PAD.top + cH - ((v - yLow) / yRange) * cH;
|
||||
|
||||
// Build polyline points and filled area path
|
||||
const linePoints = pts.map(p => `${x(p.t).toFixed(1)},${y(p.v).toFixed(1)}`).join(' ');
|
||||
const areaPath = `M${x(pts[0].t).toFixed(1)},${(PAD.top + cH).toFixed(1)} ` +
|
||||
pts.map(p => `L${x(p.t).toFixed(1)},${y(p.v).toFixed(1)}`).join(' ') +
|
||||
` L${x(pts[pts.length-1].t).toFixed(1)},${(PAD.top + cH).toFixed(1)} Z`;
|
||||
|
||||
// Color based on latest absolute CPU %
|
||||
const latest = pts[pts.length - 1].v;
|
||||
const strokeColor = latest > 90 ? '#e53935' : latest > 70 ? '#fb8c00' : '#43a047';
|
||||
const fillColor = latest > 90 ? '#ffcdd2' : latest > 70 ? '#ffe0b2' : '#c8e6c9';
|
||||
|
||||
// Compute nice tick step for ~3-5 grid lines
|
||||
// Compute nice tick step for ~3-5 grid lines, then size the left
|
||||
// margin to fit the widest label (values/units can run to 3+ digits,
|
||||
// e.g. RTT samples above 99ms) instead of a fixed guess that clips them.
|
||||
const rawStep = yRange / 4;
|
||||
const mag = Math.pow(10, Math.floor(Math.log10(rawStep || 1)));
|
||||
const niceStep = [1, 2, 5, 10].map(f => f * mag).find(s => yRange / s <= 5) || mag * 10;
|
||||
const tickStart = Math.ceil(yLow / niceStep) * niceStep;
|
||||
let gridLines = '';
|
||||
for (let v = tickStart; v <= yHigh + 0.001; v += niceStep) {
|
||||
const yy = y(v).toFixed(1);
|
||||
const label = Number.isInteger(v) ? v : v.toFixed(1);
|
||||
gridLines += `<line x1="${PAD.left}" y1="${yy}" x2="${PAD.left + cW}" y2="${yy}" stroke="#e0e0e0" stroke-width="1"/>`;
|
||||
gridLines += `<text x="${(PAD.left - 3).toFixed(1)}" y="${yy}" text-anchor="end" dominant-baseline="middle" font-size="8" fill="#999">${label}</text>`;
|
||||
const yTicks = [];
|
||||
for (let v = tickStart; v <= yHigh + 0.001; v += niceStep) yTicks.push(v);
|
||||
const yTickLabels = yTicks.map(v => (Number.isInteger(v) ? v : v.toFixed(1)) + unitSuffix);
|
||||
const maxLabelLen = yTickLabels.reduce((m, s) => Math.max(m, s.length), 0);
|
||||
PAD.left = Math.max(28, Math.ceil(maxLabelLen * 5.5) + 10);
|
||||
|
||||
const cW = W - PAD.left - PAD.right;
|
||||
const cH = H - PAD.top - PAD.bottom;
|
||||
const x = t => PAD.left + ((t - tMin) / tRange) * cW;
|
||||
const y = v => PAD.top + cH - ((v - yLow) / yRange) * cH;
|
||||
|
||||
// Color based on latest value
|
||||
const latest = pts[pts.length - 1].v;
|
||||
const { stroke: strokeColor, fill: fillColor } = opts.colorFor(latest);
|
||||
|
||||
// Split into segments wherever the gap between consecutive samples is
|
||||
// much larger than the typical spacing (e.g. the host was overdue/down
|
||||
// for a while) — draw each segment separately so missing data reads as
|
||||
// a visual gap instead of an interpolated line across dead time.
|
||||
const deltas = [];
|
||||
for (let i = 1; i < pts.length; i++) deltas.push(pts[i].t - pts[i - 1].t);
|
||||
deltas.sort((a, b) => a - b);
|
||||
const medianDelta = deltas[Math.floor(deltas.length / 2)];
|
||||
const gapThreshold = medianDelta * 2.5;
|
||||
|
||||
const segments = [[pts[0]]];
|
||||
for (let i = 1; i < pts.length; i++) {
|
||||
if (pts[i].t - pts[i - 1].t > gapThreshold) segments.push([]);
|
||||
segments[segments.length - 1].push(pts[i]);
|
||||
}
|
||||
|
||||
// X-axis time labels
|
||||
let linePolylines = '';
|
||||
let areaPaths = '';
|
||||
for (const seg of segments) {
|
||||
if (seg.length < 2) continue;
|
||||
const segPoints = seg.map(p => `${x(p.t).toFixed(1)},${y(p.v).toFixed(1)}`).join(' ');
|
||||
linePolylines += `<polyline points="${segPoints}" fill="none" stroke="${strokeColor}" stroke-width="1.5" stroke-linejoin="round"/>`;
|
||||
const segArea = `M${x(seg[0].t).toFixed(1)},${(PAD.top + cH).toFixed(1)} ` +
|
||||
seg.map(p => `L${x(p.t).toFixed(1)},${y(p.v).toFixed(1)}`).join(' ') +
|
||||
` L${x(seg[seg.length-1].t).toFixed(1)},${(PAD.top + cH).toFixed(1)} Z`;
|
||||
areaPaths += `<path d="${segArea}" fill="${fillColor}" opacity="0.6"/>`;
|
||||
}
|
||||
|
||||
let gridLines = '';
|
||||
yTicks.forEach((v, i) => {
|
||||
const yy = y(v).toFixed(1);
|
||||
gridLines += `<line x1="${PAD.left}" y1="${yy}" x2="${PAD.left + cW}" y2="${yy}" stroke="#e0e0e0" stroke-width="1"/>`;
|
||||
gridLines += `<text x="${(PAD.left - 3).toFixed(1)}" y="${yy}" text-anchor="end" dominant-baseline="middle" font-size="8" fill="#999">${yTickLabels[i]}</text>`;
|
||||
});
|
||||
|
||||
// X-axis: start/end plus evenly spaced intermediate tickmarks + labels
|
||||
const fmt = ts => {
|
||||
const d = new Date(ts * 1000);
|
||||
return d.toLocaleTimeString([], { hour: '2-digit', minute: '2-digit' });
|
||||
};
|
||||
const xLabels = `
|
||||
<text x="${PAD.left}" y="${H - 2}" text-anchor="start" font-size="8" fill="#999">${fmt(pts[0].t)}</text>
|
||||
<text x="${PAD.left + cW}" y="${H - 2}" text-anchor="end" font-size="8" fill="#999">${fmt(pts[pts.length-1].t)}</text>`;
|
||||
const xTickCount = 5;
|
||||
const xAxisY = (PAD.top + cH).toFixed(1);
|
||||
let xAxisMarks = '';
|
||||
let xLabels = '';
|
||||
for (let i = 0; i < xTickCount; i++) {
|
||||
const t = tMin + (tRange * i) / (xTickCount - 1);
|
||||
const xx = x(t).toFixed(1);
|
||||
const anchor = i === 0 ? 'start' : (i === xTickCount - 1 ? 'end' : 'middle');
|
||||
xAxisMarks += `<line x1="${xx}" y1="${xAxisY}" x2="${xx}" y2="${(PAD.top + cH + 3).toFixed(1)}" stroke="#ccc" stroke-width="1"/>`;
|
||||
xLabels += `<text x="${xx}" y="${H - 2}" text-anchor="${anchor}" font-size="8" fill="#999">${fmt(t)}</text>`;
|
||||
}
|
||||
|
||||
el.innerHTML = `<svg viewBox="0 0 ${W} ${H}" preserveAspectRatio="none"
|
||||
style="width:100%;height:${H}px;display:block;">
|
||||
<defs>
|
||||
<clipPath id="cpu-clip-${hostname}">
|
||||
<clipPath id="${opts.clipId}">
|
||||
<rect x="${PAD.left}" y="${PAD.top}" width="${cW}" height="${cH}"/>
|
||||
</clipPath>
|
||||
</defs>
|
||||
${gridLines}
|
||||
<line x1="${PAD.left}" y1="${PAD.top}" x2="${PAD.left}" y2="${PAD.top + cH}" stroke="#ccc" stroke-width="1"/>
|
||||
<line x1="${PAD.left}" y1="${PAD.top + cH}" x2="${PAD.left + cW}" y2="${PAD.top + cH}" stroke="#ccc" stroke-width="1"/>
|
||||
<g clip-path="url(#cpu-clip-${hostname})">
|
||||
<path d="${areaPath}" fill="${fillColor}" opacity="0.6"/>
|
||||
<polyline points="${linePoints}" fill="none" stroke="${strokeColor}" stroke-width="1.5" stroke-linejoin="round"/>
|
||||
<g clip-path="url(#${opts.clipId})">
|
||||
${areaPaths}
|
||||
${linePolylines}
|
||||
</g>
|
||||
${xAxisMarks}
|
||||
${xLabels}
|
||||
</svg>`;
|
||||
}
|
||||
|
||||
function renderCpuChart(hostname, samples) {
|
||||
const pts = samples
|
||||
.filter(s => s.data.cpu_percent != null)
|
||||
.map(s => ({ t: s.timestamp, v: s.data.cpu_percent }));
|
||||
|
||||
renderTimeSeriesChart(`cpu-chart-${hostname}`, pts, {
|
||||
yDomain: [0, 100],
|
||||
clipId: `cpu-clip-${hostname}`,
|
||||
colorFor: (latest) => ({
|
||||
stroke: latest > 90 ? '#e53935' : latest > 70 ? '#fb8c00' : '#43a047',
|
||||
fill: latest > 90 ? '#ffcdd2' : latest > 70 ? '#ffe0b2' : '#c8e6c9',
|
||||
}),
|
||||
});
|
||||
}
|
||||
|
||||
async function fetchRttHistory(hostname, family) {
|
||||
const plugin = `rtt_${family.toLowerCase()}`;
|
||||
const r = await fetch(`/api/0/hosts/${encodeURIComponent(hostname)}/plugins/${plugin}?limit=100`);
|
||||
if (!r.ok) return [];
|
||||
const json = await r.json();
|
||||
return json.samples || [];
|
||||
}
|
||||
|
||||
function renderRttChart(hostname, family, samples, rttThresholds) {
|
||||
const pts = samples
|
||||
.filter(s => s.data.rtt != null && s.data.rtt > 0)
|
||||
.map(s => ({ t: s.timestamp, v: s.data.rtt }));
|
||||
|
||||
renderTimeSeriesChart(`rtt-chart-${hostname}-${family}`, pts, {
|
||||
yDomain: 'auto',
|
||||
clipId: `rtt-clip-${hostname}-${family}`,
|
||||
unitSuffix: ' ms',
|
||||
colorFor: (latest) => {
|
||||
if (rttThresholds?.critical != null && latest > rttThresholds.critical) {
|
||||
return { stroke: '#e53935', fill: '#ffcdd2' };
|
||||
}
|
||||
if (rttThresholds?.warning != null && latest > rttThresholds.warning) {
|
||||
return { stroke: '#fb8c00', fill: '#ffe0b2' };
|
||||
}
|
||||
return { stroke: '#1976d2', fill: '#bbdefb' };
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
function renderCpuTable(hostname, d) {
|
||||
const KEYS = [
|
||||
['cpu_percent', 'CPU Usage', 'bar'],
|
||||
@@ -1050,6 +1143,24 @@
|
||||
function renderNetworkTables(d) {
|
||||
let html = '';
|
||||
|
||||
if (d.connectivity) {
|
||||
let t = '<table class="data-table"><thead><tr><th>Protocol</th><th>Internet</th>'
|
||||
+ '<th>Address</th><th>Gateway</th><th class="num">RTT</th></tr></thead><tbody>';
|
||||
for (const fam of ['ipv4', 'ipv6']) {
|
||||
const c = d.connectivity[fam];
|
||||
if (!c) continue;
|
||||
const known = c.present === true || c.present === false;
|
||||
const cls = c.present === true ? 'status-up' : '';
|
||||
const txt = c.present === true ? '✓ present' : (c.present === false ? 'absent' : '?');
|
||||
t += `<tr><td>${fam === 'ipv4' ? 'IPv4' : 'IPv6'}</td>
|
||||
<td class="${cls}">${txt}</td>
|
||||
<td>${escHtml(c.addr || '—')}</td><td>${escHtml(c.gateway || '—')}</td>
|
||||
<td class="num">${known && c.rtt != null ? Math.round(c.rtt) + ' ms' : '—'}</td></tr>`;
|
||||
}
|
||||
t += '</tbody></table>';
|
||||
html += `<div class="table-section-label">Connectivity</div>${t}`;
|
||||
}
|
||||
|
||||
if (d.interfaces && Object.keys(d.interfaces).length) {
|
||||
const sample = Object.values(d.interfaces)[0];
|
||||
const hasDelta = 'bytes_sent_delta' in sample;
|
||||
@@ -1293,6 +1404,11 @@
|
||||
document.querySelectorAll('.host-card:not(.collapsed)').forEach(card => {
|
||||
const hostname = card.dataset.hostname;
|
||||
|
||||
fetchHostInfo(hostname).then(data => {
|
||||
infoCache[hostname] = data;
|
||||
renderInfoSection(hostname, data);
|
||||
}).catch(() => {});
|
||||
|
||||
card.querySelectorAll('.plugin-accordion:not(.collapsed)').forEach(acc => {
|
||||
const pname = acc.dataset.plugin;
|
||||
if (!GLANCE_PLUGINS.includes(pname)) {
|
||||
|
||||
+125
-29
@@ -194,6 +194,40 @@ def _set_connectivity_alert(host, afam, level_name):
|
||||
state.update(level, level_name)
|
||||
|
||||
|
||||
_PRESENCE_KEYS = {"p4": "IPv4", "p6": "IPv6"}
|
||||
|
||||
|
||||
def _apply_presence(host, msg, now, uname, ctx, skip_conn):
|
||||
"""Record client-reported protocol presence and settle the affected connections.
|
||||
|
||||
A family reported absent whose connection is already overdue/unknown is parked as
|
||||
ABSENT (timer cancelled, connectivity alert cleared). One still UP is left alone; its
|
||||
timer expiry parks it. A family reported present again while ABSENT restarts a grace
|
||||
timer. The connection that carried this very heartbeat (skip_conn) is never touched.
|
||||
"""
|
||||
cfg = ctx.get("config", {})
|
||||
for key, afam in _PRESENCE_KEYS.items():
|
||||
if key not in msg:
|
||||
continue
|
||||
present = bool(msg[key])
|
||||
host.set_presence(afam, present, now)
|
||||
conn = host.connections.get(afam)
|
||||
if conn is None or conn is skip_conn:
|
||||
continue
|
||||
cls = conn.__class__
|
||||
if not present:
|
||||
if conn.getstate() in (cls.OVERDUE, cls.UNKNOWN):
|
||||
conn.cancel_overdue_timer()
|
||||
conn.newstate(cls.ABSENT, now)
|
||||
_set_connectivity_alert(host, afam, "OK")
|
||||
elif conn.getstate() == cls.ABSENT:
|
||||
conn.newstate(cls.UNKNOWN, now)
|
||||
interval = int(msg.get("interval", 0) or 0) or host.interval
|
||||
if interval > 0:
|
||||
on_overdue, _ = _make_timer_callbacks(uname, host, ctx)
|
||||
conn.reset_overdue_timer(interval + cfg.get("grace", 2), on_overdue)
|
||||
|
||||
|
||||
def _make_timer_callbacks(uname, host, ctx):
|
||||
"""Return (on_overdue, on_unknown) async callbacks for connection timer logic.
|
||||
|
||||
@@ -210,26 +244,52 @@ def _make_timer_callbacks(uname, host, ctx):
|
||||
msg_to_websockets("host", host.stateinfo())
|
||||
|
||||
async def on_overdue(connection):
|
||||
if connection.getstate() != connection.__class__.UP:
|
||||
cls = connection.__class__
|
||||
if connection.getstate() not in (cls.UP, cls.UNKNOWN):
|
||||
return
|
||||
now = time.time()
|
||||
connection.newstate(connection.__class__.OVERDUE, now, cfg.get("grace", 2))
|
||||
msg = f"{connection.afam} overdue"
|
||||
eventlog(uname, "CRITICAL", msg)
|
||||
if host.watched:
|
||||
asyncio.create_task(notify_mod.send_notification(
|
||||
uname,
|
||||
notify_mod.Notification(title=f"[CRITICAL] {uname}", body=msg, level="CRITICAL"),
|
||||
))
|
||||
# Track in alert_states so the Alerts Dashboard shows this
|
||||
_set_connectivity_alert(host, connection.afam, "CRITICAL")
|
||||
if threshold_checker:
|
||||
threshold_checker.check_value(
|
||||
host_name=uname,
|
||||
metric_path="rtt",
|
||||
value=float("inf"),
|
||||
alert_states=host.alert_states,
|
||||
)
|
||||
afam = connection.afam
|
||||
# Park only while some other family is still present: the last monitored family
|
||||
# must never be parked silently, or a host whose client reports no internet on any
|
||||
# protocol would die without an alert.
|
||||
if not host.flag_present(afam) and any(
|
||||
af != afam and host.family_present(af) for af in host.connections):
|
||||
# The client says this protocol has no connectivity: not an outage.
|
||||
connection.newstate(cls.ABSENT, now)
|
||||
connection.cancel_overdue_timer()
|
||||
_set_connectivity_alert(host, afam, "OK")
|
||||
if msg_to_websockets:
|
||||
msg_to_websockets("host", host.stateinfo())
|
||||
return
|
||||
|
||||
# If any connection is DOWN (clean shutdown reported) Host.status() is "down", so a
|
||||
# later overdue of the other family raises no separate event/alert; the shutdown did.
|
||||
prev = host.status()
|
||||
connection.newstate(cls.OVERDUE, now, cfg.get("grace", 2))
|
||||
new = host.status()
|
||||
if new == "overdue" and prev != "overdue":
|
||||
names = ", ".join(sorted(af for af, c in host.connections.items()
|
||||
if c.getstate() == cls.OVERDUE))
|
||||
msg = f"{names} overdue"
|
||||
eventlog(uname, "CRITICAL", msg)
|
||||
if host.watched:
|
||||
asyncio.create_task(notify_mod.send_notification(
|
||||
uname,
|
||||
notify_mod.Notification(title=f"[CRITICAL] {uname}", body=msg, level="CRITICAL"),
|
||||
))
|
||||
# Track in alert_states so the Alerts Dashboard shows this
|
||||
for af, c in host.connections.items():
|
||||
if c.getstate() == cls.OVERDUE:
|
||||
_set_connectivity_alert(host, af, "CRITICAL")
|
||||
if threshold_checker:
|
||||
threshold_checker.check_value(
|
||||
host_name=uname,
|
||||
metric_path="rtt",
|
||||
value=float("inf"),
|
||||
alert_states=host.alert_states,
|
||||
)
|
||||
elif new == "simplex":
|
||||
eventlog(uname, "INFO", f"{afam} overdue (simplex)")
|
||||
if msg_to_websockets:
|
||||
msg_to_websockets("host", host.stateinfo())
|
||||
connection.reset_overdue_timer(DROPOVERDUE, on_unknown)
|
||||
@@ -270,6 +330,8 @@ def restore_connection_timers(hbdclass, ctx):
|
||||
interval = host.interval
|
||||
for afam, conn in list(host.connections.items()):
|
||||
state = conn.getstate()
|
||||
if state == hbdclass.Connection.ABSENT:
|
||||
continue
|
||||
if state == hbdclass.Connection.DOWN:
|
||||
_set_connectivity_alert(host, afam, "CRITICAL")
|
||||
continue
|
||||
@@ -295,6 +357,7 @@ def restore_connection_timers(hbdclass, ctx):
|
||||
restored += 1
|
||||
|
||||
elif state == hbdclass.Connection.OVERDUE:
|
||||
host_overdue = host.status() == "overdue"
|
||||
elapsed_overdue = now - conn.statetime
|
||||
remaining = DROPOVERDUE - elapsed_overdue
|
||||
if remaining <= 1:
|
||||
@@ -304,6 +367,9 @@ def restore_connection_timers(hbdclass, ctx):
|
||||
"Marking %s/%s UNKNOWN (overdue %.1f days)",
|
||||
uname, afam, elapsed_overdue / 86400,
|
||||
)
|
||||
# A connection dropped to UNKNOWN keeps its alert, like the
|
||||
# UNKNOWN branch above and on_unknown do.
|
||||
host_overdue = True
|
||||
else:
|
||||
conn.reset_overdue_timer(remaining, on_unknown)
|
||||
logger.debug(
|
||||
@@ -313,12 +379,23 @@ def restore_connection_timers(hbdclass, ctx):
|
||||
# Ensure the connectivity alert is set — it may be missing if
|
||||
# hbd was shut down before the on_overdue callback had a chance
|
||||
# to record it.
|
||||
_set_connectivity_alert(host, afam, "CRITICAL")
|
||||
if host_overdue:
|
||||
_set_connectivity_alert(host, afam, "CRITICAL")
|
||||
restored += 1
|
||||
|
||||
logger.info("Restored timers for %d connection(s)", restored)
|
||||
|
||||
|
||||
def _is_rtt_key(plugin_name: str) -> bool:
|
||||
"""True for synthetic RTT-history keys (rtt_ipv4/rtt_ipv6), not real plugin data."""
|
||||
return plugin_name.startswith("rtt_")
|
||||
|
||||
|
||||
def _has_real_plugin_data(plugin_data: dict) -> bool:
|
||||
"""True if plugin_data holds any real (non-RTT) client-collected plugin data."""
|
||||
return any(not _is_rtt_key(k) for k in plugin_data)
|
||||
|
||||
|
||||
def handle_datagram(msg: dict, addr, transport, ctx: dict):
|
||||
"""Handle a parsed datagram message.
|
||||
|
||||
@@ -385,7 +462,7 @@ def handle_datagram(msg: dict, addr, transport, ctx: dict):
|
||||
host.doesack = msg.get("acks", -1)
|
||||
# send ACK back; ask client to resend plugin info when we have none yet
|
||||
rmsg = {"time": time.time()}
|
||||
if not host.plugin_data:
|
||||
if not _has_real_plugin_data(host.plugin_data):
|
||||
rmsg["request_update"] = 1
|
||||
opkt = dicttos("ACK", rmsg)
|
||||
try:
|
||||
@@ -507,24 +584,36 @@ def handle_datagram(msg: dict, addr, transport, ctx: dict):
|
||||
if message:
|
||||
eventlog(uname, "INFO", message, service=service)
|
||||
|
||||
_apply_presence(host, msg, now, uname, ctx, conn)
|
||||
prev_status = host.status()
|
||||
|
||||
if conn.getstate() != hbdcls.Connection.UP:
|
||||
# Transition to UP and log/notify if appropriate
|
||||
lasts = conn.state
|
||||
d = conn.newstate(hbdcls.Connection.UP, now)
|
||||
# On reboot, pre-boot plugin data and derived alerts are stale.
|
||||
# Cancel all plugin timers and wipe plugin state so timers restart
|
||||
# cleanly from the first two post-boot samples.
|
||||
for pname in list(host.plugin_timers):
|
||||
host.cancel_plugin_timer(pname)
|
||||
host.plugin_data.clear()
|
||||
if boot:
|
||||
# On reboot, pre-boot plugin data and derived alerts are stale.
|
||||
# Cancel all plugin timers and wipe plugin state so timers restart
|
||||
# cleanly from the first two post-boot samples. An ordinary
|
||||
# reconnect (no boot flag) doesn't invalidate the client's
|
||||
# already-collected data, so it's left alone — this keeps chart
|
||||
# history intact across a transient network blip.
|
||||
for pname in list(host.plugin_timers):
|
||||
host.cancel_plugin_timer(pname)
|
||||
for pname in [k for k in host.plugin_data if not _is_rtt_key(k)]:
|
||||
del host.plugin_data[pname]
|
||||
stale_plugin_keys = [
|
||||
k for k in host.alert_states
|
||||
if k not in ("rtt",) and not k.startswith("connectivity.")
|
||||
]
|
||||
for k in stale_plugin_keys:
|
||||
del host.alert_states[k]
|
||||
# Clear connectivity alert now that the host is back up
|
||||
# Clear connectivity alert now that the host is back up; once the host is no longer
|
||||
# overdue no family may keep a connectivity alert (e.g. the simplex case).
|
||||
_set_connectivity_alert(host, conn.afam, "OK")
|
||||
if host.status() in ("up", "simplex"):
|
||||
for af in host.connections:
|
||||
_set_connectivity_alert(host, af, "OK")
|
||||
# Don't log/notify RECOVER for a brand-new host seen for the first time —
|
||||
# it was never down, it just hasn't been seen before.
|
||||
if not newh:
|
||||
@@ -536,8 +625,11 @@ def handle_datagram(msg: dict, addr, transport, ctx: dict):
|
||||
else:
|
||||
m = "%s back after being %s for %s" % (conn.afam, lasts, dur(d))
|
||||
if m:
|
||||
eventlog(uname, "RECOVER", m)
|
||||
if host.watched:
|
||||
# RECOVER (and a notification) only if the host itself was overdue/down;
|
||||
# a single family coming back while another stayed up is informational.
|
||||
level = "RECOVER" if prev_status in ("overdue", "down", "unknown") else "INFO"
|
||||
eventlog(uname, level, m)
|
||||
if host.watched and level == "RECOVER":
|
||||
asyncio.create_task(notify_mod.send_notification(
|
||||
uname,
|
||||
notify_mod.Notification(title=f"[RECOVER] {uname}", body=m, level="RECOVER"),
|
||||
@@ -562,6 +654,10 @@ def handle_datagram(msg: dict, addr, transport, ctx: dict):
|
||||
if interval > 0:
|
||||
host.interval = interval
|
||||
|
||||
# Record RTT history for charting
|
||||
if rtt is not None:
|
||||
host.add_plugin_data(f"rtt_{conn.afam.lower()}", {"rtt": rtt}, timestamp=now)
|
||||
|
||||
# Timer-based reachability monitoring
|
||||
# Reset overdue timer on every heartbeat
|
||||
if interval > 0 and conn.getstate() != hbdcls.Connection.DOWN:
|
||||
|
||||
+1
-1
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
|
||||
|
||||
[project]
|
||||
name = "hbd"
|
||||
version = "5.4.0"
|
||||
version = "5.4.5"
|
||||
description = "Heartbeat monitoring system — client (hbc) and server (hbd)"
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.11"
|
||||
|
||||
@@ -9,8 +9,10 @@ sed -i".bak" "s/\*\*Package:\*\* \`hbd\` v[0-9.]*/\*\*Package:\*\* \`hbd\` v$VE
|
||||
|
||||
# Update CHANGELOG.md with commits since last tag
|
||||
LASTTAG=$(git describe --tags --abbrev=0 2>/dev/null || true)
|
||||
ADDED=$(git log "${LASTTAG:+$LASTTAG..}HEAD" --pretty="%s" | grep "^feat:" | sed 's/^feat: /- /')
|
||||
FIXED=$(git log "${LASTTAG:+$LASTTAG..}HEAD" --pretty="%s" | grep "^fix:" | sed 's/^fix: /- /')
|
||||
# Matches both unscoped ("feat: ...") and scoped ("feat(server): ...") conventional-commit
|
||||
# subjects; the scope is dropped from the changelog line.
|
||||
ADDED=$(git log "${LASTTAG:+$LASTTAG..}HEAD" --pretty="%s" | grep -E "^feat(\([^)]*\))?:" | sed -E 's/^feat(\([^)]*\))?: /- /')
|
||||
FIXED=$(git log "${LASTTAG:+$LASTTAG..}HEAD" --pretty="%s" | grep -E "^fix(\([^)]*\))?:" | sed -E 's/^fix(\([^)]*\))?: /- /')
|
||||
{
|
||||
printf "## [%s]\n" "$VER"
|
||||
[ -n "$ADDED" ] && printf "\n### Added\n%s\n" "$ADDED"
|
||||
|
||||
@@ -1,2 +1,3 @@
|
||||
hbc_mini
|
||||
hbc_mini_dbg
|
||||
test_probe
|
||||
|
||||
+8
-2
@@ -7,7 +7,7 @@ SRC = hbc_mini.c
|
||||
# FreeBSD/NetBSD keep zlib in base; no extra flags needed.
|
||||
# On some NetBSD installs pthreads may need -lpthread from pkgsrc.
|
||||
|
||||
.PHONY: all clean debug
|
||||
.PHONY: all clean debug test
|
||||
|
||||
all: $(TARGET)
|
||||
|
||||
@@ -17,5 +17,11 @@ $(TARGET): $(SRC)
|
||||
debug: $(SRC)
|
||||
$(CC) -g -fsanitize=address,undefined -o $(TARGET)_dbg $< $(LDFLAGS)
|
||||
|
||||
test: test_probe
|
||||
./test_probe
|
||||
|
||||
test_probe: test_probe.c $(SRC)
|
||||
$(CC) $(CFLAGS) -o $@ test_probe.c $(LDFLAGS)
|
||||
|
||||
clean:
|
||||
rm -f $(TARGET) $(TARGET)_dbg
|
||||
rm -f $(TARGET) $(TARGET)_dbg test_probe
|
||||
|
||||
+381
-5
@@ -6,6 +6,12 @@
|
||||
*
|
||||
* Drop on any host and run: ./hbc_mini <hbd-host>
|
||||
* Config: ~/.hbc.json (same keys as Python version)
|
||||
*
|
||||
* Presence probe (built in): reports per protocol (IPv4/IPv6) whether the host has internet
|
||||
* connectivity, as p4/p6 flags on every heartbeat, so hbd stops treating a missing IPv6 (or
|
||||
* IPv4) path as an outage. Config, in the network_monitor section of ~/.hbc.json:
|
||||
* probe_enabled (default true), probe_v4 (default 1.1.1.1), probe_v6 (default
|
||||
* 2606:4700:4700::1111; IP literals only), probe_interval (default 60 s)
|
||||
*/
|
||||
|
||||
#ifdef __linux__
|
||||
@@ -396,6 +402,10 @@ typedef struct {
|
||||
|
||||
char net_skip[MAX_NET_SKIP][64];
|
||||
int net_skip_count;
|
||||
|
||||
bool probe_enabled;
|
||||
int probe_interval;
|
||||
char probe_v4[64], probe_v6[64];
|
||||
} config_t;
|
||||
|
||||
static void config_defaults(config_t *c) {
|
||||
@@ -410,6 +420,10 @@ static void config_defaults(config_t *c) {
|
||||
c->net_interval = 300;
|
||||
snprintf(c->net_skip[0], 64, "lo");
|
||||
c->net_skip_count = 1;
|
||||
c->probe_enabled = true;
|
||||
c->probe_interval = 60;
|
||||
snprintf(c->probe_v4, sizeof(c->probe_v4), "1.1.1.1");
|
||||
snprintf(c->probe_v6, sizeof(c->probe_v6), "2606:4700:4700::1111");
|
||||
}
|
||||
|
||||
static void config_load(config_t *cfg, const char *path) {
|
||||
@@ -498,6 +512,10 @@ static void config_load(config_t *cfg, const char *path) {
|
||||
jval_t *net = plugins ? jget(plugins,"network_monitor") : jget(root,"network_monitor");
|
||||
if (net && net->type == JT_OBJECT) {
|
||||
if ((v=jget(net,"interval"))) cfg->net_interval = jint(v, cfg->net_interval);
|
||||
if ((v=jget(net,"probe_enabled"))) cfg->probe_enabled = jint(v, 1) != 0;
|
||||
if ((v=jget(net,"probe_interval"))) cfg->probe_interval = jint(v, cfg->probe_interval);
|
||||
if ((v=jget(net,"probe_v4"))) snprintf(cfg->probe_v4, sizeof(cfg->probe_v4), "%s", jstr(v, cfg->probe_v4));
|
||||
if ((v=jget(net,"probe_v6"))) snprintf(cfg->probe_v6, sizeof(cfg->probe_v6), "%s", jstr(v, cfg->probe_v6));
|
||||
jval_t *sk = jget(net,"skip_interfaces");
|
||||
if (sk && sk->type == JT_ARRAY) {
|
||||
cfg->net_skip_count = 0;
|
||||
@@ -984,6 +1002,341 @@ static void plugin_disk_monitor(conn_t *c, const config_t *cfg) {
|
||||
LOGD("sent disk_monitor");
|
||||
}
|
||||
|
||||
/* ============================================================
|
||||
* Connectivity probe: per-protocol internet presence (IPv4 / IPv6)
|
||||
*
|
||||
* Tier 1 (every PROBE_TIER1 s, sends no packets): a UDP connect() to the probe address shows
|
||||
* whether a route exists and which source address would be used. Tier 2 (every probe_interval
|
||||
* s, sooner after a failure or a change of address or gateway): one ICMP echo, only when tier 1
|
||||
* found a route. Absent = no route, or PROBE_FAIL_LIMIT consecutive failed echoes; present =
|
||||
* one success; unknown (flag omitted) when the family cannot be probed. The result rides on
|
||||
* every HTB as p4/p6 and inside the network_monitor metric. Mirrors hbd/client/connectivity.py.
|
||||
* ============================================================ */
|
||||
|
||||
#define PROBE_UNKNOWN (-1)
|
||||
#define PROBE_FAIL_LIMIT 2
|
||||
#define PROBE_TIER1 5 /* seconds between route checks */
|
||||
|
||||
typedef struct {
|
||||
int present; /* PROBE_UNKNOWN, 0 = absent, 1 = present */
|
||||
char addr[INET6_ADDRSTRLEN];
|
||||
char gateway[INET6_ADDRSTRLEN];
|
||||
double rtt; /* ms, < 0 when unknown */
|
||||
int fails; /* consecutive failed echoes */
|
||||
double next_ping; /* monotonic seconds */
|
||||
char last_key[2 * INET6_ADDRSTRLEN + 2];
|
||||
bool ping_disabled; /* no usable ping binary */
|
||||
} probe_fam_t;
|
||||
|
||||
typedef struct {
|
||||
/* 1 = route exists (source address in out), 0 = no route, -1 = family unsupported/unusable */
|
||||
int (*local_address)(int af, const char *target, char *out, size_t n);
|
||||
/* RTT in ms; -1 = no reply; -2 = ping unavailable */
|
||||
double (*ping)(int af, const char *target);
|
||||
/* fills out with the default gateway, "" when unknown */
|
||||
void (*gateway)(int af, char *out, size_t n);
|
||||
} probe_ops_t;
|
||||
|
||||
typedef struct {
|
||||
probe_fam_t fam[2]; /* [0] = IPv4, [1] = IPv6 */
|
||||
char target[2][64];
|
||||
double probe_interval, retry_interval;
|
||||
bool cheap_gateway; /* gateway read is cheap (/proc), so read every tier 1 */
|
||||
pthread_mutex_t mu; /* guards fam[] between the probe thread and readers */
|
||||
} probe_t;
|
||||
|
||||
/* Lowest-metric default gateway from /proc/net/route. The kernel prints the raw network-order
|
||||
* u32 in host order, so it can be stored straight into s_addr on any endianness. */
|
||||
static void gw_parse_proc_v4(const char *text, char *out, size_t n) {
|
||||
long best = -1; out[0] = '\0';
|
||||
const char *p = strchr(text, '\n'); /* skip the header line */
|
||||
while (p && *++p) {
|
||||
char iface[64], dest[16], gw[16]; unsigned flags; int refcnt, use, metric;
|
||||
if (sscanf(p, "%63s %15s %15s %x %d %d %d", iface, dest, gw, &flags, &refcnt, &use, &metric) == 7
|
||||
&& !strcmp(dest, "00000000") && (flags & 0x2)) {
|
||||
if (best < 0 || metric < best) {
|
||||
struct in_addr a; a.s_addr = (uint32_t)strtoul(gw, NULL, 16);
|
||||
if (inet_ntop(AF_INET, &a, out, n)) best = metric;
|
||||
}
|
||||
}
|
||||
p = strchr(p, '\n');
|
||||
}
|
||||
}
|
||||
|
||||
/* Lowest-metric default gateway from /proc/net/ipv6_route (skips RTF_REJECT and "lo"). */
|
||||
static void gw_parse_proc_v6(const char *text, char *out, size_t n) {
|
||||
long best = -1; out[0] = '\0';
|
||||
const char *zero = "00000000000000000000000000000000";
|
||||
for (const char *p = text; p && *p; ) {
|
||||
char dest[40], src[40], nh[40], iface[64];
|
||||
unsigned plen, splen, metric, refcnt, use, flags;
|
||||
if (sscanf(p, "%32s %x %32s %x %32s %x %x %x %x %63s",
|
||||
dest, &plen, src, &splen, nh, &metric, &refcnt, &use, &flags, iface) == 10
|
||||
&& !strcmp(dest, zero) && plen == 0 && strcmp(nh, zero) && strcmp(iface, "lo")
|
||||
&& (flags & 0x2) && !(flags & 0x200)) {
|
||||
if (best < 0 || (long)metric < best) {
|
||||
struct in6_addr a; memset(&a, 0, sizeof(a));
|
||||
bool ok = strlen(nh) == 32;
|
||||
for (int i = 0; ok && i < 16; i++) {
|
||||
unsigned b;
|
||||
if (sscanf(nh + 2 * i, "%2x", &b) != 1) ok = false; else a.s6_addr[i] = (uint8_t)b;
|
||||
}
|
||||
if (ok && inet_ntop(AF_INET6, &a, out, n)) best = (long)metric;
|
||||
}
|
||||
}
|
||||
p = strchr(p, '\n'); if (p) p++;
|
||||
}
|
||||
}
|
||||
|
||||
/* Gateway from `route -n get [-inet6] default` (the BSDs). */
|
||||
static void gw_parse_route_get(const char *text, char *out, size_t n) {
|
||||
out[0] = '\0';
|
||||
const char *p = strstr(text, "gateway:");
|
||||
if (!p) return;
|
||||
p += 8;
|
||||
while (*p == ' ' || *p == '\t') p++;
|
||||
size_t i = 0;
|
||||
while (*p && !isspace((unsigned char)*p) && i + 1 < n) out[i++] = *p++;
|
||||
out[i] = '\0';
|
||||
}
|
||||
|
||||
/* Average RTT from ping output: -1.0 no reply, -2.0 ping could not be run. Works for Linux
|
||||
* ("rtt min/avg/max/mdev") and BSD ("round-trip min/avg/max/stddev"). */
|
||||
static double ping_parse_rtt(const char *out) {
|
||||
if (strstr(out, "not found") || strstr(out, "No such file")) return -2.0;
|
||||
const char *p = strstr(out, "min/avg/max");
|
||||
if (!p) return -1.0;
|
||||
p = strchr(p, '=');
|
||||
double mn, avg, mx;
|
||||
if (p && sscanf(p + 1, " %lf/%lf/%lf", &mn, &avg, &mx) == 3) return avg;
|
||||
return -1.0;
|
||||
}
|
||||
|
||||
/* One probe step for one family (f: 0 = IPv4, 1 = IPv6). Works on *s only, so the probe thread
|
||||
* can run it on a private copy without holding the mutex during the (blocking) echo. */
|
||||
static void probe_step(const probe_t *p, probe_fam_t *s, int f, const probe_ops_t *ops, double now) {
|
||||
int af = f ? AF_INET6 : AF_INET;
|
||||
char addr[INET6_ADDRSTRLEN] = "", gw[INET6_ADDRSTRLEN] = "";
|
||||
|
||||
int r = ops->local_address(af, p->target[f], addr, sizeof(addr));
|
||||
if (r < 0) { s->present = PROBE_UNKNOWN; return; }
|
||||
if (r == 0) { /* tier 1: no route => absent immediately */
|
||||
s->fails = 0; s->next_ping = 0; s->last_key[0] = '\0';
|
||||
s->present = 0; s->addr[0] = '\0'; s->gateway[0] = '\0'; s->rtt = -1;
|
||||
return;
|
||||
}
|
||||
|
||||
if (p->cheap_gateway) ops->gateway(af, gw, sizeof(gw));
|
||||
else snprintf(gw, sizeof(gw), "%s", s->gateway);
|
||||
char key[sizeof(s->last_key)];
|
||||
snprintf(key, sizeof(key), "%s|%s", addr, gw);
|
||||
bool due = strcmp(key, s->last_key) != 0 || now >= s->next_ping;
|
||||
if (due && !p->cheap_gateway) {
|
||||
ops->gateway(af, gw, sizeof(gw));
|
||||
snprintf(key, sizeof(key), "%s|%s", addr, gw);
|
||||
}
|
||||
snprintf(s->addr, sizeof(s->addr), "%s", addr);
|
||||
snprintf(s->gateway, sizeof(s->gateway), "%s", gw);
|
||||
snprintf(s->last_key, sizeof(s->last_key), "%s", key);
|
||||
|
||||
if (s->ping_disabled) { s->present = PROBE_UNKNOWN; return; } /* cannot probe: unknown, never absent, while a route exists */
|
||||
if (!due) return;
|
||||
double rtt = ops->ping(af, p->target[f]);
|
||||
if (rtt < -1.5) { s->ping_disabled = true; s->present = PROBE_UNKNOWN; return; }
|
||||
if (rtt < 0) {
|
||||
s->fails++;
|
||||
s->next_ping = now + p->retry_interval;
|
||||
if (s->fails >= PROBE_FAIL_LIMIT) { s->present = 0; s->rtt = -1; }
|
||||
} else {
|
||||
s->fails = 0;
|
||||
s->next_ping = now + p->probe_interval;
|
||||
s->present = 1; s->rtt = rtt;
|
||||
}
|
||||
}
|
||||
|
||||
/* HTB flags for the families whose presence is known. */
|
||||
static void probe_flags(probe_t *p, kvdict_t *d) {
|
||||
pthread_mutex_lock(&p->mu);
|
||||
if (p->fam[0].present >= 0) kv_set_int(d, "p4", p->fam[0].present);
|
||||
if (p->fam[1].present >= 0) kv_set_int(d, "p6", p->fam[1].present);
|
||||
pthread_mutex_unlock(&p->mu);
|
||||
}
|
||||
|
||||
/* The `connectivity` metric as a JSON object; returns its length or -1 on overflow. */
|
||||
static int probe_metric_json(probe_t *p, char *out, size_t n) {
|
||||
static const char *names[2] = { "ipv4", "ipv6" };
|
||||
size_t len = 0;
|
||||
len += (size_t)snprintf(out + len, n - len, "{");
|
||||
pthread_mutex_lock(&p->mu);
|
||||
for (int f = 0; f < 2 && len < n; f++) {
|
||||
const probe_fam_t *s = &p->fam[f];
|
||||
char pres[8], addr[64], gw[64], rtt[32];
|
||||
snprintf(pres, sizeof(pres), "%s", s->present < 0 ? "null" : s->present ? "true" : "false");
|
||||
if (s->addr[0]) snprintf(addr, sizeof(addr), "\"%s\"", s->addr); else snprintf(addr, sizeof(addr), "null");
|
||||
if (s->gateway[0]) snprintf(gw, sizeof(gw), "\"%s\"", s->gateway); else snprintf(gw, sizeof(gw), "null");
|
||||
if (s->rtt >= 0) snprintf(rtt, sizeof(rtt), "%.3f", s->rtt); else snprintf(rtt, sizeof(rtt), "null");
|
||||
len += (size_t)snprintf(out + len, n - len,
|
||||
"%s\"%s\":{\"present\":%s,\"addr\":%s,\"gateway\":%s,\"rtt\":%s}",
|
||||
f ? "," : "", names[f], pres, addr, gw, rtt);
|
||||
}
|
||||
pthread_mutex_unlock(&p->mu);
|
||||
if (len + 2 > n) { out[0] = '\0'; return -1; }
|
||||
out[len++] = '}'; out[len] = '\0';
|
||||
return (int)len;
|
||||
}
|
||||
|
||||
static double mono_ts(void) {
|
||||
struct timespec ts;
|
||||
clock_gettime(CLOCK_MONOTONIC, &ts);
|
||||
return ts.tv_sec + ts.tv_nsec * 1e-9;
|
||||
}
|
||||
|
||||
/* Source address the kernel would use to reach target (an IP literal). A UDP connect() sends
|
||||
* nothing. Returns 1 = route (address in out), 0 = no route, -1 = family unsupported or the
|
||||
* target is not an IP literal (unknown, never absent). */
|
||||
static int sys_local_address(int af, const char *target, char *out, size_t n) {
|
||||
int fd = socket(af, SOCK_DGRAM, 0);
|
||||
if (fd < 0) return -1;
|
||||
struct sockaddr_storage sa; socklen_t sl;
|
||||
memset(&sa, 0, sizeof(sa));
|
||||
if (af == AF_INET) {
|
||||
struct sockaddr_in *s4 = (struct sockaddr_in *)&sa;
|
||||
s4->sin_family = AF_INET; s4->sin_port = htons(9);
|
||||
if (inet_pton(AF_INET, target, &s4->sin_addr) != 1) { close(fd); return -1; }
|
||||
sl = sizeof(*s4);
|
||||
} else {
|
||||
struct sockaddr_in6 *s6 = (struct sockaddr_in6 *)&sa;
|
||||
s6->sin6_family = AF_INET6; s6->sin6_port = htons(9);
|
||||
if (inet_pton(AF_INET6, target, &s6->sin6_addr) != 1) { close(fd); return -1; }
|
||||
sl = sizeof(*s6);
|
||||
}
|
||||
int rc = 0;
|
||||
if (connect(fd, (struct sockaddr *)&sa, sl) == 0) {
|
||||
struct sockaddr_storage me; socklen_t ml = sizeof(me);
|
||||
if (getsockname(fd, (struct sockaddr *)&me, &ml) == 0) {
|
||||
const void *src = (af == AF_INET)
|
||||
? (const void *)&((struct sockaddr_in *)&me)->sin_addr
|
||||
: (const void *)&((struct sockaddr_in6 *)&me)->sin6_addr;
|
||||
if (inet_ntop(af, src, out, (socklen_t)n)) rc = 1;
|
||||
}
|
||||
}
|
||||
close(fd);
|
||||
return rc;
|
||||
}
|
||||
|
||||
static char *slurp(FILE *f) {
|
||||
size_t cap = 4096, len = 0;
|
||||
char *buf = malloc(cap);
|
||||
if (!buf) return NULL;
|
||||
size_t r;
|
||||
while ((r = fread(buf + len, 1, cap - len - 1, f)) > 0) {
|
||||
len += r;
|
||||
if (len + 1 >= cap) {
|
||||
if (cap >= 65536) break;
|
||||
cap *= 2;
|
||||
char *nb = realloc(buf, cap);
|
||||
if (!nb) { free(buf); return NULL; }
|
||||
buf = nb;
|
||||
}
|
||||
}
|
||||
buf[len] = '\0';
|
||||
return buf;
|
||||
}
|
||||
|
||||
static bool g_have_timeout = false; /* timeout(1) exists: bounds ping6 on the BSDs */
|
||||
|
||||
/* One echo request: RTT in ms, -1.0 no reply, -2.0 ping could not be run. The target is an IP
|
||||
* literal from the config; anything else is refused before it can reach the shell. */
|
||||
static double sys_ping(int af, const char *target) {
|
||||
for (const char *c = target; *c; c++)
|
||||
if (!(isxdigit((unsigned char)*c) || *c == '.' || *c == ':')) return -2.0;
|
||||
char cmd[256];
|
||||
#ifdef __linux__
|
||||
snprintf(cmd, sizeof(cmd), "ping -c 1 -W 2 -w 3 %s %s 2>&1", af == AF_INET ? "-4" : "-6", target);
|
||||
#else
|
||||
/* BSD ping: -W is in milliseconds, and IPv6 is reached through ping6 */
|
||||
if (af == AF_INET)
|
||||
snprintf(cmd, sizeof(cmd), "%sping -c 1 -W 2000 %s 2>&1", g_have_timeout ? "timeout 4 " : "", target);
|
||||
else
|
||||
snprintf(cmd, sizeof(cmd), "%sping6 -c 1 %s 2>&1", g_have_timeout ? "timeout 4 " : "", target);
|
||||
#endif
|
||||
FILE *f = popen(cmd, "r");
|
||||
if (!f) return -2.0;
|
||||
char *out = slurp(f);
|
||||
pclose(f);
|
||||
if (!out) return -1.0;
|
||||
double rtt = ping_parse_rtt(out);
|
||||
free(out);
|
||||
return rtt;
|
||||
}
|
||||
|
||||
/* Default gateway for the family, "" when unknown. Best effort; never affects presence. */
|
||||
static void sys_gateway(int af, char *out, size_t n) {
|
||||
out[0] = '\0';
|
||||
#ifdef __linux__
|
||||
FILE *f = fopen(af == AF_INET ? "/proc/net/route" : "/proc/net/ipv6_route", "r");
|
||||
if (!f) return;
|
||||
char *text = slurp(f);
|
||||
fclose(f);
|
||||
if (!text) return;
|
||||
if (af == AF_INET) gw_parse_proc_v4(text, out, n); else gw_parse_proc_v6(text, out, n);
|
||||
free(text);
|
||||
#else
|
||||
FILE *f = popen(af == AF_INET ? "route -n get default 2>/dev/null"
|
||||
: "route -n get -inet6 default 2>/dev/null", "r");
|
||||
if (!f) return;
|
||||
char *text = slurp(f);
|
||||
pclose(f);
|
||||
if (!text) return;
|
||||
gw_parse_route_get(text, out, n);
|
||||
free(text);
|
||||
#endif
|
||||
}
|
||||
|
||||
static probe_t g_probe;
|
||||
static bool g_probe_enabled = false;
|
||||
|
||||
static void probe_init(probe_t *p, const config_t *cfg) {
|
||||
memset(p, 0, sizeof(*p));
|
||||
snprintf(p->target[0], sizeof(p->target[0]), "%s", cfg->probe_v4);
|
||||
snprintf(p->target[1], sizeof(p->target[1]), "%s", cfg->probe_v6);
|
||||
p->probe_interval = cfg->probe_interval > 0 ? cfg->probe_interval : 60;
|
||||
p->retry_interval = 10;
|
||||
#ifdef __linux__
|
||||
p->cheap_gateway = true; /* /proc read; elsewhere a subprocess only when due */
|
||||
#endif
|
||||
p->fam[0].present = p->fam[1].present = PROBE_UNKNOWN;
|
||||
p->fam[0].rtt = p->fam[1].rtt = -1;
|
||||
pthread_mutex_init(&p->mu, NULL);
|
||||
g_have_timeout = system("command -v timeout >/dev/null 2>&1") == 0;
|
||||
g_probe_enabled = cfg->probe_enabled;
|
||||
|
||||
/* Diagnostic only: sys_local_address already turns a non-literal target into PROBE_UNKNOWN
|
||||
* (via inet_pton failing there), so this does not change behavior — it just stops that from
|
||||
* being silent. */
|
||||
struct in_addr a4;
|
||||
struct in6_addr a6;
|
||||
if (inet_pton(AF_INET, p->target[0], &a4) != 1)
|
||||
LOGW("probe_v4 is not an IPv4 literal: %s", p->target[0]);
|
||||
if (inet_pton(AF_INET6, p->target[1], &a6) != 1)
|
||||
LOGW("probe_v6 is not an IPv6 literal: %s", p->target[1]);
|
||||
}
|
||||
|
||||
static void *probe_thread(void *arg) {
|
||||
probe_t *p = arg;
|
||||
static const probe_ops_t ops = { sys_local_address, sys_ping, sys_gateway };
|
||||
while (g_running) {
|
||||
double now = mono_ts();
|
||||
for (int f = 0; f < 2 && g_running; f++) {
|
||||
pthread_mutex_lock(&p->mu); probe_fam_t tmp = p->fam[f]; pthread_mutex_unlock(&p->mu);
|
||||
probe_step(p, &tmp, f, &ops, now); /* may block in ping: no lock held */
|
||||
pthread_mutex_lock(&p->mu); p->fam[f] = tmp; pthread_mutex_unlock(&p->mu);
|
||||
}
|
||||
for (int i = 0; i < PROBE_TIER1 && g_running; i++) sleep(1);
|
||||
}
|
||||
return NULL;
|
||||
}
|
||||
|
||||
/* ============================================================
|
||||
* Plugin: network_monitor
|
||||
* Linux: /proc/net/dev
|
||||
@@ -1077,16 +1430,26 @@ static void plugin_network_monitor(conn_t *c, const config_t *cfg) {
|
||||
json[jlen++] = '}'; json[jlen] = '\0';
|
||||
memcpy(g_net_prev, curr, cn * sizeof(net_stat_t));
|
||||
g_net_prev_n = cn; g_net_prev_ts = now;
|
||||
if (first) { free(json); return; }
|
||||
|
||||
kvdict_t d; kv_clear(&d);
|
||||
kv_set(&d, "plugin", "network_monitor");
|
||||
char *jval = malloc(MAX_VAL + 1);
|
||||
if (jval) { snprintf(jval, MAX_VAL, "@%s", json); kv_set(&d, "interfaces", jval); free(jval); }
|
||||
if (!first) {
|
||||
char *jval = malloc(MAX_VAL + 1);
|
||||
if (jval) { snprintf(jval, MAX_VAL, "@%s", json); kv_set(&d, "interfaces", jval); free(jval); }
|
||||
}
|
||||
bool have_conn = false;
|
||||
if (g_probe_enabled) {
|
||||
char cj[512];
|
||||
cj[0] = '@';
|
||||
if (probe_metric_json(&g_probe, cj + 1, sizeof(cj) - 1) > 0) {
|
||||
kv_set(&d, "connectivity", cj);
|
||||
have_conn = true;
|
||||
}
|
||||
}
|
||||
free(json);
|
||||
if (first && !have_conn) return;
|
||||
kv_set_dbl(&d, "_timestamp", now_ts());
|
||||
kv_set_int(&d, "_interval", cfg->net_interval);
|
||||
conn_send(c, "PLG", &d);
|
||||
free(json);
|
||||
LOGD("sent network_monitor");
|
||||
}
|
||||
|
||||
@@ -1323,6 +1686,7 @@ int main(int argc, char **argv) {
|
||||
|
||||
config_t cfg;
|
||||
config_load(&cfg, cfgpath);
|
||||
probe_init(&g_probe, &cfg);
|
||||
|
||||
if (do_daemon) {
|
||||
daemonize();
|
||||
@@ -1405,6 +1769,13 @@ int main(int argc, char **argv) {
|
||||
pthread_t tid;
|
||||
pthread_create(&tid, NULL, monitor_thread, &tctx);
|
||||
|
||||
pthread_t ptid; bool probe_running = false;
|
||||
if (g_probe_enabled) {
|
||||
int prc = pthread_create(&ptid, NULL, probe_thread, &g_probe);
|
||||
if (prc == 0) probe_running = true;
|
||||
else LOGW("pthread_create(probe): %s", strerror(prc));
|
||||
}
|
||||
|
||||
/* Main heartbeat + receive loop */
|
||||
time_t last_hb = 0;
|
||||
while (g_running) {
|
||||
@@ -1415,6 +1786,7 @@ int main(int argc, char **argv) {
|
||||
kv_set_int(&hb, "acks", g_conns[i].ack_count);
|
||||
kv_set_dbl(&hb, "rtt", g_conns[i].rtt);
|
||||
kv_set_int(&hb, "interval", cfg.interval);
|
||||
if (g_probe_enabled) probe_flags(&g_probe, &hb);
|
||||
conn_send(&g_conns[i], "HTB", &hb);
|
||||
}
|
||||
last_hb = now;
|
||||
@@ -1449,6 +1821,10 @@ int main(int argc, char **argv) {
|
||||
conn_send(primary, "HTB", &sh);
|
||||
usleep(300000);
|
||||
}
|
||||
/* Joined after the shutdown HTB goes out: the probe thread is never cancelled (it can be
|
||||
* inside popen), so joining it can block for a few seconds — long enough to lose the
|
||||
* clean-shutdown notification to an impatient supervisor's SIGKILL. */
|
||||
if (probe_running) pthread_join(ptid, NULL);
|
||||
for (int i = 0; i < g_nconns; i++) conn_close(&g_conns[i]);
|
||||
if (g_use_syslog) closelog();
|
||||
if (g_restart) { LOGI("restarting..."); execv(argv[0], argv); }
|
||||
|
||||
@@ -0,0 +1,243 @@
|
||||
/* Unit tests for the connectivity-probe helpers in hbc_mini.c.
|
||||
* Build and run: make test (live check of this host: ./test_probe --live)
|
||||
* The source file is included directly so its static functions can be exercised; its main()
|
||||
* is renamed out of the way.
|
||||
*/
|
||||
#define main hbc_mini_main
|
||||
#include "hbc_mini.c"
|
||||
#undef main
|
||||
|
||||
static int g_run = 0, g_fail = 0;
|
||||
#define CHECK(cond) do { g_run++; if (!(cond)) { g_fail++; \
|
||||
fprintf(stderr, "FAIL %s:%d: %s\n", __FILE__, __LINE__, #cond); } } while (0)
|
||||
#define CHECK_STR(a, b) do { g_run++; if (strcmp((a), (b)) != 0) { g_fail++; \
|
||||
fprintf(stderr, "FAIL %s:%d: \"%s\" != \"%s\"\n", __FILE__, __LINE__, (a), (b)); } } while (0)
|
||||
|
||||
static const char PROC_V4[] =
|
||||
"Iface\tDestination\tGateway \tFlags\tRefCnt\tUse\tMetric\tMask\t\tMTU\tWindow\tIRTT\n"
|
||||
"wlan0\t00000000\t0100A8C0\t0003\t0\t0\t600\t00000000\t0\t0\t0\n"
|
||||
"eth0\t00000000\t0101A8C0\t0003\t0\t0\t100\t00000000\t0\t0\t0\n"
|
||||
"wlan0\t0000A8C0\t00000000\t0001\t0\t0\t600\t00FFFFFF\t0\t0\t0\n";
|
||||
|
||||
static const char PROC_V6[] =
|
||||
"00000000000000000000000000000000 00 00000000000000000000000000000000 00 "
|
||||
"fe800000000000000000000000000001 00000400 00000001 00000000 00000003 wlan0\n"
|
||||
"00000000000000000000000000000000 00 00000000000000000000000000000000 00 "
|
||||
"00000000000000000000000000000000 ffffffff 00000001 00000000 00200200 lo\n"
|
||||
"fe800000000000000000000000000000 40 00000000000000000000000000000000 00 "
|
||||
"00000000000000000000000000000000 00000100 00000001 00000000 00000001 wlan0\n";
|
||||
|
||||
static const char ROUTE_GET[] =
|
||||
" route to: 0.0.0.0\ndestination: 0.0.0.0\n mask: 0.0.0.0\n"
|
||||
" gateway: 192.0.2.1\n fib: 0\n interface: vtnet0\n";
|
||||
|
||||
static const char LINUX_PING[] =
|
||||
"PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.\n"
|
||||
"64 bytes from 1.1.1.1: icmp_seq=1 ttl=57 time=11.2 ms\n\n"
|
||||
"--- 1.1.1.1 ping statistics ---\n"
|
||||
"1 packets transmitted, 1 received, 0% packet loss, time 0ms\n"
|
||||
"rtt min/avg/max/mdev = 11.234/11.234/11.234/0.000 ms\n";
|
||||
|
||||
static const char FREEBSD_PING[] =
|
||||
"PING 1.1.1.1 (1.1.1.1): 56 data bytes\n"
|
||||
"64 bytes from 1.1.1.1: icmp_seq=0 ttl=59 time=4.006 ms\n\n"
|
||||
"--- 1.1.1.1 ping statistics ---\n"
|
||||
"1 packets transmitted, 1 packets received, 0.0% packet loss, 1 packets out of wait time\n"
|
||||
"round-trip min/avg/max/stddev = 4.006/4.006/4.006/0.000 ms\n";
|
||||
|
||||
static void test_gateway_parsers(void) {
|
||||
char gw[64];
|
||||
gw_parse_proc_v4(PROC_V4, gw, sizeof(gw));
|
||||
CHECK_STR(gw, "192.168.1.1"); /* eth0, metric 100 beats wlan0, 600 */
|
||||
gw_parse_proc_v4("Iface\tDestination\n", gw, sizeof(gw));
|
||||
CHECK_STR(gw, "");
|
||||
gw_parse_proc_v6(PROC_V6, gw, sizeof(gw));
|
||||
CHECK_STR(gw, "fe80::1"); /* reject and non-default lines skipped */
|
||||
gw_parse_route_get(ROUTE_GET, gw, sizeof(gw));
|
||||
CHECK_STR(gw, "192.0.2.1");
|
||||
gw_parse_route_get("route: writing to routing socket: not in table\n", gw, sizeof(gw));
|
||||
CHECK_STR(gw, "");
|
||||
}
|
||||
|
||||
static void test_ping_parser(void) {
|
||||
CHECK(ping_parse_rtt(LINUX_PING) > 11.23 && ping_parse_rtt(LINUX_PING) < 11.24);
|
||||
CHECK(ping_parse_rtt(FREEBSD_PING) > 4.0 && ping_parse_rtt(FREEBSD_PING) < 4.01);
|
||||
CHECK(ping_parse_rtt("1 packets transmitted, 0 received, 100% packet loss\n") == -1.0);
|
||||
CHECK(ping_parse_rtt("sh: 1: ping: not found\n") == -2.0);
|
||||
CHECK(ping_parse_rtt("ping: command not found\n") == -2.0);
|
||||
}
|
||||
|
||||
/* ---- scriptable network for probe_step ------------------------------------------------ */
|
||||
static struct {
|
||||
char addr[2][INET6_ADDRSTRLEN]; /* "" = no route */
|
||||
char gw[2][INET6_ADDRSTRLEN];
|
||||
double rtt[2]; /* -1 no reply, -2 ping unavailable */
|
||||
int unsupported[2];
|
||||
int npings[2];
|
||||
} net;
|
||||
|
||||
static int fake_local(int af, const char *t, char *out, size_t n) {
|
||||
int f = af == AF_INET6; (void)t;
|
||||
if (net.unsupported[f]) return -1;
|
||||
if (!net.addr[f][0]) return 0;
|
||||
snprintf(out, n, "%s", net.addr[f]);
|
||||
return 1;
|
||||
}
|
||||
static double fake_ping(int af, const char *t) { int f = af == AF_INET6; (void)t; net.npings[f]++; return net.rtt[f]; }
|
||||
static void fake_gw(int af, char *out, size_t n) { snprintf(out, n, "%s", net.gw[af == AF_INET6]); }
|
||||
static const probe_ops_t FAKE = { fake_local, fake_ping, fake_gw };
|
||||
|
||||
static void net_reset(void) {
|
||||
memset(&net, 0, sizeof(net));
|
||||
snprintf(net.addr[0], INET6_ADDRSTRLEN, "192.0.2.5");
|
||||
snprintf(net.addr[1], INET6_ADDRSTRLEN, "2001:db8::5");
|
||||
snprintf(net.gw[0], INET6_ADDRSTRLEN, "192.0.2.1");
|
||||
snprintf(net.gw[1], INET6_ADDRSTRLEN, "fe80::1");
|
||||
net.rtt[0] = 10.0; net.rtt[1] = 20.0;
|
||||
}
|
||||
|
||||
static void probe_reset(probe_t *p) {
|
||||
memset(p, 0, sizeof(*p));
|
||||
snprintf(p->target[0], sizeof(p->target[0]), "1.1.1.1");
|
||||
snprintf(p->target[1], sizeof(p->target[1]), "2606:4700:4700::1111");
|
||||
p->probe_interval = 60; p->retry_interval = 10; p->cheap_gateway = true;
|
||||
p->fam[0].present = p->fam[1].present = PROBE_UNKNOWN;
|
||||
p->fam[0].rtt = p->fam[1].rtt = -1;
|
||||
pthread_mutex_init(&p->mu, NULL);
|
||||
}
|
||||
|
||||
static void step_both(probe_t *p, double now) {
|
||||
probe_step(p, &p->fam[0], 0, &FAKE, now);
|
||||
probe_step(p, &p->fam[1], 1, &FAKE, now);
|
||||
}
|
||||
|
||||
static void test_present_with_details(void) {
|
||||
net_reset(); probe_t p; probe_reset(&p);
|
||||
step_both(&p, 1000);
|
||||
CHECK(p.fam[0].present == 1 && p.fam[1].present == 1);
|
||||
CHECK_STR(p.fam[0].addr, "192.0.2.5");
|
||||
CHECK_STR(p.fam[0].gateway, "192.0.2.1");
|
||||
CHECK(p.fam[0].rtt == 10.0);
|
||||
kvdict_t d; kv_clear(&d);
|
||||
probe_flags(&p, &d);
|
||||
CHECK_STR(kv_get(&d, "p4") ? kv_get(&d, "p4") : "(null)", "1");
|
||||
CHECK_STR(kv_get(&d, "p6") ? kv_get(&d, "p6") : "(null)", "1");
|
||||
char js[512];
|
||||
CHECK(probe_metric_json(&p, js, sizeof(js)) > 0);
|
||||
CHECK(strstr(js, "\"ipv6\":{\"present\":true,\"addr\":\"2001:db8::5\",\"gateway\":\"fe80::1\"") != NULL);
|
||||
}
|
||||
|
||||
static void test_no_route_is_absent_and_wastes_no_ping(void) {
|
||||
net_reset(); net.addr[1][0] = '\0'; probe_t p; probe_reset(&p);
|
||||
step_both(&p, 1000);
|
||||
CHECK(p.fam[1].present == 0 && net.npings[1] == 0 && net.npings[0] == 1);
|
||||
kvdict_t d; kv_clear(&d);
|
||||
probe_flags(&p, &d);
|
||||
CHECK_STR(kv_get(&d, "p6") ? kv_get(&d, "p6") : "(null)", "0");
|
||||
char js[512];
|
||||
probe_metric_json(&p, js, sizeof(js));
|
||||
CHECK(strstr(js, "\"ipv6\":{\"present\":false,\"addr\":null,\"gateway\":null,\"rtt\":null}") != NULL);
|
||||
}
|
||||
|
||||
static void test_debounce_and_rate_limit(void) {
|
||||
net_reset(); probe_t p; probe_reset(&p);
|
||||
step_both(&p, 1000);
|
||||
step_both(&p, 1000); /* cached: no second ping */
|
||||
CHECK(net.npings[0] == 1 && net.npings[1] == 1);
|
||||
net.rtt[1] = -1;
|
||||
step_both(&p, 1060); /* first failure: still present */
|
||||
CHECK(p.fam[1].present == 1);
|
||||
step_both(&p, 1070); /* retry interval elapsed: flips */
|
||||
CHECK(p.fam[1].present == 0);
|
||||
net.rtt[1] = 25;
|
||||
step_both(&p, 1080); /* first success recovers */
|
||||
CHECK(p.fam[1].present == 1);
|
||||
}
|
||||
|
||||
static void test_first_failure_is_unknown_and_omitted(void) {
|
||||
net_reset(); net.rtt[0] = -1; probe_t p; probe_reset(&p);
|
||||
step_both(&p, 1000);
|
||||
CHECK(p.fam[0].present == PROBE_UNKNOWN);
|
||||
kvdict_t d; kv_clear(&d);
|
||||
probe_flags(&p, &d);
|
||||
CHECK(kv_get(&d, "p4") == NULL && kv_get(&d, "p6") != NULL);
|
||||
char js[512];
|
||||
probe_metric_json(&p, js, sizeof(js));
|
||||
CHECK(strstr(js, "\"ipv4\":{\"present\":null") != NULL);
|
||||
}
|
||||
|
||||
static void test_address_or_gateway_change_repings(void) {
|
||||
net_reset(); probe_t p; probe_reset(&p);
|
||||
step_both(&p, 1000);
|
||||
snprintf(net.addr[0], INET6_ADDRSTRLEN, "198.51.100.7");
|
||||
step_both(&p, 1000);
|
||||
CHECK(net.npings[0] == 2);
|
||||
snprintf(net.gw[0], INET6_ADDRSTRLEN, "192.0.2.254");
|
||||
step_both(&p, 1000);
|
||||
CHECK(net.npings[0] == 3);
|
||||
}
|
||||
|
||||
static void test_route_loss_then_return_pings_immediately(void) {
|
||||
net_reset(); probe_t p; probe_reset(&p);
|
||||
step_both(&p, 1000);
|
||||
net.addr[1][0] = '\0';
|
||||
step_both(&p, 1000);
|
||||
CHECK(p.fam[1].present == 0);
|
||||
snprintf(net.addr[1], INET6_ADDRSTRLEN, "2001:db8::5");
|
||||
step_both(&p, 1000);
|
||||
CHECK(net.npings[1] == 2 && p.fam[1].present == 1);
|
||||
}
|
||||
|
||||
static void test_missing_ping_stays_unknown_even_after_route_loss(void) {
|
||||
net_reset(); net.rtt[0] = -2; probe_t p; probe_reset(&p);
|
||||
step_both(&p, 1000);
|
||||
step_both(&p, 1600);
|
||||
CHECK(p.fam[0].present == PROBE_UNKNOWN && net.npings[0] == 1);
|
||||
net.addr[0][0] = '\0';
|
||||
step_both(&p, 1700);
|
||||
CHECK(p.fam[0].present == 0); /* no route is definitive */
|
||||
snprintf(net.addr[0], INET6_ADDRSTRLEN, "192.0.2.5");
|
||||
step_both(&p, 1800);
|
||||
CHECK(p.fam[0].present == PROBE_UNKNOWN && net.npings[0] == 1);
|
||||
}
|
||||
|
||||
static void test_unsupported_family_is_unknown(void) {
|
||||
net_reset(); net.unsupported[1] = 1; probe_t p; probe_reset(&p);
|
||||
step_both(&p, 1000);
|
||||
CHECK(p.fam[1].present == PROBE_UNKNOWN && p.fam[0].present == 1);
|
||||
}
|
||||
|
||||
static int live_mode(void) {
|
||||
static const probe_ops_t sys_ops = { sys_local_address, sys_ping, sys_gateway };
|
||||
probe_t p; probe_reset(&p);
|
||||
p.cheap_gateway = false;
|
||||
snprintf(p.target[0], sizeof(p.target[0]), "1.1.1.1");
|
||||
for (int round = 0; round < 3; round++) {
|
||||
double now = mono_ts();
|
||||
for (int f = 0; f < 2; f++) probe_step(&p, &p.fam[f], f, &sys_ops, now);
|
||||
sleep(round < 2 ? 11 : 0);
|
||||
}
|
||||
char js[512];
|
||||
probe_metric_json(&p, js, sizeof(js));
|
||||
kvdict_t d; kv_clear(&d);
|
||||
probe_flags(&p, &d);
|
||||
printf("%s\np4=%s p6=%s\n", js, kv_get(&d, "p4") ? kv_get(&d, "p4") : "-",
|
||||
kv_get(&d, "p6") ? kv_get(&d, "p6") : "-");
|
||||
return 0;
|
||||
}
|
||||
|
||||
int main(int argc, char **argv) {
|
||||
if (argc > 1 && !strcmp(argv[1], "--live")) return live_mode();
|
||||
test_gateway_parsers();
|
||||
test_ping_parser();
|
||||
test_present_with_details();
|
||||
test_no_route_is_absent_and_wastes_no_ping();
|
||||
test_debounce_and_rate_limit();
|
||||
test_first_failure_is_unknown_and_omitted();
|
||||
test_address_or_gateway_change_repings();
|
||||
test_route_loss_then_return_pings_immediately();
|
||||
test_missing_ping_stays_unknown_even_after_route_loss();
|
||||
test_unsupported_family_is_unknown();
|
||||
printf("%d checks, %d failed\n", g_run, g_fail);
|
||||
return g_fail ? 1 : 0;
|
||||
}
|
||||
+353
-19
@@ -10,14 +10,20 @@ Built-in plugins (always available):
|
||||
os_info — OS name, kernel, distro, Python version
|
||||
ping_monitor — ICMP ping RTT to configured hosts
|
||||
nagios_runner — run Nagios-compatible check scripts
|
||||
network_monitor — per-interface TX/RX bytes/s (Linux); per-protocol connectivity (all platforms)
|
||||
|
||||
Linux-only plugins (via /proc, no extra tools needed):
|
||||
cpu_monitor — CPU %, load average, core count
|
||||
memory_monitor — RAM and swap usage
|
||||
network_monitor — per-interface TX/RX bytes/s
|
||||
|
||||
Cross-platform (via df subprocess):
|
||||
disk_monitor — per-mount usage and capacity
|
||||
|
||||
Presence probe (built in): reports per protocol (IPv4/IPv6) whether the host has internet
|
||||
connectivity, as p4/p6 flags on every heartbeat, so hbd stops treating a missing IPv6 (or
|
||||
IPv4) path as an outage. Config, in the network_monitor section of ~/.hbc.json:
|
||||
probe_enabled (default true), probe_v4 (default 1.1.1.1), probe_v6 (default
|
||||
2606:4700:4700::1111; IP literals only), probe_interval (default 60 s)
|
||||
"""
|
||||
|
||||
import argparse
|
||||
@@ -36,12 +42,13 @@ import time
|
||||
import zlib
|
||||
from abc import ABC, abstractmethod
|
||||
from collections import defaultdict
|
||||
from dataclasses import asdict, dataclass
|
||||
from logging.handlers import SysLogHandler
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List, Optional, Tuple
|
||||
from typing import Any, Awaitable, Callable, Dict, List, Optional, Set, Tuple
|
||||
|
||||
# updated by scripts/bumpminor.sh
|
||||
__version__ = "5.4.0"
|
||||
__version__ = "5.4.5"
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Protocol (mirrors hbd/common/proto.py)
|
||||
@@ -637,6 +644,315 @@ class DiskMonitorPlugin(MonitorPlugin):
|
||||
return {"partitions": partitions} if partitions else {}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Connectivity probe — per-protocol internet presence (IPv4 / IPv6)
|
||||
#
|
||||
# Tier 1 (every tier1_interval s, sends no packets): a UDP connect() to the probe address
|
||||
# shows whether a route exists and which source address would be used. Tier 2 (every
|
||||
# probe_interval s, sooner after a failure or a change of address or gateway): one ICMP echo,
|
||||
# only when tier 1 found a route. A family is absent after no route at all or fail_limit
|
||||
# consecutive failed echoes, present after one success, and unknown (flag omitted) when it
|
||||
# cannot be probed. The result rides on every HTB as p4/p6 and, when network_monitor runs, as
|
||||
# its `connectivity` metric. Mirrors hbd/client/connectivity.py.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_FAMILIES = ("ipv4", "ipv6")
|
||||
_FLAG_KEYS = {"ipv4": "p4", "ipv6": "p6"}
|
||||
_DEFAULT_PROBES = {"ipv4": "1.1.1.1", "ipv6": "2606:4700:4700::1111"}
|
||||
_BSD_PLATFORMS = ("darwin", "freebsd", "netbsd", "openbsd", "dragonfly")
|
||||
|
||||
|
||||
class _ProbeUnavailable(Exception):
|
||||
"""The address family cannot be probed on this host (e.g. no IPv6 support)."""
|
||||
|
||||
|
||||
class _PingUnavailable(Exception):
|
||||
"""The system ping binary could not be executed."""
|
||||
|
||||
|
||||
def _parse_proc_route_v4(text: str) -> Optional[str]:
|
||||
"""Default gateway from /proc/net/route (lowest metric wins)."""
|
||||
best: Optional[Tuple[int, str]] = None
|
||||
for line in text.splitlines()[1:]:
|
||||
f = line.split()
|
||||
if len(f) < 8 or f[1] != "00000000" or not int(f[3], 16) & 0x2:
|
||||
continue
|
||||
metric = int(f[6])
|
||||
if best is None or metric < best[0]:
|
||||
# the kernel prints the raw network-order u32 in host order: use the host byte order
|
||||
best = (metric, socket.inet_ntoa(int(f[2], 16).to_bytes(4, sys.byteorder)))
|
||||
return best[1] if best else None
|
||||
|
||||
|
||||
def _parse_proc_route_v6(text: str) -> Optional[str]:
|
||||
"""Default gateway from /proc/net/ipv6_route (lowest metric wins)."""
|
||||
best: Optional[Tuple[int, str]] = None
|
||||
for line in text.splitlines():
|
||||
f = line.split()
|
||||
if len(f) < 10:
|
||||
continue
|
||||
dest, plen, nexthop, metric, flags, iface = f[0], f[1], f[4], f[5], f[8], f[9]
|
||||
if dest != "0" * 32 or plen != "00" or nexthop == "0" * 32 or iface == "lo":
|
||||
continue
|
||||
fl = int(flags, 16)
|
||||
if not fl & 0x2 or fl & 0x200: # need RTF_GATEWAY, skip RTF_REJECT
|
||||
continue
|
||||
m = int(metric, 16)
|
||||
if best is None or m < best[0]:
|
||||
best = (m, socket.inet_ntop(socket.AF_INET6, bytes.fromhex(nexthop)))
|
||||
return best[1] if best else None
|
||||
|
||||
|
||||
def _parse_route_get(text: str) -> Optional[str]:
|
||||
"""Gateway from `route -n get [-inet6] default` (macOS and the BSDs)."""
|
||||
m = re.search(r"^\s*gateway:\s*(\S+)", text, re.MULTILINE)
|
||||
return m.group(1) if m else None
|
||||
|
||||
|
||||
def _parse_windows_route(text: str, family: str) -> Optional[str]:
|
||||
"""Default gateway from `route print -4` / `route print -6` (lowest metric wins)."""
|
||||
best: Optional[Tuple[int, str]] = None
|
||||
for line in text.splitlines():
|
||||
if family == "ipv4":
|
||||
f = line.split()
|
||||
if not (len(f) >= 5 and f[0] == "0.0.0.0" and f[1] == "0.0.0.0" and f[2] != "On-link"):
|
||||
continue
|
||||
cand = (int(f[4]), f[2])
|
||||
else:
|
||||
m = re.match(r"^\s*\d+\s+(\d+)\s+::/0\s+(\S+)\s*$", line)
|
||||
if not m or m.group(2) == "On-link":
|
||||
continue
|
||||
cand = (int(m.group(1)), m.group(2))
|
||||
if best is None or cand[0] < best[0]:
|
||||
best = cand
|
||||
return best[1] if best else None
|
||||
|
||||
|
||||
async def _run_cmd(cmd: List[str], timeout: float = 5.0) -> str:
|
||||
proc = await asyncio.create_subprocess_exec(
|
||||
*cmd, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.DEVNULL)
|
||||
try:
|
||||
out, _ = await asyncio.wait_for(proc.communicate(), timeout=timeout)
|
||||
except asyncio.TimeoutError:
|
||||
proc.kill()
|
||||
await proc.wait()
|
||||
raise
|
||||
return out.decode(errors="replace")
|
||||
|
||||
|
||||
async def _default_gateway(family: str) -> Optional[str]:
|
||||
"""Default gateway for "ipv4"/"ipv6", or None on any failure (informational only)."""
|
||||
v4 = family == "ipv4"
|
||||
try:
|
||||
if sys.platform.startswith("linux"):
|
||||
with open("/proc/net/route" if v4 else "/proc/net/ipv6_route") as fh:
|
||||
text = fh.read()
|
||||
return _parse_proc_route_v4(text) if v4 else _parse_proc_route_v6(text)
|
||||
if sys.platform.startswith(_BSD_PLATFORMS):
|
||||
cmd = ["route", "-n", "get"] + ([] if v4 else ["-inet6"]) + ["default"]
|
||||
return _parse_route_get(await _run_cmd(cmd))
|
||||
if sys.platform == "win32":
|
||||
text = await _run_cmd(["route", "print", "-4" if v4 else "-6"])
|
||||
return _parse_windows_route(text, family)
|
||||
except Exception:
|
||||
return None
|
||||
return None
|
||||
|
||||
|
||||
def _probe_ping_command(host: str, family: str) -> List[str]:
|
||||
"""Command for ONE echo request with a ~2 s reply wait."""
|
||||
v4 = family == "ipv4"
|
||||
if sys.platform == "win32":
|
||||
return ["ping", "-n", "1", "-w", "2000", "-4" if v4 else "-6", host]
|
||||
if sys.platform.startswith(_BSD_PLATFORMS):
|
||||
# BSD/macOS -W is in milliseconds, and IPv6 is reached through ping6.
|
||||
return ["ping", "-c", "1", "-W", "2000", host] if v4 else ["ping6", "-c", "1", host]
|
||||
return ["ping", "-c", "1", "-W", "2", "-4" if v4 else "-6", host]
|
||||
|
||||
|
||||
def _parse_ping_rtt(output: str) -> Optional[float]:
|
||||
"""Average RTT in ms from ping output, or None when there was no reply."""
|
||||
m = re.search(r"(?:rtt|round-trip)\s+min/avg/max/\S+\s*=\s*([\d.]+)/([\d.]+)/([\d.]+)", output)
|
||||
if m:
|
||||
return float(m.group(2))
|
||||
if "TTL=" in output: # Windows reply line (also when localized)
|
||||
m = re.search(r"[=<]\s*(\d+)\s*ms", output)
|
||||
return float(m.group(1)) if m else 0.0
|
||||
return None
|
||||
|
||||
|
||||
async def _ping_rtt(target: str, family: str) -> Optional[float]:
|
||||
"""One ICMP echo; RTT in ms, or None if no reply. Raises _PingUnavailable without ping."""
|
||||
try:
|
||||
proc = await asyncio.create_subprocess_exec(
|
||||
*_probe_ping_command(target, family),
|
||||
stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE)
|
||||
except FileNotFoundError as e:
|
||||
raise _PingUnavailable(str(e))
|
||||
try:
|
||||
out, _ = await asyncio.wait_for(proc.communicate(), timeout=6)
|
||||
except asyncio.TimeoutError:
|
||||
proc.kill()
|
||||
await proc.wait()
|
||||
return None
|
||||
return _parse_ping_rtt(out.decode(errors="replace"))
|
||||
|
||||
|
||||
def _local_address(family: str, target: str) -> Optional[str]:
|
||||
"""Source address the kernel would use to reach target, or None if there is no route.
|
||||
|
||||
A UDP connect() sends nothing. Raises _ProbeUnavailable if the family is unsupported.
|
||||
"""
|
||||
af = socket.AF_INET if family == "ipv4" else socket.AF_INET6
|
||||
try:
|
||||
sock = socket.socket(af, socket.SOCK_DGRAM)
|
||||
except OSError as e:
|
||||
raise _ProbeUnavailable(str(e))
|
||||
try:
|
||||
sock.connect((target, 9))
|
||||
return str(sock.getsockname()[0])
|
||||
except OSError:
|
||||
return None
|
||||
finally:
|
||||
sock.close()
|
||||
|
||||
|
||||
@dataclass
|
||||
class _FamilyState:
|
||||
present: Optional[bool] = None # None = unknown (probe cannot run yet/at all)
|
||||
addr: Optional[str] = None
|
||||
gateway: Optional[str] = None
|
||||
rtt: Optional[float] = None
|
||||
|
||||
|
||||
class _Connectivity:
|
||||
def __init__(
|
||||
self,
|
||||
probes: Optional[Dict[str, str]] = None,
|
||||
probe_interval: float = 60.0,
|
||||
retry_interval: float = 10.0,
|
||||
tier1_interval: float = 5.0,
|
||||
fail_limit: int = 2,
|
||||
local_address_fn: Callable[[str, str], Optional[str]] = _local_address,
|
||||
ping_fn: Callable[[str, str], Awaitable[Optional[float]]] = _ping_rtt,
|
||||
gateway_fn: Callable[[str], Awaitable[Optional[str]]] = _default_gateway,
|
||||
clock: Callable[[], float] = time.monotonic,
|
||||
):
|
||||
self.probes = dict(_DEFAULT_PROBES)
|
||||
self.probes.update(probes or {})
|
||||
self.probe_interval = probe_interval
|
||||
self.retry_interval = retry_interval
|
||||
self.tier1_interval = tier1_interval
|
||||
self.fail_limit = fail_limit
|
||||
self._local_address = local_address_fn
|
||||
self._ping = ping_fn
|
||||
self._gateway_fn = gateway_fn
|
||||
self._clock = clock
|
||||
# Reading the gateway is a cheap /proc read on Linux but a subprocess elsewhere, so
|
||||
# elsewhere it is refreshed only when a tier-2 probe is due.
|
||||
self._cheap_gateway = sys.platform.startswith("linux")
|
||||
self.states: Dict[str, _FamilyState] = {f: _FamilyState() for f in _FAMILIES}
|
||||
self._fails: Dict[str, int] = {f: 0 for f in _FAMILIES}
|
||||
self._next_ping: Dict[str, float] = {f: 0.0 for f in _FAMILIES}
|
||||
self._last_key: Dict[str, Optional[Tuple[Optional[str], Optional[str]]]] = {
|
||||
f: None for f in _FAMILIES}
|
||||
self._ping_disabled: Set[str] = set()
|
||||
|
||||
@classmethod
|
||||
def from_config(cls, cfg: Optional[Dict[str, Any]]) -> "_Connectivity":
|
||||
cfg = cfg or {}
|
||||
probes = {}
|
||||
if cfg.get("probe_v4"):
|
||||
probes["ipv4"] = str(cfg["probe_v4"])
|
||||
if cfg.get("probe_v6"):
|
||||
probes["ipv6"] = str(cfg["probe_v6"])
|
||||
return cls(probes=probes, probe_interval=float(cfg.get("probe_interval", 60)))
|
||||
|
||||
# ---- readers (cheap; safe to call on every heartbeat) ---------------------------
|
||||
def snapshot(self) -> Dict[str, _FamilyState]:
|
||||
return {f: _FamilyState(**asdict(s)) for f, s in self.states.items()}
|
||||
|
||||
def flags(self) -> Dict[str, bool]:
|
||||
"""HTB flags for families whose presence is known."""
|
||||
return {_FLAG_KEYS[f]: s.present for f, s in self.states.items() if s.present is not None}
|
||||
|
||||
def as_metric(self) -> Dict[str, Dict[str, Any]]:
|
||||
return {f: asdict(s) for f, s in self.states.items()}
|
||||
|
||||
# ---- probing ----------------------------------------------------------------------
|
||||
async def refresh(self) -> None:
|
||||
for fam in _FAMILIES:
|
||||
try:
|
||||
await self._refresh_family(fam)
|
||||
except Exception:
|
||||
logging.getLogger("hbc.probe").exception("connectivity probe failed for %s", fam)
|
||||
|
||||
async def run(self) -> None:
|
||||
while _running:
|
||||
await self.refresh()
|
||||
await _sleep(self.tier1_interval)
|
||||
|
||||
async def _gateway(self, fam: str) -> Optional[str]:
|
||||
try:
|
||||
return await self._gateway_fn(fam)
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
async def _refresh_family(self, fam: str) -> None:
|
||||
st = self.states[fam]
|
||||
try:
|
||||
addr = self._local_address(fam, self.probes[fam])
|
||||
except _ProbeUnavailable:
|
||||
st.present = None
|
||||
return
|
||||
|
||||
if addr is None: # tier 1: no route => absent immediately
|
||||
self._fails[fam] = 0
|
||||
self._next_ping[fam] = 0.0
|
||||
self._last_key[fam] = None
|
||||
st.present, st.addr, st.gateway, st.rtt = False, None, None, None
|
||||
return
|
||||
|
||||
now = self._clock()
|
||||
gw = await self._gateway(fam) if self._cheap_gateway else st.gateway
|
||||
key = (addr, gw)
|
||||
due = key != self._last_key[fam] or now >= self._next_ping[fam]
|
||||
if due and not self._cheap_gateway:
|
||||
gw = await self._gateway(fam)
|
||||
key = (addr, gw)
|
||||
st.addr, st.gateway = addr, gw
|
||||
self._last_key[fam] = key
|
||||
if fam in self._ping_disabled:
|
||||
st.present = None # cannot probe: unknown, never absent, while a route exists
|
||||
elif due:
|
||||
await self._tier2(fam, st, now)
|
||||
|
||||
async def _tier2(self, fam: str, st: _FamilyState, now: float) -> None:
|
||||
try:
|
||||
rtt = await self._ping(self.probes[fam], fam)
|
||||
except _PingUnavailable:
|
||||
logging.getLogger("hbc.probe").warning("ping unavailable; %s presence stays unknown", fam)
|
||||
self._ping_disabled.add(fam)
|
||||
st.present = None
|
||||
return
|
||||
except (asyncio.TimeoutError, OSError):
|
||||
rtt = None
|
||||
if rtt is None:
|
||||
self._fails[fam] += 1
|
||||
self._next_ping[fam] = now + self.retry_interval
|
||||
if self._fails[fam] >= self.fail_limit:
|
||||
st.present, st.rtt = False, None
|
||||
else:
|
||||
self._fails[fam] = 0
|
||||
self._next_ping[fam] = now + self.probe_interval
|
||||
st.present, st.rtt = True, rtt
|
||||
|
||||
|
||||
# The running probe (None when probe_enabled is false); shared by the heartbeat sender and
|
||||
# the network_monitor plugin. Set in _async_main.
|
||||
_probe: Optional[_Connectivity] = None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Plugin: network_monitor (Linux /proc/net/dev)
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -658,7 +974,7 @@ def _read_net_dev() -> Dict[str, Tuple[int, int]]:
|
||||
|
||||
class NetworkMonitorPlugin(MonitorPlugin):
|
||||
name = "network_monitor"
|
||||
description = "Network I/O rates via /proc/net/dev (Linux)"
|
||||
description = "Network I/O rates via /proc/net/dev (Linux) and per-protocol internet connectivity"
|
||||
interval = 300
|
||||
|
||||
def __init__(self, config: Optional[Dict[str, Any]] = None):
|
||||
@@ -667,19 +983,25 @@ class NetworkMonitorPlugin(MonitorPlugin):
|
||||
self.interval = cfg.get("interval", 300)
|
||||
self.skip_ifaces: set = set(cfg.get("skip_interfaces", ["lo"]))
|
||||
self._prev: Optional[Tuple[float, Dict[str, Tuple[int, int]]]] = None
|
||||
self._have_dev = False
|
||||
|
||||
async def initialize(self) -> bool:
|
||||
if platform.system() != "Linux":
|
||||
self.skip_reason = "Linux only (/proc/net/dev not available)"
|
||||
dev = _read_net_dev() if platform.system() == "Linux" else {}
|
||||
self._have_dev = bool(dev)
|
||||
if not self._have_dev and _probe is None:
|
||||
self.skip_reason = "no /proc/net/dev (Linux only) and the connectivity probe is disabled"
|
||||
return False
|
||||
dev = _read_net_dev()
|
||||
if not dev:
|
||||
self.skip_reason = "/proc/net/dev not readable"
|
||||
return False
|
||||
self._prev = (time.time(), dev) # prime first delta
|
||||
if self._have_dev:
|
||||
self._prev = (time.time(), dev) # prime first delta
|
||||
return True
|
||||
|
||||
async def _collect_metrics(self) -> Dict[str, Any]:
|
||||
data: Dict[str, Any] = await self._collect_interfaces() if self._have_dev else {}
|
||||
if _probe is not None:
|
||||
data["connectivity"] = _probe.as_metric()
|
||||
return data
|
||||
|
||||
async def _collect_interfaces(self) -> Dict[str, Any]:
|
||||
now = time.time()
|
||||
curr = _read_net_dev()
|
||||
if not curr or self._prev is None:
|
||||
@@ -940,15 +1262,21 @@ async def _handle_update(conn: AsyncConnection):
|
||||
# Heartbeat sender
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
async def _heartbeat_sender(conn: AsyncConnection, interval: int):
|
||||
def _build_heartbeat_msg(conn: AsyncConnection, interval: int,
|
||||
probe: Optional[_Connectivity] = None) -> Dict[str, Any]:
|
||||
"""HTB payload; carries p4/p6 presence flags when the probe knows them."""
|
||||
msg: Dict[str, Any] = {"acks": conn.ackcount, "rtt": conn.rtts[-1], "interval": interval}
|
||||
if probe is not None:
|
||||
msg.update(probe.flags())
|
||||
return msg
|
||||
|
||||
|
||||
async def _heartbeat_sender(conn: AsyncConnection, interval: int,
|
||||
probe: Optional[_Connectivity] = None):
|
||||
log = logging.getLogger("hbc.hb")
|
||||
while _running:
|
||||
try:
|
||||
await conn.sendto({
|
||||
"acks": conn.ackcount,
|
||||
"rtt": conn.rtts[-1],
|
||||
"interval": interval,
|
||||
})
|
||||
await conn.sendto(_build_heartbeat_msg(conn, interval, probe))
|
||||
except Exception as e:
|
||||
log.error("send: %s", e)
|
||||
await _sleep(interval)
|
||||
@@ -1056,7 +1384,7 @@ def _reconfigure_syslog(level: int):
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
async def _async_main(args, cfg: Dict[str, Any]) -> int:
|
||||
global _running, _shutdown_event, _active_tasks, send_shutdown
|
||||
global _running, _shutdown_event, _active_tasks, send_shutdown, _probe
|
||||
_running = True
|
||||
_shutdown_event = asyncio.Event()
|
||||
_active_tasks = []
|
||||
@@ -1113,6 +1441,10 @@ async def _async_main(args, cfg: Dict[str, Any]) -> int:
|
||||
c.close()
|
||||
return 0
|
||||
|
||||
# Per-protocol internet presence probe (shared with the network_monitor plugin)
|
||||
net_cfg = dict((cfg.get("plugins") or {}).get("network_monitor") or cfg.get("network_monitor") or {})
|
||||
_probe = _Connectivity.from_config(net_cfg) if net_cfg.get("probe_enabled", True) else None
|
||||
|
||||
plugins = await _load_plugins(cfg)
|
||||
|
||||
loop = asyncio.get_event_loop()
|
||||
@@ -1126,8 +1458,10 @@ async def _async_main(args, cfg: Dict[str, Any]) -> int:
|
||||
|
||||
loop.add_signal_handler(signal.SIGHUP, _sighup)
|
||||
|
||||
if _probe is not None:
|
||||
_active_tasks.append(asyncio.create_task(_probe.run()))
|
||||
for conn in connections:
|
||||
_active_tasks.append(asyncio.create_task(_heartbeat_sender(conn, interval)))
|
||||
_active_tasks.append(asyncio.create_task(_heartbeat_sender(conn, interval, _probe)))
|
||||
|
||||
if plugins and connections:
|
||||
_active_tasks.append(asyncio.create_task(_plugin_collector(connections[0], plugins)))
|
||||
|
||||
@@ -0,0 +1,28 @@
|
||||
from hbd.client.main import build_heartbeat_msg
|
||||
|
||||
|
||||
class _Conn:
|
||||
ackcount = 3
|
||||
rtts = [0, 12.5]
|
||||
|
||||
|
||||
class _Conn2:
|
||||
def __init__(self, flags):
|
||||
self._flags = flags
|
||||
|
||||
def flags(self):
|
||||
return self._flags
|
||||
|
||||
|
||||
def test_heartbeat_without_connectivity_is_unchanged():
|
||||
assert build_heartbeat_msg(_Conn(), 60) == {"acks": 3, "rtt": 12.5, "interval": 60}
|
||||
|
||||
|
||||
def test_heartbeat_carries_presence_flags():
|
||||
msg = build_heartbeat_msg(_Conn(), 60, _Conn2({"p4": True, "p6": False}))
|
||||
assert msg == {"acks": 3, "rtt": 12.5, "interval": 60, "p4": True, "p6": False}
|
||||
|
||||
|
||||
def test_heartbeat_omits_unknown_flags():
|
||||
msg = build_heartbeat_msg(_Conn(), 60, _Conn2({}))
|
||||
assert "p4" not in msg and "p6" not in msg
|
||||
@@ -0,0 +1,173 @@
|
||||
import asyncio
|
||||
|
||||
from hbd.client.connectivity import Connectivity, ProbeUnavailable
|
||||
from hbd.client.pingutil import PingUnavailable
|
||||
|
||||
|
||||
class Net:
|
||||
"""Scriptable network: addr None = no route, 'unavailable' = family unsupported."""
|
||||
|
||||
def __init__(self):
|
||||
self.addr = {"ipv4": "192.0.2.5", "ipv6": "2001:db8::5"}
|
||||
self.gw = {"ipv4": "192.0.2.1", "ipv6": "fe80::1"}
|
||||
self.rtt = {"ipv4": 10.0, "ipv6": 20.0} # None = ping fails, 'unavailable' = no ping binary
|
||||
self.pings = []
|
||||
self.now = 1000.0
|
||||
|
||||
def local(self, fam, target):
|
||||
a = self.addr[fam]
|
||||
if a == "unavailable":
|
||||
raise ProbeUnavailable("no such family")
|
||||
return a
|
||||
|
||||
async def ping(self, target, fam):
|
||||
self.pings.append(fam)
|
||||
r = self.rtt[fam]
|
||||
if r == "unavailable":
|
||||
raise PingUnavailable("no ping")
|
||||
return r
|
||||
|
||||
async def gateway(self, fam):
|
||||
return self.gw[fam]
|
||||
|
||||
def clock(self):
|
||||
return self.now
|
||||
|
||||
|
||||
def make(net, **kw):
|
||||
c = Connectivity(local_address_fn=net.local, ping_fn=net.ping, gateway_fn=net.gateway,
|
||||
clock=net.clock, **kw)
|
||||
c._cheap_gateway = True
|
||||
return c
|
||||
|
||||
|
||||
def refresh(c):
|
||||
asyncio.run(c.refresh())
|
||||
|
||||
|
||||
def test_present_with_addr_gateway_rtt():
|
||||
net = Net()
|
||||
c = make(net)
|
||||
refresh(c)
|
||||
s = c.snapshot()["ipv4"]
|
||||
assert (s.present, s.addr, s.gateway, s.rtt) == (True, "192.0.2.5", "192.0.2.1", 10.0)
|
||||
assert c.flags() == {"p4": True, "p6": True}
|
||||
assert c.as_metric()["ipv6"]["gateway"] == "fe80::1"
|
||||
|
||||
|
||||
def test_no_route_is_immediately_absent():
|
||||
net = Net()
|
||||
net.addr["ipv6"] = None
|
||||
c = make(net)
|
||||
refresh(c)
|
||||
assert c.flags() == {"p4": True, "p6": False}
|
||||
assert net.pings == ["ipv4"] # no ping wasted on a family without a route
|
||||
assert c.snapshot()["ipv6"].addr is None
|
||||
|
||||
|
||||
def test_ping_is_debounced_and_rate_limited():
|
||||
net = Net()
|
||||
c = make(net, probe_interval=60.0, retry_interval=10.0)
|
||||
refresh(c)
|
||||
refresh(c) # same instant: cached, no second ping
|
||||
assert net.pings == ["ipv4", "ipv6"]
|
||||
net.now += 60
|
||||
net.rtt["ipv6"] = None # first failure: still present (debounce)
|
||||
refresh(c)
|
||||
assert c.flags()["p6"] is True
|
||||
net.now += 10 # retry_interval elapsed: second failure flips it
|
||||
refresh(c)
|
||||
assert c.flags()["p6"] is False
|
||||
|
||||
|
||||
def test_first_probe_failing_once_is_unknown():
|
||||
net = Net()
|
||||
net.rtt["ipv4"] = None
|
||||
c = make(net)
|
||||
refresh(c)
|
||||
assert "p4" not in c.flags() and c.flags()["p6"] is True
|
||||
|
||||
|
||||
def test_recovers_on_first_success():
|
||||
net = Net()
|
||||
net.rtt["ipv6"] = None
|
||||
c = make(net, retry_interval=10.0)
|
||||
refresh(c)
|
||||
net.now += 10
|
||||
refresh(c)
|
||||
assert c.flags()["p6"] is False
|
||||
net.rtt["ipv6"] = 25.0
|
||||
net.now += 10
|
||||
refresh(c)
|
||||
assert c.flags()["p6"] is True
|
||||
|
||||
|
||||
def test_address_change_invalidates_cache():
|
||||
net = Net()
|
||||
c = make(net)
|
||||
refresh(c)
|
||||
net.addr["ipv4"] = "198.51.100.7" # moved networks, no time has passed
|
||||
refresh(c)
|
||||
assert net.pings.count("ipv4") == 2
|
||||
assert c.snapshot()["ipv4"].addr == "198.51.100.7"
|
||||
|
||||
|
||||
def test_gateway_change_invalidates_cache():
|
||||
net = Net()
|
||||
c = make(net)
|
||||
refresh(c)
|
||||
net.gw["ipv4"] = "192.0.2.254"
|
||||
refresh(c)
|
||||
assert net.pings.count("ipv4") == 2
|
||||
|
||||
|
||||
def test_route_loss_then_return_pings_immediately():
|
||||
net = Net()
|
||||
c = make(net)
|
||||
refresh(c)
|
||||
net.addr["ipv6"] = None
|
||||
refresh(c)
|
||||
net.addr["ipv6"] = "2001:db8::5"
|
||||
refresh(c)
|
||||
assert net.pings.count("ipv6") == 2 and c.flags()["p6"] is True
|
||||
|
||||
|
||||
def test_missing_ping_binary_leaves_presence_unknown_and_stops_retrying():
|
||||
net = Net()
|
||||
net.rtt["ipv4"] = "unavailable"
|
||||
c = make(net)
|
||||
refresh(c)
|
||||
net.now += 600
|
||||
refresh(c)
|
||||
assert "p4" not in c.flags()
|
||||
assert net.pings.count("ipv4") == 1
|
||||
|
||||
|
||||
def test_unsupported_family_is_unknown():
|
||||
net = Net()
|
||||
net.addr["ipv6"] = "unavailable"
|
||||
c = make(net)
|
||||
refresh(c)
|
||||
assert "p6" not in c.flags() and c.flags()["p4"] is True
|
||||
|
||||
|
||||
def test_from_config():
|
||||
c = Connectivity.from_config({"probe_v4": "9.9.9.9", "probe_interval": 30})
|
||||
assert c.probes["ipv4"] == "9.9.9.9"
|
||||
assert c.probes["ipv6"] == "2606:4700:4700::1111"
|
||||
assert c.probe_interval == 30.0
|
||||
|
||||
|
||||
def test_ping_less_family_stays_unknown_after_route_returns():
|
||||
net = Net()
|
||||
net.rtt["ipv4"] = "unavailable"
|
||||
c = make(net)
|
||||
refresh(c)
|
||||
assert "p4" not in c.flags()
|
||||
net.addr["ipv4"] = None
|
||||
refresh(c)
|
||||
assert c.flags()["p4"] is False # no route is definitive
|
||||
net.addr["ipv4"] = "192.0.2.5"
|
||||
refresh(c)
|
||||
assert "p4" not in c.flags()
|
||||
assert net.pings.count("ipv4") == 1 # missing binary is not retried
|
||||
@@ -98,6 +98,26 @@ def test_alert_during_the_quiet_window_keeps_it_flapping():
|
||||
assert flap.flapping_services("h1") == ["cpu"]
|
||||
|
||||
|
||||
# --- host removal ------------------------------------------------------------
|
||||
|
||||
def test_clear_host_drops_flapping_state_even_without_a_recover():
|
||||
# A host dropped mid-flap (no RECOVER ever received) must not stay
|
||||
# flapping forever: nothing will ever set ok_since for it again.
|
||||
alerts(4)
|
||||
assert flap.flapping_services("h1") == ["cpu"]
|
||||
flap.clear_host("h1")
|
||||
assert flap.flapping_services("h1") == []
|
||||
assert flap.observe("h1", "cpu", "CRITICAL") == flap.PASS
|
||||
|
||||
|
||||
def test_clear_host_only_affects_the_named_host():
|
||||
alerts(4, host="h1")
|
||||
alerts(4, host="h2")
|
||||
flap.clear_host("h1")
|
||||
assert flap.flapping_services("h1") == []
|
||||
assert flap.flapping_services("h2") == ["cpu"]
|
||||
|
||||
|
||||
# --- keying -----------------------------------------------------------------
|
||||
|
||||
def test_services_and_hosts_are_tracked_independently():
|
||||
|
||||
@@ -0,0 +1,75 @@
|
||||
import asyncio
|
||||
import struct
|
||||
|
||||
from hbd.client import gateway
|
||||
|
||||
PROC_V4 = (
|
||||
"Iface\tDestination\tGateway \tFlags\tRefCnt\tUse\tMetric\tMask\t\tMTU\tWindow\tIRTT\n"
|
||||
"wlan0\t00000000\t0100A8C0\t0003\t0\t0\t600\t00000000\t0\t0\t0\n"
|
||||
"eth0\t00000000\t0101A8C0\t0003\t0\t0\t100\t00000000\t0\t0\t0\n"
|
||||
"wlan0\t0000A8C0\t00000000\t0001\t0\t0\t600\t00FFFFFF\t0\t0\t0\n"
|
||||
)
|
||||
|
||||
PROC_V6 = (
|
||||
"00000000000000000000000000000000 00 00000000000000000000000000000000 00 "
|
||||
"fe800000000000000000000000000001 00000400 00000001 00000000 00000003 wlan0\n"
|
||||
"00000000000000000000000000000000 00 00000000000000000000000000000000 00 "
|
||||
"00000000000000000000000000000000 ffffffff 00000001 00000000 00200200 lo\n"
|
||||
"fe800000000000000000000000000000 40 00000000000000000000000000000000 00 "
|
||||
"00000000000000000000000000000000 00000100 00000001 00000000 00000001 wlan0\n"
|
||||
)
|
||||
|
||||
MACOS = """ route to: default
|
||||
destination: default
|
||||
mask: default
|
||||
gateway: 192.168.1.1
|
||||
interface: en0
|
||||
"""
|
||||
|
||||
WIN_V4 = """IPv4 Route Table
|
||||
Active Routes:
|
||||
Network Destination Netmask Gateway Interface Metric
|
||||
0.0.0.0 0.0.0.0 192.168.1.1 192.168.1.50 25
|
||||
0.0.0.0 0.0.0.0 10.0.0.1 10.0.0.9 50
|
||||
127.0.0.0 255.0.0.0 On-link 127.0.0.1 331
|
||||
"""
|
||||
|
||||
WIN_V6 = """IPv6 Route Table
|
||||
Active Routes:
|
||||
If Metric Network Destination Gateway
|
||||
12 281 ::/0 fe80::1
|
||||
1 331 ::1/128 On-link
|
||||
"""
|
||||
|
||||
|
||||
def test_proc_v4_picks_lowest_metric_gateway():
|
||||
assert gateway.parse_proc_route_v4(PROC_V4) == "192.168.1.1" # eth0, metric 100
|
||||
|
||||
|
||||
def test_proc_v4_none_without_default_route():
|
||||
assert gateway.parse_proc_route_v4(PROC_V4.splitlines()[0] + "\n") is None
|
||||
|
||||
|
||||
def test_proc_v6_skips_reject_and_non_default():
|
||||
assert gateway.parse_proc_route_v6(PROC_V6) == "fe80::1"
|
||||
|
||||
|
||||
def test_macos():
|
||||
assert gateway.parse_macos_route(MACOS) == "192.168.1.1"
|
||||
assert gateway.parse_macos_route("route: writing to routing socket: not in table") is None
|
||||
|
||||
|
||||
def test_windows_v4_lowest_metric():
|
||||
assert gateway.parse_windows_route(WIN_V4, "ipv4") == "192.168.1.1"
|
||||
|
||||
|
||||
def test_windows_v6():
|
||||
assert gateway.parse_windows_route(WIN_V6, "ipv6") == "fe80::1"
|
||||
|
||||
|
||||
def test_default_gateway_never_raises_on_parser_error(monkeypatch):
|
||||
"""default_gateway catches all exceptions; never raises even if parser fails."""
|
||||
monkeypatch.setattr("sys.platform", "linux")
|
||||
monkeypatch.setattr(gateway, "parse_proc_route_v4", lambda _: (_ for _ in ()).throw(struct.error))
|
||||
result = asyncio.run(gateway.default_gateway("ipv4"))
|
||||
assert result is None
|
||||
@@ -0,0 +1,381 @@
|
||||
"""Connectivity probe in the standalone scripts/hbc_mini.py (stdlib-only single-file client)."""
|
||||
import asyncio
|
||||
import importlib.util
|
||||
import pathlib
|
||||
import sys
|
||||
|
||||
import pytest
|
||||
|
||||
_PATH = pathlib.Path(__file__).resolve().parents[1] / "scripts" / "hbc_mini.py"
|
||||
_SPEC = importlib.util.spec_from_file_location("hbc_mini", _PATH)
|
||||
mini = importlib.util.module_from_spec(_SPEC)
|
||||
sys.modules["hbc_mini"] = mini # dataclasses resolve their module by name
|
||||
_SPEC.loader.exec_module(mini)
|
||||
|
||||
PROC_V4 = (
|
||||
"Iface\tDestination\tGateway \tFlags\tRefCnt\tUse\tMetric\tMask\t\tMTU\tWindow\tIRTT\n"
|
||||
"wlan0\t00000000\t0100A8C0\t0003\t0\t0\t600\t00000000\t0\t0\t0\n"
|
||||
"eth0\t00000000\t0101A8C0\t0003\t0\t0\t100\t00000000\t0\t0\t0\n"
|
||||
"wlan0\t0000A8C0\t00000000\t0001\t0\t0\t600\t00FFFFFF\t0\t0\t0\n"
|
||||
)
|
||||
PROC_V6 = (
|
||||
"00000000000000000000000000000000 00 00000000000000000000000000000000 00 "
|
||||
"fe800000000000000000000000000001 00000400 00000001 00000000 00000003 wlan0\n"
|
||||
"00000000000000000000000000000000 00 00000000000000000000000000000000 00 "
|
||||
"00000000000000000000000000000000 ffffffff 00000001 00000000 00200200 lo\n"
|
||||
"fe800000000000000000000000000000 40 00000000000000000000000000000000 00 "
|
||||
"00000000000000000000000000000000 00000100 00000001 00000000 00000001 wlan0\n"
|
||||
)
|
||||
ROUTE_GET = """ route to: 0.0.0.0
|
||||
destination: 0.0.0.0
|
||||
mask: 0.0.0.0
|
||||
gateway: 192.0.2.1
|
||||
fib: 0
|
||||
interface: vtnet0
|
||||
"""
|
||||
WIN_V4 = """IPv4 Route Table
|
||||
Active Routes:
|
||||
Network Destination Netmask Gateway Interface Metric
|
||||
0.0.0.0 0.0.0.0 192.168.1.1 192.168.1.50 25
|
||||
0.0.0.0 0.0.0.0 10.0.0.1 10.0.0.9 50
|
||||
127.0.0.0 255.0.0.0 On-link 127.0.0.1 331
|
||||
"""
|
||||
WIN_V6 = """IPv6 Route Table
|
||||
Active Routes:
|
||||
If Metric Network Destination Gateway
|
||||
12 281 ::/0 fe80::1
|
||||
1 331 ::1/128 On-link
|
||||
"""
|
||||
LINUX_PING = """PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
|
||||
64 bytes from 1.1.1.1: icmp_seq=1 ttl=57 time=11.2 ms
|
||||
|
||||
--- 1.1.1.1 ping statistics ---
|
||||
1 packets transmitted, 1 received, 0% packet loss, time 0ms
|
||||
rtt min/avg/max/mdev = 11.234/11.234/11.234/0.000 ms
|
||||
"""
|
||||
FREEBSD_PING = """PING 1.1.1.1 (1.1.1.1): 56 data bytes
|
||||
64 bytes from 1.1.1.1: icmp_seq=0 ttl=59 time=4.006 ms
|
||||
|
||||
--- 1.1.1.1 ping statistics ---
|
||||
1 packets transmitted, 1 packets received, 0.0% packet loss, 1 packets out of wait time
|
||||
round-trip min/avg/max/stddev = 4.006/4.006/4.006/0.000 ms
|
||||
"""
|
||||
WIN_PING = """Pinging 1.1.1.1 with 32 bytes of data:
|
||||
Reply from 1.1.1.1: bytes=32 time=12ms TTL=57
|
||||
|
||||
Ping statistics for 1.1.1.1:
|
||||
Packets: Sent = 1, Received = 1, Lost = 0 (0% loss),
|
||||
"""
|
||||
LOST_PING = """--- 1.1.1.1 ping statistics ---
|
||||
1 packets transmitted, 0 received, 100% packet loss, time 0ms
|
||||
"""
|
||||
|
||||
|
||||
# ---- gateway parsers ---------------------------------------------------------------
|
||||
|
||||
def test_proc_v4_lowest_metric_wins():
|
||||
assert mini._parse_proc_route_v4(PROC_V4) == "192.168.1.1" # eth0, metric 100
|
||||
|
||||
|
||||
def test_proc_v4_none_without_default_route():
|
||||
assert mini._parse_proc_route_v4(PROC_V4.splitlines()[0] + "\n") is None
|
||||
|
||||
|
||||
def test_proc_v6_skips_reject_and_non_default():
|
||||
assert mini._parse_proc_route_v6(PROC_V6) == "fe80::1"
|
||||
|
||||
|
||||
def test_route_get_bsd_and_macos():
|
||||
assert mini._parse_route_get(ROUTE_GET) == "192.0.2.1"
|
||||
assert mini._parse_route_get("route: writing to routing socket: not in table") is None
|
||||
|
||||
|
||||
def test_windows_routes():
|
||||
assert mini._parse_windows_route(WIN_V4, "ipv4") == "192.168.1.1"
|
||||
assert mini._parse_windows_route(WIN_V6, "ipv6") == "fe80::1"
|
||||
|
||||
|
||||
def test_default_gateway_never_raises(monkeypatch):
|
||||
monkeypatch.setattr(sys, "platform", "linux")
|
||||
monkeypatch.setattr(mini, "_parse_proc_route_v4", lambda t: 1 / 0)
|
||||
assert asyncio.run(mini._default_gateway("ipv4")) is None
|
||||
|
||||
|
||||
# ---- ping ---------------------------------------------------------------------------
|
||||
|
||||
@pytest.mark.parametrize("platform,family,expected", [
|
||||
("linux", "ipv4", ["ping", "-c", "1", "-W", "2", "-4", "h"]),
|
||||
("linux", "ipv6", ["ping", "-c", "1", "-W", "2", "-6", "h"]),
|
||||
("freebsd15", "ipv4", ["ping", "-c", "1", "-W", "2000", "h"]),
|
||||
("freebsd15", "ipv6", ["ping6", "-c", "1", "h"]),
|
||||
("darwin", "ipv4", ["ping", "-c", "1", "-W", "2000", "h"]),
|
||||
("darwin", "ipv6", ["ping6", "-c", "1", "h"]),
|
||||
("win32", "ipv4", ["ping", "-n", "1", "-w", "2000", "-4", "h"]),
|
||||
("win32", "ipv6", ["ping", "-n", "1", "-w", "2000", "-6", "h"]),
|
||||
])
|
||||
def test_probe_ping_command(monkeypatch, platform, family, expected):
|
||||
monkeypatch.setattr(sys, "platform", platform)
|
||||
assert mini._probe_ping_command("h", family) == expected
|
||||
|
||||
|
||||
def test_parse_ping_rtt():
|
||||
assert mini._parse_ping_rtt(LINUX_PING) == 11.234
|
||||
assert mini._parse_ping_rtt(FREEBSD_PING) == 4.006
|
||||
assert mini._parse_ping_rtt(WIN_PING) == 12.0
|
||||
assert mini._parse_ping_rtt(LOST_PING) is None
|
||||
assert mini._parse_ping_rtt("ping: connect: Network is unreachable") is None
|
||||
|
||||
|
||||
# ---- probe state machine ------------------------------------------------------------
|
||||
|
||||
class Net:
|
||||
"""Scriptable network: addr None = no route, 'unavailable' = family unsupported."""
|
||||
|
||||
def __init__(self):
|
||||
self.addr = {"ipv4": "192.0.2.5", "ipv6": "2001:db8::5"}
|
||||
self.gw = {"ipv4": "192.0.2.1", "ipv6": "fe80::1"}
|
||||
self.rtt = {"ipv4": 10.0, "ipv6": 20.0} # None = ping fails, 'unavailable' = no ping binary
|
||||
self.pings = []
|
||||
self.now = 1000.0
|
||||
|
||||
def local(self, fam, target):
|
||||
a = self.addr[fam]
|
||||
if a == "unavailable":
|
||||
raise mini._ProbeUnavailable("no such family")
|
||||
return a
|
||||
|
||||
async def ping(self, target, fam):
|
||||
self.pings.append(fam)
|
||||
r = self.rtt[fam]
|
||||
if r == "unavailable":
|
||||
raise mini._PingUnavailable("no ping")
|
||||
return r
|
||||
|
||||
async def gateway(self, fam):
|
||||
return self.gw[fam]
|
||||
|
||||
def clock(self):
|
||||
return self.now
|
||||
|
||||
|
||||
def make(net, **kw):
|
||||
c = mini._Connectivity(local_address_fn=net.local, ping_fn=net.ping, gateway_fn=net.gateway,
|
||||
clock=net.clock, **kw)
|
||||
c._cheap_gateway = True
|
||||
return c
|
||||
|
||||
|
||||
def refresh(c):
|
||||
asyncio.run(c.refresh())
|
||||
|
||||
|
||||
def test_present_with_addr_gateway_rtt():
|
||||
net = Net()
|
||||
c = make(net)
|
||||
refresh(c)
|
||||
s = c.snapshot()["ipv4"]
|
||||
assert (s.present, s.addr, s.gateway, s.rtt) == (True, "192.0.2.5", "192.0.2.1", 10.0)
|
||||
assert c.flags() == {"p4": True, "p6": True}
|
||||
assert c.as_metric()["ipv6"]["gateway"] == "fe80::1"
|
||||
|
||||
|
||||
def test_no_route_is_immediately_absent():
|
||||
net = Net()
|
||||
net.addr["ipv6"] = None
|
||||
c = make(net)
|
||||
refresh(c)
|
||||
assert c.flags() == {"p4": True, "p6": False}
|
||||
assert net.pings == ["ipv4"]
|
||||
assert c.snapshot()["ipv6"].addr is None
|
||||
|
||||
|
||||
def test_ping_is_debounced_and_rate_limited():
|
||||
net = Net()
|
||||
c = make(net, probe_interval=60.0, retry_interval=10.0)
|
||||
refresh(c)
|
||||
refresh(c)
|
||||
assert net.pings == ["ipv4", "ipv6"]
|
||||
net.now += 60
|
||||
net.rtt["ipv6"] = None
|
||||
refresh(c)
|
||||
assert c.flags()["p6"] is True
|
||||
net.now += 10
|
||||
refresh(c)
|
||||
assert c.flags()["p6"] is False
|
||||
|
||||
|
||||
def test_first_probe_failing_once_is_unknown():
|
||||
net = Net()
|
||||
net.rtt["ipv4"] = None
|
||||
c = make(net)
|
||||
refresh(c)
|
||||
assert "p4" not in c.flags() and c.flags()["p6"] is True
|
||||
|
||||
|
||||
def test_recovers_on_first_success():
|
||||
net = Net()
|
||||
net.rtt["ipv6"] = None
|
||||
c = make(net, retry_interval=10.0)
|
||||
refresh(c)
|
||||
net.now += 10
|
||||
refresh(c)
|
||||
assert c.flags()["p6"] is False
|
||||
net.rtt["ipv6"] = 25.0
|
||||
net.now += 10
|
||||
refresh(c)
|
||||
assert c.flags()["p6"] is True
|
||||
|
||||
|
||||
def test_address_change_invalidates_cache():
|
||||
net = Net()
|
||||
c = make(net)
|
||||
refresh(c)
|
||||
net.addr["ipv4"] = "198.51.100.7"
|
||||
refresh(c)
|
||||
assert net.pings.count("ipv4") == 2
|
||||
assert c.snapshot()["ipv4"].addr == "198.51.100.7"
|
||||
|
||||
|
||||
def test_gateway_change_invalidates_cache():
|
||||
net = Net()
|
||||
c = make(net)
|
||||
refresh(c)
|
||||
net.gw["ipv4"] = "192.0.2.254"
|
||||
refresh(c)
|
||||
assert net.pings.count("ipv4") == 2
|
||||
|
||||
|
||||
def test_route_loss_then_return_pings_immediately():
|
||||
net = Net()
|
||||
c = make(net)
|
||||
refresh(c)
|
||||
net.addr["ipv6"] = None
|
||||
refresh(c)
|
||||
net.addr["ipv6"] = "2001:db8::5"
|
||||
refresh(c)
|
||||
assert net.pings.count("ipv6") == 2 and c.flags()["p6"] is True
|
||||
|
||||
|
||||
def test_missing_ping_binary_leaves_presence_unknown_and_stops_retrying():
|
||||
net = Net()
|
||||
net.rtt["ipv4"] = "unavailable"
|
||||
c = make(net)
|
||||
refresh(c)
|
||||
net.now += 600
|
||||
refresh(c)
|
||||
assert "p4" not in c.flags()
|
||||
assert net.pings.count("ipv4") == 1
|
||||
|
||||
|
||||
def test_ping_less_family_stays_unknown_after_route_returns():
|
||||
net = Net()
|
||||
net.rtt["ipv4"] = "unavailable"
|
||||
c = make(net)
|
||||
refresh(c)
|
||||
assert "p4" not in c.flags()
|
||||
net.addr["ipv4"] = None
|
||||
refresh(c)
|
||||
assert c.flags()["p4"] is False # no route is definitive
|
||||
net.addr["ipv4"] = "192.0.2.5"
|
||||
refresh(c)
|
||||
assert "p4" not in c.flags() and net.pings.count("ipv4") == 1
|
||||
|
||||
|
||||
def test_unsupported_family_is_unknown():
|
||||
net = Net()
|
||||
net.addr["ipv6"] = "unavailable"
|
||||
c = make(net)
|
||||
refresh(c)
|
||||
assert "p6" not in c.flags() and c.flags()["p4"] is True
|
||||
|
||||
|
||||
def test_from_config():
|
||||
c = mini._Connectivity.from_config({"probe_v4": "9.9.9.9", "probe_interval": 30})
|
||||
assert c.probes["ipv4"] == "9.9.9.9"
|
||||
assert c.probes["ipv6"] == "2606:4700:4700::1111"
|
||||
assert c.probe_interval == 30.0
|
||||
|
||||
|
||||
def test_local_address_on_this_host_never_raises():
|
||||
# Real UDP connect(): either an address string, None (no route) or _ProbeUnavailable.
|
||||
for fam, target in (("ipv4", "1.1.1.1"), ("ipv6", "2606:4700:4700::1111")):
|
||||
try:
|
||||
r = mini._local_address(fam, target)
|
||||
except mini._ProbeUnavailable:
|
||||
continue
|
||||
assert r is None or isinstance(r, str)
|
||||
|
||||
|
||||
# ---- heartbeat message and plugin wiring ---------------------------------------------
|
||||
|
||||
class _Conn:
|
||||
ackcount = 3
|
||||
rtts = [0.0, 12.5]
|
||||
|
||||
|
||||
class _FlagsProbe:
|
||||
def __init__(self, flags):
|
||||
self._flags = flags
|
||||
|
||||
def flags(self):
|
||||
return self._flags
|
||||
|
||||
|
||||
def test_heartbeat_without_probe_is_unchanged():
|
||||
assert mini._build_heartbeat_msg(_Conn(), 60) == {"acks": 3, "rtt": 12.5, "interval": 60}
|
||||
|
||||
|
||||
def test_heartbeat_carries_presence_flags():
|
||||
msg = mini._build_heartbeat_msg(_Conn(), 60, _FlagsProbe({"p4": True, "p6": False}))
|
||||
assert msg == {"acks": 3, "rtt": 12.5, "interval": 60, "p4": True, "p6": False}
|
||||
|
||||
|
||||
def test_heartbeat_omits_unknown_flags():
|
||||
msg = mini._build_heartbeat_msg(_Conn(), 60, _FlagsProbe({}))
|
||||
assert "p4" not in msg and "p6" not in msg
|
||||
|
||||
|
||||
def test_flags_survive_the_wire_encoding():
|
||||
msg = mini._build_heartbeat_msg(_Conn(), 60, _FlagsProbe({"p4": True, "p6": False}))
|
||||
out = mini._stodict(mini._dicttos("HTB", dict(msg, name="h", id=1)))
|
||||
assert out["p4"] == 1 and out["p6"] == 0
|
||||
|
||||
|
||||
class _FakeProbe:
|
||||
def as_metric(self):
|
||||
return {"ipv4": {"present": True, "addr": "192.0.2.5", "gateway": "192.0.2.1", "rtt": 9.0},
|
||||
"ipv6": {"present": False, "addr": None, "gateway": None, "rtt": None}}
|
||||
|
||||
|
||||
def test_plugin_reports_connectivity_alongside_interfaces(monkeypatch):
|
||||
monkeypatch.setattr(mini, "_probe", _FakeProbe())
|
||||
monkeypatch.setattr(mini.platform, "system", lambda: "Linux")
|
||||
plugin = mini.NetworkMonitorPlugin({})
|
||||
assert asyncio.run(plugin.initialize()) is True
|
||||
data = asyncio.run(plugin._collect_metrics())
|
||||
assert data["connectivity"]["ipv4"]["present"] is True
|
||||
assert data["connectivity"]["ipv6"]["present"] is False
|
||||
|
||||
|
||||
def test_plugin_probe_only_mode_on_non_linux(monkeypatch):
|
||||
monkeypatch.setattr(mini, "_probe", _FakeProbe())
|
||||
monkeypatch.setattr(mini.platform, "system", lambda: "FreeBSD")
|
||||
plugin = mini.NetworkMonitorPlugin({})
|
||||
assert asyncio.run(plugin.initialize()) is True
|
||||
assert list(asyncio.run(plugin._collect_metrics())) == ["connectivity"]
|
||||
|
||||
|
||||
def test_plugin_still_skips_on_non_linux_without_probe(monkeypatch):
|
||||
monkeypatch.setattr(mini, "_probe", None)
|
||||
monkeypatch.setattr(mini.platform, "system", lambda: "FreeBSD")
|
||||
plugin = mini.NetworkMonitorPlugin({})
|
||||
assert asyncio.run(plugin.initialize()) is False
|
||||
assert plugin.skip_reason
|
||||
|
||||
|
||||
def test_plugin_omits_connectivity_without_probe(monkeypatch):
|
||||
monkeypatch.setattr(mini, "_probe", None)
|
||||
monkeypatch.setattr(mini.platform, "system", lambda: "Linux")
|
||||
plugin = mini.NetworkMonitorPlugin({})
|
||||
assert asyncio.run(plugin.initialize()) is True
|
||||
assert "connectivity" not in asyncio.run(plugin._collect_metrics())
|
||||
@@ -0,0 +1,120 @@
|
||||
import time
|
||||
|
||||
import pytest
|
||||
|
||||
from hbd.server import hbdclass
|
||||
from hbd.server.hbdclass import Connection as C
|
||||
|
||||
|
||||
def _host(name):
|
||||
hbdclass.Host.hosts.pop(name, None)
|
||||
return hbdclass.Host(name)
|
||||
|
||||
|
||||
def _conn(host, afam, state):
|
||||
addr = "192.0.2.1" if afam == "IPv4" else "2001:db8::1"
|
||||
conn, _ = host.conndata(0, addr, None, 1000.0)
|
||||
conn.state = state
|
||||
return conn
|
||||
|
||||
|
||||
CASES = [
|
||||
# v4 state, v6 state, presence flags, expected status
|
||||
(C.UP, C.UP, {}, "up"),
|
||||
(C.UP, C.OVERDUE, {}, "simplex"), # both present (legacy) => simplex
|
||||
(C.UP, C.OVERDUE, {"IPv6": True}, "simplex"),
|
||||
(C.UP, C.OVERDUE, {"IPv6": False}, "up"), # v6 not present => not monitored
|
||||
(C.OVERDUE, C.OVERDUE, {}, "overdue"),
|
||||
(C.OVERDUE, C.ABSENT, {"IPv6": False}, "overdue"), # only present family is overdue
|
||||
(C.UP, C.ABSENT, {"IPv6": False}, "up"),
|
||||
(C.DOWN, C.UP, {}, "down"),
|
||||
(C.UNKNOWN, C.UNKNOWN, {}, "unknown"),
|
||||
(C.OVERDUE, C.UNKNOWN, {}, "overdue"),
|
||||
(C.UP, C.UNKNOWN, {}, "up"),
|
||||
(C.OVERDUE, C.UP, {"IPv6": False}, "simplex"), # HTB arriving => present despite flag
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("s4,s6,flags,expected", CASES)
|
||||
def test_status_matrix(s4, s6, flags, expected):
|
||||
h = _host("status-matrix")
|
||||
_conn(h, "IPv4", s4)
|
||||
_conn(h, "IPv6", s6)
|
||||
for afam, present in flags.items():
|
||||
h.set_presence(afam, present, time.time())
|
||||
assert h.status() == expected
|
||||
|
||||
|
||||
def test_status_single_family_host():
|
||||
h = _host("status-v4only")
|
||||
_conn(h, "IPv4", C.UP)
|
||||
assert h.status() == "up"
|
||||
h.connections["IPv4"].state = C.OVERDUE
|
||||
assert h.status() == "overdue"
|
||||
|
||||
|
||||
def test_status_no_connections_is_unknown():
|
||||
assert _host("status-empty").status() == "unknown"
|
||||
|
||||
|
||||
def test_family_present_rules():
|
||||
h = _host("fp")
|
||||
v6 = _conn(h, "IPv6", C.OVERDUE)
|
||||
assert h.family_present("IPv6") is True # never reported => present
|
||||
h.set_presence("IPv6", False, 1.0)
|
||||
assert h.family_present("IPv6") is False
|
||||
v6.state = C.UP # real traffic overrides the flag
|
||||
assert h.family_present("IPv6") is True
|
||||
assert h.flag_present("IPv6") is False
|
||||
|
||||
|
||||
def test_stateinfo_and_statedict_expose_status():
|
||||
h = _host("status-exposed")
|
||||
_conn(h, "IPv4", C.UP)
|
||||
assert h.stateinfo()["status"] == "up"
|
||||
assert h.statedict()["status"] == "up"
|
||||
|
||||
|
||||
def test_absent_connection_statedict_is_neutral():
|
||||
h = _host("absent-dict")
|
||||
conn = _conn(h, "IPv6", C.ABSENT)
|
||||
conn.rtts.append(42.0)
|
||||
d = conn.statedict()
|
||||
assert d["state"] == "absent" and d["rtt"] == "" and d["rttstate"] == "absent"
|
||||
|
||||
|
||||
def test_presence_survives_pickle_and_old_pickles_get_default():
|
||||
import pickle
|
||||
h = _host("pickled")
|
||||
h.set_presence("IPv6", False, 5.0)
|
||||
h2 = pickle.loads(pickle.dumps(h))
|
||||
assert h2.presence == {"IPv6": {"present": False, "ts": 5.0}}
|
||||
del h2.__dict__["presence"] # simulate a pre-feature pickle
|
||||
h2.fixup()
|
||||
assert h2.presence == {}
|
||||
|
||||
|
||||
def _all_absent(name, s4, s6=None):
|
||||
h = _host(name)
|
||||
_conn(h, "IPv4", s4)
|
||||
h.set_presence("IPv4", False, time.time())
|
||||
if s6 is not None:
|
||||
_conn(h, "IPv6", s6)
|
||||
h.set_presence("IPv6", False, time.time())
|
||||
return h
|
||||
|
||||
|
||||
def test_all_absent_single_family_overdue_is_overdue():
|
||||
assert _all_absent("abs-single", C.OVERDUE).status() == "overdue"
|
||||
|
||||
|
||||
def test_all_absent_dual_stack_one_absent_one_overdue_is_overdue():
|
||||
assert _all_absent("abs-dual-od", C.ABSENT, C.OVERDUE).status() == "overdue"
|
||||
|
||||
|
||||
def test_all_absent_dual_stack_v6_up_is_up():
|
||||
assert _all_absent("abs-dual-up", C.ABSENT, C.UP).status() == "up"
|
||||
|
||||
|
||||
def test_all_absent_all_unknown_is_unknown():
|
||||
assert _all_absent("abs-unknown", C.UNKNOWN, C.UNKNOWN).status() == "unknown"
|
||||
@@ -1,5 +1,4 @@
|
||||
"""Tests for _build_host_info helper in http.py."""
|
||||
import pytest
|
||||
from unittest.mock import MagicMock
|
||||
from hbd.server.http import _build_host_info
|
||||
|
||||
@@ -175,11 +174,11 @@ def test_build_host_info_covers_empty_when_exact_matches_only():
|
||||
|
||||
|
||||
class _FakeConnFull:
|
||||
def __init__(self, afam, addr, state, rtt, statetime, lastbeat):
|
||||
def __init__(self, afam, addr, state, rtt, statetime, lastbeat, rtts=None):
|
||||
self.afam = afam
|
||||
self.addr = addr
|
||||
self.state = state
|
||||
self.rtts = [rtt]
|
||||
self.rtts = rtts if rtts is not None else [rtt]
|
||||
self.statetime = statetime
|
||||
self.lastbeat = lastbeat
|
||||
|
||||
@@ -200,3 +199,14 @@ def test_build_host_info_connections_empty():
|
||||
host = _FakeHost()
|
||||
result = _build_host_info(host)
|
||||
assert result["connections"] == []
|
||||
|
||||
|
||||
def test_build_host_info_connection_rtt_uses_latest_sample():
|
||||
"""rtt should reflect the most recent sample, not the oldest retained one."""
|
||||
host = _FakeHost(connections={
|
||||
"IPv4": _FakeConnFull("IPv4", "10.0.0.5", "up", None, 1000.0, 2000.0,
|
||||
rtts=[100.0, 50.0, 14.2]),
|
||||
})
|
||||
result = _build_host_info(host)
|
||||
conns = {c["family"]: c for c in result["connections"]}
|
||||
assert conns["IPv4"]["rtt"] == 14.2
|
||||
|
||||
@@ -0,0 +1,23 @@
|
||||
"""Tests for the plugin-accordion visibility filter in http.py."""
|
||||
from hbd.server.http import _visible_plugin_names
|
||||
|
||||
|
||||
def test_visible_plugin_names_excludes_rtt_history_keys():
|
||||
plugin_data = {
|
||||
"cpu_monitor": [],
|
||||
"rtt_ipv4": [],
|
||||
"rtt_ipv6": [],
|
||||
"network_monitor": [],
|
||||
}
|
||||
result = _visible_plugin_names(plugin_data)
|
||||
assert sorted(result) == ["cpu_monitor", "network_monitor"]
|
||||
|
||||
|
||||
def test_visible_plugin_names_empty_when_no_plugins():
|
||||
assert _visible_plugin_names({}) == []
|
||||
|
||||
|
||||
def test_visible_plugin_names_keeps_names_not_prefixed_with_rtt_():
|
||||
plugin_data = {"nagios_runner": [], "os_info": []}
|
||||
result = _visible_plugin_names(plugin_data)
|
||||
assert sorted(result) == ["nagios_runner", "os_info"]
|
||||
@@ -0,0 +1,26 @@
|
||||
import asyncio
|
||||
|
||||
from hbd.client import connectivity
|
||||
from hbd.client.plugins.network_monitor import NetworkMonitorPlugin
|
||||
|
||||
|
||||
class _FakeShared:
|
||||
def as_metric(self):
|
||||
return {"ipv4": {"present": True, "addr": "192.0.2.5", "gateway": "192.0.2.1", "rtt": 9.0},
|
||||
"ipv6": {"present": False, "addr": None, "gateway": None, "rtt": None}}
|
||||
|
||||
|
||||
def test_collect_includes_connectivity_when_probe_running():
|
||||
connectivity.set_shared(_FakeShared())
|
||||
try:
|
||||
data = asyncio.run(NetworkMonitorPlugin()._collect_metrics())
|
||||
finally:
|
||||
connectivity.set_shared(None)
|
||||
assert data["connectivity"]["ipv4"]["present"] is True
|
||||
assert data["connectivity"]["ipv6"]["present"] is False
|
||||
|
||||
|
||||
def test_collect_omits_connectivity_without_probe():
|
||||
connectivity.set_shared(None)
|
||||
data = asyncio.run(NetworkMonitorPlugin()._collect_metrics())
|
||||
assert "connectivity" not in data
|
||||
@@ -0,0 +1,60 @@
|
||||
import sys
|
||||
|
||||
import pytest
|
||||
|
||||
from hbd.client import pingutil
|
||||
|
||||
LINUX_OK = """PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
|
||||
64 bytes from 1.1.1.1: icmp_seq=1 ttl=57 time=11.2 ms
|
||||
|
||||
--- 1.1.1.1 ping statistics ---
|
||||
1 packets transmitted, 1 received, 0% packet loss, time 0ms
|
||||
rtt min/avg/max/mdev = 11.234/11.234/11.234/0.000 ms
|
||||
"""
|
||||
|
||||
MACOS_OK = """PING 1.1.1.1 (1.1.1.1): 56 data bytes
|
||||
64 bytes from 1.1.1.1: icmp_seq=0 ttl=57 time=12.3 ms
|
||||
|
||||
--- 1.1.1.1 ping statistics ---
|
||||
1 packets transmitted, 1 packets received, 0.0% packet loss
|
||||
round-trip min/avg/max/stddev = 12.3/12.3/12.3/0.000 ms
|
||||
"""
|
||||
|
||||
LOST = """--- 1.1.1.1 ping statistics ---
|
||||
1 packets transmitted, 0 received, 100% packet loss, time 0ms
|
||||
"""
|
||||
|
||||
|
||||
def test_parse_linux():
|
||||
r = pingutil.parse_ping_output(LINUX_OK)
|
||||
assert r == {"rtt_min": 11.234, "rtt_avg": 11.234, "rtt_max": 11.234, "loss": 0.0}
|
||||
|
||||
|
||||
def test_parse_macos():
|
||||
r = pingutil.parse_ping_output(MACOS_OK)
|
||||
assert r["rtt_avg"] == 12.3 and r["loss"] == 0.0
|
||||
|
||||
|
||||
def test_parse_lost():
|
||||
r = pingutil.parse_ping_output(LOST)
|
||||
assert r["rtt_avg"] == float("inf") and r["loss"] == 100.0
|
||||
|
||||
|
||||
def test_parse_garbage_is_unreachable():
|
||||
r = pingutil.parse_ping_output("ping: connect: Network is unreachable")
|
||||
assert r["rtt_avg"] == float("inf") and r["loss"] == 100.0
|
||||
|
||||
|
||||
@pytest.mark.parametrize("platform,family,expected", [
|
||||
("linux", None, ["ping", "-c", "3", "-W", "5", "h"]),
|
||||
("linux", 6, ["ping", "-c", "3", "-W", "5", "-6", "h"]),
|
||||
("linux", 4, ["ping", "-c", "3", "-W", "5", "-4", "h"]),
|
||||
("win32", None, ["ping", "-n", "3", "-w", "5000", "h"]),
|
||||
("win32", 6, ["ping", "-n", "3", "-w", "5000", "-6", "h"]),
|
||||
("darwin", None, ["ping", "-c", "3", "-W", "5", "h"]),
|
||||
("darwin", 4, ["ping", "-c", "3", "-W", "5000", "h"]),
|
||||
("darwin", 6, ["ping6", "-c", "3", "h"]),
|
||||
])
|
||||
def test_ping_command(monkeypatch, platform, family, expected):
|
||||
monkeypatch.setattr(sys, "platform", platform)
|
||||
assert pingutil.ping_command("h", 3, 5, family) == expected
|
||||
@@ -0,0 +1,250 @@
|
||||
"""Host-level overdue alerting driven by client-reported protocol presence."""
|
||||
import asyncio
|
||||
import time
|
||||
|
||||
from hbd.common.proto import dicttos
|
||||
from hbd.server import hbdclass, udp
|
||||
from hbd.server.hbdclass import Connection as C
|
||||
from hbd.server.udp import handle_datagram, parse_message
|
||||
|
||||
|
||||
class _FakeTransport:
|
||||
def sendto(self, data, addr):
|
||||
pass
|
||||
|
||||
|
||||
def _ctx():
|
||||
return {"config": {}, "hbdclass": hbdclass, "msg_to_websockets": None,
|
||||
"DEBUG": 0, "verbose": False}
|
||||
|
||||
|
||||
def _fresh(name):
|
||||
hbdclass.Host.hosts.pop(name, None)
|
||||
|
||||
|
||||
def _send(name, addr, ctx, **fields):
|
||||
msg = parse_message(dicttos("HTB", {"name": name, "id": 0, **fields}))
|
||||
handle_datagram(msg, (addr, 50000), _FakeTransport(), ctx)
|
||||
|
||||
|
||||
def _events(monkeypatch):
|
||||
events = []
|
||||
monkeypatch.setattr(udp, "eventlog", lambda u, level, msg, **kw: events.append((level, msg)))
|
||||
return events
|
||||
|
||||
|
||||
def _dual_stack(name, ctx):
|
||||
_fresh(name)
|
||||
_send(name, "127.0.0.1", ctx)
|
||||
_send(name, "::1", ctx)
|
||||
return hbdclass.Host.hosts[name]
|
||||
|
||||
|
||||
def test_flags_are_stored_on_host():
|
||||
ctx = _ctx()
|
||||
_fresh("pres-store")
|
||||
_send("pres-store", "127.0.0.1", ctx, p4=1, p6=0)
|
||||
h = hbdclass.Host.hosts["pres-store"]
|
||||
assert h.presence["IPv4"]["present"] is True
|
||||
assert h.presence["IPv6"]["present"] is False
|
||||
|
||||
|
||||
def test_no_flags_leaves_presence_unknown():
|
||||
ctx = _ctx()
|
||||
_fresh("pres-legacy")
|
||||
_send("pres-legacy", "127.0.0.1", ctx)
|
||||
assert hbdclass.Host.hosts["pres-legacy"].presence == {}
|
||||
|
||||
|
||||
def test_absent_flag_moves_overdue_family_to_absent_and_clears_alert(monkeypatch):
|
||||
_events(monkeypatch)
|
||||
ctx = _ctx()
|
||||
h = _dual_stack("pres-absent", ctx)
|
||||
v6 = h.connections["IPv6"]
|
||||
v6.state = C.OVERDUE
|
||||
udp._set_connectivity_alert(h, "IPv6", "CRITICAL")
|
||||
_send("pres-absent", "127.0.0.1", ctx, p4=1, p6=0)
|
||||
assert v6.state == C.ABSENT
|
||||
assert "connectivity.IPv6" not in h.alert_states
|
||||
assert h.status() == "up"
|
||||
|
||||
|
||||
def test_absent_flag_does_not_touch_family_that_is_up():
|
||||
ctx = _ctx()
|
||||
h = _dual_stack("pres-up-stays", ctx)
|
||||
_send("pres-up-stays", "127.0.0.1", ctx, p4=1, p6=0)
|
||||
assert h.connections["IPv6"].state == C.UP # traffic overrides the flag
|
||||
|
||||
|
||||
def test_present_again_arms_grace_timer():
|
||||
ctx = _ctx()
|
||||
|
||||
async def scenario():
|
||||
_fresh("pres-back")
|
||||
_send("pres-back", "127.0.0.1", ctx, interval=10)
|
||||
_send("pres-back", "::1", ctx, interval=10)
|
||||
h = hbdclass.Host.hosts["pres-back"]
|
||||
v6 = h.connections["IPv6"]
|
||||
v6.cancel_overdue_timer()
|
||||
v6.state = C.ABSENT
|
||||
_send("pres-back", "127.0.0.1", ctx, interval=10, p6=1)
|
||||
assert v6.state == C.UNKNOWN and v6.overdue_timer is not None
|
||||
for c in h.connections.values():
|
||||
c.cancel_overdue_timer()
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_one_family_overdue_is_simplex_all_overdue_is_critical(monkeypatch):
|
||||
events = _events(monkeypatch)
|
||||
ctx = _ctx()
|
||||
h = _dual_stack("pres-simplex", ctx)
|
||||
on_overdue, _ = udp._make_timer_callbacks("pres-simplex", h, ctx)
|
||||
|
||||
asyncio.run(on_overdue(h.connections["IPv6"]))
|
||||
assert h.status() == "simplex"
|
||||
assert ("INFO", "IPv6 overdue (simplex)") in events
|
||||
assert not any(level == "CRITICAL" for level, _ in events)
|
||||
assert "connectivity.IPv6" not in h.alert_states
|
||||
|
||||
asyncio.run(on_overdue(h.connections["IPv4"]))
|
||||
assert h.status() == "overdue"
|
||||
assert [e for e in events if e[0] == "CRITICAL"] == [("CRITICAL", "IPv4, IPv6 overdue")]
|
||||
assert {"connectivity.IPv4", "connectivity.IPv6"} <= set(h.alert_states)
|
||||
|
||||
|
||||
def test_timer_expiry_on_absent_family_is_silent(monkeypatch):
|
||||
events = _events(monkeypatch)
|
||||
ctx = _ctx()
|
||||
h = _dual_stack("pres-quiet", ctx)
|
||||
_send("pres-quiet", "127.0.0.1", ctx, p4=1, p6=0) # v6 still UP => left alone
|
||||
events.clear()
|
||||
on_overdue, _ = udp._make_timer_callbacks("pres-quiet", h, ctx)
|
||||
asyncio.run(on_overdue(h.connections["IPv6"]))
|
||||
assert h.connections["IPv6"].state == C.ABSENT
|
||||
assert events == []
|
||||
assert h.status() == "up"
|
||||
|
||||
|
||||
def test_recovery_from_simplex_is_info_only(monkeypatch):
|
||||
events = _events(monkeypatch)
|
||||
ctx = _ctx()
|
||||
h = _dual_stack("pres-rec-simplex", ctx)
|
||||
v6 = h.connections["IPv6"]
|
||||
v6.state, v6.statetime = C.OVERDUE, time.time() - 100
|
||||
events.clear()
|
||||
_send("pres-rec-simplex", "::1", ctx)
|
||||
assert v6.state == C.UP
|
||||
assert [e[0] for e in events] == ["INFO"] and "back after" in events[0][1]
|
||||
|
||||
|
||||
def test_recovery_from_overdue_is_recover_and_clears_all_alerts(monkeypatch):
|
||||
events = _events(monkeypatch)
|
||||
ctx = _ctx()
|
||||
h = _dual_stack("pres-rec-overdue", ctx)
|
||||
for afam in ("IPv4", "IPv6"):
|
||||
c = h.connections[afam]
|
||||
c.state, c.statetime = C.OVERDUE, time.time() - 100
|
||||
udp._set_connectivity_alert(h, afam, "CRITICAL")
|
||||
events.clear()
|
||||
_send("pres-rec-overdue", "127.0.0.1", ctx)
|
||||
assert [e[0] for e in events] == ["RECOVER"]
|
||||
assert h.status() == "simplex"
|
||||
assert "connectivity.IPv6" not in h.alert_states # left overdue => all cleared
|
||||
assert "connectivity.IPv4" not in h.alert_states
|
||||
|
||||
|
||||
def test_restore_skips_absent_and_sets_no_alert_for_simplex():
|
||||
ctx = _ctx()
|
||||
|
||||
async def scenario():
|
||||
h = _dual_stack("pres-restore", ctx)
|
||||
h.connections["IPv6"].state = C.OVERDUE # v4 UP + v6 OVERDUE => simplex
|
||||
udp.restore_connection_timers(hbdclass, ctx)
|
||||
assert "connectivity.IPv6" not in h.alert_states
|
||||
h.connections["IPv6"].cancel_overdue_timer()
|
||||
h.connections["IPv6"].state = C.ABSENT
|
||||
h.set_presence("IPv6", False, time.time())
|
||||
udp.restore_connection_timers(hbdclass, ctx)
|
||||
assert h.connections["IPv6"].state == C.ABSENT
|
||||
for c in h.connections.values():
|
||||
c.cancel_overdue_timer()
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def _past_drop(conn):
|
||||
conn.state, conn.statetime = C.OVERDUE, time.time() - (udp.DROPOVERDUE + 10)
|
||||
|
||||
|
||||
def test_restore_past_drop_window_keeps_alert_single_family():
|
||||
ctx = _ctx()
|
||||
|
||||
async def scenario():
|
||||
_fresh("pres-drop-single")
|
||||
_send("pres-drop-single", "127.0.0.1", ctx)
|
||||
h = hbdclass.Host.hosts["pres-drop-single"]
|
||||
_past_drop(h.connections["IPv4"])
|
||||
udp.restore_connection_timers(hbdclass, ctx)
|
||||
assert h.connections["IPv4"].state == C.UNKNOWN
|
||||
assert "connectivity.IPv4" in h.alert_states
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_restore_past_drop_window_keeps_alerts_dual_family():
|
||||
ctx = _ctx()
|
||||
|
||||
async def scenario():
|
||||
h = _dual_stack("pres-drop-dual", ctx)
|
||||
for c in h.connections.values():
|
||||
_past_drop(c)
|
||||
udp.restore_connection_timers(hbdclass, ctx)
|
||||
assert {"connectivity.IPv4", "connectivity.IPv6"} <= set(h.alert_states)
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_v4_only_host_overdue_is_single_critical(monkeypatch):
|
||||
events = _events(monkeypatch)
|
||||
ctx = _ctx()
|
||||
_fresh("pres-v4only")
|
||||
_send("pres-v4only", "127.0.0.1", ctx)
|
||||
h = hbdclass.Host.hosts["pres-v4only"]
|
||||
on_overdue, _ = udp._make_timer_callbacks("pres-v4only", h, ctx)
|
||||
asyncio.run(on_overdue(h.connections["IPv4"]))
|
||||
assert [e for e in events if e[0] == "CRITICAL"] == [("CRITICAL", "IPv4 overdue")]
|
||||
assert "connectivity.IPv4" in h.alert_states
|
||||
asyncio.run(on_overdue(h.connections["IPv4"]))
|
||||
assert len([e for e in events if e[0] == "CRITICAL"]) == 1
|
||||
|
||||
|
||||
def test_all_families_absent_v4_only_host_still_alerts_when_it_dies(monkeypatch):
|
||||
events = _events(monkeypatch)
|
||||
ctx = _ctx()
|
||||
_fresh("pres-allabsent")
|
||||
_send("pres-allabsent", "127.0.0.1", ctx, p4=0)
|
||||
h = hbdclass.Host.hosts["pres-allabsent"]
|
||||
on_overdue, _ = udp._make_timer_callbacks("pres-allabsent", h, ctx)
|
||||
asyncio.run(on_overdue(h.connections["IPv4"]))
|
||||
assert [e for e in events if e[0] == "CRITICAL"] == [("CRITICAL", "IPv4 overdue")]
|
||||
assert h.connections["IPv4"].state == C.OVERDUE
|
||||
assert h.status() == "overdue"
|
||||
assert "connectivity.IPv4" in h.alert_states
|
||||
|
||||
|
||||
def test_last_present_family_is_not_parked_when_other_is_absent(monkeypatch):
|
||||
events = _events(monkeypatch)
|
||||
ctx = _ctx()
|
||||
h = _dual_stack("pres-lastfam", ctx)
|
||||
_send("pres-lastfam", "127.0.0.1", ctx, p4=0, p6=0)
|
||||
events.clear()
|
||||
on_overdue, _ = udp._make_timer_callbacks("pres-lastfam", h, ctx)
|
||||
asyncio.run(on_overdue(h.connections["IPv4"]))
|
||||
assert h.connections["IPv4"].state == C.ABSENT
|
||||
assert not any(level == "CRITICAL" for level, _ in events)
|
||||
assert h.status() == "up"
|
||||
asyncio.run(on_overdue(h.connections["IPv6"]))
|
||||
assert [e for e in events if e[0] == "CRITICAL"] == [("CRITICAL", "IPv6 overdue")]
|
||||
assert h.connections["IPv6"].state == C.OVERDUE
|
||||
assert h.status() == "overdue"
|
||||
@@ -0,0 +1,153 @@
|
||||
"""Tests for RTT history capture in udp.py's handle_datagram."""
|
||||
import time
|
||||
|
||||
from hbd.common.proto import dicttos
|
||||
from hbd.server import hbdclass
|
||||
from hbd.server.udp import handle_datagram, parse_message
|
||||
|
||||
|
||||
class _FakeTransport:
|
||||
def __init__(self):
|
||||
self.sent = []
|
||||
|
||||
def sendto(self, data, addr):
|
||||
self.sent.append((data, addr))
|
||||
|
||||
|
||||
def _htb(name, rtt=None, interval=0, boot=0):
|
||||
d = {"name": name, "interval": interval, "id": 0}
|
||||
if rtt is not None:
|
||||
d["rtt"] = rtt
|
||||
if boot:
|
||||
d["boot"] = boot
|
||||
return parse_message(dicttos("HTB", d))
|
||||
|
||||
|
||||
def _base_ctx():
|
||||
return {
|
||||
"config": {},
|
||||
"hbdclass": hbdclass,
|
||||
"msg_to_websockets": None,
|
||||
"DEBUG": 0,
|
||||
"verbose": False,
|
||||
}
|
||||
|
||||
|
||||
def test_handle_datagram_records_rtt_history_for_new_connection():
|
||||
hbdclass.Host.hosts.pop("rtt-hist-host", None)
|
||||
handle_datagram(_htb("rtt-hist-host", rtt=42.5), ("127.0.0.1", 50000),
|
||||
_FakeTransport(), _base_ctx())
|
||||
|
||||
host = hbdclass.Host.hosts["rtt-hist-host"]
|
||||
samples = host.plugin_data.get("rtt_ipv4")
|
||||
assert samples is not None
|
||||
assert len(samples) == 1
|
||||
ts, data = samples[0]
|
||||
assert data == {"rtt": 42.5}
|
||||
assert isinstance(ts, float)
|
||||
|
||||
|
||||
def test_handle_datagram_appends_rtt_history_across_heartbeats():
|
||||
hbdclass.Host.hosts.pop("rtt-hist-host2", None)
|
||||
transport = _FakeTransport()
|
||||
ctx = _base_ctx()
|
||||
handle_datagram(_htb("rtt-hist-host2", rtt=10.0), ("127.0.0.1", 50000), transport, ctx)
|
||||
handle_datagram(_htb("rtt-hist-host2", rtt=20.0), ("127.0.0.1", 50000), transport, ctx)
|
||||
|
||||
host = hbdclass.Host.hosts["rtt-hist-host2"]
|
||||
samples = host.plugin_data["rtt_ipv4"]
|
||||
assert [d["rtt"] for _, d in samples] == [10.0, 20.0]
|
||||
|
||||
|
||||
def test_handle_datagram_skips_rtt_history_when_rtt_missing():
|
||||
hbdclass.Host.hosts.pop("rtt-hist-host3", None)
|
||||
handle_datagram(_htb("rtt-hist-host3", rtt=None), ("127.0.0.1", 50000),
|
||||
_FakeTransport(), _base_ctx())
|
||||
|
||||
host = hbdclass.Host.hosts["rtt-hist-host3"]
|
||||
assert "rtt_ipv4" not in host.plugin_data
|
||||
|
||||
|
||||
def test_request_update_fires_on_recovery_even_with_rtt_history():
|
||||
"""Regression for Finding 1: rtt_* keys must not permanently disable the
|
||||
request_update gate. A connection recovering from a non-UP state must
|
||||
still be asked to resend real plugin data, even though rtt_ipv4 already
|
||||
holds samples from before the drop.
|
||||
"""
|
||||
hbdclass.Host.hosts.pop("rtt-hist-host4", None)
|
||||
transport = _FakeTransport()
|
||||
ctx = _base_ctx()
|
||||
|
||||
# First heartbeat: brand-new host, no plugin data at all yet.
|
||||
handle_datagram(_htb("rtt-hist-host4", rtt=15.0), ("127.0.0.1", 50000), transport, ctx)
|
||||
host = hbdclass.Host.hosts["rtt-hist-host4"]
|
||||
assert host.plugin_data.get("rtt_ipv4") # rtt history now non-empty
|
||||
|
||||
# Simulate a recovery: connection was dropped (e.g. OVERDUE->UP after a
|
||||
# missed heartbeat) and is about to come back UP on the next heartbeat.
|
||||
conn = host.connections["IPv4"]
|
||||
conn.state = hbdclass.Connection.DOWN
|
||||
|
||||
transport.sent.clear()
|
||||
handle_datagram(_htb("rtt-hist-host4", rtt=16.0), ("127.0.0.1", 50000), transport, ctx)
|
||||
|
||||
ack_data, _ = transport.sent[0]
|
||||
ack = parse_message(ack_data)
|
||||
assert ack.get("request_update")
|
||||
|
||||
|
||||
def test_ordinary_recovery_preserves_real_plugin_data_and_rtt_history():
|
||||
"""An ordinary reconnect (no boot flag) — e.g. OVERDUE->UP after a
|
||||
transient network blip — must NOT wipe already-collected real plugin
|
||||
data (e.g. cpu_monitor, os_info) or rtt_* history. Only an actual
|
||||
client reboot invalidates that data (see the boot-flag test below).
|
||||
"""
|
||||
hbdclass.Host.hosts.pop("rtt-hist-host5", None)
|
||||
transport = _FakeTransport()
|
||||
ctx = _base_ctx()
|
||||
|
||||
for rtt in (10.0, 11.0, 12.0):
|
||||
handle_datagram(_htb("rtt-hist-host5", rtt=rtt), ("127.0.0.1", 50000), transport, ctx)
|
||||
|
||||
host = hbdclass.Host.hosts["rtt-hist-host5"]
|
||||
assert len(host.plugin_data["rtt_ipv4"]) == 3
|
||||
|
||||
# Simulate a drop, and pretend the client had previously sent real
|
||||
# plugin data (collected before the connection went down).
|
||||
conn = host.connections["IPv4"]
|
||||
conn.state = hbdclass.Connection.DOWN
|
||||
host.add_plugin_data("os_info", {"os": "linux"}, timestamp=time.time())
|
||||
assert "os_info" in host.plugin_data
|
||||
|
||||
# Ordinary recovery heartbeat — no boot flag.
|
||||
handle_datagram(_htb("rtt-hist-host5", rtt=13.0), ("127.0.0.1", 50000), transport, ctx)
|
||||
|
||||
assert "os_info" in host.plugin_data
|
||||
assert len(host.plugin_data["rtt_ipv4"]) == 4
|
||||
|
||||
|
||||
def test_boot_recovery_clears_real_plugin_data_but_preserves_rtt_history():
|
||||
"""A recovery heartbeat carrying the boot flag (client process actually
|
||||
restarted) must still wipe stale real plugin data, while rtt_* history
|
||||
(still a valid measurement, unaffected by a client reboot) survives.
|
||||
"""
|
||||
hbdclass.Host.hosts.pop("rtt-hist-host6", None)
|
||||
transport = _FakeTransport()
|
||||
ctx = _base_ctx()
|
||||
|
||||
for rtt in (10.0, 11.0, 12.0):
|
||||
handle_datagram(_htb("rtt-hist-host6", rtt=rtt), ("127.0.0.1", 50000), transport, ctx)
|
||||
|
||||
host = hbdclass.Host.hosts["rtt-hist-host6"]
|
||||
assert len(host.plugin_data["rtt_ipv4"]) == 3
|
||||
|
||||
conn = host.connections["IPv4"]
|
||||
conn.state = hbdclass.Connection.DOWN
|
||||
host.add_plugin_data("os_info", {"os": "linux"}, timestamp=time.time())
|
||||
assert "os_info" in host.plugin_data
|
||||
|
||||
# Recovery heartbeat with boot=1 — client process actually restarted.
|
||||
handle_datagram(_htb("rtt-hist-host6", rtt=13.0, boot=1), ("127.0.0.1", 50000), transport, ctx)
|
||||
|
||||
assert "os_info" not in host.plugin_data
|
||||
assert len(host.plugin_data["rtt_ipv4"]) == 4
|
||||
Reference in New Issue
Block a user