Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
13f9916693 | ||
|
|
80b830fb07 | ||
|
|
6790ff53a8 | ||
|
|
445ff6b474 | ||
|
|
748349a7b9 | ||
|
|
85c6a4eebe | ||
|
|
952adb68f8 | ||
|
|
3e2d343fe7 | ||
|
|
885d5dc937 | ||
|
|
e8d9c4a958 | ||
|
|
328f248c43 | ||
|
|
747a8c27c9 | ||
|
|
5243f0a834 | ||
|
|
3c6f7fc868 | ||
|
|
adbd2f9b46 | ||
|
|
248b17edca | ||
|
|
3639464ba6 | ||
|
|
997279dce8 |
@@ -2,6 +2,35 @@
|
||||
|
||||
All notable changes to this project are documented here, organized by release.
|
||||
|
||||
## [5.4.7]
|
||||
|
||||
### Fixed
|
||||
- clear all mypy errors
|
||||
- honour the legacy thresholds: config; repair the stale test suite
|
||||
- ignore datagrams that are not named HTB/PLG messages
|
||||
|
||||
---
|
||||
|
||||
## [5.4.6]
|
||||
|
||||
### Added
|
||||
- re-resolve the server hostnames at the DNS TTL
|
||||
|
||||
---
|
||||
|
||||
## [5.4.5]
|
||||
|
||||
### Added
|
||||
- `hbc_mini.py` and `hbc_mini.c` (the dependency-free clients) get the same per-protocol connectivity probe as `hbc`: `p4`/`p6` flags on every heartbeat and a `connectivity` metric (presence, own address, gateway, RTT) via `network_monitor`, with gateway lookup on Linux, FreeBSD/NetBSD/DragonFly (both clients) and macOS/Windows (`hbc_mini.py` only)
|
||||
- `scripts/c/test_probe.c` and a `make test` target — the C client's first automated test suite
|
||||
|
||||
### Fixed
|
||||
- `hbc_mini.c`: the clean-shutdown heartbeat could be delayed or lost behind an in-flight connectivity probe on SIGTERM; it is now sent before the probe thread is joined
|
||||
- `hbc_mini.c`: a connectivity probe that fails to start, or is configured with a non-IP-literal target, now logs a warning instead of failing silently
|
||||
- `hbc_mini.py`: removed an unused import
|
||||
|
||||
---
|
||||
|
||||
## [5.4.4]
|
||||
|
||||
### Added
|
||||
|
||||
@@ -33,7 +33,7 @@ hbd passwd <username>
|
||||
|
||||
Tests live in `tests/` (pytest, imported as a package). A second directory `test/` contains only TLS fixtures (`.pem` files), not test code.
|
||||
|
||||
Line length: 111 characters (`black`, `flake8`, both configured).
|
||||
Line length: `black` and `flake8` both use 165 characters (set in `pyproject.toml`, `tox.ini` and `.flake8`).
|
||||
|
||||
## Architecture
|
||||
|
||||
|
||||
@@ -20,7 +20,7 @@ A lightweight UDP-based host monitoring system. Monitored hosts run a client (`h
|
||||
└────────────────────┘ └────────────────────────────┘
|
||||
```
|
||||
|
||||
**Package:** `hbd` v5.4.4
|
||||
**Package:** `hbd` v5.4.7
|
||||
**Python:** 3.11+
|
||||
|
||||
### Subpackages
|
||||
@@ -242,6 +242,7 @@ plugins:
|
||||
### Connection behaviour
|
||||
|
||||
- The client sends heartbeats over UDP to each server address resolved from the hostname (IPv4 and IPv6).
|
||||
- The server hostname is re-resolved every TTL seconds (the smallest A/AAAA TTL, at least 30s; 300s when the TTL cannot be determined, e.g. an IP literal), so an address change is followed without restarting the client. A failed lookup keeps the current addresses.
|
||||
- If a connection fails to open at startup, IPv6 connections are dropped after 3 consecutive failures. IPv4 connections retry indefinitely.
|
||||
- In daemon mode (`-d`), all log output goes to syslog (`LOG_DAEMON` facility).
|
||||
|
||||
@@ -659,7 +660,28 @@ Config: `~/.hbc.json` (JSON format, same keys as `~/.hbc.yaml`).
|
||||
| `cpu_monitor` | Linux (`/proc/stat`; no per-core, no frequency) |
|
||||
| `memory_monitor` | Linux (`/proc/meminfo`) |
|
||||
| `disk_monitor` | Linux, macOS, BSD (`df -P`) |
|
||||
| `network_monitor` | Linux (`/proc/net/dev`) |
|
||||
| `network_monitor` | Linux (`/proc/net/dev`); per-protocol connectivity probe (all platforms) |
|
||||
|
||||
`network_monitor` also reports per-protocol (IPv4/IPv6) internet presence — whether the host
|
||||
has a working route and reply to a well-known address, plus its own address, the local
|
||||
gateway, and RTT — as `p4`/`p6` flags on every heartbeat and a `connectivity` object in the
|
||||
plugin data. Configurable, defaults shown:
|
||||
|
||||
```json
|
||||
{
|
||||
"plugins": {
|
||||
"network_monitor": {
|
||||
"probe_enabled": true,
|
||||
"probe_v4": "1.1.1.1",
|
||||
"probe_v6": "2606:4700:4700::1111",
|
||||
"probe_interval": 60
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
`probe_v4`/`probe_v6` must be IP literals (no hostnames). Gateway lookup works on Linux,
|
||||
FreeBSD/NetBSD/DragonFly/macOS (`route -n get`), and Windows (`route print`).
|
||||
|
||||
Not available vs full `hbc`: no YAML config, no `filesystem_info`, no `zfs_monitor`, no IPv6 early-fail protection.
|
||||
|
||||
@@ -712,6 +734,7 @@ SIGHUP causes the process to restart itself via `execv()`. SIGTERM/SIGINT trigge
|
||||
| `disk_monitor` | All | `df -P` subprocess |
|
||||
| `network_monitor` | Linux | `/proc/net/dev` |
|
||||
| `network_monitor` | FreeBSD, NetBSD, DragonFly | `getifaddrs()` + `AF_LINK` |
|
||||
| `network_monitor` (connectivity probe) | Linux, FreeBSD, NetBSD, DragonFly | UDP `connect()` route check + one ICMP echo per protocol |
|
||||
| `ping_monitor` | All | `ping` subprocess |
|
||||
| `nagios_runner` | All | `popen()` subprocess |
|
||||
|
||||
@@ -743,12 +766,33 @@ SIGHUP causes the process to restart itself via `execv()`. SIGTERM/SIGINT trigge
|
||||
}
|
||||
```
|
||||
|
||||
`network_monitor` also runs the per-protocol connectivity probe described above, in a
|
||||
dedicated thread (its ICMP echo can block for a few seconds). Same config keys and defaults
|
||||
as `hbc_mini.py`:
|
||||
|
||||
```json
|
||||
{
|
||||
"plugins": {
|
||||
"network_monitor": {
|
||||
"probe_enabled": true,
|
||||
"probe_v4": "1.1.1.1",
|
||||
"probe_v6": "2606:4700:4700::1111",
|
||||
"probe_interval": 60
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Gateway lookup on the C client covers Linux and FreeBSD/NetBSD/DragonFly only (no macOS or
|
||||
Windows build of `hbc_mini.c` exists).
|
||||
|
||||
### Differences from `hbc_mini.py`
|
||||
|
||||
- No `filesystem_info` or `zfs_monitor` plugins
|
||||
- `UPD` (self-update) messages are logged but not acted on
|
||||
- No IPv6 early-fail protection
|
||||
- Config is JSON only (`~/.hbc.json`), no YAML
|
||||
- No macOS or Windows build (Linux and the BSDs only)
|
||||
|
||||
---
|
||||
|
||||
|
||||
+1
-1
@@ -14,4 +14,4 @@ Install options:
|
||||
"""
|
||||
|
||||
__all__ = ["__version__"]
|
||||
__version__ = "5.4.4"
|
||||
__version__ = "5.4.7"
|
||||
|
||||
@@ -1,3 +1,3 @@
|
||||
"""HeartBeat Client (hbc) - System monitoring client."""
|
||||
|
||||
from hbd import __version__
|
||||
from hbd import __version__ # noqa: F401
|
||||
|
||||
@@ -2,16 +2,16 @@
|
||||
|
||||
import logging
|
||||
import os
|
||||
import logging
|
||||
from typing import Any, Dict
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
try:
|
||||
import yaml
|
||||
except Exception:
|
||||
yaml = None
|
||||
yaml = None # type: ignore[assignment]
|
||||
|
||||
CLIENT_DEFAULTS = {
|
||||
CLIENT_DEFAULTS: Dict[str, Any] = {
|
||||
# Network settings
|
||||
"hb_port": 50003, # Port where hbd servers listen
|
||||
"interval": 10, # Heartbeat interval in seconds
|
||||
|
||||
@@ -0,0 +1,170 @@
|
||||
"""Re-resolution of the server hostnames at the TTL of their A/AAAA records.
|
||||
|
||||
getaddrinfo() hides the TTL, so a minimal stub-resolver query (resolv.conf nameservers, search
|
||||
list) is made only to learn it. The addresses themselves still come from getaddrinfo(). When
|
||||
the TTL cannot be learned (IP literal, /etc/hosts name, no resolv.conf, resolver timeout)
|
||||
DEFAULT_TTL is used.
|
||||
"""
|
||||
|
||||
import ipaddress
|
||||
import logging
|
||||
import os
|
||||
import socket
|
||||
import struct
|
||||
from typing import List, Optional, Tuple
|
||||
|
||||
logger = logging.getLogger("hbc.dns")
|
||||
|
||||
DEFAULT_TTL = 300 # used when the real TTL is unknown
|
||||
MIN_TTL = 30 # a TTL of 0 must not turn into a resolution loop
|
||||
FAILED_RETRY = 60 # re-try delay after a failed resolution
|
||||
QUERY_TIMEOUT = 2.0
|
||||
|
||||
TYPE_A, TYPE_CNAME, TYPE_AAAA = 1, 5, 28
|
||||
|
||||
|
||||
def parse_resolv_conf(path: str = "/etc/resolv.conf") -> Tuple[List[str], List[str]]:
|
||||
"""Return (nameservers, search domains)."""
|
||||
servers: List[str] = []
|
||||
search: List[str] = []
|
||||
try:
|
||||
with open(path) as f:
|
||||
for line in f:
|
||||
parts = line.split()
|
||||
if len(parts) < 2:
|
||||
continue
|
||||
if parts[0] == "nameserver":
|
||||
servers.append(parts[1].split("%")[0])
|
||||
elif parts[0] in ("search", "domain"):
|
||||
search = parts[1:]
|
||||
except OSError:
|
||||
pass
|
||||
return servers, search
|
||||
|
||||
|
||||
def build_query(name: str, qtype: int, qid: int) -> bytes:
|
||||
out = struct.pack("!HHHHHH", qid, 0x0100, 1, 0, 0, 0)
|
||||
for label in name.rstrip(".").split("."):
|
||||
out += bytes([len(label)]) + label.encode("idna")
|
||||
return out + b"\x00" + struct.pack("!HH", qtype, 1)
|
||||
|
||||
|
||||
def _skip_name(buf: bytes, off: int) -> int:
|
||||
while True:
|
||||
n = buf[off]
|
||||
if n == 0:
|
||||
return off + 1
|
||||
if n & 0xC0:
|
||||
return off + 2
|
||||
off += n + 1
|
||||
|
||||
|
||||
def min_answer_ttl(resp: bytes, qid: int, qtype: int) -> Optional[int]:
|
||||
"""Smallest TTL among the answer RRs (CNAMEs included), or None if there is no qtype answer."""
|
||||
try:
|
||||
rid, flags, qd, an = struct.unpack("!HHHH", resp[:8])
|
||||
if rid != qid or flags & 0x000F:
|
||||
return None
|
||||
off = 12
|
||||
for _ in range(qd):
|
||||
off = _skip_name(resp, off) + 4
|
||||
ttls: List[int] = []
|
||||
found = False
|
||||
for _ in range(an):
|
||||
off = _skip_name(resp, off)
|
||||
rtype, _cls, ttl, rdlen = struct.unpack("!HHIH", resp[off:off + 10])
|
||||
off += 10 + rdlen
|
||||
if rtype in (qtype, TYPE_CNAME):
|
||||
ttls.append(ttl)
|
||||
found = found or rtype == qtype
|
||||
return min(ttls) if found else None
|
||||
except (struct.error, IndexError):
|
||||
return None
|
||||
|
||||
|
||||
def _query_ttl(server: str, name: str, qtype: int) -> Optional[int]:
|
||||
qid = struct.unpack("!H", os.urandom(2))[0]
|
||||
try:
|
||||
af = socket.AF_INET6 if ":" in server else socket.AF_INET
|
||||
with socket.socket(af, socket.SOCK_DGRAM) as s:
|
||||
s.settimeout(QUERY_TIMEOUT)
|
||||
s.sendto(build_query(name, qtype, qid), (server, 53))
|
||||
return min_answer_ttl(s.recv(4096), qid, qtype)
|
||||
except (OSError, UnicodeError):
|
||||
return None
|
||||
|
||||
|
||||
def lookup_ttl(host: str, af_filter: int = 0) -> Optional[int]:
|
||||
"""Blocking. Smallest A/AAAA TTL for host, or None when it cannot be determined."""
|
||||
try:
|
||||
ipaddress.ip_address(host)
|
||||
return None
|
||||
except ValueError:
|
||||
pass
|
||||
servers, search = parse_resolv_conf()
|
||||
if host.endswith("."):
|
||||
names = [host]
|
||||
elif "." in host:
|
||||
names = [host] + [f"{host}.{d}" for d in search]
|
||||
else:
|
||||
names = [f"{host}.{d}" for d in search] + [host]
|
||||
qtypes = [t for t, af in ((TYPE_A, socket.AF_INET), (TYPE_AAAA, socket.AF_INET6))
|
||||
if af_filter in (0, af)]
|
||||
for name in names:
|
||||
ttls = []
|
||||
for qtype in qtypes:
|
||||
for server in servers:
|
||||
ttl = _query_ttl(server, name, qtype)
|
||||
if ttl is not None:
|
||||
ttls.append(ttl)
|
||||
break
|
||||
if ttls:
|
||||
return min(ttls)
|
||||
return None
|
||||
|
||||
|
||||
def refresh_delay(ttls: List[Optional[int]]) -> int:
|
||||
"""Seconds until the next resolution: the smallest known TTL, floored at MIN_TTL."""
|
||||
known = [t for t in ttls if t is not None]
|
||||
return max(min(known), MIN_TTL) if known else DEFAULT_TTL
|
||||
|
||||
|
||||
def resolve(hosts: List[str], port: int, af_filter: int) -> Optional[List[Tuple[int, str]]]:
|
||||
"""Blocking. Every (family, address) the hosts resolve to, deduplicated, or None if any lookup fails."""
|
||||
found: List[Tuple[int, str]] = []
|
||||
for host in hosts:
|
||||
try:
|
||||
infos = socket.getaddrinfo(host, port, af_filter, 0, socket.SOL_UDP)
|
||||
except socket.gaierror as e:
|
||||
logger.warning(f"Cannot resolve {host}: {e}")
|
||||
return None
|
||||
for info in infos:
|
||||
entry = (info[0], str(info[4][0]))
|
||||
if entry not in found:
|
||||
found.append(entry)
|
||||
return found or None
|
||||
|
||||
|
||||
def retarget(connections: list, resolved: List[Tuple[int, str]]) -> Tuple[List[Tuple[int, str]], list]:
|
||||
"""Point the connections at the resolved addresses, in place.
|
||||
|
||||
A connection whose address is gone is re-pointed at an address that is new, in list order, so
|
||||
the first connection (the one plugin data uses) survives as long as anything resolves.
|
||||
Returns (new addresses still needing a connection, connections to drop).
|
||||
"""
|
||||
current = {c.addr for c in connections}
|
||||
wanted = {addr for _, addr in resolved}
|
||||
new = [(af, addr) for af, addr in resolved if addr not in current]
|
||||
gone = [c for c in connections if c.addr not in wanted]
|
||||
paired = min(len(gone), len(new))
|
||||
for conn, (af, addr) in zip(gone, new):
|
||||
logger.info(f"Server address {conn.addr} -> {addr}")
|
||||
conn.set_target(addr, af)
|
||||
drop = gone[paired:]
|
||||
if connections and connections[0] in drop:
|
||||
# plugin data goes out on the first connection: hand it a surviving address and drop that one instead
|
||||
keep = next(c for c in connections if c not in gone)
|
||||
connections[0].set_target(keep.addr, keep.af)
|
||||
drop.remove(connections[0])
|
||||
drop.append(keep)
|
||||
return new[paired:], drop
|
||||
+57
-3
@@ -16,7 +16,7 @@ import sys
|
||||
import time
|
||||
from logging.handlers import SysLogHandler
|
||||
from pathlib import Path
|
||||
from typing import Dict, List, Optional
|
||||
from typing import List, Optional
|
||||
|
||||
# Import protocol and config
|
||||
from .config import load_config
|
||||
@@ -25,6 +25,7 @@ from .. import __version__
|
||||
|
||||
# Import plugin system
|
||||
from .plugin import PluginRegistry, PluginLoader, InfoPlugin, MonitorPlugin
|
||||
from . import dnsttl
|
||||
from .connectivity import Connectivity, set_shared
|
||||
|
||||
# Constants
|
||||
@@ -35,6 +36,7 @@ MAXRECV = 32767
|
||||
# Global state
|
||||
running = True
|
||||
dorestart = False
|
||||
send_shutdown = False
|
||||
shutdown_event: Optional[asyncio.Event] = None
|
||||
active_tasks: List[asyncio.Task] = []
|
||||
|
||||
@@ -96,6 +98,14 @@ class AsyncConnection:
|
||||
self.transport = None
|
||||
self.protocol = None
|
||||
|
||||
def set_target(self, addr: str, af: int):
|
||||
"""Re-point at a new server address; the socket is reopened on the next send if the family changed."""
|
||||
if af != self.af:
|
||||
self.close()
|
||||
self.af = af
|
||||
self.addr = addr
|
||||
self.logger = logging.getLogger(f"hbc.conn.{addr}")
|
||||
|
||||
async def sendto(self, msg: dict, msg_id: str = "HTB"):
|
||||
"""Send a message to the server.
|
||||
|
||||
@@ -500,6 +510,48 @@ def stop():
|
||||
task.cancel()
|
||||
|
||||
|
||||
async def dns_refresher(connections: List[AsyncConnection], hosts: List[str], port: int, af_filter: int,
|
||||
name: str, interval: int, connectivity: Optional[Connectivity], next_id: int):
|
||||
"""Re-resolve the server hosts every TTL seconds and follow address changes.
|
||||
|
||||
A failed lookup leaves the connections untouched. Addresses that disappear are re-pointed at
|
||||
new ones where possible; the rest are retired, and surplus new ones get a connection.
|
||||
"""
|
||||
logger = logging.getLogger("hbc.dns")
|
||||
loop = asyncio.get_running_loop()
|
||||
delay = await loop.run_in_executor(None, _ttl_delay, hosts, af_filter)
|
||||
while running:
|
||||
try:
|
||||
if shutdown_event:
|
||||
await asyncio.wait_for(shutdown_event.wait(), timeout=delay)
|
||||
break
|
||||
await asyncio.sleep(delay)
|
||||
except asyncio.TimeoutError:
|
||||
pass
|
||||
resolved = await loop.run_in_executor(None, dnsttl.resolve, hosts, port, af_filter)
|
||||
if resolved is None:
|
||||
delay = dnsttl.FAILED_RETRY
|
||||
continue
|
||||
new, gone = dnsttl.retarget(connections, resolved)
|
||||
for conn in gone:
|
||||
if len(connections) > 1:
|
||||
logger.info(f"Dropping server address {conn.addr}")
|
||||
conn._dead = True
|
||||
conn.close()
|
||||
connections.remove(conn)
|
||||
for af, addr in new:
|
||||
conn = AsyncConnection(next_id, addr, port, af, name)
|
||||
next_id += 1
|
||||
logger.info(f"Adding server address {addr}")
|
||||
connections.append(conn)
|
||||
active_tasks.append(asyncio.create_task(heartbeat_sender(conn, interval, connectivity)))
|
||||
delay = await loop.run_in_executor(None, _ttl_delay, hosts, af_filter)
|
||||
|
||||
|
||||
def _ttl_delay(hosts: List[str], af_filter: int) -> int:
|
||||
return dnsttl.refresh_delay([dnsttl.lookup_ttl(h, af_filter) for h in hosts])
|
||||
|
||||
|
||||
async def cleanup(connections: List[AsyncConnection]):
|
||||
"""Cleanup connections on shutdown."""
|
||||
logger = logging.getLogger("hbc.cleanup")
|
||||
@@ -521,7 +573,7 @@ async def cleanup(connections: List[AsyncConnection]):
|
||||
|
||||
async def async_main(args, config):
|
||||
"""Async main function."""
|
||||
global running, shutdown_event, active_tasks, send_shutdown, allow_remote_command
|
||||
global shutdown_event, active_tasks, send_shutdown, allow_remote_command
|
||||
|
||||
# Create shutdown event
|
||||
shutdown_event = asyncio.Event()
|
||||
@@ -640,6 +692,9 @@ async def async_main(args, config):
|
||||
task = asyncio.create_task(heartbeat_sender(conn, interval, connectivity))
|
||||
active_tasks.append(task)
|
||||
|
||||
active_tasks.append(asyncio.create_task(
|
||||
dns_refresher(connections, hb_hosts, hb_port, af_filter, iam, interval, connectivity, conn_id)))
|
||||
|
||||
# Plugin collector (uses all connections, but we'll use first one)
|
||||
if connections and registry.get_enabled():
|
||||
task = asyncio.create_task(plugin_collector(connections[0], registry))
|
||||
@@ -784,7 +839,6 @@ def build_parser():
|
||||
|
||||
def main(argv=None):
|
||||
"""Main entry point."""
|
||||
global running, dorestart
|
||||
|
||||
parser = build_parser()
|
||||
args = parser.parse_args(argv)
|
||||
|
||||
@@ -17,7 +17,7 @@ import logging
|
||||
import sys
|
||||
from abc import ABC, abstractmethod
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List, Optional, Type
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
|
||||
class Plugin(ABC):
|
||||
@@ -256,7 +256,7 @@ class PluginRegistry:
|
||||
"""Get all enabled plugins."""
|
||||
return [p for p in self._plugins.values() if p.enabled]
|
||||
|
||||
def get_by_type(self, plugin_type: Type[Plugin]) -> List[Plugin]:
|
||||
def get_by_type(self, plugin_type: type) -> List[Plugin]:
|
||||
"""Get all plugins of a specific type.
|
||||
|
||||
Args:
|
||||
|
||||
@@ -5,8 +5,6 @@ load average, and process counts.
|
||||
"""
|
||||
|
||||
from typing import Any, Dict, Optional
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
# Import from parent package
|
||||
from hbd.client.plugin import MonitorPlugin
|
||||
@@ -30,7 +28,7 @@ class CPUMonitorPlugin(MonitorPlugin):
|
||||
|
||||
def __init__(self, config: Optional[Dict[str, Any]] = None):
|
||||
super().__init__(config)
|
||||
self.psutil = None
|
||||
self.psutil: Any = None
|
||||
self.per_core = config.get("per_core", False) if config else False
|
||||
self.interval = config.get("interval", 300) if config else 300
|
||||
|
||||
|
||||
@@ -5,12 +5,12 @@ Collects disk usage and I/O statistics using psutil.
|
||||
"""
|
||||
|
||||
import logging
|
||||
from typing import Dict, Any, Optional, List
|
||||
from typing import Dict, Any, Optional
|
||||
|
||||
try:
|
||||
import psutil
|
||||
except ImportError:
|
||||
psutil = None
|
||||
psutil = None # type: ignore[assignment]
|
||||
|
||||
from hbd.client.plugin import MonitorPlugin
|
||||
|
||||
@@ -58,7 +58,7 @@ class DiskMonitorPlugin(MonitorPlugin):
|
||||
raise ImportError("psutil library is required for disk_monitor plugin")
|
||||
|
||||
# Store previous I/O counters for delta calculation
|
||||
self._prev_io = {}
|
||||
self._prev_io: Dict[str, Any] = {}
|
||||
|
||||
async def initialize(self):
|
||||
"""Initialize the plugin (check psutil availability)."""
|
||||
@@ -114,7 +114,7 @@ class DiskMonitorPlugin(MonitorPlugin):
|
||||
|
||||
async def _collect_metrics(self) -> Dict[str, Any]:
|
||||
"""Collect disk metrics from psutil."""
|
||||
metrics = {}
|
||||
metrics: Dict[str, Any] = {}
|
||||
|
||||
# Collect partition usage
|
||||
partitions_data = {}
|
||||
|
||||
@@ -10,7 +10,7 @@ from typing import Dict, Any, Optional
|
||||
try:
|
||||
import psutil
|
||||
except ImportError:
|
||||
psutil = None
|
||||
psutil = None # type: ignore[assignment]
|
||||
|
||||
from hbd.client.plugin import InfoPlugin
|
||||
|
||||
@@ -98,8 +98,8 @@ class FilesystemInfoPlugin(InfoPlugin):
|
||||
|
||||
async def _collect_info(self) -> Dict[str, Any]:
|
||||
"""Collect filesystem information from psutil."""
|
||||
info = {}
|
||||
filesystems = []
|
||||
info: Dict[str, Any] = {}
|
||||
filesystems: list = []
|
||||
filesystem_types = set()
|
||||
|
||||
# Get mounted disk partitions
|
||||
@@ -112,7 +112,7 @@ class FilesystemInfoPlugin(InfoPlugin):
|
||||
if partition.fstype in self.exclude_types:
|
||||
continue
|
||||
|
||||
fs_info = {
|
||||
fs_info: Dict[str, Any] = {
|
||||
'device': partition.device,
|
||||
'mountpoint': partition.mountpoint,
|
||||
'fstype': partition.fstype,
|
||||
|
||||
@@ -10,7 +10,7 @@ from typing import Dict, Any, Optional
|
||||
try:
|
||||
import psutil
|
||||
except ImportError:
|
||||
psutil = None
|
||||
psutil = None # type: ignore[assignment]
|
||||
|
||||
from hbd.client.plugin import MonitorPlugin
|
||||
|
||||
@@ -32,6 +32,7 @@ def _zfs_arc_bytes() -> int:
|
||||
pass
|
||||
return 0
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@@ -115,7 +116,7 @@ class MemoryMonitorPlugin(MonitorPlugin):
|
||||
|
||||
async def _collect_metrics(self) -> Dict[str, Any]:
|
||||
"""Collect memory metrics from psutil."""
|
||||
metrics = {}
|
||||
metrics: Dict[str, Any] = {}
|
||||
|
||||
# Virtual (physical) memory statistics
|
||||
vmem = psutil.virtual_memory()
|
||||
|
||||
@@ -124,7 +124,7 @@ class NagiosRunnerPlugin(MonitorPlugin):
|
||||
Returns:
|
||||
Dictionary with results from all plugins
|
||||
"""
|
||||
results = {}
|
||||
results: Dict[str, Any] = {}
|
||||
|
||||
for cmd_config in self.commands:
|
||||
name = cmd_config.get("name")
|
||||
@@ -181,7 +181,8 @@ class NagiosRunnerPlugin(MonitorPlugin):
|
||||
self.logger.error(f"Command timed out: {command}")
|
||||
return NAGIOS_UNKNOWN, f"Command timed out after {self.timeout}s", {}
|
||||
|
||||
status_code = proc.returncode
|
||||
# communicate() has finished, so the process has exited and returncode is set
|
||||
status_code = proc.returncode if proc.returncode is not None else NAGIOS_UNKNOWN
|
||||
|
||||
if status_code < 0:
|
||||
return NAGIOS_UNKNOWN, f"Process killed by signal {-status_code}", {}
|
||||
@@ -225,7 +226,7 @@ class NagiosRunnerPlugin(MonitorPlugin):
|
||||
Returns:
|
||||
Dictionary of metric_name: value
|
||||
"""
|
||||
perfdata = {}
|
||||
perfdata: Dict[str, Any] = {}
|
||||
|
||||
# Performance data comes after the pipe character
|
||||
if '|' not in output:
|
||||
|
||||
@@ -5,12 +5,12 @@ Collects network interface statistics and connection information using psutil.
|
||||
"""
|
||||
|
||||
import logging
|
||||
from typing import Dict, Any, Optional, List
|
||||
from typing import Dict, Any, Optional
|
||||
|
||||
try:
|
||||
import psutil
|
||||
except ImportError:
|
||||
psutil = None
|
||||
psutil = None # type: ignore[assignment]
|
||||
|
||||
from hbd.client.plugin import MonitorPlugin
|
||||
from hbd.client.connectivity import get_shared
|
||||
@@ -63,7 +63,7 @@ class NetworkMonitorPlugin(MonitorPlugin):
|
||||
raise ImportError("psutil library is required for network_monitor plugin")
|
||||
|
||||
# Store previous I/O counters for delta calculation
|
||||
self._prev_io = {}
|
||||
self._prev_io: Dict[str, Any] = {}
|
||||
|
||||
async def initialize(self):
|
||||
"""Initialize the plugin (check psutil availability)."""
|
||||
@@ -124,7 +124,7 @@ class NetworkMonitorPlugin(MonitorPlugin):
|
||||
|
||||
async def _collect_metrics(self) -> Dict[str, Any]:
|
||||
"""Collect network metrics from psutil."""
|
||||
metrics = {}
|
||||
metrics: Dict[str, Any] = {}
|
||||
|
||||
# Collect per-interface I/O counters
|
||||
try:
|
||||
@@ -169,7 +169,7 @@ class NetworkMonitorPlugin(MonitorPlugin):
|
||||
if self.include_connections:
|
||||
try:
|
||||
connections = psutil.net_connections(kind='inet')
|
||||
conn_stats = {}
|
||||
conn_stats: Dict[str, int] = {}
|
||||
|
||||
# Count connections by state
|
||||
for conn in connections:
|
||||
|
||||
@@ -5,7 +5,6 @@ kernel, architecture, and distribution details.
|
||||
"""
|
||||
|
||||
import platform
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, Optional
|
||||
|
||||
|
||||
@@ -67,6 +67,7 @@ class PingMonitorPlugin(MonitorPlugin):
|
||||
self.timeout = int(cfg.get("timeout", 5))
|
||||
# hosts: dict of {hostname: {warning: x, critical: y}} or list of hostnames
|
||||
raw_hosts = cfg.get("hosts", {})
|
||||
self.hosts: Dict[str, Any]
|
||||
if isinstance(raw_hosts, list):
|
||||
self.hosts = {h: {} for h in raw_hosts}
|
||||
else:
|
||||
|
||||
@@ -1,3 +1,3 @@
|
||||
"""Common utilities shared between hbc and hbd."""
|
||||
|
||||
from hbd import __version__
|
||||
from hbd import __version__ # noqa: F401
|
||||
|
||||
+2
-3
@@ -8,7 +8,7 @@ Message Types:
|
||||
PLG: Plugin data message (client -> server)
|
||||
"""
|
||||
|
||||
from typing import Dict, Any, Union
|
||||
from typing import Dict, Any
|
||||
import json
|
||||
import zlib
|
||||
|
||||
@@ -92,7 +92,7 @@ def stodict(msg: bytes):
|
||||
'!' and decodes accordingly. Returns a dict with key 'ID' set to the
|
||||
message ID and the parsed key/value pairs.
|
||||
"""
|
||||
d = {}
|
||||
d: Dict[str, Any] = {}
|
||||
if len(msg) > 0 and chr(msg[0]) == "!":
|
||||
# message is: b'!ID:' + compressed_payload
|
||||
# original code used msg[1:4].decode() for ID (3 bytes including colon)
|
||||
@@ -159,4 +159,3 @@ def decode_plugin_data(msg: bytes) -> Dict[str, Any]:
|
||||
Dictionary with 'ID', 'plugin', and plugin data fields
|
||||
"""
|
||||
return stodict(msg)
|
||||
|
||||
|
||||
@@ -1,3 +1,3 @@
|
||||
"""HeartBeat Daemon (hbd) - Server/daemon component."""
|
||||
|
||||
from hbd import __version__
|
||||
from hbd import __version__ # noqa: F401
|
||||
|
||||
+10
-4
@@ -120,7 +120,7 @@ def cmd_passwd(args):
|
||||
def cmd_notify(args):
|
||||
"""Send a test message via a single notification channel."""
|
||||
from .config import load_config
|
||||
from .notify import Notification, _dispatch_to_channel, setup
|
||||
from .notify import Notification, setup
|
||||
|
||||
config = load_config(args.configfile)
|
||||
setup(config)
|
||||
@@ -193,7 +193,9 @@ def _read_pid(configfile) -> int | None:
|
||||
|
||||
|
||||
def cmd_stop(args):
|
||||
import os, signal as _signal, time
|
||||
import os
|
||||
import signal as _signal
|
||||
import time
|
||||
pid = _read_pid(args.configfile)
|
||||
if pid is None:
|
||||
sys.exit(1)
|
||||
@@ -212,7 +214,8 @@ def cmd_stop(args):
|
||||
|
||||
|
||||
def cmd_reload(args):
|
||||
import os, signal as _signal
|
||||
import os
|
||||
import signal as _signal
|
||||
pid = _read_pid(args.configfile)
|
||||
if pid is None:
|
||||
sys.exit(1)
|
||||
@@ -222,7 +225,10 @@ def cmd_reload(args):
|
||||
|
||||
|
||||
def cmd_restart(args):
|
||||
import os, signal as _signal, time, subprocess
|
||||
import os
|
||||
import signal as _signal
|
||||
import time
|
||||
import subprocess
|
||||
pid = _read_pid(args.configfile)
|
||||
if pid is not None:
|
||||
print(f"Stopping hbd (pid {pid})...")
|
||||
|
||||
@@ -7,7 +7,7 @@ import os
|
||||
try:
|
||||
import yaml
|
||||
except Exception:
|
||||
yaml = None
|
||||
yaml = None # type: ignore[assignment]
|
||||
|
||||
SERVER_DEFAULTS = {
|
||||
# Network settings
|
||||
@@ -205,9 +205,6 @@ class ReloadableConfig:
|
||||
# Load new config
|
||||
new_config = load_config(path)
|
||||
|
||||
# Store old config for rollback if needed
|
||||
old_config = self._config
|
||||
|
||||
# Update config
|
||||
self._config = new_config
|
||||
self._logger.info(f"Configuration reloaded from {path}")
|
||||
|
||||
@@ -15,6 +15,7 @@ def _make_yaml() -> YAML:
|
||||
y.preserve_quotes = True
|
||||
return y
|
||||
|
||||
|
||||
# Top-level keys managed by the 'server' logical section
|
||||
_SERVER_KEYS = [
|
||||
"hbd_port", "hbd_host", "ws_port", "wss_port", "hb_port",
|
||||
|
||||
+3
-1
@@ -1,4 +1,6 @@
|
||||
msgs = [] # in-memory list of recent messages for new websocket clients; also logged to file via notify.eventlog
|
||||
msgs: list = [] # in-memory list of recent messages for new websocket clients; also logged to file via notify.eventlog
|
||||
|
||||
|
||||
class Data:
|
||||
|
||||
def __init__(self, config):
|
||||
|
||||
+5
-5
@@ -2,7 +2,7 @@
|
||||
|
||||
from __future__ import annotations
|
||||
from subprocess import Popen, PIPE, STDOUT
|
||||
from typing import Optional
|
||||
from typing import Any, Callable, Optional
|
||||
import asyncio
|
||||
import logging
|
||||
|
||||
@@ -79,8 +79,8 @@ async def dns_update_worker(
|
||||
hbdclass,
|
||||
cfg: dict,
|
||||
async_queue=None,
|
||||
log: Optional[callable] = None,
|
||||
pushmsg: Optional[callable] = None,
|
||||
log: Optional[Callable[..., Any]] = None,
|
||||
pushmsg: Optional[Callable[..., Any]] = None,
|
||||
loop: Optional[asyncio.AbstractEventLoop] = None,
|
||||
):
|
||||
"""Pure async DNS worker that processes updates from asyncio.Queue.
|
||||
@@ -164,7 +164,7 @@ async def dns_update_worker(
|
||||
def start_dns_worker(
|
||||
hbdclass,
|
||||
cfg: dict,
|
||||
log: Optional[callable] = None,
|
||||
log: Optional[Callable[..., Any]] = None,
|
||||
loop: Optional[asyncio.AbstractEventLoop] = None,
|
||||
):
|
||||
"""Start the async DNS worker and return the Task.
|
||||
@@ -176,7 +176,7 @@ def start_dns_worker(
|
||||
loop = asyncio.get_event_loop()
|
||||
|
||||
# Create asyncio.Queue and wrap in a bridge for thread-safe puts
|
||||
async_q = asyncio.Queue()
|
||||
async_q: asyncio.Queue[Any] = asyncio.Queue()
|
||||
|
||||
class _QueueBridge:
|
||||
"""Thread-safe wrapper around asyncio.Queue for synchronous callers."""
|
||||
|
||||
@@ -8,6 +8,7 @@ import time
|
||||
import json
|
||||
import copy
|
||||
import queue
|
||||
from typing import Any, Dict
|
||||
|
||||
num = 0
|
||||
|
||||
@@ -24,7 +25,7 @@ def log(host, m):
|
||||
class Connection:
|
||||
# map of addrs to names
|
||||
|
||||
htab = {}
|
||||
htab: Dict[str, Any] = {}
|
||||
UNKNOWN = "unknown"
|
||||
UP = "up"
|
||||
DOWN = "down"
|
||||
@@ -278,8 +279,8 @@ class Connection:
|
||||
#
|
||||
class Host:
|
||||
# Table of Hosts
|
||||
hosts = {}
|
||||
dnsQ = queue.Queue()
|
||||
hosts: Dict[str, "Host"] = {}
|
||||
dnsQ: "queue.Queue[Any]" = queue.Queue()
|
||||
|
||||
def __init__(self, name):
|
||||
global num
|
||||
|
||||
+1
-1
@@ -109,6 +109,7 @@ def _insert_threshold_metric(thresholds: dict, metric_path: str, values: dict) -
|
||||
else:
|
||||
thresholds[plugin].setdefault(intermediate, {})[leaf] = cfg
|
||||
|
||||
|
||||
def _render_template(html_str: str, **context) -> str:
|
||||
tmpl = jinja2.Template(html_str)
|
||||
return tmpl.render(**context)
|
||||
@@ -1132,7 +1133,6 @@ async def start(
|
||||
elif is_mon:
|
||||
monitored.append(hostname)
|
||||
|
||||
|
||||
# Resolve notification channel configs for display
|
||||
notif_channels = []
|
||||
if current_user:
|
||||
|
||||
@@ -11,7 +11,7 @@ import os
|
||||
import asyncio
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Dict, Any, Optional, Callable, Iterable, List, Tuple, Union
|
||||
from typing import Dict, Any, Optional, Callable, Iterable, List, TextIO, Tuple, Union
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -52,7 +52,7 @@ class MessageJournal:
|
||||
self.enabled = self.config.get('journal_enabled', True)
|
||||
|
||||
# Runtime state
|
||||
self._file_handle = None
|
||||
self._file_handle: Optional[TextIO] = None
|
||||
self._current_size = 0
|
||||
self._lock = asyncio.Lock()
|
||||
self._initialized = False
|
||||
|
||||
+1
-2
@@ -5,8 +5,6 @@ import logging
|
||||
import socket
|
||||
import time
|
||||
import signal
|
||||
import sys
|
||||
import ssl
|
||||
from . import __version__
|
||||
|
||||
from . import udp
|
||||
@@ -23,6 +21,7 @@ eventlog = notify_mod.eventlog
|
||||
|
||||
# shared runtime collections and helpers
|
||||
|
||||
|
||||
def save_state(config, hbdclass):
|
||||
"""Save current state to pickle file. Safe to call at any time."""
|
||||
import pickle
|
||||
|
||||
@@ -8,8 +8,6 @@ This module can be extended for additional monitoring tasks.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
import asyncio
|
||||
import time
|
||||
from . import notify as notify_mod
|
||||
|
||||
DROPOVERDUE = 7 * 24 * 3600
|
||||
@@ -25,4 +23,3 @@ async def cleanup_connections(hbdclass):
|
||||
for conn_type, conn in host.connections.items():
|
||||
if hasattr(conn, 'cancel_overdue_timer'):
|
||||
conn.cancel_overdue_timer()
|
||||
|
||||
|
||||
@@ -21,7 +21,7 @@ import smtplib
|
||||
import subprocess
|
||||
import time
|
||||
import sys
|
||||
from dataclasses import dataclass, field
|
||||
from dataclasses import dataclass
|
||||
from typing import Optional
|
||||
|
||||
from . import data
|
||||
@@ -49,6 +49,7 @@ logf = None
|
||||
|
||||
_LEVEL_ORDER = {"RECOVER": 0, "INFO": 0, "WARNING": 1, "CRITICAL": 2}
|
||||
|
||||
|
||||
def _level_value(level: str) -> int:
|
||||
return _LEVEL_ORDER.get(level.upper(), 0)
|
||||
|
||||
@@ -103,7 +104,6 @@ def initlog(logfile):
|
||||
|
||||
|
||||
def closelog():
|
||||
global logf
|
||||
if logf and logf != sys.stderr:
|
||||
try:
|
||||
logf.close()
|
||||
@@ -331,8 +331,6 @@ async def _send_sms_voipms_async(channel_cfg: dict, notif: Notification) -> bool
|
||||
return False
|
||||
|
||||
|
||||
|
||||
|
||||
async def _send_matrix_async(channel_cfg: dict, notif: Notification) -> bool:
|
||||
"""Send a Matrix message using matrix-nio."""
|
||||
try:
|
||||
@@ -341,7 +339,6 @@ async def _send_matrix_async(channel_cfg: dict, notif: Notification) -> bool:
|
||||
logger.error("matrix-nio not installed; pip install matrix-nio")
|
||||
return False
|
||||
|
||||
from nio import AsyncClient
|
||||
homeserver = channel_cfg.get("homeserver", "")
|
||||
access_token = channel_cfg.get("access_token", "")
|
||||
room_id = channel_cfg.get("room_id", "")
|
||||
@@ -359,7 +356,6 @@ async def _send_matrix_async(channel_cfg: dict, notif: Notification) -> bool:
|
||||
client = AsyncClient(homeserver)
|
||||
client.access_token = access_token
|
||||
try:
|
||||
from nio import RoomSendResponse
|
||||
content = {
|
||||
"msgtype": "m.text",
|
||||
"body": text,
|
||||
|
||||
+1
-1
@@ -128,7 +128,7 @@ def get_providers(config: dict) -> list[ResolvedProvider]:
|
||||
Entries with missing required fields or unknown types are skipped with
|
||||
a warning log. Order follows config declaration order.
|
||||
"""
|
||||
result = []
|
||||
result: list = []
|
||||
oauth_cfg = config.get("oauth", {})
|
||||
if not isinstance(oauth_cfg, dict):
|
||||
return result
|
||||
|
||||
@@ -420,7 +420,7 @@ def get_settings_sections(config: dict, threshold_checker=None, user=None) -> li
|
||||
"fields": [
|
||||
field("journal_enabled", "Enabled", "boolean",
|
||||
"Turn journalling on or off.", editable=True),
|
||||
field("journal_dir", "Journal directory","path",
|
||||
field("journal_dir", "Journal directory", "path",
|
||||
"Directory where journal files are written.", editable=True),
|
||||
field("journal_file", "Journal filename", "text",
|
||||
"Base filename for the journal (rotated copies get a numeric suffix)."),
|
||||
|
||||
+14
-11
@@ -13,13 +13,14 @@ import asyncio
|
||||
import logging
|
||||
import time
|
||||
from enum import Enum
|
||||
from typing import Dict, List, Any, Optional, Tuple, Callable
|
||||
from typing import Dict, List, Any, Optional, Tuple
|
||||
from . import notify as notify_mod
|
||||
from .config import THRESHOLD_DEFAULTS
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
eventlog = notify_mod.eventlog
|
||||
|
||||
|
||||
class AlertLevel(Enum):
|
||||
"""Alert severity levels."""
|
||||
OK = 0
|
||||
@@ -52,16 +53,16 @@ class AlertState:
|
||||
self.metric_path = metric_path
|
||||
self.level = AlertLevel.OK
|
||||
self.since = time.time()
|
||||
self.last_value = None
|
||||
self.last_value: Any = None
|
||||
self.last_check = time.time()
|
||||
self.notification_count = 0
|
||||
self.last_notification = None
|
||||
self.threshold_value = None # The threshold value that triggered alert
|
||||
self.operator = None # The comparison operator (>, <, >=, etc.)
|
||||
self.last_notification: Optional[float] = None
|
||||
self.threshold_value: Optional[float] = None # The threshold value that triggered alert
|
||||
self.operator: Optional[str] = None # The comparison operator (>, <, >=, etc.)
|
||||
self.hysteresis: Optional[float] = None # Hysteresis fraction used for recovery
|
||||
self.formatted_message = None # Formatted display message for UI
|
||||
self.formatted_message: Optional[str] = None # Formatted display message for UI
|
||||
self.acknowledged = False # Whether alert has been acknowledged
|
||||
self.acknowledged_at = None # Timestamp when acknowledged
|
||||
self.acknowledged_at: Optional[float] = None # Timestamp when acknowledged
|
||||
self.consecutive_count = 0 # Consecutive exceedances while still OK (for count gating)
|
||||
self.pending_since: Optional[float] = None # non-None while waiting out grace period before notifying
|
||||
|
||||
@@ -182,6 +183,7 @@ class AlertState:
|
||||
def __str__(self):
|
||||
return self.to_dict().__str__()
|
||||
|
||||
|
||||
class ThresholdConfig:
|
||||
"""Configuration for a single threshold check."""
|
||||
|
||||
@@ -360,13 +362,13 @@ class ThresholdChecker:
|
||||
journal: Optional MessageJournal instance for logging threshold events
|
||||
"""
|
||||
# Named threshold configurations (pre-merged: defaults + overrides): {config_name: {metric_path: ThresholdConfig}}
|
||||
self.threshold_configs = {}
|
||||
self.threshold_configs: Dict[str, Dict[str, ThresholdConfig]] = {}
|
||||
|
||||
# Raw overrides only for each named config (no defaults baked in): {config_name: {metric_path: ThresholdConfig}}
|
||||
self.threshold_raw_configs: Dict[str, Dict[str, ThresholdConfig]] = {}
|
||||
|
||||
# Single threshold set for backward compatibility: {metric_path: ThresholdConfig}
|
||||
self.thresholds = {}
|
||||
self.thresholds: Dict[str, ThresholdConfig] = {}
|
||||
|
||||
# Host to ordered list of config names: {host_name: [config_name, ...]}
|
||||
self.host_config_mapping: Dict[str, List[str]] = {}
|
||||
@@ -827,7 +829,7 @@ class ThresholdChecker:
|
||||
Dictionary of thresholds for this host
|
||||
"""
|
||||
# Legacy mode: single threshold set for all hosts
|
||||
if self.thresholds and not self.threshold_configs:
|
||||
if self.thresholds:
|
||||
return self.thresholds
|
||||
|
||||
if not self.threshold_configs:
|
||||
@@ -946,6 +948,7 @@ class ThresholdChecker:
|
||||
self._check_pending_or_renotify(host_name, alert_state, metric_path, value, threshold, None)
|
||||
|
||||
return None
|
||||
|
||||
def _find_threshold(
|
||||
self, thresholds: Dict[str, "ThresholdConfig"], metric_path: str
|
||||
) -> Tuple[Optional["ThresholdConfig"], Optional[str]]:
|
||||
@@ -1273,7 +1276,7 @@ class ThresholdChecker:
|
||||
|
||||
def _format_display(
|
||||
self,
|
||||
display_format: str,
|
||||
display_format: Optional[str],
|
||||
value: Any,
|
||||
threshold_value: Optional[float],
|
||||
op_symbol: str,
|
||||
|
||||
+15
-7
@@ -5,6 +5,7 @@ import socket
|
||||
import struct
|
||||
import time
|
||||
import zlib
|
||||
from typing import Optional
|
||||
import logging
|
||||
|
||||
from platform import system as platform_system
|
||||
@@ -25,6 +26,7 @@ _STALE_INTERVAL_MULTIPLIER = 3
|
||||
# Supported on Linux, FreeBSD, and macOS. The constant is not exposed by
|
||||
# Python's socket module on all platforms
|
||||
platform = platform_system()
|
||||
_SO_TIMESTAMP: Optional[int]
|
||||
if platform == "Darwin":
|
||||
_SO_TIMESTAMP = 1024 # SO_TIMESTAMP on macOS (not in Python's socket module)
|
||||
elif platform == "Linux":
|
||||
@@ -45,6 +47,8 @@ def enable_kernel_timestamps(sock) -> bool:
|
||||
Returns True if the kernel will supply receive timestamps, False otherwise
|
||||
(unsupported platform, older kernel, or insufficient permissions).
|
||||
"""
|
||||
if _SO_TIMESTAMP is None:
|
||||
return False
|
||||
try:
|
||||
sock.setsockopt(socket.SOL_SOCKET, _SO_TIMESTAMP, 1)
|
||||
return True
|
||||
@@ -71,6 +75,7 @@ class RecvmsgTransport:
|
||||
Exposes the same sendto() / close() interface as asyncio's DatagramTransport
|
||||
so the rest of the code does not need to know which path is in use.
|
||||
"""
|
||||
|
||||
def __init__(self, loop, sock):
|
||||
self._loop = loop
|
||||
self._sock = sock
|
||||
@@ -422,15 +427,20 @@ def handle_datagram(msg: dict, addr, transport, ctx: dict):
|
||||
except Exception as e:
|
||||
logger.debug(f"Failed to log message to journal: {e}")
|
||||
|
||||
# Only named HTB/PLG datagrams come from clients; anything else (e.g. an internet scanner's
|
||||
# SIP probe) is journaled above but must not sign on a host.
|
||||
if msg.get("ID") not in ("HTB", "PLG") or not msg.get("name"):
|
||||
logger.debug("Ignoring non-client datagram from %s", addr)
|
||||
return
|
||||
|
||||
cfg = ctx.get("config", {})
|
||||
hbdcls = ctx.get("hbdclass")
|
||||
hbdcls = ctx["hbdclass"]
|
||||
msg_to_websockets = ctx.get("msg_to_websockets")
|
||||
DEBUG = ctx.get("DEBUG", 0)
|
||||
verbose = ctx.get("verbose", False)
|
||||
|
||||
# normalize addr (ip, port)
|
||||
ip = addr[0] if isinstance(addr, (list, tuple)) else addr
|
||||
name = msg.get("name", "unknown")
|
||||
name = msg["name"]
|
||||
from ..common.utils import shortname
|
||||
from . import config as config_mod
|
||||
|
||||
@@ -453,10 +463,8 @@ def handle_datagram(msg: dict, addr, transport, ctx: dict):
|
||||
newh = False
|
||||
|
||||
cid = msg.get("id", 0)
|
||||
try:
|
||||
rtt = float(msg.get("rtt"))
|
||||
except TypeError:
|
||||
rtt = None
|
||||
rtt_raw = msg.get("rtt")
|
||||
rtt = float(rtt_raw) if rtt_raw is not None else None
|
||||
|
||||
if msg.get("ID") == "HTB":
|
||||
host.doesack = msg.get("acks", -1)
|
||||
|
||||
+14
-3
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
|
||||
|
||||
[project]
|
||||
name = "hbd"
|
||||
version = "5.4.4"
|
||||
version = "5.4.7"
|
||||
description = "Heartbeat monitoring system — client (hbc) and server (hbd)"
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.11"
|
||||
@@ -58,9 +58,12 @@ all = [
|
||||
# Development dependencies
|
||||
dev = [
|
||||
"pytest>=7.0",
|
||||
"pytest-asyncio>=0.23",
|
||||
"pytest-cov>=4.0",
|
||||
"flake8>=5.0",
|
||||
"mypy>=1.10",
|
||||
"types-PyYAML",
|
||||
"types-psutil",
|
||||
"black>=23.0",
|
||||
"isort>=5.0",
|
||||
"tox>=4.0",
|
||||
@@ -83,10 +86,18 @@ include = ["hbd*"]
|
||||
|
||||
|
||||
[tool.black]
|
||||
line-length = 111
|
||||
line-length = 165
|
||||
|
||||
[tool.flake8]
|
||||
max-line-length = 111
|
||||
max-line-length = 165
|
||||
|
||||
[tool.pylint.format]
|
||||
max-line-length = 111
|
||||
|
||||
[[tool.mypy.overrides]]
|
||||
# optional notification libraries that ship neither stubs nor py.typed
|
||||
module = ["nio", "nio.*", "mattermostdriver", "mattermostdriver.*"]
|
||||
ignore_missing_imports = true
|
||||
|
||||
[tool.pytest.ini_options]
|
||||
testpaths = ["tests"]
|
||||
|
||||
@@ -9,8 +9,10 @@ sed -i".bak" "s/\*\*Package:\*\* \`hbd\` v[0-9.]*/\*\*Package:\*\* \`hbd\` v$VE
|
||||
|
||||
# Update CHANGELOG.md with commits since last tag
|
||||
LASTTAG=$(git describe --tags --abbrev=0 2>/dev/null || true)
|
||||
ADDED=$(git log "${LASTTAG:+$LASTTAG..}HEAD" --pretty="%s" | grep "^feat:" | sed 's/^feat: /- /')
|
||||
FIXED=$(git log "${LASTTAG:+$LASTTAG..}HEAD" --pretty="%s" | grep "^fix:" | sed 's/^fix: /- /')
|
||||
# Matches both unscoped ("feat: ...") and scoped ("feat(server): ...") conventional-commit
|
||||
# subjects; the scope is dropped from the changelog line.
|
||||
ADDED=$(git log "${LASTTAG:+$LASTTAG..}HEAD" --pretty="%s" | grep -E "^feat(\([^)]*\))?:" | sed -E 's/^feat(\([^)]*\))?: /- /')
|
||||
FIXED=$(git log "${LASTTAG:+$LASTTAG..}HEAD" --pretty="%s" | grep -E "^fix(\([^)]*\))?:" | sed -E 's/^fix(\([^)]*\))?: /- /')
|
||||
{
|
||||
printf "## [%s]\n" "$VER"
|
||||
[ -n "$ADDED" ] && printf "\n### Added\n%s\n" "$ADDED"
|
||||
|
||||
@@ -1,2 +1,3 @@
|
||||
hbc_mini
|
||||
hbc_mini_dbg
|
||||
test_probe
|
||||
|
||||
+10
-2
@@ -1,13 +1,15 @@
|
||||
CC ?= cc
|
||||
CFLAGS = -O2 -Wall -Wextra -std=c11
|
||||
LDFLAGS = -lz -lpthread -lm
|
||||
# res_search() lives in libresolv on older glibc; in libc on newer glibc, musl and the BSDs
|
||||
LDFLAGS += $(shell echo 'int main(void){return 0;}' | $(CC) -x c - -lresolv -o /dev/null 2>/dev/null && echo -lresolv)
|
||||
TARGET = hbc_mini
|
||||
SRC = hbc_mini.c
|
||||
|
||||
# FreeBSD/NetBSD keep zlib in base; no extra flags needed.
|
||||
# On some NetBSD installs pthreads may need -lpthread from pkgsrc.
|
||||
|
||||
.PHONY: all clean debug
|
||||
.PHONY: all clean debug test
|
||||
|
||||
all: $(TARGET)
|
||||
|
||||
@@ -17,5 +19,11 @@ $(TARGET): $(SRC)
|
||||
debug: $(SRC)
|
||||
$(CC) -g -fsanitize=address,undefined -o $(TARGET)_dbg $< $(LDFLAGS)
|
||||
|
||||
test: test_probe
|
||||
./test_probe
|
||||
|
||||
test_probe: test_probe.c $(SRC)
|
||||
$(CC) $(CFLAGS) -o $@ test_probe.c $(LDFLAGS)
|
||||
|
||||
clean:
|
||||
rm -f $(TARGET) $(TARGET)_dbg
|
||||
rm -f $(TARGET) $(TARGET)_dbg test_probe
|
||||
|
||||
+583
-4
@@ -6,6 +6,12 @@
|
||||
*
|
||||
* Drop on any host and run: ./hbc_mini <hbd-host>
|
||||
* Config: ~/.hbc.json (same keys as Python version)
|
||||
*
|
||||
* Presence probe (built in): reports per protocol (IPv4/IPv6) whether the host has internet
|
||||
* connectivity, as p4/p6 flags on every heartbeat, so hbd stops treating a missing IPv6 (or
|
||||
* IPv4) path as an outage. Config, in the network_monitor section of ~/.hbc.json:
|
||||
* probe_enabled (default true), probe_v4 (default 1.1.1.1), probe_v6 (default
|
||||
* 2606:4700:4700::1111; IP literals only), probe_interval (default 60 s)
|
||||
*/
|
||||
|
||||
#ifdef __linux__
|
||||
@@ -14,6 +20,7 @@
|
||||
#endif
|
||||
|
||||
#include <arpa/inet.h>
|
||||
#include <arpa/nameser.h>
|
||||
#include <netinet/in.h>
|
||||
#include <ctype.h>
|
||||
#include <errno.h>
|
||||
@@ -21,6 +28,7 @@
|
||||
#include <math.h>
|
||||
#include <netdb.h>
|
||||
#include <pthread.h>
|
||||
#include <resolv.h>
|
||||
#include <signal.h>
|
||||
#include <stdarg.h>
|
||||
#include <stdbool.h>
|
||||
@@ -396,6 +404,10 @@ typedef struct {
|
||||
|
||||
char net_skip[MAX_NET_SKIP][64];
|
||||
int net_skip_count;
|
||||
|
||||
bool probe_enabled;
|
||||
int probe_interval;
|
||||
char probe_v4[64], probe_v6[64];
|
||||
} config_t;
|
||||
|
||||
static void config_defaults(config_t *c) {
|
||||
@@ -410,6 +422,10 @@ static void config_defaults(config_t *c) {
|
||||
c->net_interval = 300;
|
||||
snprintf(c->net_skip[0], 64, "lo");
|
||||
c->net_skip_count = 1;
|
||||
c->probe_enabled = true;
|
||||
c->probe_interval = 60;
|
||||
snprintf(c->probe_v4, sizeof(c->probe_v4), "1.1.1.1");
|
||||
snprintf(c->probe_v6, sizeof(c->probe_v6), "2606:4700:4700::1111");
|
||||
}
|
||||
|
||||
static void config_load(config_t *cfg, const char *path) {
|
||||
@@ -498,6 +514,10 @@ static void config_load(config_t *cfg, const char *path) {
|
||||
jval_t *net = plugins ? jget(plugins,"network_monitor") : jget(root,"network_monitor");
|
||||
if (net && net->type == JT_OBJECT) {
|
||||
if ((v=jget(net,"interval"))) cfg->net_interval = jint(v, cfg->net_interval);
|
||||
if ((v=jget(net,"probe_enabled"))) cfg->probe_enabled = jint(v, 1) != 0;
|
||||
if ((v=jget(net,"probe_interval"))) cfg->probe_interval = jint(v, cfg->probe_interval);
|
||||
if ((v=jget(net,"probe_v4"))) snprintf(cfg->probe_v4, sizeof(cfg->probe_v4), "%s", jstr(v, cfg->probe_v4));
|
||||
if ((v=jget(net,"probe_v6"))) snprintf(cfg->probe_v6, sizeof(cfg->probe_v6), "%s", jstr(v, cfg->probe_v6));
|
||||
jval_t *sk = jget(net,"skip_interfaces");
|
||||
if (sk && sk->type == JT_ARRAY) {
|
||||
cfg->net_skip_count = 0;
|
||||
@@ -558,6 +578,7 @@ static int conn_send(conn_t *c, const char *msg_id, kvdict_t *d) {
|
||||
|
||||
struct sockaddr_storage sa; socklen_t salen = 0;
|
||||
memset(&sa, 0, sizeof(sa));
|
||||
pthread_mutex_lock(&c->mu); /* conn_retarget() may change addr/af/sockfd */
|
||||
if (c->af == AF_INET) {
|
||||
struct sockaddr_in *s4 = (struct sockaddr_in *)&sa;
|
||||
s4->sin_family = AF_INET;
|
||||
@@ -572,7 +593,6 @@ static int conn_send(conn_t *c, const char *msg_id, kvdict_t *d) {
|
||||
salen = sizeof(*s6);
|
||||
}
|
||||
|
||||
pthread_mutex_lock(&c->mu);
|
||||
c->last_send = now_ts();
|
||||
ssize_t sent = sendto(c->sockfd, wire, wlen, 0, (struct sockaddr *)&sa, salen);
|
||||
pthread_mutex_unlock(&c->mu);
|
||||
@@ -984,6 +1004,341 @@ static void plugin_disk_monitor(conn_t *c, const config_t *cfg) {
|
||||
LOGD("sent disk_monitor");
|
||||
}
|
||||
|
||||
/* ============================================================
|
||||
* Connectivity probe: per-protocol internet presence (IPv4 / IPv6)
|
||||
*
|
||||
* Tier 1 (every PROBE_TIER1 s, sends no packets): a UDP connect() to the probe address shows
|
||||
* whether a route exists and which source address would be used. Tier 2 (every probe_interval
|
||||
* s, sooner after a failure or a change of address or gateway): one ICMP echo, only when tier 1
|
||||
* found a route. Absent = no route, or PROBE_FAIL_LIMIT consecutive failed echoes; present =
|
||||
* one success; unknown (flag omitted) when the family cannot be probed. The result rides on
|
||||
* every HTB as p4/p6 and inside the network_monitor metric. Mirrors hbd/client/connectivity.py.
|
||||
* ============================================================ */
|
||||
|
||||
#define PROBE_UNKNOWN (-1)
|
||||
#define PROBE_FAIL_LIMIT 2
|
||||
#define PROBE_TIER1 5 /* seconds between route checks */
|
||||
|
||||
typedef struct {
|
||||
int present; /* PROBE_UNKNOWN, 0 = absent, 1 = present */
|
||||
char addr[INET6_ADDRSTRLEN];
|
||||
char gateway[INET6_ADDRSTRLEN];
|
||||
double rtt; /* ms, < 0 when unknown */
|
||||
int fails; /* consecutive failed echoes */
|
||||
double next_ping; /* monotonic seconds */
|
||||
char last_key[2 * INET6_ADDRSTRLEN + 2];
|
||||
bool ping_disabled; /* no usable ping binary */
|
||||
} probe_fam_t;
|
||||
|
||||
typedef struct {
|
||||
/* 1 = route exists (source address in out), 0 = no route, -1 = family unsupported/unusable */
|
||||
int (*local_address)(int af, const char *target, char *out, size_t n);
|
||||
/* RTT in ms; -1 = no reply; -2 = ping unavailable */
|
||||
double (*ping)(int af, const char *target);
|
||||
/* fills out with the default gateway, "" when unknown */
|
||||
void (*gateway)(int af, char *out, size_t n);
|
||||
} probe_ops_t;
|
||||
|
||||
typedef struct {
|
||||
probe_fam_t fam[2]; /* [0] = IPv4, [1] = IPv6 */
|
||||
char target[2][64];
|
||||
double probe_interval, retry_interval;
|
||||
bool cheap_gateway; /* gateway read is cheap (/proc), so read every tier 1 */
|
||||
pthread_mutex_t mu; /* guards fam[] between the probe thread and readers */
|
||||
} probe_t;
|
||||
|
||||
/* Lowest-metric default gateway from /proc/net/route. The kernel prints the raw network-order
|
||||
* u32 in host order, so it can be stored straight into s_addr on any endianness. */
|
||||
static void gw_parse_proc_v4(const char *text, char *out, size_t n) {
|
||||
long best = -1; out[0] = '\0';
|
||||
const char *p = strchr(text, '\n'); /* skip the header line */
|
||||
while (p && *++p) {
|
||||
char iface[64], dest[16], gw[16]; unsigned flags; int refcnt, use, metric;
|
||||
if (sscanf(p, "%63s %15s %15s %x %d %d %d", iface, dest, gw, &flags, &refcnt, &use, &metric) == 7
|
||||
&& !strcmp(dest, "00000000") && (flags & 0x2)) {
|
||||
if (best < 0 || metric < best) {
|
||||
struct in_addr a; a.s_addr = (uint32_t)strtoul(gw, NULL, 16);
|
||||
if (inet_ntop(AF_INET, &a, out, n)) best = metric;
|
||||
}
|
||||
}
|
||||
p = strchr(p, '\n');
|
||||
}
|
||||
}
|
||||
|
||||
/* Lowest-metric default gateway from /proc/net/ipv6_route (skips RTF_REJECT and "lo"). */
|
||||
static void gw_parse_proc_v6(const char *text, char *out, size_t n) {
|
||||
long best = -1; out[0] = '\0';
|
||||
const char *zero = "00000000000000000000000000000000";
|
||||
for (const char *p = text; p && *p; ) {
|
||||
char dest[40], src[40], nh[40], iface[64];
|
||||
unsigned plen, splen, metric, refcnt, use, flags;
|
||||
if (sscanf(p, "%32s %x %32s %x %32s %x %x %x %x %63s",
|
||||
dest, &plen, src, &splen, nh, &metric, &refcnt, &use, &flags, iface) == 10
|
||||
&& !strcmp(dest, zero) && plen == 0 && strcmp(nh, zero) && strcmp(iface, "lo")
|
||||
&& (flags & 0x2) && !(flags & 0x200)) {
|
||||
if (best < 0 || (long)metric < best) {
|
||||
struct in6_addr a; memset(&a, 0, sizeof(a));
|
||||
bool ok = strlen(nh) == 32;
|
||||
for (int i = 0; ok && i < 16; i++) {
|
||||
unsigned b;
|
||||
if (sscanf(nh + 2 * i, "%2x", &b) != 1) ok = false; else a.s6_addr[i] = (uint8_t)b;
|
||||
}
|
||||
if (ok && inet_ntop(AF_INET6, &a, out, n)) best = (long)metric;
|
||||
}
|
||||
}
|
||||
p = strchr(p, '\n'); if (p) p++;
|
||||
}
|
||||
}
|
||||
|
||||
/* Gateway from `route -n get [-inet6] default` (the BSDs). */
|
||||
static void gw_parse_route_get(const char *text, char *out, size_t n) {
|
||||
out[0] = '\0';
|
||||
const char *p = strstr(text, "gateway:");
|
||||
if (!p) return;
|
||||
p += 8;
|
||||
while (*p == ' ' || *p == '\t') p++;
|
||||
size_t i = 0;
|
||||
while (*p && !isspace((unsigned char)*p) && i + 1 < n) out[i++] = *p++;
|
||||
out[i] = '\0';
|
||||
}
|
||||
|
||||
/* Average RTT from ping output: -1.0 no reply, -2.0 ping could not be run. Works for Linux
|
||||
* ("rtt min/avg/max/mdev") and BSD ("round-trip min/avg/max/stddev"). */
|
||||
static double ping_parse_rtt(const char *out) {
|
||||
if (strstr(out, "not found") || strstr(out, "No such file")) return -2.0;
|
||||
const char *p = strstr(out, "min/avg/max");
|
||||
if (!p) return -1.0;
|
||||
p = strchr(p, '=');
|
||||
double mn, avg, mx;
|
||||
if (p && sscanf(p + 1, " %lf/%lf/%lf", &mn, &avg, &mx) == 3) return avg;
|
||||
return -1.0;
|
||||
}
|
||||
|
||||
/* One probe step for one family (f: 0 = IPv4, 1 = IPv6). Works on *s only, so the probe thread
|
||||
* can run it on a private copy without holding the mutex during the (blocking) echo. */
|
||||
static void probe_step(const probe_t *p, probe_fam_t *s, int f, const probe_ops_t *ops, double now) {
|
||||
int af = f ? AF_INET6 : AF_INET;
|
||||
char addr[INET6_ADDRSTRLEN] = "", gw[INET6_ADDRSTRLEN] = "";
|
||||
|
||||
int r = ops->local_address(af, p->target[f], addr, sizeof(addr));
|
||||
if (r < 0) { s->present = PROBE_UNKNOWN; return; }
|
||||
if (r == 0) { /* tier 1: no route => absent immediately */
|
||||
s->fails = 0; s->next_ping = 0; s->last_key[0] = '\0';
|
||||
s->present = 0; s->addr[0] = '\0'; s->gateway[0] = '\0'; s->rtt = -1;
|
||||
return;
|
||||
}
|
||||
|
||||
if (p->cheap_gateway) ops->gateway(af, gw, sizeof(gw));
|
||||
else snprintf(gw, sizeof(gw), "%s", s->gateway);
|
||||
char key[sizeof(s->last_key)];
|
||||
snprintf(key, sizeof(key), "%s|%s", addr, gw);
|
||||
bool due = strcmp(key, s->last_key) != 0 || now >= s->next_ping;
|
||||
if (due && !p->cheap_gateway) {
|
||||
ops->gateway(af, gw, sizeof(gw));
|
||||
snprintf(key, sizeof(key), "%s|%s", addr, gw);
|
||||
}
|
||||
snprintf(s->addr, sizeof(s->addr), "%s", addr);
|
||||
snprintf(s->gateway, sizeof(s->gateway), "%s", gw);
|
||||
snprintf(s->last_key, sizeof(s->last_key), "%s", key);
|
||||
|
||||
if (s->ping_disabled) { s->present = PROBE_UNKNOWN; return; } /* cannot probe: unknown, never absent, while a route exists */
|
||||
if (!due) return;
|
||||
double rtt = ops->ping(af, p->target[f]);
|
||||
if (rtt < -1.5) { s->ping_disabled = true; s->present = PROBE_UNKNOWN; return; }
|
||||
if (rtt < 0) {
|
||||
s->fails++;
|
||||
s->next_ping = now + p->retry_interval;
|
||||
if (s->fails >= PROBE_FAIL_LIMIT) { s->present = 0; s->rtt = -1; }
|
||||
} else {
|
||||
s->fails = 0;
|
||||
s->next_ping = now + p->probe_interval;
|
||||
s->present = 1; s->rtt = rtt;
|
||||
}
|
||||
}
|
||||
|
||||
/* HTB flags for the families whose presence is known. */
|
||||
static void probe_flags(probe_t *p, kvdict_t *d) {
|
||||
pthread_mutex_lock(&p->mu);
|
||||
if (p->fam[0].present >= 0) kv_set_int(d, "p4", p->fam[0].present);
|
||||
if (p->fam[1].present >= 0) kv_set_int(d, "p6", p->fam[1].present);
|
||||
pthread_mutex_unlock(&p->mu);
|
||||
}
|
||||
|
||||
/* The `connectivity` metric as a JSON object; returns its length or -1 on overflow. */
|
||||
static int probe_metric_json(probe_t *p, char *out, size_t n) {
|
||||
static const char *names[2] = { "ipv4", "ipv6" };
|
||||
size_t len = 0;
|
||||
len += (size_t)snprintf(out + len, n - len, "{");
|
||||
pthread_mutex_lock(&p->mu);
|
||||
for (int f = 0; f < 2 && len < n; f++) {
|
||||
const probe_fam_t *s = &p->fam[f];
|
||||
char pres[8], addr[64], gw[64], rtt[32];
|
||||
snprintf(pres, sizeof(pres), "%s", s->present < 0 ? "null" : s->present ? "true" : "false");
|
||||
if (s->addr[0]) snprintf(addr, sizeof(addr), "\"%s\"", s->addr); else snprintf(addr, sizeof(addr), "null");
|
||||
if (s->gateway[0]) snprintf(gw, sizeof(gw), "\"%s\"", s->gateway); else snprintf(gw, sizeof(gw), "null");
|
||||
if (s->rtt >= 0) snprintf(rtt, sizeof(rtt), "%.3f", s->rtt); else snprintf(rtt, sizeof(rtt), "null");
|
||||
len += (size_t)snprintf(out + len, n - len,
|
||||
"%s\"%s\":{\"present\":%s,\"addr\":%s,\"gateway\":%s,\"rtt\":%s}",
|
||||
f ? "," : "", names[f], pres, addr, gw, rtt);
|
||||
}
|
||||
pthread_mutex_unlock(&p->mu);
|
||||
if (len + 2 > n) { out[0] = '\0'; return -1; }
|
||||
out[len++] = '}'; out[len] = '\0';
|
||||
return (int)len;
|
||||
}
|
||||
|
||||
static double mono_ts(void) {
|
||||
struct timespec ts;
|
||||
clock_gettime(CLOCK_MONOTONIC, &ts);
|
||||
return ts.tv_sec + ts.tv_nsec * 1e-9;
|
||||
}
|
||||
|
||||
/* Source address the kernel would use to reach target (an IP literal). A UDP connect() sends
|
||||
* nothing. Returns 1 = route (address in out), 0 = no route, -1 = family unsupported or the
|
||||
* target is not an IP literal (unknown, never absent). */
|
||||
static int sys_local_address(int af, const char *target, char *out, size_t n) {
|
||||
int fd = socket(af, SOCK_DGRAM, 0);
|
||||
if (fd < 0) return -1;
|
||||
struct sockaddr_storage sa; socklen_t sl;
|
||||
memset(&sa, 0, sizeof(sa));
|
||||
if (af == AF_INET) {
|
||||
struct sockaddr_in *s4 = (struct sockaddr_in *)&sa;
|
||||
s4->sin_family = AF_INET; s4->sin_port = htons(9);
|
||||
if (inet_pton(AF_INET, target, &s4->sin_addr) != 1) { close(fd); return -1; }
|
||||
sl = sizeof(*s4);
|
||||
} else {
|
||||
struct sockaddr_in6 *s6 = (struct sockaddr_in6 *)&sa;
|
||||
s6->sin6_family = AF_INET6; s6->sin6_port = htons(9);
|
||||
if (inet_pton(AF_INET6, target, &s6->sin6_addr) != 1) { close(fd); return -1; }
|
||||
sl = sizeof(*s6);
|
||||
}
|
||||
int rc = 0;
|
||||
if (connect(fd, (struct sockaddr *)&sa, sl) == 0) {
|
||||
struct sockaddr_storage me; socklen_t ml = sizeof(me);
|
||||
if (getsockname(fd, (struct sockaddr *)&me, &ml) == 0) {
|
||||
const void *src = (af == AF_INET)
|
||||
? (const void *)&((struct sockaddr_in *)&me)->sin_addr
|
||||
: (const void *)&((struct sockaddr_in6 *)&me)->sin6_addr;
|
||||
if (inet_ntop(af, src, out, (socklen_t)n)) rc = 1;
|
||||
}
|
||||
}
|
||||
close(fd);
|
||||
return rc;
|
||||
}
|
||||
|
||||
static char *slurp(FILE *f) {
|
||||
size_t cap = 4096, len = 0;
|
||||
char *buf = malloc(cap);
|
||||
if (!buf) return NULL;
|
||||
size_t r;
|
||||
while ((r = fread(buf + len, 1, cap - len - 1, f)) > 0) {
|
||||
len += r;
|
||||
if (len + 1 >= cap) {
|
||||
if (cap >= 65536) break;
|
||||
cap *= 2;
|
||||
char *nb = realloc(buf, cap);
|
||||
if (!nb) { free(buf); return NULL; }
|
||||
buf = nb;
|
||||
}
|
||||
}
|
||||
buf[len] = '\0';
|
||||
return buf;
|
||||
}
|
||||
|
||||
static bool g_have_timeout = false; /* timeout(1) exists: bounds ping6 on the BSDs */
|
||||
|
||||
/* One echo request: RTT in ms, -1.0 no reply, -2.0 ping could not be run. The target is an IP
|
||||
* literal from the config; anything else is refused before it can reach the shell. */
|
||||
static double sys_ping(int af, const char *target) {
|
||||
for (const char *c = target; *c; c++)
|
||||
if (!(isxdigit((unsigned char)*c) || *c == '.' || *c == ':')) return -2.0;
|
||||
char cmd[256];
|
||||
#ifdef __linux__
|
||||
snprintf(cmd, sizeof(cmd), "ping -c 1 -W 2 -w 3 %s %s 2>&1", af == AF_INET ? "-4" : "-6", target);
|
||||
#else
|
||||
/* BSD ping: -W is in milliseconds, and IPv6 is reached through ping6 */
|
||||
if (af == AF_INET)
|
||||
snprintf(cmd, sizeof(cmd), "%sping -c 1 -W 2000 %s 2>&1", g_have_timeout ? "timeout 4 " : "", target);
|
||||
else
|
||||
snprintf(cmd, sizeof(cmd), "%sping6 -c 1 %s 2>&1", g_have_timeout ? "timeout 4 " : "", target);
|
||||
#endif
|
||||
FILE *f = popen(cmd, "r");
|
||||
if (!f) return -2.0;
|
||||
char *out = slurp(f);
|
||||
pclose(f);
|
||||
if (!out) return -1.0;
|
||||
double rtt = ping_parse_rtt(out);
|
||||
free(out);
|
||||
return rtt;
|
||||
}
|
||||
|
||||
/* Default gateway for the family, "" when unknown. Best effort; never affects presence. */
|
||||
static void sys_gateway(int af, char *out, size_t n) {
|
||||
out[0] = '\0';
|
||||
#ifdef __linux__
|
||||
FILE *f = fopen(af == AF_INET ? "/proc/net/route" : "/proc/net/ipv6_route", "r");
|
||||
if (!f) return;
|
||||
char *text = slurp(f);
|
||||
fclose(f);
|
||||
if (!text) return;
|
||||
if (af == AF_INET) gw_parse_proc_v4(text, out, n); else gw_parse_proc_v6(text, out, n);
|
||||
free(text);
|
||||
#else
|
||||
FILE *f = popen(af == AF_INET ? "route -n get default 2>/dev/null"
|
||||
: "route -n get -inet6 default 2>/dev/null", "r");
|
||||
if (!f) return;
|
||||
char *text = slurp(f);
|
||||
pclose(f);
|
||||
if (!text) return;
|
||||
gw_parse_route_get(text, out, n);
|
||||
free(text);
|
||||
#endif
|
||||
}
|
||||
|
||||
static probe_t g_probe;
|
||||
static bool g_probe_enabled = false;
|
||||
|
||||
static void probe_init(probe_t *p, const config_t *cfg) {
|
||||
memset(p, 0, sizeof(*p));
|
||||
snprintf(p->target[0], sizeof(p->target[0]), "%s", cfg->probe_v4);
|
||||
snprintf(p->target[1], sizeof(p->target[1]), "%s", cfg->probe_v6);
|
||||
p->probe_interval = cfg->probe_interval > 0 ? cfg->probe_interval : 60;
|
||||
p->retry_interval = 10;
|
||||
#ifdef __linux__
|
||||
p->cheap_gateway = true; /* /proc read; elsewhere a subprocess only when due */
|
||||
#endif
|
||||
p->fam[0].present = p->fam[1].present = PROBE_UNKNOWN;
|
||||
p->fam[0].rtt = p->fam[1].rtt = -1;
|
||||
pthread_mutex_init(&p->mu, NULL);
|
||||
g_have_timeout = system("command -v timeout >/dev/null 2>&1") == 0;
|
||||
g_probe_enabled = cfg->probe_enabled;
|
||||
|
||||
/* Diagnostic only: sys_local_address already turns a non-literal target into PROBE_UNKNOWN
|
||||
* (via inet_pton failing there), so this does not change behavior — it just stops that from
|
||||
* being silent. */
|
||||
struct in_addr a4;
|
||||
struct in6_addr a6;
|
||||
if (inet_pton(AF_INET, p->target[0], &a4) != 1)
|
||||
LOGW("probe_v4 is not an IPv4 literal: %s", p->target[0]);
|
||||
if (inet_pton(AF_INET6, p->target[1], &a6) != 1)
|
||||
LOGW("probe_v6 is not an IPv6 literal: %s", p->target[1]);
|
||||
}
|
||||
|
||||
static void *probe_thread(void *arg) {
|
||||
probe_t *p = arg;
|
||||
static const probe_ops_t ops = { sys_local_address, sys_ping, sys_gateway };
|
||||
while (g_running) {
|
||||
double now = mono_ts();
|
||||
for (int f = 0; f < 2 && g_running; f++) {
|
||||
pthread_mutex_lock(&p->mu); probe_fam_t tmp = p->fam[f]; pthread_mutex_unlock(&p->mu);
|
||||
probe_step(p, &tmp, f, &ops, now); /* may block in ping: no lock held */
|
||||
pthread_mutex_lock(&p->mu); p->fam[f] = tmp; pthread_mutex_unlock(&p->mu);
|
||||
}
|
||||
for (int i = 0; i < PROBE_TIER1 && g_running; i++) sleep(1);
|
||||
}
|
||||
return NULL;
|
||||
}
|
||||
|
||||
/* ============================================================
|
||||
* Plugin: network_monitor
|
||||
* Linux: /proc/net/dev
|
||||
@@ -1077,16 +1432,26 @@ static void plugin_network_monitor(conn_t *c, const config_t *cfg) {
|
||||
json[jlen++] = '}'; json[jlen] = '\0';
|
||||
memcpy(g_net_prev, curr, cn * sizeof(net_stat_t));
|
||||
g_net_prev_n = cn; g_net_prev_ts = now;
|
||||
if (first) { free(json); return; }
|
||||
|
||||
kvdict_t d; kv_clear(&d);
|
||||
kv_set(&d, "plugin", "network_monitor");
|
||||
if (!first) {
|
||||
char *jval = malloc(MAX_VAL + 1);
|
||||
if (jval) { snprintf(jval, MAX_VAL, "@%s", json); kv_set(&d, "interfaces", jval); free(jval); }
|
||||
}
|
||||
bool have_conn = false;
|
||||
if (g_probe_enabled) {
|
||||
char cj[512];
|
||||
cj[0] = '@';
|
||||
if (probe_metric_json(&g_probe, cj + 1, sizeof(cj) - 1) > 0) {
|
||||
kv_set(&d, "connectivity", cj);
|
||||
have_conn = true;
|
||||
}
|
||||
}
|
||||
free(json);
|
||||
if (first && !have_conn) return;
|
||||
kv_set_dbl(&d, "_timestamp", now_ts());
|
||||
kv_set_int(&d, "_interval", cfg->net_interval);
|
||||
conn_send(c, "PLG", &d);
|
||||
free(json);
|
||||
LOGD("sent network_monitor");
|
||||
}
|
||||
|
||||
@@ -1292,6 +1657,199 @@ static void usage(const char *prog) {
|
||||
" -h Help\n", prog);
|
||||
}
|
||||
|
||||
/* ============================================================
|
||||
* Re-resolution of the server hosts at the TTL of their A/AAAA records
|
||||
* getaddrinfo() hides the TTL, so a res_search() query is made only to learn it; the
|
||||
* addresses still come from getaddrinfo(). TTL unknown (IP literal, /etc/hosts name, no
|
||||
* resolv.conf, timeout) -> DNS_DEFAULT_TTL. Runs in a detached thread so a slow resolver
|
||||
* cannot delay heartbeats; the main loop applies the result.
|
||||
* ============================================================ */
|
||||
|
||||
#define DNS_DEFAULT_TTL 300 /* used when the real TTL is unknown */
|
||||
#define DNS_MIN_TTL 30 /* a TTL of 0 must not turn into a resolution loop */
|
||||
#define DNS_FAILED_RETRY 60
|
||||
|
||||
typedef struct { int af; char addr[INET6_ADDRSTRLEN]; } dns_addr_t;
|
||||
|
||||
typedef struct {
|
||||
const char **hosts;
|
||||
int nhost, port, af_filter;
|
||||
} dns_ctx_t;
|
||||
|
||||
static struct {
|
||||
pthread_mutex_t mu;
|
||||
bool ready; /* a result is waiting for the main loop */
|
||||
int n;
|
||||
dns_addr_t addrs[MAX_HOSTS];
|
||||
} g_dns = { PTHREAD_MUTEX_INITIALIZER, false, 0, {{0, ""}} };
|
||||
|
||||
/* Smallest A/AAAA TTL (CNAMEs included) of host, or -1 if it cannot be determined. */
|
||||
static long dns_host_ttl(const char *host, int af_filter) {
|
||||
unsigned char probe[sizeof(struct in6_addr)];
|
||||
if (inet_pton(AF_INET, host, probe) == 1 || inet_pton(AF_INET6, host, probe) == 1) return -1;
|
||||
|
||||
long best = -1;
|
||||
int types[2] = { ns_t_a, ns_t_aaaa };
|
||||
for (int t = 0; t < 2; t++) {
|
||||
if (af_filter == AF_INET && t == 1) continue;
|
||||
if (af_filter == AF_INET6 && t == 0) continue;
|
||||
unsigned char ans[NS_PACKETSZ * 2];
|
||||
int len = res_search(host, ns_c_in, types[t], ans, sizeof(ans));
|
||||
ns_msg msg;
|
||||
if (len < 0 || ns_initparse(ans, len, &msg) < 0) continue;
|
||||
long min = -1; bool found = false;
|
||||
for (int i = 0; i < ns_msg_count(msg, ns_s_an); i++) {
|
||||
ns_rr rr;
|
||||
if (ns_parserr(&msg, ns_s_an, i, &rr) < 0) continue;
|
||||
int ty = ns_rr_type(rr);
|
||||
if (ty != types[t] && ty != ns_t_cname) continue;
|
||||
if (ty == types[t]) found = true;
|
||||
if (min < 0 || (long)ns_rr_ttl(rr) < min) min = (long)ns_rr_ttl(rr);
|
||||
}
|
||||
if (found && (best < 0 || min < best)) best = min;
|
||||
}
|
||||
return best;
|
||||
}
|
||||
|
||||
/* Seconds until the next resolution: the smallest known TTL, floored at DNS_MIN_TTL. */
|
||||
static int dns_refresh_delay(const dns_ctx_t *x) {
|
||||
long best = -1;
|
||||
for (int i = 0; i < x->nhost; i++) {
|
||||
long t = dns_host_ttl(x->hosts[i], x->af_filter);
|
||||
if (t >= 0 && (best < 0 || t < best)) best = t;
|
||||
}
|
||||
if (best < 0) return DNS_DEFAULT_TTL;
|
||||
return best < DNS_MIN_TTL ? DNS_MIN_TTL : (best > 86400 ? 86400 : (int)best);
|
||||
}
|
||||
|
||||
/* Every (family, address) the hosts resolve to, deduplicated; -1 if any lookup fails. */
|
||||
static int dns_resolve_all(const dns_ctx_t *x, dns_addr_t *out, int max) {
|
||||
int n = 0;
|
||||
for (int i = 0; i < x->nhost; i++) {
|
||||
struct addrinfo hints = {0}, *res = NULL;
|
||||
hints.ai_socktype = SOCK_DGRAM;
|
||||
hints.ai_protocol = IPPROTO_UDP;
|
||||
hints.ai_family = x->af_filter;
|
||||
char ps[16]; snprintf(ps, sizeof(ps), "%d", x->port);
|
||||
if (getaddrinfo(x->hosts[i], ps, &hints, &res) != 0) {
|
||||
LOGW("cannot resolve %s", x->hosts[i]);
|
||||
return -1;
|
||||
}
|
||||
for (struct addrinfo *ai = res; ai && n < max; ai = ai->ai_next) {
|
||||
dns_addr_t a; a.af = ai->ai_family;
|
||||
void *ip = (ai->ai_family == AF_INET)
|
||||
? (void *)&((struct sockaddr_in *)ai->ai_addr)->sin_addr
|
||||
: (void *)&((struct sockaddr_in6 *)ai->ai_addr)->sin6_addr;
|
||||
inet_ntop(ai->ai_family, ip, a.addr, sizeof(a.addr));
|
||||
bool dup = false;
|
||||
for (int k = 0; k < n; k++) if (strcmp(out[k].addr, a.addr) == 0) dup = true;
|
||||
if (!dup) out[n++] = a;
|
||||
}
|
||||
freeaddrinfo(res);
|
||||
}
|
||||
return n;
|
||||
}
|
||||
|
||||
static void *dns_thread(void *arg) {
|
||||
const dns_ctx_t *x = arg;
|
||||
int delay = dns_refresh_delay(x);
|
||||
while (g_running) {
|
||||
for (int s = 0; s < delay && g_running; s++) sleep(1);
|
||||
if (!g_running) break;
|
||||
dns_addr_t tmp[MAX_HOSTS];
|
||||
int n = dns_resolve_all(x, tmp, MAX_HOSTS);
|
||||
if (n <= 0) { delay = DNS_FAILED_RETRY; continue; }
|
||||
pthread_mutex_lock(&g_dns.mu);
|
||||
memcpy(g_dns.addrs, tmp, sizeof(tmp));
|
||||
g_dns.n = n; g_dns.ready = true;
|
||||
pthread_mutex_unlock(&g_dns.mu);
|
||||
delay = dns_refresh_delay(x);
|
||||
}
|
||||
return NULL;
|
||||
}
|
||||
|
||||
/* Point c at a new server address; the socket is replaced if the family changed. */
|
||||
static void conn_retarget(conn_t *c, int af, const char *addr) {
|
||||
LOGI("server address %s -> %s", c->addr, addr);
|
||||
pthread_mutex_lock(&c->mu);
|
||||
if (af != c->af) {
|
||||
int fd = socket(af, SOCK_DGRAM, 0);
|
||||
if (fd < 0) { LOGE("socket: %s", strerror(errno)); pthread_mutex_unlock(&c->mu); return; }
|
||||
fcntl(fd, F_SETFL, fcntl(fd, F_GETFL, 0) | O_NONBLOCK);
|
||||
if (c->sockfd >= 0) close(c->sockfd);
|
||||
c->sockfd = fd; c->af = af;
|
||||
}
|
||||
snprintf(c->addr, sizeof(c->addr), "%s", addr);
|
||||
pthread_mutex_unlock(&c->mu);
|
||||
}
|
||||
|
||||
/* Main-loop side: follow the latest resolution. Mirrors hbd.client.dnsttl.retarget(). Gone
|
||||
* addresses are re-pointed at new ones in list order, so g_conns[0] (the one plugin data
|
||||
* uses) survives as long as anything resolves; the rest are retired, surplus new ones added. */
|
||||
static void dns_apply(int *conn_id, const char *iam, int port) {
|
||||
dns_addr_t r[MAX_HOSTS]; int nr;
|
||||
pthread_mutex_lock(&g_dns.mu);
|
||||
if (!g_dns.ready) { pthread_mutex_unlock(&g_dns.mu); return; }
|
||||
memcpy(r, g_dns.addrs, sizeof(r)); nr = g_dns.n; g_dns.ready = false;
|
||||
pthread_mutex_unlock(&g_dns.mu);
|
||||
|
||||
int gone[MAX_HOSTS], ng = 0, fresh[MAX_HOSTS], nf = 0;
|
||||
for (int i = 0; i < g_nconns; i++) {
|
||||
bool wanted = false;
|
||||
for (int j = 0; j < nr; j++) if (strcmp(g_conns[i].addr, r[j].addr) == 0) wanted = true;
|
||||
if (!wanted) gone[ng++] = i;
|
||||
}
|
||||
for (int j = 0; j < nr; j++) {
|
||||
bool have = false;
|
||||
for (int i = 0; i < g_nconns; i++) if (strcmp(g_conns[i].addr, r[j].addr) == 0) have = true;
|
||||
if (!have) fresh[nf++] = j;
|
||||
}
|
||||
|
||||
int paired = ng < nf ? ng : nf;
|
||||
for (int k = 0; k < paired; k++) conn_retarget(&g_conns[gone[k]], r[fresh[k]].af, r[fresh[k]].addr);
|
||||
|
||||
/* drop[] holds connection indexes to retire, in ascending order */
|
||||
int drop[MAX_HOSTS], nd = 0;
|
||||
for (int k = paired; k < ng; k++) drop[nd++] = gone[k];
|
||||
if (nd > 0 && drop[0] == 0) {
|
||||
int keep = -1;
|
||||
for (int i = 1; i < g_nconns && keep < 0; i++) {
|
||||
bool isgone = false;
|
||||
for (int k = 0; k < ng; k++) if (gone[k] == i) isgone = true;
|
||||
if (!isgone) keep = i;
|
||||
}
|
||||
if (keep > 0) {
|
||||
conn_retarget(&g_conns[0], g_conns[keep].af, g_conns[keep].addr);
|
||||
drop[0] = keep;
|
||||
for (int a = 0; a < nd - 1 && drop[a] > drop[a + 1]; a++) {
|
||||
int tmp = drop[a]; drop[a] = drop[a + 1]; drop[a + 1] = tmp;
|
||||
}
|
||||
}
|
||||
}
|
||||
for (int k = nd - 1; k >= 0; k--) {
|
||||
int i = drop[k];
|
||||
if (i == 0) continue;
|
||||
LOGI("dropping server address %s", g_conns[i].addr);
|
||||
conn_close(&g_conns[i]);
|
||||
pthread_mutex_destroy(&g_conns[i].mu);
|
||||
int last = --g_nconns;
|
||||
if (i != last) {
|
||||
memcpy(&g_conns[i], &g_conns[last], sizeof(conn_t));
|
||||
pthread_mutex_init(&g_conns[i].mu, NULL);
|
||||
}
|
||||
}
|
||||
|
||||
for (int k = paired; k < nf && g_nconns < MAX_HOSTS; k++) {
|
||||
conn_t *c = &g_conns[g_nconns];
|
||||
memset(c, 0, sizeof(*c));
|
||||
c->conn_id = (*conn_id)++; c->port = port;
|
||||
c->af = r[fresh[k]].af; c->sockfd = -1;
|
||||
snprintf(c->name, sizeof(c->name), "%s", iam);
|
||||
snprintf(c->addr, sizeof(c->addr), "%s", r[fresh[k]].addr);
|
||||
if (conn_open(c)) { g_nconns++; LOGI("adding server address %s", c->addr); }
|
||||
}
|
||||
}
|
||||
|
||||
int main(int argc, char **argv) {
|
||||
bool do_boot = false;
|
||||
bool do_daemon = false;
|
||||
@@ -1323,6 +1881,7 @@ int main(int argc, char **argv) {
|
||||
|
||||
config_t cfg;
|
||||
config_load(&cfg, cfgpath);
|
||||
probe_init(&g_probe, &cfg);
|
||||
|
||||
if (do_daemon) {
|
||||
daemonize();
|
||||
@@ -1405,9 +1964,24 @@ int main(int argc, char **argv) {
|
||||
pthread_t tid;
|
||||
pthread_create(&tid, NULL, monitor_thread, &tctx);
|
||||
|
||||
pthread_t ptid; bool probe_running = false;
|
||||
if (g_probe_enabled) {
|
||||
int prc = pthread_create(&ptid, NULL, probe_thread, &g_probe);
|
||||
if (prc == 0) probe_running = true;
|
||||
else LOGW("pthread_create(probe): %s", strerror(prc));
|
||||
}
|
||||
|
||||
/* Re-resolution at the DNS TTL; detached, the process exit reaps it */
|
||||
dns_ctx_t dns_ctx = { hosts, nhost, cfg.hb_port, af_filter };
|
||||
pthread_t dtid;
|
||||
int drc = pthread_create(&dtid, NULL, dns_thread, &dns_ctx);
|
||||
if (drc == 0) pthread_detach(dtid);
|
||||
else LOGW("pthread_create(dns): %s", strerror(drc));
|
||||
|
||||
/* Main heartbeat + receive loop */
|
||||
time_t last_hb = 0;
|
||||
while (g_running) {
|
||||
dns_apply(&conn_id, iam, cfg.hb_port);
|
||||
time_t now = time(NULL);
|
||||
if (now - last_hb >= cfg.interval) {
|
||||
for (int i = 0; i < g_nconns; i++) {
|
||||
@@ -1415,6 +1989,7 @@ int main(int argc, char **argv) {
|
||||
kv_set_int(&hb, "acks", g_conns[i].ack_count);
|
||||
kv_set_dbl(&hb, "rtt", g_conns[i].rtt);
|
||||
kv_set_int(&hb, "interval", cfg.interval);
|
||||
if (g_probe_enabled) probe_flags(&g_probe, &hb);
|
||||
conn_send(&g_conns[i], "HTB", &hb);
|
||||
}
|
||||
last_hb = now;
|
||||
@@ -1449,6 +2024,10 @@ int main(int argc, char **argv) {
|
||||
conn_send(primary, "HTB", &sh);
|
||||
usleep(300000);
|
||||
}
|
||||
/* Joined after the shutdown HTB goes out: the probe thread is never cancelled (it can be
|
||||
* inside popen), so joining it can block for a few seconds — long enough to lose the
|
||||
* clean-shutdown notification to an impatient supervisor's SIGKILL. */
|
||||
if (probe_running) pthread_join(ptid, NULL);
|
||||
for (int i = 0; i < g_nconns; i++) conn_close(&g_conns[i]);
|
||||
if (g_use_syslog) closelog();
|
||||
if (g_restart) { LOGI("restarting..."); execv(argv[0], argv); }
|
||||
|
||||
@@ -0,0 +1,294 @@
|
||||
/* Unit tests for the connectivity-probe helpers in hbc_mini.c.
|
||||
* Build and run: make test (live check of this host: ./test_probe --live)
|
||||
* The source file is included directly so its static functions can be exercised; its main()
|
||||
* is renamed out of the way.
|
||||
*/
|
||||
#define main hbc_mini_main
|
||||
#include "hbc_mini.c"
|
||||
#undef main
|
||||
|
||||
static int g_run = 0, g_fail = 0;
|
||||
#define CHECK(cond) do { g_run++; if (!(cond)) { g_fail++; \
|
||||
fprintf(stderr, "FAIL %s:%d: %s\n", __FILE__, __LINE__, #cond); } } while (0)
|
||||
#define CHECK_STR(a, b) do { g_run++; if (strcmp((a), (b)) != 0) { g_fail++; \
|
||||
fprintf(stderr, "FAIL %s:%d: \"%s\" != \"%s\"\n", __FILE__, __LINE__, (a), (b)); } } while (0)
|
||||
|
||||
static const char PROC_V4[] =
|
||||
"Iface\tDestination\tGateway \tFlags\tRefCnt\tUse\tMetric\tMask\t\tMTU\tWindow\tIRTT\n"
|
||||
"wlan0\t00000000\t0100A8C0\t0003\t0\t0\t600\t00000000\t0\t0\t0\n"
|
||||
"eth0\t00000000\t0101A8C0\t0003\t0\t0\t100\t00000000\t0\t0\t0\n"
|
||||
"wlan0\t0000A8C0\t00000000\t0001\t0\t0\t600\t00FFFFFF\t0\t0\t0\n";
|
||||
|
||||
static const char PROC_V6[] =
|
||||
"00000000000000000000000000000000 00 00000000000000000000000000000000 00 "
|
||||
"fe800000000000000000000000000001 00000400 00000001 00000000 00000003 wlan0\n"
|
||||
"00000000000000000000000000000000 00 00000000000000000000000000000000 00 "
|
||||
"00000000000000000000000000000000 ffffffff 00000001 00000000 00200200 lo\n"
|
||||
"fe800000000000000000000000000000 40 00000000000000000000000000000000 00 "
|
||||
"00000000000000000000000000000000 00000100 00000001 00000000 00000001 wlan0\n";
|
||||
|
||||
static const char ROUTE_GET[] =
|
||||
" route to: 0.0.0.0\ndestination: 0.0.0.0\n mask: 0.0.0.0\n"
|
||||
" gateway: 192.0.2.1\n fib: 0\n interface: vtnet0\n";
|
||||
|
||||
static const char LINUX_PING[] =
|
||||
"PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.\n"
|
||||
"64 bytes from 1.1.1.1: icmp_seq=1 ttl=57 time=11.2 ms\n\n"
|
||||
"--- 1.1.1.1 ping statistics ---\n"
|
||||
"1 packets transmitted, 1 received, 0% packet loss, time 0ms\n"
|
||||
"rtt min/avg/max/mdev = 11.234/11.234/11.234/0.000 ms\n";
|
||||
|
||||
static const char FREEBSD_PING[] =
|
||||
"PING 1.1.1.1 (1.1.1.1): 56 data bytes\n"
|
||||
"64 bytes from 1.1.1.1: icmp_seq=0 ttl=59 time=4.006 ms\n\n"
|
||||
"--- 1.1.1.1 ping statistics ---\n"
|
||||
"1 packets transmitted, 1 packets received, 0.0% packet loss, 1 packets out of wait time\n"
|
||||
"round-trip min/avg/max/stddev = 4.006/4.006/4.006/0.000 ms\n";
|
||||
|
||||
static void test_gateway_parsers(void) {
|
||||
char gw[64];
|
||||
gw_parse_proc_v4(PROC_V4, gw, sizeof(gw));
|
||||
CHECK_STR(gw, "192.168.1.1"); /* eth0, metric 100 beats wlan0, 600 */
|
||||
gw_parse_proc_v4("Iface\tDestination\n", gw, sizeof(gw));
|
||||
CHECK_STR(gw, "");
|
||||
gw_parse_proc_v6(PROC_V6, gw, sizeof(gw));
|
||||
CHECK_STR(gw, "fe80::1"); /* reject and non-default lines skipped */
|
||||
gw_parse_route_get(ROUTE_GET, gw, sizeof(gw));
|
||||
CHECK_STR(gw, "192.0.2.1");
|
||||
gw_parse_route_get("route: writing to routing socket: not in table\n", gw, sizeof(gw));
|
||||
CHECK_STR(gw, "");
|
||||
}
|
||||
|
||||
static void test_ping_parser(void) {
|
||||
CHECK(ping_parse_rtt(LINUX_PING) > 11.23 && ping_parse_rtt(LINUX_PING) < 11.24);
|
||||
CHECK(ping_parse_rtt(FREEBSD_PING) > 4.0 && ping_parse_rtt(FREEBSD_PING) < 4.01);
|
||||
CHECK(ping_parse_rtt("1 packets transmitted, 0 received, 100% packet loss\n") == -1.0);
|
||||
CHECK(ping_parse_rtt("sh: 1: ping: not found\n") == -2.0);
|
||||
CHECK(ping_parse_rtt("ping: command not found\n") == -2.0);
|
||||
}
|
||||
|
||||
/* ---- scriptable network for probe_step ------------------------------------------------ */
|
||||
static struct {
|
||||
char addr[2][INET6_ADDRSTRLEN]; /* "" = no route */
|
||||
char gw[2][INET6_ADDRSTRLEN];
|
||||
double rtt[2]; /* -1 no reply, -2 ping unavailable */
|
||||
int unsupported[2];
|
||||
int npings[2];
|
||||
} net;
|
||||
|
||||
static int fake_local(int af, const char *t, char *out, size_t n) {
|
||||
int f = af == AF_INET6; (void)t;
|
||||
if (net.unsupported[f]) return -1;
|
||||
if (!net.addr[f][0]) return 0;
|
||||
snprintf(out, n, "%s", net.addr[f]);
|
||||
return 1;
|
||||
}
|
||||
static double fake_ping(int af, const char *t) { int f = af == AF_INET6; (void)t; net.npings[f]++; return net.rtt[f]; }
|
||||
static void fake_gw(int af, char *out, size_t n) { snprintf(out, n, "%s", net.gw[af == AF_INET6]); }
|
||||
static const probe_ops_t FAKE = { fake_local, fake_ping, fake_gw };
|
||||
|
||||
static void net_reset(void) {
|
||||
memset(&net, 0, sizeof(net));
|
||||
snprintf(net.addr[0], INET6_ADDRSTRLEN, "192.0.2.5");
|
||||
snprintf(net.addr[1], INET6_ADDRSTRLEN, "2001:db8::5");
|
||||
snprintf(net.gw[0], INET6_ADDRSTRLEN, "192.0.2.1");
|
||||
snprintf(net.gw[1], INET6_ADDRSTRLEN, "fe80::1");
|
||||
net.rtt[0] = 10.0; net.rtt[1] = 20.0;
|
||||
}
|
||||
|
||||
static void probe_reset(probe_t *p) {
|
||||
memset(p, 0, sizeof(*p));
|
||||
snprintf(p->target[0], sizeof(p->target[0]), "1.1.1.1");
|
||||
snprintf(p->target[1], sizeof(p->target[1]), "2606:4700:4700::1111");
|
||||
p->probe_interval = 60; p->retry_interval = 10; p->cheap_gateway = true;
|
||||
p->fam[0].present = p->fam[1].present = PROBE_UNKNOWN;
|
||||
p->fam[0].rtt = p->fam[1].rtt = -1;
|
||||
pthread_mutex_init(&p->mu, NULL);
|
||||
}
|
||||
|
||||
static void step_both(probe_t *p, double now) {
|
||||
probe_step(p, &p->fam[0], 0, &FAKE, now);
|
||||
probe_step(p, &p->fam[1], 1, &FAKE, now);
|
||||
}
|
||||
|
||||
static void test_present_with_details(void) {
|
||||
net_reset(); probe_t p; probe_reset(&p);
|
||||
step_both(&p, 1000);
|
||||
CHECK(p.fam[0].present == 1 && p.fam[1].present == 1);
|
||||
CHECK_STR(p.fam[0].addr, "192.0.2.5");
|
||||
CHECK_STR(p.fam[0].gateway, "192.0.2.1");
|
||||
CHECK(p.fam[0].rtt == 10.0);
|
||||
kvdict_t d; kv_clear(&d);
|
||||
probe_flags(&p, &d);
|
||||
CHECK_STR(kv_get(&d, "p4") ? kv_get(&d, "p4") : "(null)", "1");
|
||||
CHECK_STR(kv_get(&d, "p6") ? kv_get(&d, "p6") : "(null)", "1");
|
||||
char js[512];
|
||||
CHECK(probe_metric_json(&p, js, sizeof(js)) > 0);
|
||||
CHECK(strstr(js, "\"ipv6\":{\"present\":true,\"addr\":\"2001:db8::5\",\"gateway\":\"fe80::1\"") != NULL);
|
||||
}
|
||||
|
||||
static void test_no_route_is_absent_and_wastes_no_ping(void) {
|
||||
net_reset(); net.addr[1][0] = '\0'; probe_t p; probe_reset(&p);
|
||||
step_both(&p, 1000);
|
||||
CHECK(p.fam[1].present == 0 && net.npings[1] == 0 && net.npings[0] == 1);
|
||||
kvdict_t d; kv_clear(&d);
|
||||
probe_flags(&p, &d);
|
||||
CHECK_STR(kv_get(&d, "p6") ? kv_get(&d, "p6") : "(null)", "0");
|
||||
char js[512];
|
||||
probe_metric_json(&p, js, sizeof(js));
|
||||
CHECK(strstr(js, "\"ipv6\":{\"present\":false,\"addr\":null,\"gateway\":null,\"rtt\":null}") != NULL);
|
||||
}
|
||||
|
||||
static void test_debounce_and_rate_limit(void) {
|
||||
net_reset(); probe_t p; probe_reset(&p);
|
||||
step_both(&p, 1000);
|
||||
step_both(&p, 1000); /* cached: no second ping */
|
||||
CHECK(net.npings[0] == 1 && net.npings[1] == 1);
|
||||
net.rtt[1] = -1;
|
||||
step_both(&p, 1060); /* first failure: still present */
|
||||
CHECK(p.fam[1].present == 1);
|
||||
step_both(&p, 1070); /* retry interval elapsed: flips */
|
||||
CHECK(p.fam[1].present == 0);
|
||||
net.rtt[1] = 25;
|
||||
step_both(&p, 1080); /* first success recovers */
|
||||
CHECK(p.fam[1].present == 1);
|
||||
}
|
||||
|
||||
static void test_first_failure_is_unknown_and_omitted(void) {
|
||||
net_reset(); net.rtt[0] = -1; probe_t p; probe_reset(&p);
|
||||
step_both(&p, 1000);
|
||||
CHECK(p.fam[0].present == PROBE_UNKNOWN);
|
||||
kvdict_t d; kv_clear(&d);
|
||||
probe_flags(&p, &d);
|
||||
CHECK(kv_get(&d, "p4") == NULL && kv_get(&d, "p6") != NULL);
|
||||
char js[512];
|
||||
probe_metric_json(&p, js, sizeof(js));
|
||||
CHECK(strstr(js, "\"ipv4\":{\"present\":null") != NULL);
|
||||
}
|
||||
|
||||
static void test_address_or_gateway_change_repings(void) {
|
||||
net_reset(); probe_t p; probe_reset(&p);
|
||||
step_both(&p, 1000);
|
||||
snprintf(net.addr[0], INET6_ADDRSTRLEN, "198.51.100.7");
|
||||
step_both(&p, 1000);
|
||||
CHECK(net.npings[0] == 2);
|
||||
snprintf(net.gw[0], INET6_ADDRSTRLEN, "192.0.2.254");
|
||||
step_both(&p, 1000);
|
||||
CHECK(net.npings[0] == 3);
|
||||
}
|
||||
|
||||
static void test_route_loss_then_return_pings_immediately(void) {
|
||||
net_reset(); probe_t p; probe_reset(&p);
|
||||
step_both(&p, 1000);
|
||||
net.addr[1][0] = '\0';
|
||||
step_both(&p, 1000);
|
||||
CHECK(p.fam[1].present == 0);
|
||||
snprintf(net.addr[1], INET6_ADDRSTRLEN, "2001:db8::5");
|
||||
step_both(&p, 1000);
|
||||
CHECK(net.npings[1] == 2 && p.fam[1].present == 1);
|
||||
}
|
||||
|
||||
static void test_missing_ping_stays_unknown_even_after_route_loss(void) {
|
||||
net_reset(); net.rtt[0] = -2; probe_t p; probe_reset(&p);
|
||||
step_both(&p, 1000);
|
||||
step_both(&p, 1600);
|
||||
CHECK(p.fam[0].present == PROBE_UNKNOWN && net.npings[0] == 1);
|
||||
net.addr[0][0] = '\0';
|
||||
step_both(&p, 1700);
|
||||
CHECK(p.fam[0].present == 0); /* no route is definitive */
|
||||
snprintf(net.addr[0], INET6_ADDRSTRLEN, "192.0.2.5");
|
||||
step_both(&p, 1800);
|
||||
CHECK(p.fam[0].present == PROBE_UNKNOWN && net.npings[0] == 1);
|
||||
}
|
||||
|
||||
static void test_unsupported_family_is_unknown(void) {
|
||||
net_reset(); net.unsupported[1] = 1; probe_t p; probe_reset(&p);
|
||||
step_both(&p, 1000);
|
||||
CHECK(p.fam[1].present == PROBE_UNKNOWN && p.fam[0].present == 1);
|
||||
}
|
||||
|
||||
static int live_mode(void) {
|
||||
static const probe_ops_t sys_ops = { sys_local_address, sys_ping, sys_gateway };
|
||||
probe_t p; probe_reset(&p);
|
||||
p.cheap_gateway = false;
|
||||
snprintf(p.target[0], sizeof(p.target[0]), "1.1.1.1");
|
||||
for (int round = 0; round < 3; round++) {
|
||||
double now = mono_ts();
|
||||
for (int f = 0; f < 2; f++) probe_step(&p, &p.fam[f], f, &sys_ops, now);
|
||||
sleep(round < 2 ? 11 : 0);
|
||||
}
|
||||
char js[512];
|
||||
probe_metric_json(&p, js, sizeof(js));
|
||||
kvdict_t d; kv_clear(&d);
|
||||
probe_flags(&p, &d);
|
||||
printf("%s\np4=%s p6=%s\n", js, kv_get(&d, "p4") ? kv_get(&d, "p4") : "-",
|
||||
kv_get(&d, "p6") ? kv_get(&d, "p6") : "-");
|
||||
return 0;
|
||||
}
|
||||
|
||||
/* ---- server address re-resolution (dns_apply) ---- */
|
||||
|
||||
static void dns_conn(int i, int af, const char *addr) {
|
||||
memset(&g_conns[i], 0, sizeof(conn_t));
|
||||
g_conns[i].conn_id = i + 1; g_conns[i].af = af; g_conns[i].port = 5;
|
||||
snprintf(g_conns[i].addr, sizeof(g_conns[i].addr), "%s", addr);
|
||||
conn_open(&g_conns[i]);
|
||||
}
|
||||
|
||||
static void dns_push(int n, const dns_addr_t *a) {
|
||||
memcpy(g_dns.addrs, a, n * sizeof(*a)); g_dns.n = n; g_dns.ready = true;
|
||||
}
|
||||
|
||||
static void test_dns_apply_follows_changes(void) {
|
||||
int id = 3;
|
||||
dns_conn(0, AF_INET, "1.1.1.1"); dns_conn(1, AF_INET6, "2001:db8::1"); g_nconns = 2;
|
||||
|
||||
dns_addr_t moved[2] = {{AF_INET, "9.9.9.9"}, {AF_INET6, "2001:db8::9"}};
|
||||
dns_push(2, moved); dns_apply(&id, "me", 5);
|
||||
CHECK(g_nconns == 2);
|
||||
CHECK_STR(g_conns[0].addr, "9.9.9.9");
|
||||
CHECK_STR(g_conns[1].addr, "2001:db8::9");
|
||||
|
||||
/* the first connection's address vanishes and nothing replaces it: it survives, the other goes */
|
||||
dns_addr_t v6only[1] = {{AF_INET6, "2001:db8::9"}};
|
||||
dns_push(1, v6only); dns_apply(&id, "me", 5);
|
||||
CHECK(g_nconns == 1);
|
||||
CHECK_STR(g_conns[0].addr, "2001:db8::9");
|
||||
CHECK(g_conns[0].af == AF_INET6);
|
||||
|
||||
dns_addr_t grown[2] = {{AF_INET6, "2001:db8::9"}, {AF_INET, "5.5.5.5"}};
|
||||
dns_push(2, grown); dns_apply(&id, "me", 5);
|
||||
CHECK(g_nconns == 2);
|
||||
CHECK_STR(g_conns[1].addr, "5.5.5.5");
|
||||
CHECK(g_conns[1].conn_id == 3);
|
||||
|
||||
dns_push(2, grown); dns_apply(&id, "me", 5); /* unchanged: nothing happens */
|
||||
CHECK(g_nconns == 2 && id == 4);
|
||||
for (int i = 0; i < g_nconns; i++) conn_close(&g_conns[i]);
|
||||
g_nconns = 0;
|
||||
}
|
||||
|
||||
static void test_dns_ttl_unknown_for_ip_literals(void) {
|
||||
CHECK(dns_host_ttl("192.0.2.1", 0) == -1);
|
||||
CHECK(dns_host_ttl("2001:db8::1", 0) == -1);
|
||||
const char *h[] = {"192.0.2.1"}; dns_ctx_t x = {h, 1, 5, 0};
|
||||
CHECK(dns_refresh_delay(&x) == DNS_DEFAULT_TTL);
|
||||
}
|
||||
|
||||
int main(int argc, char **argv) {
|
||||
if (argc > 1 && !strcmp(argv[1], "--live")) return live_mode();
|
||||
test_gateway_parsers();
|
||||
test_ping_parser();
|
||||
test_present_with_details();
|
||||
test_no_route_is_absent_and_wastes_no_ping();
|
||||
test_debounce_and_rate_limit();
|
||||
test_first_failure_is_unknown_and_omitted();
|
||||
test_address_or_gateway_change_repings();
|
||||
test_route_loss_then_return_pings_immediately();
|
||||
test_missing_ping_stays_unknown_even_after_route_loss();
|
||||
test_unsupported_family_is_unknown();
|
||||
test_dns_apply_follows_changes();
|
||||
test_dns_ttl_unknown_for_ip_literals();
|
||||
printf("%d checks, %d failed\n", g_run, g_fail);
|
||||
return g_fail ? 1 : 0;
|
||||
}
|
||||
+566
-18
@@ -10,18 +10,25 @@ Built-in plugins (always available):
|
||||
os_info — OS name, kernel, distro, Python version
|
||||
ping_monitor — ICMP ping RTT to configured hosts
|
||||
nagios_runner — run Nagios-compatible check scripts
|
||||
network_monitor — per-interface TX/RX bytes/s (Linux); per-protocol connectivity (all platforms)
|
||||
|
||||
Linux-only plugins (via /proc, no extra tools needed):
|
||||
cpu_monitor — CPU %, load average, core count
|
||||
memory_monitor — RAM and swap usage
|
||||
network_monitor — per-interface TX/RX bytes/s
|
||||
|
||||
Cross-platform (via df subprocess):
|
||||
disk_monitor — per-mount usage and capacity
|
||||
|
||||
Presence probe (built in): reports per protocol (IPv4/IPv6) whether the host has internet
|
||||
connectivity, as p4/p6 flags on every heartbeat, so hbd stops treating a missing IPv6 (or
|
||||
IPv4) path as an outage. Config, in the network_monitor section of ~/.hbc.json:
|
||||
probe_enabled (default true), probe_v4 (default 1.1.1.1), probe_v6 (default
|
||||
2606:4700:4700::1111; IP literals only), probe_interval (default 60 s)
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import asyncio
|
||||
import ipaddress
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
@@ -30,18 +37,20 @@ import re
|
||||
import shutil
|
||||
import signal
|
||||
import socket
|
||||
import struct
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
import zlib
|
||||
from abc import ABC, abstractmethod
|
||||
from collections import defaultdict
|
||||
from dataclasses import asdict, dataclass
|
||||
from logging.handlers import SysLogHandler
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List, Optional, Tuple
|
||||
from typing import Any, Awaitable, Callable, Dict, List, Optional, Set, Tuple
|
||||
|
||||
# updated by scripts/bumpminor.sh
|
||||
__version__ = "5.4.4"
|
||||
__version__ = "5.4.7"
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Protocol (mirrors hbd/common/proto.py)
|
||||
@@ -637,6 +646,315 @@ class DiskMonitorPlugin(MonitorPlugin):
|
||||
return {"partitions": partitions} if partitions else {}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Connectivity probe — per-protocol internet presence (IPv4 / IPv6)
|
||||
#
|
||||
# Tier 1 (every tier1_interval s, sends no packets): a UDP connect() to the probe address
|
||||
# shows whether a route exists and which source address would be used. Tier 2 (every
|
||||
# probe_interval s, sooner after a failure or a change of address or gateway): one ICMP echo,
|
||||
# only when tier 1 found a route. A family is absent after no route at all or fail_limit
|
||||
# consecutive failed echoes, present after one success, and unknown (flag omitted) when it
|
||||
# cannot be probed. The result rides on every HTB as p4/p6 and, when network_monitor runs, as
|
||||
# its `connectivity` metric. Mirrors hbd/client/connectivity.py.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_FAMILIES = ("ipv4", "ipv6")
|
||||
_FLAG_KEYS = {"ipv4": "p4", "ipv6": "p6"}
|
||||
_DEFAULT_PROBES = {"ipv4": "1.1.1.1", "ipv6": "2606:4700:4700::1111"}
|
||||
_BSD_PLATFORMS = ("darwin", "freebsd", "netbsd", "openbsd", "dragonfly")
|
||||
|
||||
|
||||
class _ProbeUnavailable(Exception):
|
||||
"""The address family cannot be probed on this host (e.g. no IPv6 support)."""
|
||||
|
||||
|
||||
class _PingUnavailable(Exception):
|
||||
"""The system ping binary could not be executed."""
|
||||
|
||||
|
||||
def _parse_proc_route_v4(text: str) -> Optional[str]:
|
||||
"""Default gateway from /proc/net/route (lowest metric wins)."""
|
||||
best: Optional[Tuple[int, str]] = None
|
||||
for line in text.splitlines()[1:]:
|
||||
f = line.split()
|
||||
if len(f) < 8 or f[1] != "00000000" or not int(f[3], 16) & 0x2:
|
||||
continue
|
||||
metric = int(f[6])
|
||||
if best is None or metric < best[0]:
|
||||
# the kernel prints the raw network-order u32 in host order: use the host byte order
|
||||
best = (metric, socket.inet_ntoa(int(f[2], 16).to_bytes(4, sys.byteorder)))
|
||||
return best[1] if best else None
|
||||
|
||||
|
||||
def _parse_proc_route_v6(text: str) -> Optional[str]:
|
||||
"""Default gateway from /proc/net/ipv6_route (lowest metric wins)."""
|
||||
best: Optional[Tuple[int, str]] = None
|
||||
for line in text.splitlines():
|
||||
f = line.split()
|
||||
if len(f) < 10:
|
||||
continue
|
||||
dest, plen, nexthop, metric, flags, iface = f[0], f[1], f[4], f[5], f[8], f[9]
|
||||
if dest != "0" * 32 or plen != "00" or nexthop == "0" * 32 or iface == "lo":
|
||||
continue
|
||||
fl = int(flags, 16)
|
||||
if not fl & 0x2 or fl & 0x200: # need RTF_GATEWAY, skip RTF_REJECT
|
||||
continue
|
||||
m = int(metric, 16)
|
||||
if best is None or m < best[0]:
|
||||
best = (m, socket.inet_ntop(socket.AF_INET6, bytes.fromhex(nexthop)))
|
||||
return best[1] if best else None
|
||||
|
||||
|
||||
def _parse_route_get(text: str) -> Optional[str]:
|
||||
"""Gateway from `route -n get [-inet6] default` (macOS and the BSDs)."""
|
||||
m = re.search(r"^\s*gateway:\s*(\S+)", text, re.MULTILINE)
|
||||
return m.group(1) if m else None
|
||||
|
||||
|
||||
def _parse_windows_route(text: str, family: str) -> Optional[str]:
|
||||
"""Default gateway from `route print -4` / `route print -6` (lowest metric wins)."""
|
||||
best: Optional[Tuple[int, str]] = None
|
||||
for line in text.splitlines():
|
||||
if family == "ipv4":
|
||||
f = line.split()
|
||||
if not (len(f) >= 5 and f[0] == "0.0.0.0" and f[1] == "0.0.0.0" and f[2] != "On-link"):
|
||||
continue
|
||||
cand = (int(f[4]), f[2])
|
||||
else:
|
||||
m = re.match(r"^\s*\d+\s+(\d+)\s+::/0\s+(\S+)\s*$", line)
|
||||
if not m or m.group(2) == "On-link":
|
||||
continue
|
||||
cand = (int(m.group(1)), m.group(2))
|
||||
if best is None or cand[0] < best[0]:
|
||||
best = cand
|
||||
return best[1] if best else None
|
||||
|
||||
|
||||
async def _run_cmd(cmd: List[str], timeout: float = 5.0) -> str:
|
||||
proc = await asyncio.create_subprocess_exec(
|
||||
*cmd, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.DEVNULL)
|
||||
try:
|
||||
out, _ = await asyncio.wait_for(proc.communicate(), timeout=timeout)
|
||||
except asyncio.TimeoutError:
|
||||
proc.kill()
|
||||
await proc.wait()
|
||||
raise
|
||||
return out.decode(errors="replace")
|
||||
|
||||
|
||||
async def _default_gateway(family: str) -> Optional[str]:
|
||||
"""Default gateway for "ipv4"/"ipv6", or None on any failure (informational only)."""
|
||||
v4 = family == "ipv4"
|
||||
try:
|
||||
if sys.platform.startswith("linux"):
|
||||
with open("/proc/net/route" if v4 else "/proc/net/ipv6_route") as fh:
|
||||
text = fh.read()
|
||||
return _parse_proc_route_v4(text) if v4 else _parse_proc_route_v6(text)
|
||||
if sys.platform.startswith(_BSD_PLATFORMS):
|
||||
cmd = ["route", "-n", "get"] + ([] if v4 else ["-inet6"]) + ["default"]
|
||||
return _parse_route_get(await _run_cmd(cmd))
|
||||
if sys.platform == "win32":
|
||||
text = await _run_cmd(["route", "print", "-4" if v4 else "-6"])
|
||||
return _parse_windows_route(text, family)
|
||||
except Exception:
|
||||
return None
|
||||
return None
|
||||
|
||||
|
||||
def _probe_ping_command(host: str, family: str) -> List[str]:
|
||||
"""Command for ONE echo request with a ~2 s reply wait."""
|
||||
v4 = family == "ipv4"
|
||||
if sys.platform == "win32":
|
||||
return ["ping", "-n", "1", "-w", "2000", "-4" if v4 else "-6", host]
|
||||
if sys.platform.startswith(_BSD_PLATFORMS):
|
||||
# BSD/macOS -W is in milliseconds, and IPv6 is reached through ping6.
|
||||
return ["ping", "-c", "1", "-W", "2000", host] if v4 else ["ping6", "-c", "1", host]
|
||||
return ["ping", "-c", "1", "-W", "2", "-4" if v4 else "-6", host]
|
||||
|
||||
|
||||
def _parse_ping_rtt(output: str) -> Optional[float]:
|
||||
"""Average RTT in ms from ping output, or None when there was no reply."""
|
||||
m = re.search(r"(?:rtt|round-trip)\s+min/avg/max/\S+\s*=\s*([\d.]+)/([\d.]+)/([\d.]+)", output)
|
||||
if m:
|
||||
return float(m.group(2))
|
||||
if "TTL=" in output: # Windows reply line (also when localized)
|
||||
m = re.search(r"[=<]\s*(\d+)\s*ms", output)
|
||||
return float(m.group(1)) if m else 0.0
|
||||
return None
|
||||
|
||||
|
||||
async def _ping_rtt(target: str, family: str) -> Optional[float]:
|
||||
"""One ICMP echo; RTT in ms, or None if no reply. Raises _PingUnavailable without ping."""
|
||||
try:
|
||||
proc = await asyncio.create_subprocess_exec(
|
||||
*_probe_ping_command(target, family),
|
||||
stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE)
|
||||
except FileNotFoundError as e:
|
||||
raise _PingUnavailable(str(e))
|
||||
try:
|
||||
out, _ = await asyncio.wait_for(proc.communicate(), timeout=6)
|
||||
except asyncio.TimeoutError:
|
||||
proc.kill()
|
||||
await proc.wait()
|
||||
return None
|
||||
return _parse_ping_rtt(out.decode(errors="replace"))
|
||||
|
||||
|
||||
def _local_address(family: str, target: str) -> Optional[str]:
|
||||
"""Source address the kernel would use to reach target, or None if there is no route.
|
||||
|
||||
A UDP connect() sends nothing. Raises _ProbeUnavailable if the family is unsupported.
|
||||
"""
|
||||
af = socket.AF_INET if family == "ipv4" else socket.AF_INET6
|
||||
try:
|
||||
sock = socket.socket(af, socket.SOCK_DGRAM)
|
||||
except OSError as e:
|
||||
raise _ProbeUnavailable(str(e))
|
||||
try:
|
||||
sock.connect((target, 9))
|
||||
return str(sock.getsockname()[0])
|
||||
except OSError:
|
||||
return None
|
||||
finally:
|
||||
sock.close()
|
||||
|
||||
|
||||
@dataclass
|
||||
class _FamilyState:
|
||||
present: Optional[bool] = None # None = unknown (probe cannot run yet/at all)
|
||||
addr: Optional[str] = None
|
||||
gateway: Optional[str] = None
|
||||
rtt: Optional[float] = None
|
||||
|
||||
|
||||
class _Connectivity:
|
||||
def __init__(
|
||||
self,
|
||||
probes: Optional[Dict[str, str]] = None,
|
||||
probe_interval: float = 60.0,
|
||||
retry_interval: float = 10.0,
|
||||
tier1_interval: float = 5.0,
|
||||
fail_limit: int = 2,
|
||||
local_address_fn: Callable[[str, str], Optional[str]] = _local_address,
|
||||
ping_fn: Callable[[str, str], Awaitable[Optional[float]]] = _ping_rtt,
|
||||
gateway_fn: Callable[[str], Awaitable[Optional[str]]] = _default_gateway,
|
||||
clock: Callable[[], float] = time.monotonic,
|
||||
):
|
||||
self.probes = dict(_DEFAULT_PROBES)
|
||||
self.probes.update(probes or {})
|
||||
self.probe_interval = probe_interval
|
||||
self.retry_interval = retry_interval
|
||||
self.tier1_interval = tier1_interval
|
||||
self.fail_limit = fail_limit
|
||||
self._local_address = local_address_fn
|
||||
self._ping = ping_fn
|
||||
self._gateway_fn = gateway_fn
|
||||
self._clock = clock
|
||||
# Reading the gateway is a cheap /proc read on Linux but a subprocess elsewhere, so
|
||||
# elsewhere it is refreshed only when a tier-2 probe is due.
|
||||
self._cheap_gateway = sys.platform.startswith("linux")
|
||||
self.states: Dict[str, _FamilyState] = {f: _FamilyState() for f in _FAMILIES}
|
||||
self._fails: Dict[str, int] = {f: 0 for f in _FAMILIES}
|
||||
self._next_ping: Dict[str, float] = {f: 0.0 for f in _FAMILIES}
|
||||
self._last_key: Dict[str, Optional[Tuple[Optional[str], Optional[str]]]] = {
|
||||
f: None for f in _FAMILIES}
|
||||
self._ping_disabled: Set[str] = set()
|
||||
|
||||
@classmethod
|
||||
def from_config(cls, cfg: Optional[Dict[str, Any]]) -> "_Connectivity":
|
||||
cfg = cfg or {}
|
||||
probes = {}
|
||||
if cfg.get("probe_v4"):
|
||||
probes["ipv4"] = str(cfg["probe_v4"])
|
||||
if cfg.get("probe_v6"):
|
||||
probes["ipv6"] = str(cfg["probe_v6"])
|
||||
return cls(probes=probes, probe_interval=float(cfg.get("probe_interval", 60)))
|
||||
|
||||
# ---- readers (cheap; safe to call on every heartbeat) ---------------------------
|
||||
def snapshot(self) -> Dict[str, _FamilyState]:
|
||||
return {f: _FamilyState(**asdict(s)) for f, s in self.states.items()}
|
||||
|
||||
def flags(self) -> Dict[str, bool]:
|
||||
"""HTB flags for families whose presence is known."""
|
||||
return {_FLAG_KEYS[f]: s.present for f, s in self.states.items() if s.present is not None}
|
||||
|
||||
def as_metric(self) -> Dict[str, Dict[str, Any]]:
|
||||
return {f: asdict(s) for f, s in self.states.items()}
|
||||
|
||||
# ---- probing ----------------------------------------------------------------------
|
||||
async def refresh(self) -> None:
|
||||
for fam in _FAMILIES:
|
||||
try:
|
||||
await self._refresh_family(fam)
|
||||
except Exception:
|
||||
logging.getLogger("hbc.probe").exception("connectivity probe failed for %s", fam)
|
||||
|
||||
async def run(self) -> None:
|
||||
while _running:
|
||||
await self.refresh()
|
||||
await _sleep(self.tier1_interval)
|
||||
|
||||
async def _gateway(self, fam: str) -> Optional[str]:
|
||||
try:
|
||||
return await self._gateway_fn(fam)
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
async def _refresh_family(self, fam: str) -> None:
|
||||
st = self.states[fam]
|
||||
try:
|
||||
addr = self._local_address(fam, self.probes[fam])
|
||||
except _ProbeUnavailable:
|
||||
st.present = None
|
||||
return
|
||||
|
||||
if addr is None: # tier 1: no route => absent immediately
|
||||
self._fails[fam] = 0
|
||||
self._next_ping[fam] = 0.0
|
||||
self._last_key[fam] = None
|
||||
st.present, st.addr, st.gateway, st.rtt = False, None, None, None
|
||||
return
|
||||
|
||||
now = self._clock()
|
||||
gw = await self._gateway(fam) if self._cheap_gateway else st.gateway
|
||||
key = (addr, gw)
|
||||
due = key != self._last_key[fam] or now >= self._next_ping[fam]
|
||||
if due and not self._cheap_gateway:
|
||||
gw = await self._gateway(fam)
|
||||
key = (addr, gw)
|
||||
st.addr, st.gateway = addr, gw
|
||||
self._last_key[fam] = key
|
||||
if fam in self._ping_disabled:
|
||||
st.present = None # cannot probe: unknown, never absent, while a route exists
|
||||
elif due:
|
||||
await self._tier2(fam, st, now)
|
||||
|
||||
async def _tier2(self, fam: str, st: _FamilyState, now: float) -> None:
|
||||
try:
|
||||
rtt = await self._ping(self.probes[fam], fam)
|
||||
except _PingUnavailable:
|
||||
logging.getLogger("hbc.probe").warning("ping unavailable; %s presence stays unknown", fam)
|
||||
self._ping_disabled.add(fam)
|
||||
st.present = None
|
||||
return
|
||||
except (asyncio.TimeoutError, OSError):
|
||||
rtt = None
|
||||
if rtt is None:
|
||||
self._fails[fam] += 1
|
||||
self._next_ping[fam] = now + self.retry_interval
|
||||
if self._fails[fam] >= self.fail_limit:
|
||||
st.present, st.rtt = False, None
|
||||
else:
|
||||
self._fails[fam] = 0
|
||||
self._next_ping[fam] = now + self.probe_interval
|
||||
st.present, st.rtt = True, rtt
|
||||
|
||||
|
||||
# The running probe (None when probe_enabled is false); shared by the heartbeat sender and
|
||||
# the network_monitor plugin. Set in _async_main.
|
||||
_probe: Optional[_Connectivity] = None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Plugin: network_monitor (Linux /proc/net/dev)
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -658,7 +976,7 @@ def _read_net_dev() -> Dict[str, Tuple[int, int]]:
|
||||
|
||||
class NetworkMonitorPlugin(MonitorPlugin):
|
||||
name = "network_monitor"
|
||||
description = "Network I/O rates via /proc/net/dev (Linux)"
|
||||
description = "Network I/O rates via /proc/net/dev (Linux) and per-protocol internet connectivity"
|
||||
interval = 300
|
||||
|
||||
def __init__(self, config: Optional[Dict[str, Any]] = None):
|
||||
@@ -667,19 +985,25 @@ class NetworkMonitorPlugin(MonitorPlugin):
|
||||
self.interval = cfg.get("interval", 300)
|
||||
self.skip_ifaces: set = set(cfg.get("skip_interfaces", ["lo"]))
|
||||
self._prev: Optional[Tuple[float, Dict[str, Tuple[int, int]]]] = None
|
||||
self._have_dev = False
|
||||
|
||||
async def initialize(self) -> bool:
|
||||
if platform.system() != "Linux":
|
||||
self.skip_reason = "Linux only (/proc/net/dev not available)"
|
||||
return False
|
||||
dev = _read_net_dev()
|
||||
if not dev:
|
||||
self.skip_reason = "/proc/net/dev not readable"
|
||||
dev = _read_net_dev() if platform.system() == "Linux" else {}
|
||||
self._have_dev = bool(dev)
|
||||
if not self._have_dev and _probe is None:
|
||||
self.skip_reason = "no /proc/net/dev (Linux only) and the connectivity probe is disabled"
|
||||
return False
|
||||
if self._have_dev:
|
||||
self._prev = (time.time(), dev) # prime first delta
|
||||
return True
|
||||
|
||||
async def _collect_metrics(self) -> Dict[str, Any]:
|
||||
data: Dict[str, Any] = await self._collect_interfaces() if self._have_dev else {}
|
||||
if _probe is not None:
|
||||
data["connectivity"] = _probe.as_metric()
|
||||
return data
|
||||
|
||||
async def _collect_interfaces(self) -> Dict[str, Any]:
|
||||
now = time.time()
|
||||
curr = _read_net_dev()
|
||||
if not curr or self._prev is None:
|
||||
@@ -842,6 +1166,14 @@ class AsyncConnection:
|
||||
self._transport.close()
|
||||
self._transport = None
|
||||
|
||||
def set_target(self, addr: str, af: int):
|
||||
"""Re-point at a new server address; the socket is reopened on the next send if the family changed."""
|
||||
if af != self.af:
|
||||
self.close()
|
||||
self.af = af
|
||||
self.addr = addr
|
||||
self._log = logging.getLogger(f"hbc.conn.{addr}")
|
||||
|
||||
def _handle_ack(self, msg: Dict[str, Any], now: float):
|
||||
rtt = (now - self.lastsend) * 1000.0
|
||||
self.rtts.append(rtt)
|
||||
@@ -866,6 +1198,207 @@ class AsyncConnection:
|
||||
self.lastsend = time.time()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Re-resolution of the server hosts at the TTL of their A/AAAA records
|
||||
# getaddrinfo() hides the TTL, so a minimal stub-resolver query is made only to learn it; the
|
||||
# addresses still come from getaddrinfo(). TTL unknown (IP literal, /etc/hosts name, no
|
||||
# resolv.conf, timeout) -> _DNS_DEFAULT_TTL.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
log = logging.getLogger("hbc.dns")
|
||||
|
||||
_DNS_DEFAULT_TTL = 300 # used when the real TTL is unknown
|
||||
_DNS_MIN_TTL = 30 # a TTL of 0 must not turn into a resolution loop
|
||||
_DNS_FAILED_RETRY = 60 # re-try delay after a failed resolution
|
||||
_DNS_QUERY_TIMEOUT = 2.0
|
||||
|
||||
_DNS_TYPE_A, _DNS_TYPE_CNAME, _DNS_TYPE_AAAA = 1, 5, 28
|
||||
|
||||
|
||||
def _dns_parse_resolv_conf(path: str = "/etc/resolv.conf") -> Tuple[List[str], List[str]]:
|
||||
"""Return (nameservers, search domains)."""
|
||||
servers: List[str] = []
|
||||
search: List[str] = []
|
||||
try:
|
||||
with open(path) as f:
|
||||
for line in f:
|
||||
parts = line.split()
|
||||
if len(parts) < 2:
|
||||
continue
|
||||
if parts[0] == "nameserver":
|
||||
servers.append(parts[1].split("%")[0])
|
||||
elif parts[0] in ("search", "domain"):
|
||||
search = parts[1:]
|
||||
except OSError:
|
||||
pass
|
||||
return servers, search
|
||||
|
||||
|
||||
def _dns_build_query(name: str, qtype: int, qid: int) -> bytes:
|
||||
out = struct.pack("!HHHHHH", qid, 0x0100, 1, 0, 0, 0)
|
||||
for label in name.rstrip(".").split("."):
|
||||
out += bytes([len(label)]) + label.encode("idna")
|
||||
return out + b"\x00" + struct.pack("!HH", qtype, 1)
|
||||
|
||||
|
||||
def _dns_skip_name(buf: bytes, off: int) -> int:
|
||||
while True:
|
||||
n = buf[off]
|
||||
if n == 0:
|
||||
return off + 1
|
||||
if n & 0xC0:
|
||||
return off + 2
|
||||
off += n + 1
|
||||
|
||||
|
||||
def _dns_min_answer_ttl(resp: bytes, qid: int, qtype: int) -> Optional[int]:
|
||||
"""Smallest TTL among the answer RRs (CNAMEs included), or None if there is no qtype answer."""
|
||||
try:
|
||||
rid, flags, qd, an = struct.unpack("!HHHH", resp[:8])
|
||||
if rid != qid or flags & 0x000F:
|
||||
return None
|
||||
off = 12
|
||||
for _ in range(qd):
|
||||
off = _dns_skip_name(resp, off) + 4
|
||||
ttls: List[int] = []
|
||||
found = False
|
||||
for _ in range(an):
|
||||
off = _dns_skip_name(resp, off)
|
||||
rtype, _cls, ttl, rdlen = struct.unpack("!HHIH", resp[off:off + 10])
|
||||
off += 10 + rdlen
|
||||
if rtype in (qtype, _DNS_TYPE_CNAME):
|
||||
ttls.append(ttl)
|
||||
found = found or rtype == qtype
|
||||
return min(ttls) if found else None
|
||||
except (struct.error, IndexError):
|
||||
return None
|
||||
|
||||
|
||||
def _dns_query_ttl(server: str, name: str, qtype: int) -> Optional[int]:
|
||||
qid = struct.unpack("!H", os.urandom(2))[0]
|
||||
try:
|
||||
af = socket.AF_INET6 if ":" in server else socket.AF_INET
|
||||
with socket.socket(af, socket.SOCK_DGRAM) as s:
|
||||
s.settimeout(_DNS_QUERY_TIMEOUT)
|
||||
s.sendto(_dns_build_query(name, qtype, qid), (server, 53))
|
||||
return _dns_min_answer_ttl(s.recv(4096), qid, qtype)
|
||||
except (OSError, UnicodeError):
|
||||
return None
|
||||
|
||||
|
||||
def _dns_lookup_ttl(host: str, af_filter: int = 0) -> Optional[int]:
|
||||
"""Blocking. Smallest A/AAAA TTL for host, or None when it cannot be determined."""
|
||||
try:
|
||||
ipaddress.ip_address(host)
|
||||
return None
|
||||
except ValueError:
|
||||
pass
|
||||
servers, search = _dns_parse_resolv_conf()
|
||||
if host.endswith("."):
|
||||
names = [host]
|
||||
elif "." in host:
|
||||
names = [host] + [f"{host}.{d}" for d in search]
|
||||
else:
|
||||
names = [f"{host}.{d}" for d in search] + [host]
|
||||
qtypes = [t for t, af in ((_DNS_TYPE_A, socket.AF_INET), (_DNS_TYPE_AAAA, socket.AF_INET6))
|
||||
if af_filter in (0, af)]
|
||||
for name in names:
|
||||
ttls = []
|
||||
for qtype in qtypes:
|
||||
for server in servers:
|
||||
ttl = _dns_query_ttl(server, name, qtype)
|
||||
if ttl is not None:
|
||||
ttls.append(ttl)
|
||||
break
|
||||
if ttls:
|
||||
return min(ttls)
|
||||
return None
|
||||
|
||||
|
||||
def _dns_refresh_delay(ttls: List[Optional[int]]) -> int:
|
||||
"""Seconds until the next resolution: the smallest known TTL, floored at _DNS_MIN_TTL."""
|
||||
known = [t for t in ttls if t is not None]
|
||||
return max(min(known), _DNS_MIN_TTL) if known else _DNS_DEFAULT_TTL
|
||||
|
||||
|
||||
def _dns_resolve(hosts: List[str], port: int, af_filter: int) -> Optional[List[Tuple[int, str]]]:
|
||||
"""Blocking. Every (family, address) the hosts resolve to, deduplicated, or None if any lookup fails."""
|
||||
found: List[Tuple[int, str]] = []
|
||||
for host in hosts:
|
||||
try:
|
||||
infos = socket.getaddrinfo(host, port, af_filter, 0, socket.SOL_UDP)
|
||||
except socket.gaierror as e:
|
||||
log.warning("cannot resolve %s: %s", host, e)
|
||||
return None
|
||||
for info in infos:
|
||||
entry = (info[0], str(info[4][0]))
|
||||
if entry not in found:
|
||||
found.append(entry)
|
||||
return found or None
|
||||
|
||||
|
||||
def _dns_retarget(connections: list, resolved: List[Tuple[int, str]]) -> Tuple[List[Tuple[int, str]], list]:
|
||||
"""Point the connections at the resolved addresses, in place.
|
||||
|
||||
A connection whose address is gone is re-pointed at an address that is new, in list order, so
|
||||
the first connection (the one plugin data uses) survives as long as anything resolves.
|
||||
Returns (new addresses still needing a connection, connections to drop).
|
||||
"""
|
||||
current = {c.addr for c in connections}
|
||||
wanted = {addr for _, addr in resolved}
|
||||
new = [(af, addr) for af, addr in resolved if addr not in current]
|
||||
gone = [c for c in connections if c.addr not in wanted]
|
||||
paired = min(len(gone), len(new))
|
||||
for conn, (af, addr) in zip(gone, new):
|
||||
log.info("server address %s -> %s", conn.addr, addr)
|
||||
conn.set_target(addr, af)
|
||||
drop = gone[paired:]
|
||||
if connections and connections[0] in drop:
|
||||
# plugin data goes out on the first connection: hand it a surviving address and drop that one instead
|
||||
keep = next(c for c in connections if c not in gone)
|
||||
connections[0].set_target(keep.addr, keep.af)
|
||||
drop.remove(connections[0])
|
||||
drop.append(keep)
|
||||
return new[paired:], drop
|
||||
|
||||
|
||||
async def _dns_refresher(connections: List["AsyncConnection"], hosts: List[str], port: int, af_filter: int,
|
||||
name: str, interval: int, probe: Optional[_Connectivity], next_id: int):
|
||||
"""Re-resolve the server hosts every TTL seconds and follow address changes.
|
||||
|
||||
A failed lookup leaves the connections untouched. Addresses that disappear are re-pointed at
|
||||
new ones where possible; the rest are retired, and surplus new ones get a connection.
|
||||
"""
|
||||
loop = asyncio.get_running_loop()
|
||||
|
||||
def ttl_delay() -> int:
|
||||
return _dns_refresh_delay([_dns_lookup_ttl(h, af_filter) for h in hosts])
|
||||
|
||||
delay = await loop.run_in_executor(None, ttl_delay)
|
||||
while _running:
|
||||
await _sleep(delay)
|
||||
if not _running:
|
||||
break
|
||||
resolved = await loop.run_in_executor(None, _dns_resolve, hosts, port, af_filter)
|
||||
if resolved is None:
|
||||
delay = _DNS_FAILED_RETRY
|
||||
continue
|
||||
new, gone = _dns_retarget(connections, resolved)
|
||||
for conn in gone:
|
||||
if len(connections) > 1:
|
||||
log.info("dropping server address %s", conn.addr)
|
||||
conn._dead = True
|
||||
conn.close()
|
||||
connections.remove(conn)
|
||||
for af, addr in new:
|
||||
conn = AsyncConnection(next_id, addr, port, af, name)
|
||||
next_id += 1
|
||||
log.info("adding server address %s", addr)
|
||||
connections.append(conn)
|
||||
_active_tasks.append(asyncio.create_task(_heartbeat_sender(conn, interval, probe)))
|
||||
delay = await loop.run_in_executor(None, ttl_delay)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Server command handlers
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -940,15 +1473,21 @@ async def _handle_update(conn: AsyncConnection):
|
||||
# Heartbeat sender
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
async def _heartbeat_sender(conn: AsyncConnection, interval: int):
|
||||
def _build_heartbeat_msg(conn: AsyncConnection, interval: int,
|
||||
probe: Optional[_Connectivity] = None) -> Dict[str, Any]:
|
||||
"""HTB payload; carries p4/p6 presence flags when the probe knows them."""
|
||||
msg: Dict[str, Any] = {"acks": conn.ackcount, "rtt": conn.rtts[-1], "interval": interval}
|
||||
if probe is not None:
|
||||
msg.update(probe.flags())
|
||||
return msg
|
||||
|
||||
|
||||
async def _heartbeat_sender(conn: AsyncConnection, interval: int,
|
||||
probe: Optional[_Connectivity] = None):
|
||||
log = logging.getLogger("hbc.hb")
|
||||
while _running:
|
||||
try:
|
||||
await conn.sendto({
|
||||
"acks": conn.ackcount,
|
||||
"rtt": conn.rtts[-1],
|
||||
"interval": interval,
|
||||
})
|
||||
await conn.sendto(_build_heartbeat_msg(conn, interval, probe))
|
||||
except Exception as e:
|
||||
log.error("send: %s", e)
|
||||
await _sleep(interval)
|
||||
@@ -1056,7 +1595,7 @@ def _reconfigure_syslog(level: int):
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
async def _async_main(args, cfg: Dict[str, Any]) -> int:
|
||||
global _running, _shutdown_event, _active_tasks, send_shutdown
|
||||
global _running, _shutdown_event, _active_tasks, send_shutdown, _probe
|
||||
_running = True
|
||||
_shutdown_event = asyncio.Event()
|
||||
_active_tasks = []
|
||||
@@ -1113,6 +1652,10 @@ async def _async_main(args, cfg: Dict[str, Any]) -> int:
|
||||
c.close()
|
||||
return 0
|
||||
|
||||
# Per-protocol internet presence probe (shared with the network_monitor plugin)
|
||||
net_cfg = dict((cfg.get("plugins") or {}).get("network_monitor") or cfg.get("network_monitor") or {})
|
||||
_probe = _Connectivity.from_config(net_cfg) if net_cfg.get("probe_enabled", True) else None
|
||||
|
||||
plugins = await _load_plugins(cfg)
|
||||
|
||||
loop = asyncio.get_event_loop()
|
||||
@@ -1126,8 +1669,13 @@ async def _async_main(args, cfg: Dict[str, Any]) -> int:
|
||||
|
||||
loop.add_signal_handler(signal.SIGHUP, _sighup)
|
||||
|
||||
if _probe is not None:
|
||||
_active_tasks.append(asyncio.create_task(_probe.run()))
|
||||
for conn in connections:
|
||||
_active_tasks.append(asyncio.create_task(_heartbeat_sender(conn, interval)))
|
||||
_active_tasks.append(asyncio.create_task(_heartbeat_sender(conn, interval, _probe)))
|
||||
|
||||
_active_tasks.append(asyncio.create_task(
|
||||
_dns_refresher(connections, args.hosts, port, af_filter, iam, interval, _probe, conn_id)))
|
||||
|
||||
if plugins and connections:
|
||||
_active_tasks.append(asyncio.create_task(_plugin_collector(connections[0], plugins)))
|
||||
|
||||
@@ -0,0 +1,119 @@
|
||||
"""TTL-driven re-resolution of the server hostnames in the full client."""
|
||||
import socket
|
||||
import struct
|
||||
|
||||
from hbd.client import dnsttl
|
||||
|
||||
|
||||
def _rr(rtype, ttl, rdata):
|
||||
return b"\xc0\x0c" + struct.pack("!HHIH", rtype, 1, ttl, len(rdata)) + rdata
|
||||
|
||||
|
||||
def _response(qid, qtype, answers, rcode=0):
|
||||
q = dnsttl.build_query("hbd.example.com", qtype, qid)[12:]
|
||||
return struct.pack("!HHHHHH", qid, 0x8180 | rcode, 1, len(answers), 0, 0) + q + b"".join(answers)
|
||||
|
||||
|
||||
def test_min_ttl_covers_cname_and_address():
|
||||
resp = _response(7, dnsttl.TYPE_A, [_rr(dnsttl.TYPE_CNAME, 60, b"\x01x\x00"),
|
||||
_rr(dnsttl.TYPE_A, 300, b"\x01\x02\x03\x04")])
|
||||
assert dnsttl.min_answer_ttl(resp, 7, dnsttl.TYPE_A) == 60
|
||||
|
||||
|
||||
def test_no_answer_of_requested_type_is_unknown():
|
||||
assert dnsttl.min_answer_ttl(_response(7, dnsttl.TYPE_AAAA, []), 7, dnsttl.TYPE_AAAA) is None
|
||||
|
||||
|
||||
def test_error_rcode_and_foreign_id_are_unknown():
|
||||
ans = [_rr(dnsttl.TYPE_A, 300, b"\x01\x02\x03\x04")]
|
||||
assert dnsttl.min_answer_ttl(_response(7, 1, ans, rcode=3), 7, 1) is None
|
||||
assert dnsttl.min_answer_ttl(_response(7, 1, ans), 8, 1) is None
|
||||
|
||||
|
||||
def test_truncated_response_is_unknown():
|
||||
assert dnsttl.min_answer_ttl(b"\x00\x07\x81", 7, 1) is None
|
||||
|
||||
|
||||
def test_refresh_delay():
|
||||
assert dnsttl.refresh_delay([300, 120, None]) == 120
|
||||
assert dnsttl.refresh_delay([0]) == dnsttl.MIN_TTL
|
||||
assert dnsttl.refresh_delay([None]) == dnsttl.DEFAULT_TTL
|
||||
|
||||
|
||||
def test_ip_literal_has_no_ttl():
|
||||
assert dnsttl.lookup_ttl("192.0.2.1") is None
|
||||
assert dnsttl.lookup_ttl("2001:db8::1") is None
|
||||
|
||||
|
||||
class _C:
|
||||
def __init__(self, addr, af=socket.AF_INET):
|
||||
self.addr, self.af = addr, af
|
||||
|
||||
def set_target(self, addr, af):
|
||||
self.addr, self.af = addr, af
|
||||
|
||||
|
||||
V4, V6 = socket.AF_INET, socket.AF_INET6
|
||||
|
||||
|
||||
def test_retarget_follows_address_change_in_place():
|
||||
first, second = _C("1.1.1.1"), _C("2001:db8::1", V6)
|
||||
new, gone = dnsttl.retarget([first, second], [(V4, "9.9.9.9"), (V6, "2001:db8::9")])
|
||||
assert (new, gone) == ([], [])
|
||||
assert (first.addr, second.addr) == ("9.9.9.9", "2001:db8::9")
|
||||
|
||||
|
||||
def test_retarget_unchanged_is_a_noop():
|
||||
conns = [_C("1.1.1.1")]
|
||||
assert dnsttl.retarget(conns, [(V4, "1.1.1.1")]) == ([], [])
|
||||
|
||||
|
||||
def test_retarget_reports_surplus_new_and_gone():
|
||||
a, b = _C("1.1.1.1"), _C("2.2.2.2")
|
||||
new, gone = dnsttl.retarget([a, b], [(V4, "3.3.3.3")])
|
||||
assert new == [] and gone == [b] and a.addr == "3.3.3.3"
|
||||
new, gone = dnsttl.retarget([a], [(V4, "3.3.3.3"), (V4, "4.4.4.4")])
|
||||
assert new == [(V4, "4.4.4.4")] and gone == []
|
||||
|
||||
|
||||
def test_retarget_never_drops_the_first_connection():
|
||||
a, b = _C("1.1.1.1"), _C("2.2.2.2")
|
||||
new, gone = dnsttl.retarget([a, b], [(V4, "2.2.2.2")])
|
||||
assert new == [] and gone == [b] and a.addr == "2.2.2.2"
|
||||
|
||||
|
||||
def test_resolve_failure_returns_none(monkeypatch):
|
||||
def boom(*a):
|
||||
raise socket.gaierror("nope")
|
||||
monkeypatch.setattr(socket, "getaddrinfo", boom)
|
||||
assert dnsttl.resolve(["x.invalid"], 1, 0) is None
|
||||
|
||||
|
||||
def test_refresher_follows_a_change_and_adds_a_family(monkeypatch):
|
||||
import asyncio
|
||||
from hbd.client import main as hbc
|
||||
|
||||
answers = [[(V4, "9.9.9.9"), (V6, "2001:db8::9")], None, [(V4, "9.9.9.9")]]
|
||||
monkeypatch.setattr(dnsttl, "resolve", lambda *a: answers.pop(0) if len(answers) > 1 else answers[0])
|
||||
monkeypatch.setattr(dnsttl, "lookup_ttl", lambda *a: None)
|
||||
monkeypatch.setattr(dnsttl, "DEFAULT_TTL", 0.01)
|
||||
monkeypatch.setattr(dnsttl, "FAILED_RETRY", 0.01)
|
||||
|
||||
async def idle(*a):
|
||||
await asyncio.sleep(10)
|
||||
monkeypatch.setattr(hbc, "heartbeat_sender", idle)
|
||||
|
||||
async def go():
|
||||
hbc.shutdown_event = asyncio.Event()
|
||||
hbc.active_tasks = []
|
||||
conns = [hbc.AsyncConnection(1, "1.1.1.1", 5, V4, "me")]
|
||||
task = asyncio.create_task(hbc.dns_refresher(conns, ["h"], 5, 0, "me", 10, None, 2))
|
||||
await asyncio.sleep(0.2)
|
||||
hbc.shutdown_event.set()
|
||||
await task
|
||||
for t in hbc.active_tasks:
|
||||
t.cancel()
|
||||
return conns
|
||||
|
||||
conns = asyncio.run(go())
|
||||
assert [(c.conn_id, c.addr) for c in conns] == [(1, "9.9.9.9")]
|
||||
@@ -1,5 +1,3 @@
|
||||
import glob
|
||||
import os
|
||||
import pytest
|
||||
from hbd.server import configio
|
||||
|
||||
|
||||
+37
-82
@@ -1,9 +1,8 @@
|
||||
import time
|
||||
import queue
|
||||
import asyncio
|
||||
import unittest
|
||||
from unittest.mock import patch, MagicMock
|
||||
|
||||
import hbd.dns as dns
|
||||
import hbd.server.dns as dns
|
||||
|
||||
|
||||
class TestDNS(unittest.TestCase):
|
||||
@@ -17,7 +16,7 @@ class TestDNS(unittest.TestCase):
|
||||
p = dns.create_nsupdate_payload("host", "2001:db8::1", "example")
|
||||
self.assertIn("update add host.dy.example 5 AAAA 2001:db8::1", p)
|
||||
|
||||
@patch("hbd.dns.Popen")
|
||||
@patch("hbd.server.dns.Popen")
|
||||
def test_nsupdate_success(self, mock_popen):
|
||||
proc = MagicMock()
|
||||
proc.communicate.return_value = (b"status: NOERROR", None)
|
||||
@@ -41,7 +40,7 @@ class TestDNS(unittest.TestCase):
|
||||
stderr=dns.STDOUT,
|
||||
)
|
||||
|
||||
@patch("hbd.dns.Popen")
|
||||
@patch("hbd.server.dns.Popen")
|
||||
def test_nsupdate_failure(self, mock_popen):
|
||||
proc = MagicMock()
|
||||
proc.communicate.return_value = (b"some error", None)
|
||||
@@ -57,83 +56,7 @@ class TestDNS(unittest.TestCase):
|
||||
self.assertIsNotNone(err)
|
||||
self.assertIn("some error", err)
|
||||
|
||||
def test_dnsupdatethread_processes_queue(self):
|
||||
# patch nsupdate to succeed
|
||||
with patch("hbd.dns.nsupdate", return_value=None):
|
||||
logs = []
|
||||
|
||||
def log(h, m):
|
||||
logs.append((h, m))
|
||||
|
||||
emails = []
|
||||
|
||||
def email(s, m):
|
||||
emails.append((s, m))
|
||||
|
||||
class FakeHost:
|
||||
dnsQ = queue.Queue()
|
||||
|
||||
class FakeHbd:
|
||||
Host = FakeHost
|
||||
|
||||
# start the thread (daemon) that processes the queue
|
||||
t = dns.start_dns_thread(
|
||||
FakeHbd, {"dyndomains": ["example"]}, log=log, email=email
|
||||
)
|
||||
self.assertTrue(t.is_alive())
|
||||
|
||||
# enqueue one item and wait for it to be processed (polling with timeout)
|
||||
FakeHbd.Host.dnsQ.put(("testhost", "1.2.3.4"))
|
||||
|
||||
for _ in range(30):
|
||||
if logs:
|
||||
break
|
||||
time.sleep(0.1)
|
||||
|
||||
self.assertTrue(logs, "dnsupdatethread did not call log")
|
||||
self.assertTrue(
|
||||
any("changed address" in m or "DNS updated" in m for (_h, m) in logs)
|
||||
)
|
||||
|
||||
def test_dnsupdatethread_calls_email_on_failure(self):
|
||||
# patch nsupdate to fail with an error message
|
||||
with patch("hbd.dns.nsupdate", return_value="error: failed"):
|
||||
logs = []
|
||||
|
||||
def log(h, m):
|
||||
logs.append((h, m))
|
||||
|
||||
emails = []
|
||||
|
||||
def email(s, m):
|
||||
emails.append((s, m))
|
||||
|
||||
class FakeHost:
|
||||
dnsQ = queue.Queue()
|
||||
|
||||
class FakeHbd:
|
||||
Host = FakeHost
|
||||
|
||||
dns.start_dns_thread(
|
||||
FakeHbd, {"dyndomains": ["example"]}, log=log, email=email
|
||||
)
|
||||
# enqueue and wait for the email to be sent
|
||||
FakeHbd.Host.dnsQ.put(("testhost", "1.2.3.4"))
|
||||
|
||||
for _ in range(30):
|
||||
if emails:
|
||||
break
|
||||
time.sleep(0.1)
|
||||
|
||||
self.assertTrue(emails, "dnsupdatethread did not call email on failure")
|
||||
self.assertTrue(
|
||||
any(
|
||||
"nsupdate failed" in s or "nsupdate failed" in m or "error" in m
|
||||
for (s, m) in emails
|
||||
)
|
||||
)
|
||||
|
||||
@patch("hbd.dns.Popen")
|
||||
@patch("hbd.server.dns.Popen")
|
||||
def test_nsupdate_raises_oserror(self, mock_popen):
|
||||
mock_popen.side_effect = OSError("noexec")
|
||||
err = dns.nsupdate(
|
||||
@@ -147,5 +70,37 @@ class TestDNS(unittest.TestCase):
|
||||
self.assertIn("execution failed", err)
|
||||
|
||||
|
||||
class TestDNSWorker(unittest.IsolatedAsyncioTestCase):
|
||||
|
||||
async def _run_worker(self, nsupdate_result):
|
||||
"""Feed one update through dns_update_worker; returns the (name, level, message) log calls."""
|
||||
logs = []
|
||||
queue = asyncio.Queue()
|
||||
await queue.put(("testhost", "1.2.3.4"))
|
||||
await queue.put(None)
|
||||
with patch("hbd.server.dns.nsupdate", return_value=nsupdate_result):
|
||||
await dns.dns_update_worker(
|
||||
None, {"dyndomains": ["example"]}, async_queue=queue,
|
||||
log=lambda name, level, msg: logs.append((name, level, msg)),
|
||||
)
|
||||
return logs
|
||||
|
||||
async def test_worker_logs_successful_update(self):
|
||||
logs = await self._run_worker(None)
|
||||
self.assertEqual(len(logs), 1)
|
||||
name, level, msg = logs[0]
|
||||
self.assertEqual((name, level), ("testhost", "INFO"))
|
||||
self.assertIn("DNS updated testhost.dy.example", msg)
|
||||
self.assertIn("1.2.3.4", msg)
|
||||
|
||||
async def test_worker_logs_error_on_failed_update(self):
|
||||
logs = await self._run_worker("error: failed")
|
||||
self.assertEqual(len(logs), 1)
|
||||
name, level, msg = logs[0]
|
||||
self.assertEqual((name, level), ("testhost", "ERROR"))
|
||||
self.assertIn("DNS update failed", msg)
|
||||
self.assertIn("error: failed", msg)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
from hbd.udp import handle_datagram, parse_message
|
||||
from hbd.proto import dicttos
|
||||
from hbd.server.udp import handle_datagram, parse_message
|
||||
from hbd.common.proto import dicttos, stodict
|
||||
|
||||
|
||||
class FakeTransport:
|
||||
@@ -17,7 +17,7 @@ def dummy_noop(*a, **k):
|
||||
def test_handle_cmd_sends_command():
|
||||
ftr = FakeTransport()
|
||||
# prepare ctx linking to hbdclass
|
||||
import hbdclass
|
||||
from hbd.server import hbdclass
|
||||
|
||||
ctx = {
|
||||
"config": {"watchhosts": []},
|
||||
@@ -34,7 +34,7 @@ def test_handle_cmd_sends_command():
|
||||
# create host by sending initial heartbeat
|
||||
msg = parse_message(dicttos("HTB", {"name": "cmdhost", "interval": 10}))
|
||||
handle_datagram(msg, ("127.0.0.1", 50000), ftr, ctx)
|
||||
assert ftr.sent[0][0] == b"ACK"
|
||||
assert ftr.sent[0][0].startswith(b"!ACK:")
|
||||
|
||||
# queue a CMD for the host and send another heartbeat; expect command sent
|
||||
h = hbdclass.Host.hosts["cmdhost"]
|
||||
@@ -42,8 +42,6 @@ def test_handle_cmd_sends_command():
|
||||
ftr.sent.clear()
|
||||
msg2 = parse_message(dicttos("HTB", {"name": "cmdhost", "interval": 10}))
|
||||
handle_datagram(msg2, ("127.0.0.1", 50000), ftr, ctx)
|
||||
# should have sent ACK and the command; last send should be non-empty
|
||||
assert len(ftr.sent) >= 1
|
||||
# the command for ??0 will be sent as raw cmd string
|
||||
# so at least one send contains b'doit' or similar
|
||||
assert any(b"doit" in s[0] for s in ftr.sent)
|
||||
# the ACK and the queued command are both sent; the command is a compressed CMD packet
|
||||
decoded = [stodict(data) for data, _addr in ftr.sent]
|
||||
assert any(d["ID"] == "CMD" and d["cmd"] == "doit" for d in decoded)
|
||||
|
||||
@@ -0,0 +1,72 @@
|
||||
"""TTL-driven re-resolution in the standalone scripts/hbc_mini.py."""
|
||||
import asyncio
|
||||
import importlib.util
|
||||
import pathlib
|
||||
import socket
|
||||
import struct
|
||||
import sys
|
||||
|
||||
_PATH = pathlib.Path(__file__).resolve().parents[1] / "scripts" / "hbc_mini.py"
|
||||
_SPEC = importlib.util.spec_from_file_location("hbc_mini", _PATH)
|
||||
mini = importlib.util.module_from_spec(_SPEC)
|
||||
sys.modules["hbc_mini"] = mini
|
||||
_SPEC.loader.exec_module(mini)
|
||||
|
||||
V4, V6 = socket.AF_INET, socket.AF_INET6
|
||||
|
||||
|
||||
def _rr(rtype, ttl, rdata):
|
||||
return b"\xc0\x0c" + struct.pack("!HHIH", rtype, 1, ttl, len(rdata)) + rdata
|
||||
|
||||
|
||||
def _response(qid, qtype, answers, rcode=0):
|
||||
q = mini._dns_build_query("hbd.example.com", qtype, qid)[12:]
|
||||
return struct.pack("!HHHHHH", qid, 0x8180 | rcode, 1, len(answers), 0, 0) + q + b"".join(answers)
|
||||
|
||||
|
||||
def test_min_ttl_covers_cname_and_address():
|
||||
resp = _response(7, 1, [_rr(5, 60, b"\x01x\x00"), _rr(1, 300, b"\x01\x02\x03\x04")])
|
||||
assert mini._dns_min_answer_ttl(resp, 7, 1) == 60
|
||||
|
||||
|
||||
def test_unknown_ttl_cases():
|
||||
assert mini._dns_min_answer_ttl(_response(7, 28, []), 7, 28) is None
|
||||
assert mini._dns_min_answer_ttl(_response(7, 1, [_rr(1, 9, b"1234")], rcode=3), 7, 1) is None
|
||||
assert mini._dns_min_answer_ttl(b"\x00\x07\x81", 7, 1) is None
|
||||
assert mini._dns_lookup_ttl("192.0.2.1") is None
|
||||
|
||||
|
||||
def test_refresh_delay():
|
||||
assert mini._dns_refresh_delay([300, 120, None]) == 120
|
||||
assert mini._dns_refresh_delay([0]) == mini._DNS_MIN_TTL
|
||||
assert mini._dns_refresh_delay([None]) == mini._DNS_DEFAULT_TTL
|
||||
|
||||
|
||||
def test_refresher_follows_a_change_and_adds_a_family(monkeypatch):
|
||||
answers = [[(V4, "9.9.9.9"), (V6, "2001:db8::9")], None, [(V4, "9.9.9.9")]]
|
||||
monkeypatch.setattr(mini, "_dns_resolve", lambda *a: answers.pop(0) if len(answers) > 1 else answers[0])
|
||||
monkeypatch.setattr(mini, "_dns_lookup_ttl", lambda *a: None)
|
||||
monkeypatch.setattr(mini, "_DNS_DEFAULT_TTL", 0.01)
|
||||
monkeypatch.setattr(mini, "_DNS_FAILED_RETRY", 0.01)
|
||||
|
||||
async def idle(*a):
|
||||
await asyncio.sleep(10)
|
||||
monkeypatch.setattr(mini, "_heartbeat_sender", idle)
|
||||
|
||||
async def go():
|
||||
mini._running = True
|
||||
mini._shutdown_event = asyncio.Event()
|
||||
mini._active_tasks = []
|
||||
conns = [mini.AsyncConnection(1, "1.1.1.1", 5, V4, "me")]
|
||||
task = asyncio.create_task(mini._dns_refresher(conns, ["h"], 5, 0, "me", 10, None, 2))
|
||||
await asyncio.sleep(0.2)
|
||||
mini._running = False
|
||||
mini._shutdown_event.set()
|
||||
await task
|
||||
for t in mini._active_tasks:
|
||||
t.cancel()
|
||||
return conns
|
||||
|
||||
conns = asyncio.run(go())
|
||||
assert [(c.conn_id, c.addr) for c in conns] == [(1, "9.9.9.9")]
|
||||
assert conns[0].af == V4
|
||||
@@ -0,0 +1,381 @@
|
||||
"""Connectivity probe in the standalone scripts/hbc_mini.py (stdlib-only single-file client)."""
|
||||
import asyncio
|
||||
import importlib.util
|
||||
import pathlib
|
||||
import sys
|
||||
|
||||
import pytest
|
||||
|
||||
_PATH = pathlib.Path(__file__).resolve().parents[1] / "scripts" / "hbc_mini.py"
|
||||
_SPEC = importlib.util.spec_from_file_location("hbc_mini", _PATH)
|
||||
mini = importlib.util.module_from_spec(_SPEC)
|
||||
sys.modules["hbc_mini"] = mini # dataclasses resolve their module by name
|
||||
_SPEC.loader.exec_module(mini)
|
||||
|
||||
PROC_V4 = (
|
||||
"Iface\tDestination\tGateway \tFlags\tRefCnt\tUse\tMetric\tMask\t\tMTU\tWindow\tIRTT\n"
|
||||
"wlan0\t00000000\t0100A8C0\t0003\t0\t0\t600\t00000000\t0\t0\t0\n"
|
||||
"eth0\t00000000\t0101A8C0\t0003\t0\t0\t100\t00000000\t0\t0\t0\n"
|
||||
"wlan0\t0000A8C0\t00000000\t0001\t0\t0\t600\t00FFFFFF\t0\t0\t0\n"
|
||||
)
|
||||
PROC_V6 = (
|
||||
"00000000000000000000000000000000 00 00000000000000000000000000000000 00 "
|
||||
"fe800000000000000000000000000001 00000400 00000001 00000000 00000003 wlan0\n"
|
||||
"00000000000000000000000000000000 00 00000000000000000000000000000000 00 "
|
||||
"00000000000000000000000000000000 ffffffff 00000001 00000000 00200200 lo\n"
|
||||
"fe800000000000000000000000000000 40 00000000000000000000000000000000 00 "
|
||||
"00000000000000000000000000000000 00000100 00000001 00000000 00000001 wlan0\n"
|
||||
)
|
||||
ROUTE_GET = """ route to: 0.0.0.0
|
||||
destination: 0.0.0.0
|
||||
mask: 0.0.0.0
|
||||
gateway: 192.0.2.1
|
||||
fib: 0
|
||||
interface: vtnet0
|
||||
"""
|
||||
WIN_V4 = """IPv4 Route Table
|
||||
Active Routes:
|
||||
Network Destination Netmask Gateway Interface Metric
|
||||
0.0.0.0 0.0.0.0 192.168.1.1 192.168.1.50 25
|
||||
0.0.0.0 0.0.0.0 10.0.0.1 10.0.0.9 50
|
||||
127.0.0.0 255.0.0.0 On-link 127.0.0.1 331
|
||||
"""
|
||||
WIN_V6 = """IPv6 Route Table
|
||||
Active Routes:
|
||||
If Metric Network Destination Gateway
|
||||
12 281 ::/0 fe80::1
|
||||
1 331 ::1/128 On-link
|
||||
"""
|
||||
LINUX_PING = """PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
|
||||
64 bytes from 1.1.1.1: icmp_seq=1 ttl=57 time=11.2 ms
|
||||
|
||||
--- 1.1.1.1 ping statistics ---
|
||||
1 packets transmitted, 1 received, 0% packet loss, time 0ms
|
||||
rtt min/avg/max/mdev = 11.234/11.234/11.234/0.000 ms
|
||||
"""
|
||||
FREEBSD_PING = """PING 1.1.1.1 (1.1.1.1): 56 data bytes
|
||||
64 bytes from 1.1.1.1: icmp_seq=0 ttl=59 time=4.006 ms
|
||||
|
||||
--- 1.1.1.1 ping statistics ---
|
||||
1 packets transmitted, 1 packets received, 0.0% packet loss, 1 packets out of wait time
|
||||
round-trip min/avg/max/stddev = 4.006/4.006/4.006/0.000 ms
|
||||
"""
|
||||
WIN_PING = """Pinging 1.1.1.1 with 32 bytes of data:
|
||||
Reply from 1.1.1.1: bytes=32 time=12ms TTL=57
|
||||
|
||||
Ping statistics for 1.1.1.1:
|
||||
Packets: Sent = 1, Received = 1, Lost = 0 (0% loss),
|
||||
"""
|
||||
LOST_PING = """--- 1.1.1.1 ping statistics ---
|
||||
1 packets transmitted, 0 received, 100% packet loss, time 0ms
|
||||
"""
|
||||
|
||||
|
||||
# ---- gateway parsers ---------------------------------------------------------------
|
||||
|
||||
def test_proc_v4_lowest_metric_wins():
|
||||
assert mini._parse_proc_route_v4(PROC_V4) == "192.168.1.1" # eth0, metric 100
|
||||
|
||||
|
||||
def test_proc_v4_none_without_default_route():
|
||||
assert mini._parse_proc_route_v4(PROC_V4.splitlines()[0] + "\n") is None
|
||||
|
||||
|
||||
def test_proc_v6_skips_reject_and_non_default():
|
||||
assert mini._parse_proc_route_v6(PROC_V6) == "fe80::1"
|
||||
|
||||
|
||||
def test_route_get_bsd_and_macos():
|
||||
assert mini._parse_route_get(ROUTE_GET) == "192.0.2.1"
|
||||
assert mini._parse_route_get("route: writing to routing socket: not in table") is None
|
||||
|
||||
|
||||
def test_windows_routes():
|
||||
assert mini._parse_windows_route(WIN_V4, "ipv4") == "192.168.1.1"
|
||||
assert mini._parse_windows_route(WIN_V6, "ipv6") == "fe80::1"
|
||||
|
||||
|
||||
def test_default_gateway_never_raises(monkeypatch):
|
||||
monkeypatch.setattr(sys, "platform", "linux")
|
||||
monkeypatch.setattr(mini, "_parse_proc_route_v4", lambda t: 1 / 0)
|
||||
assert asyncio.run(mini._default_gateway("ipv4")) is None
|
||||
|
||||
|
||||
# ---- ping ---------------------------------------------------------------------------
|
||||
|
||||
@pytest.mark.parametrize("platform,family,expected", [
|
||||
("linux", "ipv4", ["ping", "-c", "1", "-W", "2", "-4", "h"]),
|
||||
("linux", "ipv6", ["ping", "-c", "1", "-W", "2", "-6", "h"]),
|
||||
("freebsd15", "ipv4", ["ping", "-c", "1", "-W", "2000", "h"]),
|
||||
("freebsd15", "ipv6", ["ping6", "-c", "1", "h"]),
|
||||
("darwin", "ipv4", ["ping", "-c", "1", "-W", "2000", "h"]),
|
||||
("darwin", "ipv6", ["ping6", "-c", "1", "h"]),
|
||||
("win32", "ipv4", ["ping", "-n", "1", "-w", "2000", "-4", "h"]),
|
||||
("win32", "ipv6", ["ping", "-n", "1", "-w", "2000", "-6", "h"]),
|
||||
])
|
||||
def test_probe_ping_command(monkeypatch, platform, family, expected):
|
||||
monkeypatch.setattr(sys, "platform", platform)
|
||||
assert mini._probe_ping_command("h", family) == expected
|
||||
|
||||
|
||||
def test_parse_ping_rtt():
|
||||
assert mini._parse_ping_rtt(LINUX_PING) == 11.234
|
||||
assert mini._parse_ping_rtt(FREEBSD_PING) == 4.006
|
||||
assert mini._parse_ping_rtt(WIN_PING) == 12.0
|
||||
assert mini._parse_ping_rtt(LOST_PING) is None
|
||||
assert mini._parse_ping_rtt("ping: connect: Network is unreachable") is None
|
||||
|
||||
|
||||
# ---- probe state machine ------------------------------------------------------------
|
||||
|
||||
class Net:
|
||||
"""Scriptable network: addr None = no route, 'unavailable' = family unsupported."""
|
||||
|
||||
def __init__(self):
|
||||
self.addr = {"ipv4": "192.0.2.5", "ipv6": "2001:db8::5"}
|
||||
self.gw = {"ipv4": "192.0.2.1", "ipv6": "fe80::1"}
|
||||
self.rtt = {"ipv4": 10.0, "ipv6": 20.0} # None = ping fails, 'unavailable' = no ping binary
|
||||
self.pings = []
|
||||
self.now = 1000.0
|
||||
|
||||
def local(self, fam, target):
|
||||
a = self.addr[fam]
|
||||
if a == "unavailable":
|
||||
raise mini._ProbeUnavailable("no such family")
|
||||
return a
|
||||
|
||||
async def ping(self, target, fam):
|
||||
self.pings.append(fam)
|
||||
r = self.rtt[fam]
|
||||
if r == "unavailable":
|
||||
raise mini._PingUnavailable("no ping")
|
||||
return r
|
||||
|
||||
async def gateway(self, fam):
|
||||
return self.gw[fam]
|
||||
|
||||
def clock(self):
|
||||
return self.now
|
||||
|
||||
|
||||
def make(net, **kw):
|
||||
c = mini._Connectivity(local_address_fn=net.local, ping_fn=net.ping, gateway_fn=net.gateway,
|
||||
clock=net.clock, **kw)
|
||||
c._cheap_gateway = True
|
||||
return c
|
||||
|
||||
|
||||
def refresh(c):
|
||||
asyncio.run(c.refresh())
|
||||
|
||||
|
||||
def test_present_with_addr_gateway_rtt():
|
||||
net = Net()
|
||||
c = make(net)
|
||||
refresh(c)
|
||||
s = c.snapshot()["ipv4"]
|
||||
assert (s.present, s.addr, s.gateway, s.rtt) == (True, "192.0.2.5", "192.0.2.1", 10.0)
|
||||
assert c.flags() == {"p4": True, "p6": True}
|
||||
assert c.as_metric()["ipv6"]["gateway"] == "fe80::1"
|
||||
|
||||
|
||||
def test_no_route_is_immediately_absent():
|
||||
net = Net()
|
||||
net.addr["ipv6"] = None
|
||||
c = make(net)
|
||||
refresh(c)
|
||||
assert c.flags() == {"p4": True, "p6": False}
|
||||
assert net.pings == ["ipv4"]
|
||||
assert c.snapshot()["ipv6"].addr is None
|
||||
|
||||
|
||||
def test_ping_is_debounced_and_rate_limited():
|
||||
net = Net()
|
||||
c = make(net, probe_interval=60.0, retry_interval=10.0)
|
||||
refresh(c)
|
||||
refresh(c)
|
||||
assert net.pings == ["ipv4", "ipv6"]
|
||||
net.now += 60
|
||||
net.rtt["ipv6"] = None
|
||||
refresh(c)
|
||||
assert c.flags()["p6"] is True
|
||||
net.now += 10
|
||||
refresh(c)
|
||||
assert c.flags()["p6"] is False
|
||||
|
||||
|
||||
def test_first_probe_failing_once_is_unknown():
|
||||
net = Net()
|
||||
net.rtt["ipv4"] = None
|
||||
c = make(net)
|
||||
refresh(c)
|
||||
assert "p4" not in c.flags() and c.flags()["p6"] is True
|
||||
|
||||
|
||||
def test_recovers_on_first_success():
|
||||
net = Net()
|
||||
net.rtt["ipv6"] = None
|
||||
c = make(net, retry_interval=10.0)
|
||||
refresh(c)
|
||||
net.now += 10
|
||||
refresh(c)
|
||||
assert c.flags()["p6"] is False
|
||||
net.rtt["ipv6"] = 25.0
|
||||
net.now += 10
|
||||
refresh(c)
|
||||
assert c.flags()["p6"] is True
|
||||
|
||||
|
||||
def test_address_change_invalidates_cache():
|
||||
net = Net()
|
||||
c = make(net)
|
||||
refresh(c)
|
||||
net.addr["ipv4"] = "198.51.100.7"
|
||||
refresh(c)
|
||||
assert net.pings.count("ipv4") == 2
|
||||
assert c.snapshot()["ipv4"].addr == "198.51.100.7"
|
||||
|
||||
|
||||
def test_gateway_change_invalidates_cache():
|
||||
net = Net()
|
||||
c = make(net)
|
||||
refresh(c)
|
||||
net.gw["ipv4"] = "192.0.2.254"
|
||||
refresh(c)
|
||||
assert net.pings.count("ipv4") == 2
|
||||
|
||||
|
||||
def test_route_loss_then_return_pings_immediately():
|
||||
net = Net()
|
||||
c = make(net)
|
||||
refresh(c)
|
||||
net.addr["ipv6"] = None
|
||||
refresh(c)
|
||||
net.addr["ipv6"] = "2001:db8::5"
|
||||
refresh(c)
|
||||
assert net.pings.count("ipv6") == 2 and c.flags()["p6"] is True
|
||||
|
||||
|
||||
def test_missing_ping_binary_leaves_presence_unknown_and_stops_retrying():
|
||||
net = Net()
|
||||
net.rtt["ipv4"] = "unavailable"
|
||||
c = make(net)
|
||||
refresh(c)
|
||||
net.now += 600
|
||||
refresh(c)
|
||||
assert "p4" not in c.flags()
|
||||
assert net.pings.count("ipv4") == 1
|
||||
|
||||
|
||||
def test_ping_less_family_stays_unknown_after_route_returns():
|
||||
net = Net()
|
||||
net.rtt["ipv4"] = "unavailable"
|
||||
c = make(net)
|
||||
refresh(c)
|
||||
assert "p4" not in c.flags()
|
||||
net.addr["ipv4"] = None
|
||||
refresh(c)
|
||||
assert c.flags()["p4"] is False # no route is definitive
|
||||
net.addr["ipv4"] = "192.0.2.5"
|
||||
refresh(c)
|
||||
assert "p4" not in c.flags() and net.pings.count("ipv4") == 1
|
||||
|
||||
|
||||
def test_unsupported_family_is_unknown():
|
||||
net = Net()
|
||||
net.addr["ipv6"] = "unavailable"
|
||||
c = make(net)
|
||||
refresh(c)
|
||||
assert "p6" not in c.flags() and c.flags()["p4"] is True
|
||||
|
||||
|
||||
def test_from_config():
|
||||
c = mini._Connectivity.from_config({"probe_v4": "9.9.9.9", "probe_interval": 30})
|
||||
assert c.probes["ipv4"] == "9.9.9.9"
|
||||
assert c.probes["ipv6"] == "2606:4700:4700::1111"
|
||||
assert c.probe_interval == 30.0
|
||||
|
||||
|
||||
def test_local_address_on_this_host_never_raises():
|
||||
# Real UDP connect(): either an address string, None (no route) or _ProbeUnavailable.
|
||||
for fam, target in (("ipv4", "1.1.1.1"), ("ipv6", "2606:4700:4700::1111")):
|
||||
try:
|
||||
r = mini._local_address(fam, target)
|
||||
except mini._ProbeUnavailable:
|
||||
continue
|
||||
assert r is None or isinstance(r, str)
|
||||
|
||||
|
||||
# ---- heartbeat message and plugin wiring ---------------------------------------------
|
||||
|
||||
class _Conn:
|
||||
ackcount = 3
|
||||
rtts = [0.0, 12.5]
|
||||
|
||||
|
||||
class _FlagsProbe:
|
||||
def __init__(self, flags):
|
||||
self._flags = flags
|
||||
|
||||
def flags(self):
|
||||
return self._flags
|
||||
|
||||
|
||||
def test_heartbeat_without_probe_is_unchanged():
|
||||
assert mini._build_heartbeat_msg(_Conn(), 60) == {"acks": 3, "rtt": 12.5, "interval": 60}
|
||||
|
||||
|
||||
def test_heartbeat_carries_presence_flags():
|
||||
msg = mini._build_heartbeat_msg(_Conn(), 60, _FlagsProbe({"p4": True, "p6": False}))
|
||||
assert msg == {"acks": 3, "rtt": 12.5, "interval": 60, "p4": True, "p6": False}
|
||||
|
||||
|
||||
def test_heartbeat_omits_unknown_flags():
|
||||
msg = mini._build_heartbeat_msg(_Conn(), 60, _FlagsProbe({}))
|
||||
assert "p4" not in msg and "p6" not in msg
|
||||
|
||||
|
||||
def test_flags_survive_the_wire_encoding():
|
||||
msg = mini._build_heartbeat_msg(_Conn(), 60, _FlagsProbe({"p4": True, "p6": False}))
|
||||
out = mini._stodict(mini._dicttos("HTB", dict(msg, name="h", id=1)))
|
||||
assert out["p4"] == 1 and out["p6"] == 0
|
||||
|
||||
|
||||
class _FakeProbe:
|
||||
def as_metric(self):
|
||||
return {"ipv4": {"present": True, "addr": "192.0.2.5", "gateway": "192.0.2.1", "rtt": 9.0},
|
||||
"ipv6": {"present": False, "addr": None, "gateway": None, "rtt": None}}
|
||||
|
||||
|
||||
def test_plugin_reports_connectivity_alongside_interfaces(monkeypatch):
|
||||
monkeypatch.setattr(mini, "_probe", _FakeProbe())
|
||||
monkeypatch.setattr(mini.platform, "system", lambda: "Linux")
|
||||
plugin = mini.NetworkMonitorPlugin({})
|
||||
assert asyncio.run(plugin.initialize()) is True
|
||||
data = asyncio.run(plugin._collect_metrics())
|
||||
assert data["connectivity"]["ipv4"]["present"] is True
|
||||
assert data["connectivity"]["ipv6"]["present"] is False
|
||||
|
||||
|
||||
def test_plugin_probe_only_mode_on_non_linux(monkeypatch):
|
||||
monkeypatch.setattr(mini, "_probe", _FakeProbe())
|
||||
monkeypatch.setattr(mini.platform, "system", lambda: "FreeBSD")
|
||||
plugin = mini.NetworkMonitorPlugin({})
|
||||
assert asyncio.run(plugin.initialize()) is True
|
||||
assert list(asyncio.run(plugin._collect_metrics())) == ["connectivity"]
|
||||
|
||||
|
||||
def test_plugin_still_skips_on_non_linux_without_probe(monkeypatch):
|
||||
monkeypatch.setattr(mini, "_probe", None)
|
||||
monkeypatch.setattr(mini.platform, "system", lambda: "FreeBSD")
|
||||
plugin = mini.NetworkMonitorPlugin({})
|
||||
assert asyncio.run(plugin.initialize()) is False
|
||||
assert plugin.skip_reason
|
||||
|
||||
|
||||
def test_plugin_omits_connectivity_without_probe(monkeypatch):
|
||||
monkeypatch.setattr(mini, "_probe", None)
|
||||
monkeypatch.setattr(mini.platform, "system", lambda: "Linux")
|
||||
plugin = mini.NetworkMonitorPlugin({})
|
||||
assert asyncio.run(plugin.initialize()) is True
|
||||
assert "connectivity" not in asyncio.run(plugin._collect_metrics())
|
||||
@@ -1,5 +1,4 @@
|
||||
"""Tests for the config read/write API helpers in http.py."""
|
||||
import pytest
|
||||
from hbd.server import http
|
||||
|
||||
|
||||
|
||||
@@ -1,5 +1,4 @@
|
||||
"""Tests for PUT /api/0/users/me logic."""
|
||||
import pytest
|
||||
from hbd.server import users as users_mod
|
||||
|
||||
|
||||
@@ -87,7 +86,6 @@ def test_put_users_me_notification_channels(tmp_path):
|
||||
|
||||
def test_visible_channels_excludes_private_from_others():
|
||||
"""Private channels owned by another user must not appear in the visible set."""
|
||||
from hbd.server import settings as settings_mod
|
||||
|
||||
config = {
|
||||
"notification_channels": {
|
||||
|
||||
@@ -1,15 +1,11 @@
|
||||
import asyncio
|
||||
import logging
|
||||
import os
|
||||
import stat
|
||||
|
||||
from hbd.client.plugins.nagios_runner import (
|
||||
NagiosRunnerPlugin,
|
||||
NAGIOS_OK,
|
||||
NAGIOS_WARNING,
|
||||
NAGIOS_CRITICAL,
|
||||
NAGIOS_UNKNOWN,
|
||||
)
|
||||
from hbd.client.plugins.nagios_runner import NagiosRunnerPlugin, NAGIOS_UNKNOWN
|
||||
|
||||
# Nagios plugin exit codes (the module itself only needs UNKNOWN)
|
||||
NAGIOS_OK, NAGIOS_WARNING, NAGIOS_CRITICAL = 0, 1, 2
|
||||
|
||||
|
||||
def test_no_commands_sets_skip_reason():
|
||||
|
||||
@@ -1,5 +1,4 @@
|
||||
"""Tests for notification channel CRUD via configio helpers and visibility logic."""
|
||||
import pytest
|
||||
from hbd.server import configio, settings as settings_mod
|
||||
|
||||
|
||||
|
||||
@@ -37,7 +37,6 @@ def reset_users_dict():
|
||||
users_mod.users = original
|
||||
|
||||
|
||||
|
||||
def test_make_state_returns_unique_tokens():
|
||||
s1 = oauth.make_state()
|
||||
s2 = oauth.make_state()
|
||||
@@ -124,7 +123,6 @@ def test_provision_oauth_user_survives_config_reload():
|
||||
assert "oauthonly" in users_mod.users
|
||||
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Integration-style tests: callback logic chain
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
+8
-8
@@ -1,19 +1,19 @@
|
||||
from hbd.proto import dicttos, stodict, oldmtodict
|
||||
from hbd.common.proto import dicttos, stodict, oldmtodict
|
||||
|
||||
|
||||
def test_dicttos_and_stodict_uncompressed():
|
||||
def test_dicttos_and_stodict_roundtrip():
|
||||
msg = dicttos("HTB", {"name": "host.example", "interval": 10})
|
||||
assert msg.startswith(b"!HTB:")
|
||||
d = stodict(msg)
|
||||
assert d["ID"].startswith("HTB")
|
||||
assert d["ID"] == "HTB"
|
||||
assert d["name"] == "host.example"
|
||||
assert d["interval"] == 10
|
||||
|
||||
|
||||
def test_dicttos_and_stodict_compressed():
|
||||
msg = dicttos("ACK", {"time": 12345}, compress=True)
|
||||
d = stodict(msg)
|
||||
# for compressed the original code included the colon in ID slice; ensure no crash
|
||||
assert "ID" in d
|
||||
def test_dicttos_and_stodict_ack():
|
||||
d = stodict(dicttos("ACK", {"time": 12345}))
|
||||
assert d["ID"] == "ACK"
|
||||
assert d["time"] == 12345
|
||||
|
||||
|
||||
def test_oldmtodict():
|
||||
|
||||
@@ -1,4 +1,3 @@
|
||||
import pytest
|
||||
from hbd.server import settings as settings_mod
|
||||
|
||||
CFG = {
|
||||
|
||||
+19
-15
@@ -13,13 +13,12 @@ Tests cover:
|
||||
"""
|
||||
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
# Add parent directory to path for imports
|
||||
sys.path.insert(0, str(Path(__file__).parent.parent))
|
||||
|
||||
from hbd.threshold import (
|
||||
from hbd.server.threshold import ( # noqa: E402
|
||||
ThresholdChecker,
|
||||
ThresholdConfig,
|
||||
AlertLevel,
|
||||
@@ -170,23 +169,23 @@ def test_alert_state():
|
||||
|
||||
# Update to WARNING - should trigger notification
|
||||
changed = alert.update(AlertLevel.WARNING, 85.0)
|
||||
assert changed == True, "State change should return True"
|
||||
assert changed is True, "State change should return True"
|
||||
assert alert.level == AlertLevel.WARNING
|
||||
assert alert.last_value == 85.0
|
||||
|
||||
# Update with same level - no notification
|
||||
changed = alert.update(AlertLevel.WARNING, 86.0)
|
||||
assert changed == False, "No state change should return False"
|
||||
assert changed is False, "No state change should return False"
|
||||
assert alert.last_value == 86.0
|
||||
|
||||
# Escalate to CRITICAL
|
||||
changed = alert.update(AlertLevel.CRITICAL, 95.0)
|
||||
assert changed == True, "Escalation should trigger notification"
|
||||
assert changed is True, "Escalation should trigger notification"
|
||||
assert alert.level == AlertLevel.CRITICAL
|
||||
|
||||
# Recover to OK
|
||||
changed = alert.update(AlertLevel.OK, 50.0)
|
||||
assert changed == True, "Recovery should trigger notification"
|
||||
assert changed is True, "Recovery should trigger notification"
|
||||
assert alert.level == AlertLevel.OK
|
||||
|
||||
print(" ✓ Alert state tracking works correctly")
|
||||
@@ -257,7 +256,16 @@ def test_threshold_checker_parsing():
|
||||
print(f" ✓ Parsed {len(checker.thresholds)} thresholds correctly")
|
||||
|
||||
|
||||
def test_check_plugin_data():
|
||||
def _checker_with_notifications(config, notifications, monkeypatch):
|
||||
"""ThresholdChecker with no grace period that records notifications instead of sending them."""
|
||||
monkeypatch.setattr(
|
||||
ThresholdChecker, "_send_notification",
|
||||
lambda self, host, lvl, message, *a: notifications.append(f"[{lvl}] {host} {message}"),
|
||||
)
|
||||
return ThresholdChecker({"grace": 0, **config})
|
||||
|
||||
|
||||
def test_check_plugin_data(monkeypatch):
|
||||
"""Test checking plugin data against thresholds."""
|
||||
print("\nTest 6: Plugin data checking...")
|
||||
|
||||
@@ -277,11 +285,7 @@ def test_check_plugin_data():
|
||||
}
|
||||
|
||||
notifications = []
|
||||
|
||||
def notification_callback(msg):
|
||||
notifications.append(msg)
|
||||
|
||||
checker = ThresholdChecker(config, notification_callback=notification_callback)
|
||||
checker = _checker_with_notifications(config, notifications, monkeypatch)
|
||||
alert_states = {}
|
||||
|
||||
# First check - OK
|
||||
@@ -353,12 +357,12 @@ def test_check_plugin_data():
|
||||
|
||||
assert len(state_changes) == 2, "Two metrics should recover"
|
||||
assert len(notifications) == 2, "Two recovery notifications"
|
||||
assert any("RECOVERED" in n for n in notifications), "Should have recovery notification"
|
||||
assert any("RECOVER" in n for n in notifications), "Should have recovery notification"
|
||||
|
||||
print(" ✓ Plugin data checking and notifications work")
|
||||
|
||||
|
||||
def test_nested_metrics():
|
||||
def test_nested_metrics(monkeypatch):
|
||||
"""Test checking nested metrics like disk partitions."""
|
||||
print("\nTest 7: Nested metrics (partitions)...")
|
||||
|
||||
@@ -384,7 +388,7 @@ def test_nested_metrics():
|
||||
}
|
||||
|
||||
notifications = []
|
||||
checker = ThresholdChecker(config, notification_callback=lambda m: notifications.append(m))
|
||||
checker = _checker_with_notifications(config, notifications, monkeypatch)
|
||||
alert_states = {}
|
||||
|
||||
plugin_data = {
|
||||
|
||||
+2
-2
@@ -1,5 +1,5 @@
|
||||
from hbd.udp import parse_message
|
||||
from hbd.proto import dicttos
|
||||
from hbd.server.udp import parse_message
|
||||
from hbd.common.proto import dicttos
|
||||
|
||||
|
||||
def test_parse_message_uncompressed():
|
||||
|
||||
@@ -0,0 +1,40 @@
|
||||
"""Datagrams that are not heartbeat traffic (internet scanners) must not create hosts."""
|
||||
from hbd.common.proto import dicttos
|
||||
from hbd.server import hbdclass
|
||||
from hbd.server.udp import handle_datagram, parse_message
|
||||
|
||||
|
||||
class _FakeTransport:
|
||||
sent = 0
|
||||
|
||||
def sendto(self, data, addr):
|
||||
_FakeTransport.sent += 1
|
||||
|
||||
|
||||
def _ctx():
|
||||
return {"config": {}, "hbdclass": hbdclass, "msg_to_websockets": None,
|
||||
"DEBUG": 0, "verbose": False}
|
||||
|
||||
|
||||
def _handle(raw):
|
||||
handle_datagram(parse_message(raw), ("192.0.2.9", 50000), _FakeTransport(), _ctx())
|
||||
|
||||
|
||||
def test_sip_invite_does_not_create_a_host():
|
||||
hbdclass.Host.hosts.pop("unknown", None)
|
||||
_handle(b"INVITE sip:10000@192.0.2.1:50003 SIP/2.0\r\nTo: <sip:10000@192.0.2.1>\r\n\r\n")
|
||||
assert "unknown" not in hbdclass.Host.hosts
|
||||
|
||||
|
||||
def test_nameless_heartbeat_does_not_create_a_host_or_get_acked():
|
||||
hbdclass.Host.hosts.pop("unknown", None)
|
||||
_FakeTransport.sent = 0
|
||||
_handle(dicttos("HTB", {"id": 0}))
|
||||
assert "unknown" not in hbdclass.Host.hosts
|
||||
assert _FakeTransport.sent == 0
|
||||
|
||||
|
||||
def test_named_heartbeat_still_creates_its_host():
|
||||
hbdclass.Host.hosts.pop("stray-ok", None)
|
||||
_handle(dicttos("HTB", {"name": "stray-ok", "id": 0}))
|
||||
assert "stray-ok" in hbdclass.Host.hosts
|
||||
Reference in New Issue
Block a user