Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
e8d9c4a958 | ||
|
|
328f248c43 | ||
|
|
747a8c27c9 | ||
|
|
5243f0a834 | ||
|
|
3c6f7fc868 | ||
|
|
adbd2f9b46 | ||
|
|
248b17edca | ||
|
|
3639464ba6 | ||
|
|
997279dce8 |
@@ -2,6 +2,19 @@
|
|||||||
|
|
||||||
All notable changes to this project are documented here, organized by release.
|
All notable changes to this project are documented here, organized by release.
|
||||||
|
|
||||||
|
## [5.4.5]
|
||||||
|
|
||||||
|
### Added
|
||||||
|
- `hbc_mini.py` and `hbc_mini.c` (the dependency-free clients) get the same per-protocol connectivity probe as `hbc`: `p4`/`p6` flags on every heartbeat and a `connectivity` metric (presence, own address, gateway, RTT) via `network_monitor`, with gateway lookup on Linux, FreeBSD/NetBSD/DragonFly (both clients) and macOS/Windows (`hbc_mini.py` only)
|
||||||
|
- `scripts/c/test_probe.c` and a `make test` target — the C client's first automated test suite
|
||||||
|
|
||||||
|
### Fixed
|
||||||
|
- `hbc_mini.c`: the clean-shutdown heartbeat could be delayed or lost behind an in-flight connectivity probe on SIGTERM; it is now sent before the probe thread is joined
|
||||||
|
- `hbc_mini.c`: a connectivity probe that fails to start, or is configured with a non-IP-literal target, now logs a warning instead of failing silently
|
||||||
|
- `hbc_mini.py`: removed an unused import
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
## [5.4.4]
|
## [5.4.4]
|
||||||
|
|
||||||
### Added
|
### Added
|
||||||
|
|||||||
@@ -20,7 +20,7 @@ A lightweight UDP-based host monitoring system. Monitored hosts run a client (`h
|
|||||||
└────────────────────┘ └────────────────────────────┘
|
└────────────────────┘ └────────────────────────────┘
|
||||||
```
|
```
|
||||||
|
|
||||||
**Package:** `hbd` v5.4.4
|
**Package:** `hbd` v5.4.5
|
||||||
**Python:** 3.11+
|
**Python:** 3.11+
|
||||||
|
|
||||||
### Subpackages
|
### Subpackages
|
||||||
@@ -659,7 +659,28 @@ Config: `~/.hbc.json` (JSON format, same keys as `~/.hbc.yaml`).
|
|||||||
| `cpu_monitor` | Linux (`/proc/stat`; no per-core, no frequency) |
|
| `cpu_monitor` | Linux (`/proc/stat`; no per-core, no frequency) |
|
||||||
| `memory_monitor` | Linux (`/proc/meminfo`) |
|
| `memory_monitor` | Linux (`/proc/meminfo`) |
|
||||||
| `disk_monitor` | Linux, macOS, BSD (`df -P`) |
|
| `disk_monitor` | Linux, macOS, BSD (`df -P`) |
|
||||||
| `network_monitor` | Linux (`/proc/net/dev`) |
|
| `network_monitor` | Linux (`/proc/net/dev`); per-protocol connectivity probe (all platforms) |
|
||||||
|
|
||||||
|
`network_monitor` also reports per-protocol (IPv4/IPv6) internet presence — whether the host
|
||||||
|
has a working route and reply to a well-known address, plus its own address, the local
|
||||||
|
gateway, and RTT — as `p4`/`p6` flags on every heartbeat and a `connectivity` object in the
|
||||||
|
plugin data. Configurable, defaults shown:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"plugins": {
|
||||||
|
"network_monitor": {
|
||||||
|
"probe_enabled": true,
|
||||||
|
"probe_v4": "1.1.1.1",
|
||||||
|
"probe_v6": "2606:4700:4700::1111",
|
||||||
|
"probe_interval": 60
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
`probe_v4`/`probe_v6` must be IP literals (no hostnames). Gateway lookup works on Linux,
|
||||||
|
FreeBSD/NetBSD/DragonFly/macOS (`route -n get`), and Windows (`route print`).
|
||||||
|
|
||||||
Not available vs full `hbc`: no YAML config, no `filesystem_info`, no `zfs_monitor`, no IPv6 early-fail protection.
|
Not available vs full `hbc`: no YAML config, no `filesystem_info`, no `zfs_monitor`, no IPv6 early-fail protection.
|
||||||
|
|
||||||
@@ -712,6 +733,7 @@ SIGHUP causes the process to restart itself via `execv()`. SIGTERM/SIGINT trigge
|
|||||||
| `disk_monitor` | All | `df -P` subprocess |
|
| `disk_monitor` | All | `df -P` subprocess |
|
||||||
| `network_monitor` | Linux | `/proc/net/dev` |
|
| `network_monitor` | Linux | `/proc/net/dev` |
|
||||||
| `network_monitor` | FreeBSD, NetBSD, DragonFly | `getifaddrs()` + `AF_LINK` |
|
| `network_monitor` | FreeBSD, NetBSD, DragonFly | `getifaddrs()` + `AF_LINK` |
|
||||||
|
| `network_monitor` (connectivity probe) | Linux, FreeBSD, NetBSD, DragonFly | UDP `connect()` route check + one ICMP echo per protocol |
|
||||||
| `ping_monitor` | All | `ping` subprocess |
|
| `ping_monitor` | All | `ping` subprocess |
|
||||||
| `nagios_runner` | All | `popen()` subprocess |
|
| `nagios_runner` | All | `popen()` subprocess |
|
||||||
|
|
||||||
@@ -743,12 +765,33 @@ SIGHUP causes the process to restart itself via `execv()`. SIGTERM/SIGINT trigge
|
|||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
|
`network_monitor` also runs the per-protocol connectivity probe described above, in a
|
||||||
|
dedicated thread (its ICMP echo can block for a few seconds). Same config keys and defaults
|
||||||
|
as `hbc_mini.py`:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"plugins": {
|
||||||
|
"network_monitor": {
|
||||||
|
"probe_enabled": true,
|
||||||
|
"probe_v4": "1.1.1.1",
|
||||||
|
"probe_v6": "2606:4700:4700::1111",
|
||||||
|
"probe_interval": 60
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Gateway lookup on the C client covers Linux and FreeBSD/NetBSD/DragonFly only (no macOS or
|
||||||
|
Windows build of `hbc_mini.c` exists).
|
||||||
|
|
||||||
### Differences from `hbc_mini.py`
|
### Differences from `hbc_mini.py`
|
||||||
|
|
||||||
- No `filesystem_info` or `zfs_monitor` plugins
|
- No `filesystem_info` or `zfs_monitor` plugins
|
||||||
- `UPD` (self-update) messages are logged but not acted on
|
- `UPD` (self-update) messages are logged but not acted on
|
||||||
- No IPv6 early-fail protection
|
- No IPv6 early-fail protection
|
||||||
- Config is JSON only (`~/.hbc.json`), no YAML
|
- Config is JSON only (`~/.hbc.json`), no YAML
|
||||||
|
- No macOS or Windows build (Linux and the BSDs only)
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|||||||
+1
-1
@@ -14,4 +14,4 @@ Install options:
|
|||||||
"""
|
"""
|
||||||
|
|
||||||
__all__ = ["__version__"]
|
__all__ = ["__version__"]
|
||||||
__version__ = "5.4.4"
|
__version__ = "5.4.5"
|
||||||
|
|||||||
+1
-1
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
|
|||||||
|
|
||||||
[project]
|
[project]
|
||||||
name = "hbd"
|
name = "hbd"
|
||||||
version = "5.4.4"
|
version = "5.4.5"
|
||||||
description = "Heartbeat monitoring system — client (hbc) and server (hbd)"
|
description = "Heartbeat monitoring system — client (hbc) and server (hbd)"
|
||||||
readme = "README.md"
|
readme = "README.md"
|
||||||
requires-python = ">=3.11"
|
requires-python = ">=3.11"
|
||||||
|
|||||||
@@ -9,8 +9,10 @@ sed -i".bak" "s/\*\*Package:\*\* \`hbd\` v[0-9.]*/\*\*Package:\*\* \`hbd\` v$VE
|
|||||||
|
|
||||||
# Update CHANGELOG.md with commits since last tag
|
# Update CHANGELOG.md with commits since last tag
|
||||||
LASTTAG=$(git describe --tags --abbrev=0 2>/dev/null || true)
|
LASTTAG=$(git describe --tags --abbrev=0 2>/dev/null || true)
|
||||||
ADDED=$(git log "${LASTTAG:+$LASTTAG..}HEAD" --pretty="%s" | grep "^feat:" | sed 's/^feat: /- /')
|
# Matches both unscoped ("feat: ...") and scoped ("feat(server): ...") conventional-commit
|
||||||
FIXED=$(git log "${LASTTAG:+$LASTTAG..}HEAD" --pretty="%s" | grep "^fix:" | sed 's/^fix: /- /')
|
# subjects; the scope is dropped from the changelog line.
|
||||||
|
ADDED=$(git log "${LASTTAG:+$LASTTAG..}HEAD" --pretty="%s" | grep -E "^feat(\([^)]*\))?:" | sed -E 's/^feat(\([^)]*\))?: /- /')
|
||||||
|
FIXED=$(git log "${LASTTAG:+$LASTTAG..}HEAD" --pretty="%s" | grep -E "^fix(\([^)]*\))?:" | sed -E 's/^fix(\([^)]*\))?: /- /')
|
||||||
{
|
{
|
||||||
printf "## [%s]\n" "$VER"
|
printf "## [%s]\n" "$VER"
|
||||||
[ -n "$ADDED" ] && printf "\n### Added\n%s\n" "$ADDED"
|
[ -n "$ADDED" ] && printf "\n### Added\n%s\n" "$ADDED"
|
||||||
|
|||||||
@@ -1,2 +1,3 @@
|
|||||||
hbc_mini
|
hbc_mini
|
||||||
hbc_mini_dbg
|
hbc_mini_dbg
|
||||||
|
test_probe
|
||||||
|
|||||||
+8
-2
@@ -7,7 +7,7 @@ SRC = hbc_mini.c
|
|||||||
# FreeBSD/NetBSD keep zlib in base; no extra flags needed.
|
# FreeBSD/NetBSD keep zlib in base; no extra flags needed.
|
||||||
# On some NetBSD installs pthreads may need -lpthread from pkgsrc.
|
# On some NetBSD installs pthreads may need -lpthread from pkgsrc.
|
||||||
|
|
||||||
.PHONY: all clean debug
|
.PHONY: all clean debug test
|
||||||
|
|
||||||
all: $(TARGET)
|
all: $(TARGET)
|
||||||
|
|
||||||
@@ -17,5 +17,11 @@ $(TARGET): $(SRC)
|
|||||||
debug: $(SRC)
|
debug: $(SRC)
|
||||||
$(CC) -g -fsanitize=address,undefined -o $(TARGET)_dbg $< $(LDFLAGS)
|
$(CC) -g -fsanitize=address,undefined -o $(TARGET)_dbg $< $(LDFLAGS)
|
||||||
|
|
||||||
|
test: test_probe
|
||||||
|
./test_probe
|
||||||
|
|
||||||
|
test_probe: test_probe.c $(SRC)
|
||||||
|
$(CC) $(CFLAGS) -o $@ test_probe.c $(LDFLAGS)
|
||||||
|
|
||||||
clean:
|
clean:
|
||||||
rm -f $(TARGET) $(TARGET)_dbg
|
rm -f $(TARGET) $(TARGET)_dbg test_probe
|
||||||
|
|||||||
+379
-3
@@ -6,6 +6,12 @@
|
|||||||
*
|
*
|
||||||
* Drop on any host and run: ./hbc_mini <hbd-host>
|
* Drop on any host and run: ./hbc_mini <hbd-host>
|
||||||
* Config: ~/.hbc.json (same keys as Python version)
|
* Config: ~/.hbc.json (same keys as Python version)
|
||||||
|
*
|
||||||
|
* Presence probe (built in): reports per protocol (IPv4/IPv6) whether the host has internet
|
||||||
|
* connectivity, as p4/p6 flags on every heartbeat, so hbd stops treating a missing IPv6 (or
|
||||||
|
* IPv4) path as an outage. Config, in the network_monitor section of ~/.hbc.json:
|
||||||
|
* probe_enabled (default true), probe_v4 (default 1.1.1.1), probe_v6 (default
|
||||||
|
* 2606:4700:4700::1111; IP literals only), probe_interval (default 60 s)
|
||||||
*/
|
*/
|
||||||
|
|
||||||
#ifdef __linux__
|
#ifdef __linux__
|
||||||
@@ -396,6 +402,10 @@ typedef struct {
|
|||||||
|
|
||||||
char net_skip[MAX_NET_SKIP][64];
|
char net_skip[MAX_NET_SKIP][64];
|
||||||
int net_skip_count;
|
int net_skip_count;
|
||||||
|
|
||||||
|
bool probe_enabled;
|
||||||
|
int probe_interval;
|
||||||
|
char probe_v4[64], probe_v6[64];
|
||||||
} config_t;
|
} config_t;
|
||||||
|
|
||||||
static void config_defaults(config_t *c) {
|
static void config_defaults(config_t *c) {
|
||||||
@@ -410,6 +420,10 @@ static void config_defaults(config_t *c) {
|
|||||||
c->net_interval = 300;
|
c->net_interval = 300;
|
||||||
snprintf(c->net_skip[0], 64, "lo");
|
snprintf(c->net_skip[0], 64, "lo");
|
||||||
c->net_skip_count = 1;
|
c->net_skip_count = 1;
|
||||||
|
c->probe_enabled = true;
|
||||||
|
c->probe_interval = 60;
|
||||||
|
snprintf(c->probe_v4, sizeof(c->probe_v4), "1.1.1.1");
|
||||||
|
snprintf(c->probe_v6, sizeof(c->probe_v6), "2606:4700:4700::1111");
|
||||||
}
|
}
|
||||||
|
|
||||||
static void config_load(config_t *cfg, const char *path) {
|
static void config_load(config_t *cfg, const char *path) {
|
||||||
@@ -498,6 +512,10 @@ static void config_load(config_t *cfg, const char *path) {
|
|||||||
jval_t *net = plugins ? jget(plugins,"network_monitor") : jget(root,"network_monitor");
|
jval_t *net = plugins ? jget(plugins,"network_monitor") : jget(root,"network_monitor");
|
||||||
if (net && net->type == JT_OBJECT) {
|
if (net && net->type == JT_OBJECT) {
|
||||||
if ((v=jget(net,"interval"))) cfg->net_interval = jint(v, cfg->net_interval);
|
if ((v=jget(net,"interval"))) cfg->net_interval = jint(v, cfg->net_interval);
|
||||||
|
if ((v=jget(net,"probe_enabled"))) cfg->probe_enabled = jint(v, 1) != 0;
|
||||||
|
if ((v=jget(net,"probe_interval"))) cfg->probe_interval = jint(v, cfg->probe_interval);
|
||||||
|
if ((v=jget(net,"probe_v4"))) snprintf(cfg->probe_v4, sizeof(cfg->probe_v4), "%s", jstr(v, cfg->probe_v4));
|
||||||
|
if ((v=jget(net,"probe_v6"))) snprintf(cfg->probe_v6, sizeof(cfg->probe_v6), "%s", jstr(v, cfg->probe_v6));
|
||||||
jval_t *sk = jget(net,"skip_interfaces");
|
jval_t *sk = jget(net,"skip_interfaces");
|
||||||
if (sk && sk->type == JT_ARRAY) {
|
if (sk && sk->type == JT_ARRAY) {
|
||||||
cfg->net_skip_count = 0;
|
cfg->net_skip_count = 0;
|
||||||
@@ -984,6 +1002,341 @@ static void plugin_disk_monitor(conn_t *c, const config_t *cfg) {
|
|||||||
LOGD("sent disk_monitor");
|
LOGD("sent disk_monitor");
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* ============================================================
|
||||||
|
* Connectivity probe: per-protocol internet presence (IPv4 / IPv6)
|
||||||
|
*
|
||||||
|
* Tier 1 (every PROBE_TIER1 s, sends no packets): a UDP connect() to the probe address shows
|
||||||
|
* whether a route exists and which source address would be used. Tier 2 (every probe_interval
|
||||||
|
* s, sooner after a failure or a change of address or gateway): one ICMP echo, only when tier 1
|
||||||
|
* found a route. Absent = no route, or PROBE_FAIL_LIMIT consecutive failed echoes; present =
|
||||||
|
* one success; unknown (flag omitted) when the family cannot be probed. The result rides on
|
||||||
|
* every HTB as p4/p6 and inside the network_monitor metric. Mirrors hbd/client/connectivity.py.
|
||||||
|
* ============================================================ */
|
||||||
|
|
||||||
|
#define PROBE_UNKNOWN (-1)
|
||||||
|
#define PROBE_FAIL_LIMIT 2
|
||||||
|
#define PROBE_TIER1 5 /* seconds between route checks */
|
||||||
|
|
||||||
|
typedef struct {
|
||||||
|
int present; /* PROBE_UNKNOWN, 0 = absent, 1 = present */
|
||||||
|
char addr[INET6_ADDRSTRLEN];
|
||||||
|
char gateway[INET6_ADDRSTRLEN];
|
||||||
|
double rtt; /* ms, < 0 when unknown */
|
||||||
|
int fails; /* consecutive failed echoes */
|
||||||
|
double next_ping; /* monotonic seconds */
|
||||||
|
char last_key[2 * INET6_ADDRSTRLEN + 2];
|
||||||
|
bool ping_disabled; /* no usable ping binary */
|
||||||
|
} probe_fam_t;
|
||||||
|
|
||||||
|
typedef struct {
|
||||||
|
/* 1 = route exists (source address in out), 0 = no route, -1 = family unsupported/unusable */
|
||||||
|
int (*local_address)(int af, const char *target, char *out, size_t n);
|
||||||
|
/* RTT in ms; -1 = no reply; -2 = ping unavailable */
|
||||||
|
double (*ping)(int af, const char *target);
|
||||||
|
/* fills out with the default gateway, "" when unknown */
|
||||||
|
void (*gateway)(int af, char *out, size_t n);
|
||||||
|
} probe_ops_t;
|
||||||
|
|
||||||
|
typedef struct {
|
||||||
|
probe_fam_t fam[2]; /* [0] = IPv4, [1] = IPv6 */
|
||||||
|
char target[2][64];
|
||||||
|
double probe_interval, retry_interval;
|
||||||
|
bool cheap_gateway; /* gateway read is cheap (/proc), so read every tier 1 */
|
||||||
|
pthread_mutex_t mu; /* guards fam[] between the probe thread and readers */
|
||||||
|
} probe_t;
|
||||||
|
|
||||||
|
/* Lowest-metric default gateway from /proc/net/route. The kernel prints the raw network-order
|
||||||
|
* u32 in host order, so it can be stored straight into s_addr on any endianness. */
|
||||||
|
static void gw_parse_proc_v4(const char *text, char *out, size_t n) {
|
||||||
|
long best = -1; out[0] = '\0';
|
||||||
|
const char *p = strchr(text, '\n'); /* skip the header line */
|
||||||
|
while (p && *++p) {
|
||||||
|
char iface[64], dest[16], gw[16]; unsigned flags; int refcnt, use, metric;
|
||||||
|
if (sscanf(p, "%63s %15s %15s %x %d %d %d", iface, dest, gw, &flags, &refcnt, &use, &metric) == 7
|
||||||
|
&& !strcmp(dest, "00000000") && (flags & 0x2)) {
|
||||||
|
if (best < 0 || metric < best) {
|
||||||
|
struct in_addr a; a.s_addr = (uint32_t)strtoul(gw, NULL, 16);
|
||||||
|
if (inet_ntop(AF_INET, &a, out, n)) best = metric;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
p = strchr(p, '\n');
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Lowest-metric default gateway from /proc/net/ipv6_route (skips RTF_REJECT and "lo"). */
|
||||||
|
static void gw_parse_proc_v6(const char *text, char *out, size_t n) {
|
||||||
|
long best = -1; out[0] = '\0';
|
||||||
|
const char *zero = "00000000000000000000000000000000";
|
||||||
|
for (const char *p = text; p && *p; ) {
|
||||||
|
char dest[40], src[40], nh[40], iface[64];
|
||||||
|
unsigned plen, splen, metric, refcnt, use, flags;
|
||||||
|
if (sscanf(p, "%32s %x %32s %x %32s %x %x %x %x %63s",
|
||||||
|
dest, &plen, src, &splen, nh, &metric, &refcnt, &use, &flags, iface) == 10
|
||||||
|
&& !strcmp(dest, zero) && plen == 0 && strcmp(nh, zero) && strcmp(iface, "lo")
|
||||||
|
&& (flags & 0x2) && !(flags & 0x200)) {
|
||||||
|
if (best < 0 || (long)metric < best) {
|
||||||
|
struct in6_addr a; memset(&a, 0, sizeof(a));
|
||||||
|
bool ok = strlen(nh) == 32;
|
||||||
|
for (int i = 0; ok && i < 16; i++) {
|
||||||
|
unsigned b;
|
||||||
|
if (sscanf(nh + 2 * i, "%2x", &b) != 1) ok = false; else a.s6_addr[i] = (uint8_t)b;
|
||||||
|
}
|
||||||
|
if (ok && inet_ntop(AF_INET6, &a, out, n)) best = (long)metric;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
p = strchr(p, '\n'); if (p) p++;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Gateway from `route -n get [-inet6] default` (the BSDs). */
|
||||||
|
static void gw_parse_route_get(const char *text, char *out, size_t n) {
|
||||||
|
out[0] = '\0';
|
||||||
|
const char *p = strstr(text, "gateway:");
|
||||||
|
if (!p) return;
|
||||||
|
p += 8;
|
||||||
|
while (*p == ' ' || *p == '\t') p++;
|
||||||
|
size_t i = 0;
|
||||||
|
while (*p && !isspace((unsigned char)*p) && i + 1 < n) out[i++] = *p++;
|
||||||
|
out[i] = '\0';
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Average RTT from ping output: -1.0 no reply, -2.0 ping could not be run. Works for Linux
|
||||||
|
* ("rtt min/avg/max/mdev") and BSD ("round-trip min/avg/max/stddev"). */
|
||||||
|
static double ping_parse_rtt(const char *out) {
|
||||||
|
if (strstr(out, "not found") || strstr(out, "No such file")) return -2.0;
|
||||||
|
const char *p = strstr(out, "min/avg/max");
|
||||||
|
if (!p) return -1.0;
|
||||||
|
p = strchr(p, '=');
|
||||||
|
double mn, avg, mx;
|
||||||
|
if (p && sscanf(p + 1, " %lf/%lf/%lf", &mn, &avg, &mx) == 3) return avg;
|
||||||
|
return -1.0;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* One probe step for one family (f: 0 = IPv4, 1 = IPv6). Works on *s only, so the probe thread
|
||||||
|
* can run it on a private copy without holding the mutex during the (blocking) echo. */
|
||||||
|
static void probe_step(const probe_t *p, probe_fam_t *s, int f, const probe_ops_t *ops, double now) {
|
||||||
|
int af = f ? AF_INET6 : AF_INET;
|
||||||
|
char addr[INET6_ADDRSTRLEN] = "", gw[INET6_ADDRSTRLEN] = "";
|
||||||
|
|
||||||
|
int r = ops->local_address(af, p->target[f], addr, sizeof(addr));
|
||||||
|
if (r < 0) { s->present = PROBE_UNKNOWN; return; }
|
||||||
|
if (r == 0) { /* tier 1: no route => absent immediately */
|
||||||
|
s->fails = 0; s->next_ping = 0; s->last_key[0] = '\0';
|
||||||
|
s->present = 0; s->addr[0] = '\0'; s->gateway[0] = '\0'; s->rtt = -1;
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (p->cheap_gateway) ops->gateway(af, gw, sizeof(gw));
|
||||||
|
else snprintf(gw, sizeof(gw), "%s", s->gateway);
|
||||||
|
char key[sizeof(s->last_key)];
|
||||||
|
snprintf(key, sizeof(key), "%s|%s", addr, gw);
|
||||||
|
bool due = strcmp(key, s->last_key) != 0 || now >= s->next_ping;
|
||||||
|
if (due && !p->cheap_gateway) {
|
||||||
|
ops->gateway(af, gw, sizeof(gw));
|
||||||
|
snprintf(key, sizeof(key), "%s|%s", addr, gw);
|
||||||
|
}
|
||||||
|
snprintf(s->addr, sizeof(s->addr), "%s", addr);
|
||||||
|
snprintf(s->gateway, sizeof(s->gateway), "%s", gw);
|
||||||
|
snprintf(s->last_key, sizeof(s->last_key), "%s", key);
|
||||||
|
|
||||||
|
if (s->ping_disabled) { s->present = PROBE_UNKNOWN; return; } /* cannot probe: unknown, never absent, while a route exists */
|
||||||
|
if (!due) return;
|
||||||
|
double rtt = ops->ping(af, p->target[f]);
|
||||||
|
if (rtt < -1.5) { s->ping_disabled = true; s->present = PROBE_UNKNOWN; return; }
|
||||||
|
if (rtt < 0) {
|
||||||
|
s->fails++;
|
||||||
|
s->next_ping = now + p->retry_interval;
|
||||||
|
if (s->fails >= PROBE_FAIL_LIMIT) { s->present = 0; s->rtt = -1; }
|
||||||
|
} else {
|
||||||
|
s->fails = 0;
|
||||||
|
s->next_ping = now + p->probe_interval;
|
||||||
|
s->present = 1; s->rtt = rtt;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/* HTB flags for the families whose presence is known. */
|
||||||
|
static void probe_flags(probe_t *p, kvdict_t *d) {
|
||||||
|
pthread_mutex_lock(&p->mu);
|
||||||
|
if (p->fam[0].present >= 0) kv_set_int(d, "p4", p->fam[0].present);
|
||||||
|
if (p->fam[1].present >= 0) kv_set_int(d, "p6", p->fam[1].present);
|
||||||
|
pthread_mutex_unlock(&p->mu);
|
||||||
|
}
|
||||||
|
|
||||||
|
/* The `connectivity` metric as a JSON object; returns its length or -1 on overflow. */
|
||||||
|
static int probe_metric_json(probe_t *p, char *out, size_t n) {
|
||||||
|
static const char *names[2] = { "ipv4", "ipv6" };
|
||||||
|
size_t len = 0;
|
||||||
|
len += (size_t)snprintf(out + len, n - len, "{");
|
||||||
|
pthread_mutex_lock(&p->mu);
|
||||||
|
for (int f = 0; f < 2 && len < n; f++) {
|
||||||
|
const probe_fam_t *s = &p->fam[f];
|
||||||
|
char pres[8], addr[64], gw[64], rtt[32];
|
||||||
|
snprintf(pres, sizeof(pres), "%s", s->present < 0 ? "null" : s->present ? "true" : "false");
|
||||||
|
if (s->addr[0]) snprintf(addr, sizeof(addr), "\"%s\"", s->addr); else snprintf(addr, sizeof(addr), "null");
|
||||||
|
if (s->gateway[0]) snprintf(gw, sizeof(gw), "\"%s\"", s->gateway); else snprintf(gw, sizeof(gw), "null");
|
||||||
|
if (s->rtt >= 0) snprintf(rtt, sizeof(rtt), "%.3f", s->rtt); else snprintf(rtt, sizeof(rtt), "null");
|
||||||
|
len += (size_t)snprintf(out + len, n - len,
|
||||||
|
"%s\"%s\":{\"present\":%s,\"addr\":%s,\"gateway\":%s,\"rtt\":%s}",
|
||||||
|
f ? "," : "", names[f], pres, addr, gw, rtt);
|
||||||
|
}
|
||||||
|
pthread_mutex_unlock(&p->mu);
|
||||||
|
if (len + 2 > n) { out[0] = '\0'; return -1; }
|
||||||
|
out[len++] = '}'; out[len] = '\0';
|
||||||
|
return (int)len;
|
||||||
|
}
|
||||||
|
|
||||||
|
static double mono_ts(void) {
|
||||||
|
struct timespec ts;
|
||||||
|
clock_gettime(CLOCK_MONOTONIC, &ts);
|
||||||
|
return ts.tv_sec + ts.tv_nsec * 1e-9;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Source address the kernel would use to reach target (an IP literal). A UDP connect() sends
|
||||||
|
* nothing. Returns 1 = route (address in out), 0 = no route, -1 = family unsupported or the
|
||||||
|
* target is not an IP literal (unknown, never absent). */
|
||||||
|
static int sys_local_address(int af, const char *target, char *out, size_t n) {
|
||||||
|
int fd = socket(af, SOCK_DGRAM, 0);
|
||||||
|
if (fd < 0) return -1;
|
||||||
|
struct sockaddr_storage sa; socklen_t sl;
|
||||||
|
memset(&sa, 0, sizeof(sa));
|
||||||
|
if (af == AF_INET) {
|
||||||
|
struct sockaddr_in *s4 = (struct sockaddr_in *)&sa;
|
||||||
|
s4->sin_family = AF_INET; s4->sin_port = htons(9);
|
||||||
|
if (inet_pton(AF_INET, target, &s4->sin_addr) != 1) { close(fd); return -1; }
|
||||||
|
sl = sizeof(*s4);
|
||||||
|
} else {
|
||||||
|
struct sockaddr_in6 *s6 = (struct sockaddr_in6 *)&sa;
|
||||||
|
s6->sin6_family = AF_INET6; s6->sin6_port = htons(9);
|
||||||
|
if (inet_pton(AF_INET6, target, &s6->sin6_addr) != 1) { close(fd); return -1; }
|
||||||
|
sl = sizeof(*s6);
|
||||||
|
}
|
||||||
|
int rc = 0;
|
||||||
|
if (connect(fd, (struct sockaddr *)&sa, sl) == 0) {
|
||||||
|
struct sockaddr_storage me; socklen_t ml = sizeof(me);
|
||||||
|
if (getsockname(fd, (struct sockaddr *)&me, &ml) == 0) {
|
||||||
|
const void *src = (af == AF_INET)
|
||||||
|
? (const void *)&((struct sockaddr_in *)&me)->sin_addr
|
||||||
|
: (const void *)&((struct sockaddr_in6 *)&me)->sin6_addr;
|
||||||
|
if (inet_ntop(af, src, out, (socklen_t)n)) rc = 1;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
close(fd);
|
||||||
|
return rc;
|
||||||
|
}
|
||||||
|
|
||||||
|
static char *slurp(FILE *f) {
|
||||||
|
size_t cap = 4096, len = 0;
|
||||||
|
char *buf = malloc(cap);
|
||||||
|
if (!buf) return NULL;
|
||||||
|
size_t r;
|
||||||
|
while ((r = fread(buf + len, 1, cap - len - 1, f)) > 0) {
|
||||||
|
len += r;
|
||||||
|
if (len + 1 >= cap) {
|
||||||
|
if (cap >= 65536) break;
|
||||||
|
cap *= 2;
|
||||||
|
char *nb = realloc(buf, cap);
|
||||||
|
if (!nb) { free(buf); return NULL; }
|
||||||
|
buf = nb;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
buf[len] = '\0';
|
||||||
|
return buf;
|
||||||
|
}
|
||||||
|
|
||||||
|
static bool g_have_timeout = false; /* timeout(1) exists: bounds ping6 on the BSDs */
|
||||||
|
|
||||||
|
/* One echo request: RTT in ms, -1.0 no reply, -2.0 ping could not be run. The target is an IP
|
||||||
|
* literal from the config; anything else is refused before it can reach the shell. */
|
||||||
|
static double sys_ping(int af, const char *target) {
|
||||||
|
for (const char *c = target; *c; c++)
|
||||||
|
if (!(isxdigit((unsigned char)*c) || *c == '.' || *c == ':')) return -2.0;
|
||||||
|
char cmd[256];
|
||||||
|
#ifdef __linux__
|
||||||
|
snprintf(cmd, sizeof(cmd), "ping -c 1 -W 2 -w 3 %s %s 2>&1", af == AF_INET ? "-4" : "-6", target);
|
||||||
|
#else
|
||||||
|
/* BSD ping: -W is in milliseconds, and IPv6 is reached through ping6 */
|
||||||
|
if (af == AF_INET)
|
||||||
|
snprintf(cmd, sizeof(cmd), "%sping -c 1 -W 2000 %s 2>&1", g_have_timeout ? "timeout 4 " : "", target);
|
||||||
|
else
|
||||||
|
snprintf(cmd, sizeof(cmd), "%sping6 -c 1 %s 2>&1", g_have_timeout ? "timeout 4 " : "", target);
|
||||||
|
#endif
|
||||||
|
FILE *f = popen(cmd, "r");
|
||||||
|
if (!f) return -2.0;
|
||||||
|
char *out = slurp(f);
|
||||||
|
pclose(f);
|
||||||
|
if (!out) return -1.0;
|
||||||
|
double rtt = ping_parse_rtt(out);
|
||||||
|
free(out);
|
||||||
|
return rtt;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Default gateway for the family, "" when unknown. Best effort; never affects presence. */
|
||||||
|
static void sys_gateway(int af, char *out, size_t n) {
|
||||||
|
out[0] = '\0';
|
||||||
|
#ifdef __linux__
|
||||||
|
FILE *f = fopen(af == AF_INET ? "/proc/net/route" : "/proc/net/ipv6_route", "r");
|
||||||
|
if (!f) return;
|
||||||
|
char *text = slurp(f);
|
||||||
|
fclose(f);
|
||||||
|
if (!text) return;
|
||||||
|
if (af == AF_INET) gw_parse_proc_v4(text, out, n); else gw_parse_proc_v6(text, out, n);
|
||||||
|
free(text);
|
||||||
|
#else
|
||||||
|
FILE *f = popen(af == AF_INET ? "route -n get default 2>/dev/null"
|
||||||
|
: "route -n get -inet6 default 2>/dev/null", "r");
|
||||||
|
if (!f) return;
|
||||||
|
char *text = slurp(f);
|
||||||
|
pclose(f);
|
||||||
|
if (!text) return;
|
||||||
|
gw_parse_route_get(text, out, n);
|
||||||
|
free(text);
|
||||||
|
#endif
|
||||||
|
}
|
||||||
|
|
||||||
|
static probe_t g_probe;
|
||||||
|
static bool g_probe_enabled = false;
|
||||||
|
|
||||||
|
static void probe_init(probe_t *p, const config_t *cfg) {
|
||||||
|
memset(p, 0, sizeof(*p));
|
||||||
|
snprintf(p->target[0], sizeof(p->target[0]), "%s", cfg->probe_v4);
|
||||||
|
snprintf(p->target[1], sizeof(p->target[1]), "%s", cfg->probe_v6);
|
||||||
|
p->probe_interval = cfg->probe_interval > 0 ? cfg->probe_interval : 60;
|
||||||
|
p->retry_interval = 10;
|
||||||
|
#ifdef __linux__
|
||||||
|
p->cheap_gateway = true; /* /proc read; elsewhere a subprocess only when due */
|
||||||
|
#endif
|
||||||
|
p->fam[0].present = p->fam[1].present = PROBE_UNKNOWN;
|
||||||
|
p->fam[0].rtt = p->fam[1].rtt = -1;
|
||||||
|
pthread_mutex_init(&p->mu, NULL);
|
||||||
|
g_have_timeout = system("command -v timeout >/dev/null 2>&1") == 0;
|
||||||
|
g_probe_enabled = cfg->probe_enabled;
|
||||||
|
|
||||||
|
/* Diagnostic only: sys_local_address already turns a non-literal target into PROBE_UNKNOWN
|
||||||
|
* (via inet_pton failing there), so this does not change behavior — it just stops that from
|
||||||
|
* being silent. */
|
||||||
|
struct in_addr a4;
|
||||||
|
struct in6_addr a6;
|
||||||
|
if (inet_pton(AF_INET, p->target[0], &a4) != 1)
|
||||||
|
LOGW("probe_v4 is not an IPv4 literal: %s", p->target[0]);
|
||||||
|
if (inet_pton(AF_INET6, p->target[1], &a6) != 1)
|
||||||
|
LOGW("probe_v6 is not an IPv6 literal: %s", p->target[1]);
|
||||||
|
}
|
||||||
|
|
||||||
|
static void *probe_thread(void *arg) {
|
||||||
|
probe_t *p = arg;
|
||||||
|
static const probe_ops_t ops = { sys_local_address, sys_ping, sys_gateway };
|
||||||
|
while (g_running) {
|
||||||
|
double now = mono_ts();
|
||||||
|
for (int f = 0; f < 2 && g_running; f++) {
|
||||||
|
pthread_mutex_lock(&p->mu); probe_fam_t tmp = p->fam[f]; pthread_mutex_unlock(&p->mu);
|
||||||
|
probe_step(p, &tmp, f, &ops, now); /* may block in ping: no lock held */
|
||||||
|
pthread_mutex_lock(&p->mu); p->fam[f] = tmp; pthread_mutex_unlock(&p->mu);
|
||||||
|
}
|
||||||
|
for (int i = 0; i < PROBE_TIER1 && g_running; i++) sleep(1);
|
||||||
|
}
|
||||||
|
return NULL;
|
||||||
|
}
|
||||||
|
|
||||||
/* ============================================================
|
/* ============================================================
|
||||||
* Plugin: network_monitor
|
* Plugin: network_monitor
|
||||||
* Linux: /proc/net/dev
|
* Linux: /proc/net/dev
|
||||||
@@ -1077,16 +1430,26 @@ static void plugin_network_monitor(conn_t *c, const config_t *cfg) {
|
|||||||
json[jlen++] = '}'; json[jlen] = '\0';
|
json[jlen++] = '}'; json[jlen] = '\0';
|
||||||
memcpy(g_net_prev, curr, cn * sizeof(net_stat_t));
|
memcpy(g_net_prev, curr, cn * sizeof(net_stat_t));
|
||||||
g_net_prev_n = cn; g_net_prev_ts = now;
|
g_net_prev_n = cn; g_net_prev_ts = now;
|
||||||
if (first) { free(json); return; }
|
|
||||||
|
|
||||||
kvdict_t d; kv_clear(&d);
|
kvdict_t d; kv_clear(&d);
|
||||||
kv_set(&d, "plugin", "network_monitor");
|
kv_set(&d, "plugin", "network_monitor");
|
||||||
|
if (!first) {
|
||||||
char *jval = malloc(MAX_VAL + 1);
|
char *jval = malloc(MAX_VAL + 1);
|
||||||
if (jval) { snprintf(jval, MAX_VAL, "@%s", json); kv_set(&d, "interfaces", jval); free(jval); }
|
if (jval) { snprintf(jval, MAX_VAL, "@%s", json); kv_set(&d, "interfaces", jval); free(jval); }
|
||||||
|
}
|
||||||
|
bool have_conn = false;
|
||||||
|
if (g_probe_enabled) {
|
||||||
|
char cj[512];
|
||||||
|
cj[0] = '@';
|
||||||
|
if (probe_metric_json(&g_probe, cj + 1, sizeof(cj) - 1) > 0) {
|
||||||
|
kv_set(&d, "connectivity", cj);
|
||||||
|
have_conn = true;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
free(json);
|
||||||
|
if (first && !have_conn) return;
|
||||||
kv_set_dbl(&d, "_timestamp", now_ts());
|
kv_set_dbl(&d, "_timestamp", now_ts());
|
||||||
kv_set_int(&d, "_interval", cfg->net_interval);
|
kv_set_int(&d, "_interval", cfg->net_interval);
|
||||||
conn_send(c, "PLG", &d);
|
conn_send(c, "PLG", &d);
|
||||||
free(json);
|
|
||||||
LOGD("sent network_monitor");
|
LOGD("sent network_monitor");
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -1323,6 +1686,7 @@ int main(int argc, char **argv) {
|
|||||||
|
|
||||||
config_t cfg;
|
config_t cfg;
|
||||||
config_load(&cfg, cfgpath);
|
config_load(&cfg, cfgpath);
|
||||||
|
probe_init(&g_probe, &cfg);
|
||||||
|
|
||||||
if (do_daemon) {
|
if (do_daemon) {
|
||||||
daemonize();
|
daemonize();
|
||||||
@@ -1405,6 +1769,13 @@ int main(int argc, char **argv) {
|
|||||||
pthread_t tid;
|
pthread_t tid;
|
||||||
pthread_create(&tid, NULL, monitor_thread, &tctx);
|
pthread_create(&tid, NULL, monitor_thread, &tctx);
|
||||||
|
|
||||||
|
pthread_t ptid; bool probe_running = false;
|
||||||
|
if (g_probe_enabled) {
|
||||||
|
int prc = pthread_create(&ptid, NULL, probe_thread, &g_probe);
|
||||||
|
if (prc == 0) probe_running = true;
|
||||||
|
else LOGW("pthread_create(probe): %s", strerror(prc));
|
||||||
|
}
|
||||||
|
|
||||||
/* Main heartbeat + receive loop */
|
/* Main heartbeat + receive loop */
|
||||||
time_t last_hb = 0;
|
time_t last_hb = 0;
|
||||||
while (g_running) {
|
while (g_running) {
|
||||||
@@ -1415,6 +1786,7 @@ int main(int argc, char **argv) {
|
|||||||
kv_set_int(&hb, "acks", g_conns[i].ack_count);
|
kv_set_int(&hb, "acks", g_conns[i].ack_count);
|
||||||
kv_set_dbl(&hb, "rtt", g_conns[i].rtt);
|
kv_set_dbl(&hb, "rtt", g_conns[i].rtt);
|
||||||
kv_set_int(&hb, "interval", cfg.interval);
|
kv_set_int(&hb, "interval", cfg.interval);
|
||||||
|
if (g_probe_enabled) probe_flags(&g_probe, &hb);
|
||||||
conn_send(&g_conns[i], "HTB", &hb);
|
conn_send(&g_conns[i], "HTB", &hb);
|
||||||
}
|
}
|
||||||
last_hb = now;
|
last_hb = now;
|
||||||
@@ -1449,6 +1821,10 @@ int main(int argc, char **argv) {
|
|||||||
conn_send(primary, "HTB", &sh);
|
conn_send(primary, "HTB", &sh);
|
||||||
usleep(300000);
|
usleep(300000);
|
||||||
}
|
}
|
||||||
|
/* Joined after the shutdown HTB goes out: the probe thread is never cancelled (it can be
|
||||||
|
* inside popen), so joining it can block for a few seconds — long enough to lose the
|
||||||
|
* clean-shutdown notification to an impatient supervisor's SIGKILL. */
|
||||||
|
if (probe_running) pthread_join(ptid, NULL);
|
||||||
for (int i = 0; i < g_nconns; i++) conn_close(&g_conns[i]);
|
for (int i = 0; i < g_nconns; i++) conn_close(&g_conns[i]);
|
||||||
if (g_use_syslog) closelog();
|
if (g_use_syslog) closelog();
|
||||||
if (g_restart) { LOGI("restarting..."); execv(argv[0], argv); }
|
if (g_restart) { LOGI("restarting..."); execv(argv[0], argv); }
|
||||||
|
|||||||
@@ -0,0 +1,243 @@
|
|||||||
|
/* Unit tests for the connectivity-probe helpers in hbc_mini.c.
|
||||||
|
* Build and run: make test (live check of this host: ./test_probe --live)
|
||||||
|
* The source file is included directly so its static functions can be exercised; its main()
|
||||||
|
* is renamed out of the way.
|
||||||
|
*/
|
||||||
|
#define main hbc_mini_main
|
||||||
|
#include "hbc_mini.c"
|
||||||
|
#undef main
|
||||||
|
|
||||||
|
static int g_run = 0, g_fail = 0;
|
||||||
|
#define CHECK(cond) do { g_run++; if (!(cond)) { g_fail++; \
|
||||||
|
fprintf(stderr, "FAIL %s:%d: %s\n", __FILE__, __LINE__, #cond); } } while (0)
|
||||||
|
#define CHECK_STR(a, b) do { g_run++; if (strcmp((a), (b)) != 0) { g_fail++; \
|
||||||
|
fprintf(stderr, "FAIL %s:%d: \"%s\" != \"%s\"\n", __FILE__, __LINE__, (a), (b)); } } while (0)
|
||||||
|
|
||||||
|
static const char PROC_V4[] =
|
||||||
|
"Iface\tDestination\tGateway \tFlags\tRefCnt\tUse\tMetric\tMask\t\tMTU\tWindow\tIRTT\n"
|
||||||
|
"wlan0\t00000000\t0100A8C0\t0003\t0\t0\t600\t00000000\t0\t0\t0\n"
|
||||||
|
"eth0\t00000000\t0101A8C0\t0003\t0\t0\t100\t00000000\t0\t0\t0\n"
|
||||||
|
"wlan0\t0000A8C0\t00000000\t0001\t0\t0\t600\t00FFFFFF\t0\t0\t0\n";
|
||||||
|
|
||||||
|
static const char PROC_V6[] =
|
||||||
|
"00000000000000000000000000000000 00 00000000000000000000000000000000 00 "
|
||||||
|
"fe800000000000000000000000000001 00000400 00000001 00000000 00000003 wlan0\n"
|
||||||
|
"00000000000000000000000000000000 00 00000000000000000000000000000000 00 "
|
||||||
|
"00000000000000000000000000000000 ffffffff 00000001 00000000 00200200 lo\n"
|
||||||
|
"fe800000000000000000000000000000 40 00000000000000000000000000000000 00 "
|
||||||
|
"00000000000000000000000000000000 00000100 00000001 00000000 00000001 wlan0\n";
|
||||||
|
|
||||||
|
static const char ROUTE_GET[] =
|
||||||
|
" route to: 0.0.0.0\ndestination: 0.0.0.0\n mask: 0.0.0.0\n"
|
||||||
|
" gateway: 192.0.2.1\n fib: 0\n interface: vtnet0\n";
|
||||||
|
|
||||||
|
static const char LINUX_PING[] =
|
||||||
|
"PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.\n"
|
||||||
|
"64 bytes from 1.1.1.1: icmp_seq=1 ttl=57 time=11.2 ms\n\n"
|
||||||
|
"--- 1.1.1.1 ping statistics ---\n"
|
||||||
|
"1 packets transmitted, 1 received, 0% packet loss, time 0ms\n"
|
||||||
|
"rtt min/avg/max/mdev = 11.234/11.234/11.234/0.000 ms\n";
|
||||||
|
|
||||||
|
static const char FREEBSD_PING[] =
|
||||||
|
"PING 1.1.1.1 (1.1.1.1): 56 data bytes\n"
|
||||||
|
"64 bytes from 1.1.1.1: icmp_seq=0 ttl=59 time=4.006 ms\n\n"
|
||||||
|
"--- 1.1.1.1 ping statistics ---\n"
|
||||||
|
"1 packets transmitted, 1 packets received, 0.0% packet loss, 1 packets out of wait time\n"
|
||||||
|
"round-trip min/avg/max/stddev = 4.006/4.006/4.006/0.000 ms\n";
|
||||||
|
|
||||||
|
static void test_gateway_parsers(void) {
|
||||||
|
char gw[64];
|
||||||
|
gw_parse_proc_v4(PROC_V4, gw, sizeof(gw));
|
||||||
|
CHECK_STR(gw, "192.168.1.1"); /* eth0, metric 100 beats wlan0, 600 */
|
||||||
|
gw_parse_proc_v4("Iface\tDestination\n", gw, sizeof(gw));
|
||||||
|
CHECK_STR(gw, "");
|
||||||
|
gw_parse_proc_v6(PROC_V6, gw, sizeof(gw));
|
||||||
|
CHECK_STR(gw, "fe80::1"); /* reject and non-default lines skipped */
|
||||||
|
gw_parse_route_get(ROUTE_GET, gw, sizeof(gw));
|
||||||
|
CHECK_STR(gw, "192.0.2.1");
|
||||||
|
gw_parse_route_get("route: writing to routing socket: not in table\n", gw, sizeof(gw));
|
||||||
|
CHECK_STR(gw, "");
|
||||||
|
}
|
||||||
|
|
||||||
|
static void test_ping_parser(void) {
|
||||||
|
CHECK(ping_parse_rtt(LINUX_PING) > 11.23 && ping_parse_rtt(LINUX_PING) < 11.24);
|
||||||
|
CHECK(ping_parse_rtt(FREEBSD_PING) > 4.0 && ping_parse_rtt(FREEBSD_PING) < 4.01);
|
||||||
|
CHECK(ping_parse_rtt("1 packets transmitted, 0 received, 100% packet loss\n") == -1.0);
|
||||||
|
CHECK(ping_parse_rtt("sh: 1: ping: not found\n") == -2.0);
|
||||||
|
CHECK(ping_parse_rtt("ping: command not found\n") == -2.0);
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ---- scriptable network for probe_step ------------------------------------------------ */
|
||||||
|
static struct {
|
||||||
|
char addr[2][INET6_ADDRSTRLEN]; /* "" = no route */
|
||||||
|
char gw[2][INET6_ADDRSTRLEN];
|
||||||
|
double rtt[2]; /* -1 no reply, -2 ping unavailable */
|
||||||
|
int unsupported[2];
|
||||||
|
int npings[2];
|
||||||
|
} net;
|
||||||
|
|
||||||
|
static int fake_local(int af, const char *t, char *out, size_t n) {
|
||||||
|
int f = af == AF_INET6; (void)t;
|
||||||
|
if (net.unsupported[f]) return -1;
|
||||||
|
if (!net.addr[f][0]) return 0;
|
||||||
|
snprintf(out, n, "%s", net.addr[f]);
|
||||||
|
return 1;
|
||||||
|
}
|
||||||
|
static double fake_ping(int af, const char *t) { int f = af == AF_INET6; (void)t; net.npings[f]++; return net.rtt[f]; }
|
||||||
|
static void fake_gw(int af, char *out, size_t n) { snprintf(out, n, "%s", net.gw[af == AF_INET6]); }
|
||||||
|
static const probe_ops_t FAKE = { fake_local, fake_ping, fake_gw };
|
||||||
|
|
||||||
|
static void net_reset(void) {
|
||||||
|
memset(&net, 0, sizeof(net));
|
||||||
|
snprintf(net.addr[0], INET6_ADDRSTRLEN, "192.0.2.5");
|
||||||
|
snprintf(net.addr[1], INET6_ADDRSTRLEN, "2001:db8::5");
|
||||||
|
snprintf(net.gw[0], INET6_ADDRSTRLEN, "192.0.2.1");
|
||||||
|
snprintf(net.gw[1], INET6_ADDRSTRLEN, "fe80::1");
|
||||||
|
net.rtt[0] = 10.0; net.rtt[1] = 20.0;
|
||||||
|
}
|
||||||
|
|
||||||
|
static void probe_reset(probe_t *p) {
|
||||||
|
memset(p, 0, sizeof(*p));
|
||||||
|
snprintf(p->target[0], sizeof(p->target[0]), "1.1.1.1");
|
||||||
|
snprintf(p->target[1], sizeof(p->target[1]), "2606:4700:4700::1111");
|
||||||
|
p->probe_interval = 60; p->retry_interval = 10; p->cheap_gateway = true;
|
||||||
|
p->fam[0].present = p->fam[1].present = PROBE_UNKNOWN;
|
||||||
|
p->fam[0].rtt = p->fam[1].rtt = -1;
|
||||||
|
pthread_mutex_init(&p->mu, NULL);
|
||||||
|
}
|
||||||
|
|
||||||
|
static void step_both(probe_t *p, double now) {
|
||||||
|
probe_step(p, &p->fam[0], 0, &FAKE, now);
|
||||||
|
probe_step(p, &p->fam[1], 1, &FAKE, now);
|
||||||
|
}
|
||||||
|
|
||||||
|
static void test_present_with_details(void) {
|
||||||
|
net_reset(); probe_t p; probe_reset(&p);
|
||||||
|
step_both(&p, 1000);
|
||||||
|
CHECK(p.fam[0].present == 1 && p.fam[1].present == 1);
|
||||||
|
CHECK_STR(p.fam[0].addr, "192.0.2.5");
|
||||||
|
CHECK_STR(p.fam[0].gateway, "192.0.2.1");
|
||||||
|
CHECK(p.fam[0].rtt == 10.0);
|
||||||
|
kvdict_t d; kv_clear(&d);
|
||||||
|
probe_flags(&p, &d);
|
||||||
|
CHECK_STR(kv_get(&d, "p4") ? kv_get(&d, "p4") : "(null)", "1");
|
||||||
|
CHECK_STR(kv_get(&d, "p6") ? kv_get(&d, "p6") : "(null)", "1");
|
||||||
|
char js[512];
|
||||||
|
CHECK(probe_metric_json(&p, js, sizeof(js)) > 0);
|
||||||
|
CHECK(strstr(js, "\"ipv6\":{\"present\":true,\"addr\":\"2001:db8::5\",\"gateway\":\"fe80::1\"") != NULL);
|
||||||
|
}
|
||||||
|
|
||||||
|
static void test_no_route_is_absent_and_wastes_no_ping(void) {
|
||||||
|
net_reset(); net.addr[1][0] = '\0'; probe_t p; probe_reset(&p);
|
||||||
|
step_both(&p, 1000);
|
||||||
|
CHECK(p.fam[1].present == 0 && net.npings[1] == 0 && net.npings[0] == 1);
|
||||||
|
kvdict_t d; kv_clear(&d);
|
||||||
|
probe_flags(&p, &d);
|
||||||
|
CHECK_STR(kv_get(&d, "p6") ? kv_get(&d, "p6") : "(null)", "0");
|
||||||
|
char js[512];
|
||||||
|
probe_metric_json(&p, js, sizeof(js));
|
||||||
|
CHECK(strstr(js, "\"ipv6\":{\"present\":false,\"addr\":null,\"gateway\":null,\"rtt\":null}") != NULL);
|
||||||
|
}
|
||||||
|
|
||||||
|
static void test_debounce_and_rate_limit(void) {
|
||||||
|
net_reset(); probe_t p; probe_reset(&p);
|
||||||
|
step_both(&p, 1000);
|
||||||
|
step_both(&p, 1000); /* cached: no second ping */
|
||||||
|
CHECK(net.npings[0] == 1 && net.npings[1] == 1);
|
||||||
|
net.rtt[1] = -1;
|
||||||
|
step_both(&p, 1060); /* first failure: still present */
|
||||||
|
CHECK(p.fam[1].present == 1);
|
||||||
|
step_both(&p, 1070); /* retry interval elapsed: flips */
|
||||||
|
CHECK(p.fam[1].present == 0);
|
||||||
|
net.rtt[1] = 25;
|
||||||
|
step_both(&p, 1080); /* first success recovers */
|
||||||
|
CHECK(p.fam[1].present == 1);
|
||||||
|
}
|
||||||
|
|
||||||
|
static void test_first_failure_is_unknown_and_omitted(void) {
|
||||||
|
net_reset(); net.rtt[0] = -1; probe_t p; probe_reset(&p);
|
||||||
|
step_both(&p, 1000);
|
||||||
|
CHECK(p.fam[0].present == PROBE_UNKNOWN);
|
||||||
|
kvdict_t d; kv_clear(&d);
|
||||||
|
probe_flags(&p, &d);
|
||||||
|
CHECK(kv_get(&d, "p4") == NULL && kv_get(&d, "p6") != NULL);
|
||||||
|
char js[512];
|
||||||
|
probe_metric_json(&p, js, sizeof(js));
|
||||||
|
CHECK(strstr(js, "\"ipv4\":{\"present\":null") != NULL);
|
||||||
|
}
|
||||||
|
|
||||||
|
static void test_address_or_gateway_change_repings(void) {
|
||||||
|
net_reset(); probe_t p; probe_reset(&p);
|
||||||
|
step_both(&p, 1000);
|
||||||
|
snprintf(net.addr[0], INET6_ADDRSTRLEN, "198.51.100.7");
|
||||||
|
step_both(&p, 1000);
|
||||||
|
CHECK(net.npings[0] == 2);
|
||||||
|
snprintf(net.gw[0], INET6_ADDRSTRLEN, "192.0.2.254");
|
||||||
|
step_both(&p, 1000);
|
||||||
|
CHECK(net.npings[0] == 3);
|
||||||
|
}
|
||||||
|
|
||||||
|
static void test_route_loss_then_return_pings_immediately(void) {
|
||||||
|
net_reset(); probe_t p; probe_reset(&p);
|
||||||
|
step_both(&p, 1000);
|
||||||
|
net.addr[1][0] = '\0';
|
||||||
|
step_both(&p, 1000);
|
||||||
|
CHECK(p.fam[1].present == 0);
|
||||||
|
snprintf(net.addr[1], INET6_ADDRSTRLEN, "2001:db8::5");
|
||||||
|
step_both(&p, 1000);
|
||||||
|
CHECK(net.npings[1] == 2 && p.fam[1].present == 1);
|
||||||
|
}
|
||||||
|
|
||||||
|
static void test_missing_ping_stays_unknown_even_after_route_loss(void) {
|
||||||
|
net_reset(); net.rtt[0] = -2; probe_t p; probe_reset(&p);
|
||||||
|
step_both(&p, 1000);
|
||||||
|
step_both(&p, 1600);
|
||||||
|
CHECK(p.fam[0].present == PROBE_UNKNOWN && net.npings[0] == 1);
|
||||||
|
net.addr[0][0] = '\0';
|
||||||
|
step_both(&p, 1700);
|
||||||
|
CHECK(p.fam[0].present == 0); /* no route is definitive */
|
||||||
|
snprintf(net.addr[0], INET6_ADDRSTRLEN, "192.0.2.5");
|
||||||
|
step_both(&p, 1800);
|
||||||
|
CHECK(p.fam[0].present == PROBE_UNKNOWN && net.npings[0] == 1);
|
||||||
|
}
|
||||||
|
|
||||||
|
static void test_unsupported_family_is_unknown(void) {
|
||||||
|
net_reset(); net.unsupported[1] = 1; probe_t p; probe_reset(&p);
|
||||||
|
step_both(&p, 1000);
|
||||||
|
CHECK(p.fam[1].present == PROBE_UNKNOWN && p.fam[0].present == 1);
|
||||||
|
}
|
||||||
|
|
||||||
|
static int live_mode(void) {
|
||||||
|
static const probe_ops_t sys_ops = { sys_local_address, sys_ping, sys_gateway };
|
||||||
|
probe_t p; probe_reset(&p);
|
||||||
|
p.cheap_gateway = false;
|
||||||
|
snprintf(p.target[0], sizeof(p.target[0]), "1.1.1.1");
|
||||||
|
for (int round = 0; round < 3; round++) {
|
||||||
|
double now = mono_ts();
|
||||||
|
for (int f = 0; f < 2; f++) probe_step(&p, &p.fam[f], f, &sys_ops, now);
|
||||||
|
sleep(round < 2 ? 11 : 0);
|
||||||
|
}
|
||||||
|
char js[512];
|
||||||
|
probe_metric_json(&p, js, sizeof(js));
|
||||||
|
kvdict_t d; kv_clear(&d);
|
||||||
|
probe_flags(&p, &d);
|
||||||
|
printf("%s\np4=%s p6=%s\n", js, kv_get(&d, "p4") ? kv_get(&d, "p4") : "-",
|
||||||
|
kv_get(&d, "p6") ? kv_get(&d, "p6") : "-");
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
int main(int argc, char **argv) {
|
||||||
|
if (argc > 1 && !strcmp(argv[1], "--live")) return live_mode();
|
||||||
|
test_gateway_parsers();
|
||||||
|
test_ping_parser();
|
||||||
|
test_present_with_details();
|
||||||
|
test_no_route_is_absent_and_wastes_no_ping();
|
||||||
|
test_debounce_and_rate_limit();
|
||||||
|
test_first_failure_is_unknown_and_omitted();
|
||||||
|
test_address_or_gateway_change_repings();
|
||||||
|
test_route_loss_then_return_pings_immediately();
|
||||||
|
test_missing_ping_stays_unknown_even_after_route_loss();
|
||||||
|
test_unsupported_family_is_unknown();
|
||||||
|
printf("%d checks, %d failed\n", g_run, g_fail);
|
||||||
|
return g_fail ? 1 : 0;
|
||||||
|
}
|
||||||
+352
-18
@@ -10,14 +10,20 @@ Built-in plugins (always available):
|
|||||||
os_info — OS name, kernel, distro, Python version
|
os_info — OS name, kernel, distro, Python version
|
||||||
ping_monitor — ICMP ping RTT to configured hosts
|
ping_monitor — ICMP ping RTT to configured hosts
|
||||||
nagios_runner — run Nagios-compatible check scripts
|
nagios_runner — run Nagios-compatible check scripts
|
||||||
|
network_monitor — per-interface TX/RX bytes/s (Linux); per-protocol connectivity (all platforms)
|
||||||
|
|
||||||
Linux-only plugins (via /proc, no extra tools needed):
|
Linux-only plugins (via /proc, no extra tools needed):
|
||||||
cpu_monitor — CPU %, load average, core count
|
cpu_monitor — CPU %, load average, core count
|
||||||
memory_monitor — RAM and swap usage
|
memory_monitor — RAM and swap usage
|
||||||
network_monitor — per-interface TX/RX bytes/s
|
|
||||||
|
|
||||||
Cross-platform (via df subprocess):
|
Cross-platform (via df subprocess):
|
||||||
disk_monitor — per-mount usage and capacity
|
disk_monitor — per-mount usage and capacity
|
||||||
|
|
||||||
|
Presence probe (built in): reports per protocol (IPv4/IPv6) whether the host has internet
|
||||||
|
connectivity, as p4/p6 flags on every heartbeat, so hbd stops treating a missing IPv6 (or
|
||||||
|
IPv4) path as an outage. Config, in the network_monitor section of ~/.hbc.json:
|
||||||
|
probe_enabled (default true), probe_v4 (default 1.1.1.1), probe_v6 (default
|
||||||
|
2606:4700:4700::1111; IP literals only), probe_interval (default 60 s)
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import argparse
|
import argparse
|
||||||
@@ -36,12 +42,13 @@ import time
|
|||||||
import zlib
|
import zlib
|
||||||
from abc import ABC, abstractmethod
|
from abc import ABC, abstractmethod
|
||||||
from collections import defaultdict
|
from collections import defaultdict
|
||||||
|
from dataclasses import asdict, dataclass
|
||||||
from logging.handlers import SysLogHandler
|
from logging.handlers import SysLogHandler
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from typing import Any, Dict, List, Optional, Tuple
|
from typing import Any, Awaitable, Callable, Dict, List, Optional, Set, Tuple
|
||||||
|
|
||||||
# updated by scripts/bumpminor.sh
|
# updated by scripts/bumpminor.sh
|
||||||
__version__ = "5.4.4"
|
__version__ = "5.4.5"
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
# Protocol (mirrors hbd/common/proto.py)
|
# Protocol (mirrors hbd/common/proto.py)
|
||||||
@@ -637,6 +644,315 @@ class DiskMonitorPlugin(MonitorPlugin):
|
|||||||
return {"partitions": partitions} if partitions else {}
|
return {"partitions": partitions} if partitions else {}
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Connectivity probe — per-protocol internet presence (IPv4 / IPv6)
|
||||||
|
#
|
||||||
|
# Tier 1 (every tier1_interval s, sends no packets): a UDP connect() to the probe address
|
||||||
|
# shows whether a route exists and which source address would be used. Tier 2 (every
|
||||||
|
# probe_interval s, sooner after a failure or a change of address or gateway): one ICMP echo,
|
||||||
|
# only when tier 1 found a route. A family is absent after no route at all or fail_limit
|
||||||
|
# consecutive failed echoes, present after one success, and unknown (flag omitted) when it
|
||||||
|
# cannot be probed. The result rides on every HTB as p4/p6 and, when network_monitor runs, as
|
||||||
|
# its `connectivity` metric. Mirrors hbd/client/connectivity.py.
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
_FAMILIES = ("ipv4", "ipv6")
|
||||||
|
_FLAG_KEYS = {"ipv4": "p4", "ipv6": "p6"}
|
||||||
|
_DEFAULT_PROBES = {"ipv4": "1.1.1.1", "ipv6": "2606:4700:4700::1111"}
|
||||||
|
_BSD_PLATFORMS = ("darwin", "freebsd", "netbsd", "openbsd", "dragonfly")
|
||||||
|
|
||||||
|
|
||||||
|
class _ProbeUnavailable(Exception):
|
||||||
|
"""The address family cannot be probed on this host (e.g. no IPv6 support)."""
|
||||||
|
|
||||||
|
|
||||||
|
class _PingUnavailable(Exception):
|
||||||
|
"""The system ping binary could not be executed."""
|
||||||
|
|
||||||
|
|
||||||
|
def _parse_proc_route_v4(text: str) -> Optional[str]:
|
||||||
|
"""Default gateway from /proc/net/route (lowest metric wins)."""
|
||||||
|
best: Optional[Tuple[int, str]] = None
|
||||||
|
for line in text.splitlines()[1:]:
|
||||||
|
f = line.split()
|
||||||
|
if len(f) < 8 or f[1] != "00000000" or not int(f[3], 16) & 0x2:
|
||||||
|
continue
|
||||||
|
metric = int(f[6])
|
||||||
|
if best is None or metric < best[0]:
|
||||||
|
# the kernel prints the raw network-order u32 in host order: use the host byte order
|
||||||
|
best = (metric, socket.inet_ntoa(int(f[2], 16).to_bytes(4, sys.byteorder)))
|
||||||
|
return best[1] if best else None
|
||||||
|
|
||||||
|
|
||||||
|
def _parse_proc_route_v6(text: str) -> Optional[str]:
|
||||||
|
"""Default gateway from /proc/net/ipv6_route (lowest metric wins)."""
|
||||||
|
best: Optional[Tuple[int, str]] = None
|
||||||
|
for line in text.splitlines():
|
||||||
|
f = line.split()
|
||||||
|
if len(f) < 10:
|
||||||
|
continue
|
||||||
|
dest, plen, nexthop, metric, flags, iface = f[0], f[1], f[4], f[5], f[8], f[9]
|
||||||
|
if dest != "0" * 32 or plen != "00" or nexthop == "0" * 32 or iface == "lo":
|
||||||
|
continue
|
||||||
|
fl = int(flags, 16)
|
||||||
|
if not fl & 0x2 or fl & 0x200: # need RTF_GATEWAY, skip RTF_REJECT
|
||||||
|
continue
|
||||||
|
m = int(metric, 16)
|
||||||
|
if best is None or m < best[0]:
|
||||||
|
best = (m, socket.inet_ntop(socket.AF_INET6, bytes.fromhex(nexthop)))
|
||||||
|
return best[1] if best else None
|
||||||
|
|
||||||
|
|
||||||
|
def _parse_route_get(text: str) -> Optional[str]:
|
||||||
|
"""Gateway from `route -n get [-inet6] default` (macOS and the BSDs)."""
|
||||||
|
m = re.search(r"^\s*gateway:\s*(\S+)", text, re.MULTILINE)
|
||||||
|
return m.group(1) if m else None
|
||||||
|
|
||||||
|
|
||||||
|
def _parse_windows_route(text: str, family: str) -> Optional[str]:
|
||||||
|
"""Default gateway from `route print -4` / `route print -6` (lowest metric wins)."""
|
||||||
|
best: Optional[Tuple[int, str]] = None
|
||||||
|
for line in text.splitlines():
|
||||||
|
if family == "ipv4":
|
||||||
|
f = line.split()
|
||||||
|
if not (len(f) >= 5 and f[0] == "0.0.0.0" and f[1] == "0.0.0.0" and f[2] != "On-link"):
|
||||||
|
continue
|
||||||
|
cand = (int(f[4]), f[2])
|
||||||
|
else:
|
||||||
|
m = re.match(r"^\s*\d+\s+(\d+)\s+::/0\s+(\S+)\s*$", line)
|
||||||
|
if not m or m.group(2) == "On-link":
|
||||||
|
continue
|
||||||
|
cand = (int(m.group(1)), m.group(2))
|
||||||
|
if best is None or cand[0] < best[0]:
|
||||||
|
best = cand
|
||||||
|
return best[1] if best else None
|
||||||
|
|
||||||
|
|
||||||
|
async def _run_cmd(cmd: List[str], timeout: float = 5.0) -> str:
|
||||||
|
proc = await asyncio.create_subprocess_exec(
|
||||||
|
*cmd, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.DEVNULL)
|
||||||
|
try:
|
||||||
|
out, _ = await asyncio.wait_for(proc.communicate(), timeout=timeout)
|
||||||
|
except asyncio.TimeoutError:
|
||||||
|
proc.kill()
|
||||||
|
await proc.wait()
|
||||||
|
raise
|
||||||
|
return out.decode(errors="replace")
|
||||||
|
|
||||||
|
|
||||||
|
async def _default_gateway(family: str) -> Optional[str]:
|
||||||
|
"""Default gateway for "ipv4"/"ipv6", or None on any failure (informational only)."""
|
||||||
|
v4 = family == "ipv4"
|
||||||
|
try:
|
||||||
|
if sys.platform.startswith("linux"):
|
||||||
|
with open("/proc/net/route" if v4 else "/proc/net/ipv6_route") as fh:
|
||||||
|
text = fh.read()
|
||||||
|
return _parse_proc_route_v4(text) if v4 else _parse_proc_route_v6(text)
|
||||||
|
if sys.platform.startswith(_BSD_PLATFORMS):
|
||||||
|
cmd = ["route", "-n", "get"] + ([] if v4 else ["-inet6"]) + ["default"]
|
||||||
|
return _parse_route_get(await _run_cmd(cmd))
|
||||||
|
if sys.platform == "win32":
|
||||||
|
text = await _run_cmd(["route", "print", "-4" if v4 else "-6"])
|
||||||
|
return _parse_windows_route(text, family)
|
||||||
|
except Exception:
|
||||||
|
return None
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def _probe_ping_command(host: str, family: str) -> List[str]:
|
||||||
|
"""Command for ONE echo request with a ~2 s reply wait."""
|
||||||
|
v4 = family == "ipv4"
|
||||||
|
if sys.platform == "win32":
|
||||||
|
return ["ping", "-n", "1", "-w", "2000", "-4" if v4 else "-6", host]
|
||||||
|
if sys.platform.startswith(_BSD_PLATFORMS):
|
||||||
|
# BSD/macOS -W is in milliseconds, and IPv6 is reached through ping6.
|
||||||
|
return ["ping", "-c", "1", "-W", "2000", host] if v4 else ["ping6", "-c", "1", host]
|
||||||
|
return ["ping", "-c", "1", "-W", "2", "-4" if v4 else "-6", host]
|
||||||
|
|
||||||
|
|
||||||
|
def _parse_ping_rtt(output: str) -> Optional[float]:
|
||||||
|
"""Average RTT in ms from ping output, or None when there was no reply."""
|
||||||
|
m = re.search(r"(?:rtt|round-trip)\s+min/avg/max/\S+\s*=\s*([\d.]+)/([\d.]+)/([\d.]+)", output)
|
||||||
|
if m:
|
||||||
|
return float(m.group(2))
|
||||||
|
if "TTL=" in output: # Windows reply line (also when localized)
|
||||||
|
m = re.search(r"[=<]\s*(\d+)\s*ms", output)
|
||||||
|
return float(m.group(1)) if m else 0.0
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
async def _ping_rtt(target: str, family: str) -> Optional[float]:
|
||||||
|
"""One ICMP echo; RTT in ms, or None if no reply. Raises _PingUnavailable without ping."""
|
||||||
|
try:
|
||||||
|
proc = await asyncio.create_subprocess_exec(
|
||||||
|
*_probe_ping_command(target, family),
|
||||||
|
stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE)
|
||||||
|
except FileNotFoundError as e:
|
||||||
|
raise _PingUnavailable(str(e))
|
||||||
|
try:
|
||||||
|
out, _ = await asyncio.wait_for(proc.communicate(), timeout=6)
|
||||||
|
except asyncio.TimeoutError:
|
||||||
|
proc.kill()
|
||||||
|
await proc.wait()
|
||||||
|
return None
|
||||||
|
return _parse_ping_rtt(out.decode(errors="replace"))
|
||||||
|
|
||||||
|
|
||||||
|
def _local_address(family: str, target: str) -> Optional[str]:
|
||||||
|
"""Source address the kernel would use to reach target, or None if there is no route.
|
||||||
|
|
||||||
|
A UDP connect() sends nothing. Raises _ProbeUnavailable if the family is unsupported.
|
||||||
|
"""
|
||||||
|
af = socket.AF_INET if family == "ipv4" else socket.AF_INET6
|
||||||
|
try:
|
||||||
|
sock = socket.socket(af, socket.SOCK_DGRAM)
|
||||||
|
except OSError as e:
|
||||||
|
raise _ProbeUnavailable(str(e))
|
||||||
|
try:
|
||||||
|
sock.connect((target, 9))
|
||||||
|
return str(sock.getsockname()[0])
|
||||||
|
except OSError:
|
||||||
|
return None
|
||||||
|
finally:
|
||||||
|
sock.close()
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class _FamilyState:
|
||||||
|
present: Optional[bool] = None # None = unknown (probe cannot run yet/at all)
|
||||||
|
addr: Optional[str] = None
|
||||||
|
gateway: Optional[str] = None
|
||||||
|
rtt: Optional[float] = None
|
||||||
|
|
||||||
|
|
||||||
|
class _Connectivity:
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
probes: Optional[Dict[str, str]] = None,
|
||||||
|
probe_interval: float = 60.0,
|
||||||
|
retry_interval: float = 10.0,
|
||||||
|
tier1_interval: float = 5.0,
|
||||||
|
fail_limit: int = 2,
|
||||||
|
local_address_fn: Callable[[str, str], Optional[str]] = _local_address,
|
||||||
|
ping_fn: Callable[[str, str], Awaitable[Optional[float]]] = _ping_rtt,
|
||||||
|
gateway_fn: Callable[[str], Awaitable[Optional[str]]] = _default_gateway,
|
||||||
|
clock: Callable[[], float] = time.monotonic,
|
||||||
|
):
|
||||||
|
self.probes = dict(_DEFAULT_PROBES)
|
||||||
|
self.probes.update(probes or {})
|
||||||
|
self.probe_interval = probe_interval
|
||||||
|
self.retry_interval = retry_interval
|
||||||
|
self.tier1_interval = tier1_interval
|
||||||
|
self.fail_limit = fail_limit
|
||||||
|
self._local_address = local_address_fn
|
||||||
|
self._ping = ping_fn
|
||||||
|
self._gateway_fn = gateway_fn
|
||||||
|
self._clock = clock
|
||||||
|
# Reading the gateway is a cheap /proc read on Linux but a subprocess elsewhere, so
|
||||||
|
# elsewhere it is refreshed only when a tier-2 probe is due.
|
||||||
|
self._cheap_gateway = sys.platform.startswith("linux")
|
||||||
|
self.states: Dict[str, _FamilyState] = {f: _FamilyState() for f in _FAMILIES}
|
||||||
|
self._fails: Dict[str, int] = {f: 0 for f in _FAMILIES}
|
||||||
|
self._next_ping: Dict[str, float] = {f: 0.0 for f in _FAMILIES}
|
||||||
|
self._last_key: Dict[str, Optional[Tuple[Optional[str], Optional[str]]]] = {
|
||||||
|
f: None for f in _FAMILIES}
|
||||||
|
self._ping_disabled: Set[str] = set()
|
||||||
|
|
||||||
|
@classmethod
|
||||||
|
def from_config(cls, cfg: Optional[Dict[str, Any]]) -> "_Connectivity":
|
||||||
|
cfg = cfg or {}
|
||||||
|
probes = {}
|
||||||
|
if cfg.get("probe_v4"):
|
||||||
|
probes["ipv4"] = str(cfg["probe_v4"])
|
||||||
|
if cfg.get("probe_v6"):
|
||||||
|
probes["ipv6"] = str(cfg["probe_v6"])
|
||||||
|
return cls(probes=probes, probe_interval=float(cfg.get("probe_interval", 60)))
|
||||||
|
|
||||||
|
# ---- readers (cheap; safe to call on every heartbeat) ---------------------------
|
||||||
|
def snapshot(self) -> Dict[str, _FamilyState]:
|
||||||
|
return {f: _FamilyState(**asdict(s)) for f, s in self.states.items()}
|
||||||
|
|
||||||
|
def flags(self) -> Dict[str, bool]:
|
||||||
|
"""HTB flags for families whose presence is known."""
|
||||||
|
return {_FLAG_KEYS[f]: s.present for f, s in self.states.items() if s.present is not None}
|
||||||
|
|
||||||
|
def as_metric(self) -> Dict[str, Dict[str, Any]]:
|
||||||
|
return {f: asdict(s) for f, s in self.states.items()}
|
||||||
|
|
||||||
|
# ---- probing ----------------------------------------------------------------------
|
||||||
|
async def refresh(self) -> None:
|
||||||
|
for fam in _FAMILIES:
|
||||||
|
try:
|
||||||
|
await self._refresh_family(fam)
|
||||||
|
except Exception:
|
||||||
|
logging.getLogger("hbc.probe").exception("connectivity probe failed for %s", fam)
|
||||||
|
|
||||||
|
async def run(self) -> None:
|
||||||
|
while _running:
|
||||||
|
await self.refresh()
|
||||||
|
await _sleep(self.tier1_interval)
|
||||||
|
|
||||||
|
async def _gateway(self, fam: str) -> Optional[str]:
|
||||||
|
try:
|
||||||
|
return await self._gateway_fn(fam)
|
||||||
|
except Exception:
|
||||||
|
return None
|
||||||
|
|
||||||
|
async def _refresh_family(self, fam: str) -> None:
|
||||||
|
st = self.states[fam]
|
||||||
|
try:
|
||||||
|
addr = self._local_address(fam, self.probes[fam])
|
||||||
|
except _ProbeUnavailable:
|
||||||
|
st.present = None
|
||||||
|
return
|
||||||
|
|
||||||
|
if addr is None: # tier 1: no route => absent immediately
|
||||||
|
self._fails[fam] = 0
|
||||||
|
self._next_ping[fam] = 0.0
|
||||||
|
self._last_key[fam] = None
|
||||||
|
st.present, st.addr, st.gateway, st.rtt = False, None, None, None
|
||||||
|
return
|
||||||
|
|
||||||
|
now = self._clock()
|
||||||
|
gw = await self._gateway(fam) if self._cheap_gateway else st.gateway
|
||||||
|
key = (addr, gw)
|
||||||
|
due = key != self._last_key[fam] or now >= self._next_ping[fam]
|
||||||
|
if due and not self._cheap_gateway:
|
||||||
|
gw = await self._gateway(fam)
|
||||||
|
key = (addr, gw)
|
||||||
|
st.addr, st.gateway = addr, gw
|
||||||
|
self._last_key[fam] = key
|
||||||
|
if fam in self._ping_disabled:
|
||||||
|
st.present = None # cannot probe: unknown, never absent, while a route exists
|
||||||
|
elif due:
|
||||||
|
await self._tier2(fam, st, now)
|
||||||
|
|
||||||
|
async def _tier2(self, fam: str, st: _FamilyState, now: float) -> None:
|
||||||
|
try:
|
||||||
|
rtt = await self._ping(self.probes[fam], fam)
|
||||||
|
except _PingUnavailable:
|
||||||
|
logging.getLogger("hbc.probe").warning("ping unavailable; %s presence stays unknown", fam)
|
||||||
|
self._ping_disabled.add(fam)
|
||||||
|
st.present = None
|
||||||
|
return
|
||||||
|
except (asyncio.TimeoutError, OSError):
|
||||||
|
rtt = None
|
||||||
|
if rtt is None:
|
||||||
|
self._fails[fam] += 1
|
||||||
|
self._next_ping[fam] = now + self.retry_interval
|
||||||
|
if self._fails[fam] >= self.fail_limit:
|
||||||
|
st.present, st.rtt = False, None
|
||||||
|
else:
|
||||||
|
self._fails[fam] = 0
|
||||||
|
self._next_ping[fam] = now + self.probe_interval
|
||||||
|
st.present, st.rtt = True, rtt
|
||||||
|
|
||||||
|
|
||||||
|
# The running probe (None when probe_enabled is false); shared by the heartbeat sender and
|
||||||
|
# the network_monitor plugin. Set in _async_main.
|
||||||
|
_probe: Optional[_Connectivity] = None
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
# Plugin: network_monitor (Linux /proc/net/dev)
|
# Plugin: network_monitor (Linux /proc/net/dev)
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
@@ -658,7 +974,7 @@ def _read_net_dev() -> Dict[str, Tuple[int, int]]:
|
|||||||
|
|
||||||
class NetworkMonitorPlugin(MonitorPlugin):
|
class NetworkMonitorPlugin(MonitorPlugin):
|
||||||
name = "network_monitor"
|
name = "network_monitor"
|
||||||
description = "Network I/O rates via /proc/net/dev (Linux)"
|
description = "Network I/O rates via /proc/net/dev (Linux) and per-protocol internet connectivity"
|
||||||
interval = 300
|
interval = 300
|
||||||
|
|
||||||
def __init__(self, config: Optional[Dict[str, Any]] = None):
|
def __init__(self, config: Optional[Dict[str, Any]] = None):
|
||||||
@@ -667,19 +983,25 @@ class NetworkMonitorPlugin(MonitorPlugin):
|
|||||||
self.interval = cfg.get("interval", 300)
|
self.interval = cfg.get("interval", 300)
|
||||||
self.skip_ifaces: set = set(cfg.get("skip_interfaces", ["lo"]))
|
self.skip_ifaces: set = set(cfg.get("skip_interfaces", ["lo"]))
|
||||||
self._prev: Optional[Tuple[float, Dict[str, Tuple[int, int]]]] = None
|
self._prev: Optional[Tuple[float, Dict[str, Tuple[int, int]]]] = None
|
||||||
|
self._have_dev = False
|
||||||
|
|
||||||
async def initialize(self) -> bool:
|
async def initialize(self) -> bool:
|
||||||
if platform.system() != "Linux":
|
dev = _read_net_dev() if platform.system() == "Linux" else {}
|
||||||
self.skip_reason = "Linux only (/proc/net/dev not available)"
|
self._have_dev = bool(dev)
|
||||||
return False
|
if not self._have_dev and _probe is None:
|
||||||
dev = _read_net_dev()
|
self.skip_reason = "no /proc/net/dev (Linux only) and the connectivity probe is disabled"
|
||||||
if not dev:
|
|
||||||
self.skip_reason = "/proc/net/dev not readable"
|
|
||||||
return False
|
return False
|
||||||
|
if self._have_dev:
|
||||||
self._prev = (time.time(), dev) # prime first delta
|
self._prev = (time.time(), dev) # prime first delta
|
||||||
return True
|
return True
|
||||||
|
|
||||||
async def _collect_metrics(self) -> Dict[str, Any]:
|
async def _collect_metrics(self) -> Dict[str, Any]:
|
||||||
|
data: Dict[str, Any] = await self._collect_interfaces() if self._have_dev else {}
|
||||||
|
if _probe is not None:
|
||||||
|
data["connectivity"] = _probe.as_metric()
|
||||||
|
return data
|
||||||
|
|
||||||
|
async def _collect_interfaces(self) -> Dict[str, Any]:
|
||||||
now = time.time()
|
now = time.time()
|
||||||
curr = _read_net_dev()
|
curr = _read_net_dev()
|
||||||
if not curr or self._prev is None:
|
if not curr or self._prev is None:
|
||||||
@@ -940,15 +1262,21 @@ async def _handle_update(conn: AsyncConnection):
|
|||||||
# Heartbeat sender
|
# Heartbeat sender
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
async def _heartbeat_sender(conn: AsyncConnection, interval: int):
|
def _build_heartbeat_msg(conn: AsyncConnection, interval: int,
|
||||||
|
probe: Optional[_Connectivity] = None) -> Dict[str, Any]:
|
||||||
|
"""HTB payload; carries p4/p6 presence flags when the probe knows them."""
|
||||||
|
msg: Dict[str, Any] = {"acks": conn.ackcount, "rtt": conn.rtts[-1], "interval": interval}
|
||||||
|
if probe is not None:
|
||||||
|
msg.update(probe.flags())
|
||||||
|
return msg
|
||||||
|
|
||||||
|
|
||||||
|
async def _heartbeat_sender(conn: AsyncConnection, interval: int,
|
||||||
|
probe: Optional[_Connectivity] = None):
|
||||||
log = logging.getLogger("hbc.hb")
|
log = logging.getLogger("hbc.hb")
|
||||||
while _running:
|
while _running:
|
||||||
try:
|
try:
|
||||||
await conn.sendto({
|
await conn.sendto(_build_heartbeat_msg(conn, interval, probe))
|
||||||
"acks": conn.ackcount,
|
|
||||||
"rtt": conn.rtts[-1],
|
|
||||||
"interval": interval,
|
|
||||||
})
|
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
log.error("send: %s", e)
|
log.error("send: %s", e)
|
||||||
await _sleep(interval)
|
await _sleep(interval)
|
||||||
@@ -1056,7 +1384,7 @@ def _reconfigure_syslog(level: int):
|
|||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
async def _async_main(args, cfg: Dict[str, Any]) -> int:
|
async def _async_main(args, cfg: Dict[str, Any]) -> int:
|
||||||
global _running, _shutdown_event, _active_tasks, send_shutdown
|
global _running, _shutdown_event, _active_tasks, send_shutdown, _probe
|
||||||
_running = True
|
_running = True
|
||||||
_shutdown_event = asyncio.Event()
|
_shutdown_event = asyncio.Event()
|
||||||
_active_tasks = []
|
_active_tasks = []
|
||||||
@@ -1113,6 +1441,10 @@ async def _async_main(args, cfg: Dict[str, Any]) -> int:
|
|||||||
c.close()
|
c.close()
|
||||||
return 0
|
return 0
|
||||||
|
|
||||||
|
# Per-protocol internet presence probe (shared with the network_monitor plugin)
|
||||||
|
net_cfg = dict((cfg.get("plugins") or {}).get("network_monitor") or cfg.get("network_monitor") or {})
|
||||||
|
_probe = _Connectivity.from_config(net_cfg) if net_cfg.get("probe_enabled", True) else None
|
||||||
|
|
||||||
plugins = await _load_plugins(cfg)
|
plugins = await _load_plugins(cfg)
|
||||||
|
|
||||||
loop = asyncio.get_event_loop()
|
loop = asyncio.get_event_loop()
|
||||||
@@ -1126,8 +1458,10 @@ async def _async_main(args, cfg: Dict[str, Any]) -> int:
|
|||||||
|
|
||||||
loop.add_signal_handler(signal.SIGHUP, _sighup)
|
loop.add_signal_handler(signal.SIGHUP, _sighup)
|
||||||
|
|
||||||
|
if _probe is not None:
|
||||||
|
_active_tasks.append(asyncio.create_task(_probe.run()))
|
||||||
for conn in connections:
|
for conn in connections:
|
||||||
_active_tasks.append(asyncio.create_task(_heartbeat_sender(conn, interval)))
|
_active_tasks.append(asyncio.create_task(_heartbeat_sender(conn, interval, _probe)))
|
||||||
|
|
||||||
if plugins and connections:
|
if plugins and connections:
|
||||||
_active_tasks.append(asyncio.create_task(_plugin_collector(connections[0], plugins)))
|
_active_tasks.append(asyncio.create_task(_plugin_collector(connections[0], plugins)))
|
||||||
|
|||||||
@@ -0,0 +1,381 @@
|
|||||||
|
"""Connectivity probe in the standalone scripts/hbc_mini.py (stdlib-only single-file client)."""
|
||||||
|
import asyncio
|
||||||
|
import importlib.util
|
||||||
|
import pathlib
|
||||||
|
import sys
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
_PATH = pathlib.Path(__file__).resolve().parents[1] / "scripts" / "hbc_mini.py"
|
||||||
|
_SPEC = importlib.util.spec_from_file_location("hbc_mini", _PATH)
|
||||||
|
mini = importlib.util.module_from_spec(_SPEC)
|
||||||
|
sys.modules["hbc_mini"] = mini # dataclasses resolve their module by name
|
||||||
|
_SPEC.loader.exec_module(mini)
|
||||||
|
|
||||||
|
PROC_V4 = (
|
||||||
|
"Iface\tDestination\tGateway \tFlags\tRefCnt\tUse\tMetric\tMask\t\tMTU\tWindow\tIRTT\n"
|
||||||
|
"wlan0\t00000000\t0100A8C0\t0003\t0\t0\t600\t00000000\t0\t0\t0\n"
|
||||||
|
"eth0\t00000000\t0101A8C0\t0003\t0\t0\t100\t00000000\t0\t0\t0\n"
|
||||||
|
"wlan0\t0000A8C0\t00000000\t0001\t0\t0\t600\t00FFFFFF\t0\t0\t0\n"
|
||||||
|
)
|
||||||
|
PROC_V6 = (
|
||||||
|
"00000000000000000000000000000000 00 00000000000000000000000000000000 00 "
|
||||||
|
"fe800000000000000000000000000001 00000400 00000001 00000000 00000003 wlan0\n"
|
||||||
|
"00000000000000000000000000000000 00 00000000000000000000000000000000 00 "
|
||||||
|
"00000000000000000000000000000000 ffffffff 00000001 00000000 00200200 lo\n"
|
||||||
|
"fe800000000000000000000000000000 40 00000000000000000000000000000000 00 "
|
||||||
|
"00000000000000000000000000000000 00000100 00000001 00000000 00000001 wlan0\n"
|
||||||
|
)
|
||||||
|
ROUTE_GET = """ route to: 0.0.0.0
|
||||||
|
destination: 0.0.0.0
|
||||||
|
mask: 0.0.0.0
|
||||||
|
gateway: 192.0.2.1
|
||||||
|
fib: 0
|
||||||
|
interface: vtnet0
|
||||||
|
"""
|
||||||
|
WIN_V4 = """IPv4 Route Table
|
||||||
|
Active Routes:
|
||||||
|
Network Destination Netmask Gateway Interface Metric
|
||||||
|
0.0.0.0 0.0.0.0 192.168.1.1 192.168.1.50 25
|
||||||
|
0.0.0.0 0.0.0.0 10.0.0.1 10.0.0.9 50
|
||||||
|
127.0.0.0 255.0.0.0 On-link 127.0.0.1 331
|
||||||
|
"""
|
||||||
|
WIN_V6 = """IPv6 Route Table
|
||||||
|
Active Routes:
|
||||||
|
If Metric Network Destination Gateway
|
||||||
|
12 281 ::/0 fe80::1
|
||||||
|
1 331 ::1/128 On-link
|
||||||
|
"""
|
||||||
|
LINUX_PING = """PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
|
||||||
|
64 bytes from 1.1.1.1: icmp_seq=1 ttl=57 time=11.2 ms
|
||||||
|
|
||||||
|
--- 1.1.1.1 ping statistics ---
|
||||||
|
1 packets transmitted, 1 received, 0% packet loss, time 0ms
|
||||||
|
rtt min/avg/max/mdev = 11.234/11.234/11.234/0.000 ms
|
||||||
|
"""
|
||||||
|
FREEBSD_PING = """PING 1.1.1.1 (1.1.1.1): 56 data bytes
|
||||||
|
64 bytes from 1.1.1.1: icmp_seq=0 ttl=59 time=4.006 ms
|
||||||
|
|
||||||
|
--- 1.1.1.1 ping statistics ---
|
||||||
|
1 packets transmitted, 1 packets received, 0.0% packet loss, 1 packets out of wait time
|
||||||
|
round-trip min/avg/max/stddev = 4.006/4.006/4.006/0.000 ms
|
||||||
|
"""
|
||||||
|
WIN_PING = """Pinging 1.1.1.1 with 32 bytes of data:
|
||||||
|
Reply from 1.1.1.1: bytes=32 time=12ms TTL=57
|
||||||
|
|
||||||
|
Ping statistics for 1.1.1.1:
|
||||||
|
Packets: Sent = 1, Received = 1, Lost = 0 (0% loss),
|
||||||
|
"""
|
||||||
|
LOST_PING = """--- 1.1.1.1 ping statistics ---
|
||||||
|
1 packets transmitted, 0 received, 100% packet loss, time 0ms
|
||||||
|
"""
|
||||||
|
|
||||||
|
|
||||||
|
# ---- gateway parsers ---------------------------------------------------------------
|
||||||
|
|
||||||
|
def test_proc_v4_lowest_metric_wins():
|
||||||
|
assert mini._parse_proc_route_v4(PROC_V4) == "192.168.1.1" # eth0, metric 100
|
||||||
|
|
||||||
|
|
||||||
|
def test_proc_v4_none_without_default_route():
|
||||||
|
assert mini._parse_proc_route_v4(PROC_V4.splitlines()[0] + "\n") is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_proc_v6_skips_reject_and_non_default():
|
||||||
|
assert mini._parse_proc_route_v6(PROC_V6) == "fe80::1"
|
||||||
|
|
||||||
|
|
||||||
|
def test_route_get_bsd_and_macos():
|
||||||
|
assert mini._parse_route_get(ROUTE_GET) == "192.0.2.1"
|
||||||
|
assert mini._parse_route_get("route: writing to routing socket: not in table") is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_windows_routes():
|
||||||
|
assert mini._parse_windows_route(WIN_V4, "ipv4") == "192.168.1.1"
|
||||||
|
assert mini._parse_windows_route(WIN_V6, "ipv6") == "fe80::1"
|
||||||
|
|
||||||
|
|
||||||
|
def test_default_gateway_never_raises(monkeypatch):
|
||||||
|
monkeypatch.setattr(sys, "platform", "linux")
|
||||||
|
monkeypatch.setattr(mini, "_parse_proc_route_v4", lambda t: 1 / 0)
|
||||||
|
assert asyncio.run(mini._default_gateway("ipv4")) is None
|
||||||
|
|
||||||
|
|
||||||
|
# ---- ping ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("platform,family,expected", [
|
||||||
|
("linux", "ipv4", ["ping", "-c", "1", "-W", "2", "-4", "h"]),
|
||||||
|
("linux", "ipv6", ["ping", "-c", "1", "-W", "2", "-6", "h"]),
|
||||||
|
("freebsd15", "ipv4", ["ping", "-c", "1", "-W", "2000", "h"]),
|
||||||
|
("freebsd15", "ipv6", ["ping6", "-c", "1", "h"]),
|
||||||
|
("darwin", "ipv4", ["ping", "-c", "1", "-W", "2000", "h"]),
|
||||||
|
("darwin", "ipv6", ["ping6", "-c", "1", "h"]),
|
||||||
|
("win32", "ipv4", ["ping", "-n", "1", "-w", "2000", "-4", "h"]),
|
||||||
|
("win32", "ipv6", ["ping", "-n", "1", "-w", "2000", "-6", "h"]),
|
||||||
|
])
|
||||||
|
def test_probe_ping_command(monkeypatch, platform, family, expected):
|
||||||
|
monkeypatch.setattr(sys, "platform", platform)
|
||||||
|
assert mini._probe_ping_command("h", family) == expected
|
||||||
|
|
||||||
|
|
||||||
|
def test_parse_ping_rtt():
|
||||||
|
assert mini._parse_ping_rtt(LINUX_PING) == 11.234
|
||||||
|
assert mini._parse_ping_rtt(FREEBSD_PING) == 4.006
|
||||||
|
assert mini._parse_ping_rtt(WIN_PING) == 12.0
|
||||||
|
assert mini._parse_ping_rtt(LOST_PING) is None
|
||||||
|
assert mini._parse_ping_rtt("ping: connect: Network is unreachable") is None
|
||||||
|
|
||||||
|
|
||||||
|
# ---- probe state machine ------------------------------------------------------------
|
||||||
|
|
||||||
|
class Net:
|
||||||
|
"""Scriptable network: addr None = no route, 'unavailable' = family unsupported."""
|
||||||
|
|
||||||
|
def __init__(self):
|
||||||
|
self.addr = {"ipv4": "192.0.2.5", "ipv6": "2001:db8::5"}
|
||||||
|
self.gw = {"ipv4": "192.0.2.1", "ipv6": "fe80::1"}
|
||||||
|
self.rtt = {"ipv4": 10.0, "ipv6": 20.0} # None = ping fails, 'unavailable' = no ping binary
|
||||||
|
self.pings = []
|
||||||
|
self.now = 1000.0
|
||||||
|
|
||||||
|
def local(self, fam, target):
|
||||||
|
a = self.addr[fam]
|
||||||
|
if a == "unavailable":
|
||||||
|
raise mini._ProbeUnavailable("no such family")
|
||||||
|
return a
|
||||||
|
|
||||||
|
async def ping(self, target, fam):
|
||||||
|
self.pings.append(fam)
|
||||||
|
r = self.rtt[fam]
|
||||||
|
if r == "unavailable":
|
||||||
|
raise mini._PingUnavailable("no ping")
|
||||||
|
return r
|
||||||
|
|
||||||
|
async def gateway(self, fam):
|
||||||
|
return self.gw[fam]
|
||||||
|
|
||||||
|
def clock(self):
|
||||||
|
return self.now
|
||||||
|
|
||||||
|
|
||||||
|
def make(net, **kw):
|
||||||
|
c = mini._Connectivity(local_address_fn=net.local, ping_fn=net.ping, gateway_fn=net.gateway,
|
||||||
|
clock=net.clock, **kw)
|
||||||
|
c._cheap_gateway = True
|
||||||
|
return c
|
||||||
|
|
||||||
|
|
||||||
|
def refresh(c):
|
||||||
|
asyncio.run(c.refresh())
|
||||||
|
|
||||||
|
|
||||||
|
def test_present_with_addr_gateway_rtt():
|
||||||
|
net = Net()
|
||||||
|
c = make(net)
|
||||||
|
refresh(c)
|
||||||
|
s = c.snapshot()["ipv4"]
|
||||||
|
assert (s.present, s.addr, s.gateway, s.rtt) == (True, "192.0.2.5", "192.0.2.1", 10.0)
|
||||||
|
assert c.flags() == {"p4": True, "p6": True}
|
||||||
|
assert c.as_metric()["ipv6"]["gateway"] == "fe80::1"
|
||||||
|
|
||||||
|
|
||||||
|
def test_no_route_is_immediately_absent():
|
||||||
|
net = Net()
|
||||||
|
net.addr["ipv6"] = None
|
||||||
|
c = make(net)
|
||||||
|
refresh(c)
|
||||||
|
assert c.flags() == {"p4": True, "p6": False}
|
||||||
|
assert net.pings == ["ipv4"]
|
||||||
|
assert c.snapshot()["ipv6"].addr is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_ping_is_debounced_and_rate_limited():
|
||||||
|
net = Net()
|
||||||
|
c = make(net, probe_interval=60.0, retry_interval=10.0)
|
||||||
|
refresh(c)
|
||||||
|
refresh(c)
|
||||||
|
assert net.pings == ["ipv4", "ipv6"]
|
||||||
|
net.now += 60
|
||||||
|
net.rtt["ipv6"] = None
|
||||||
|
refresh(c)
|
||||||
|
assert c.flags()["p6"] is True
|
||||||
|
net.now += 10
|
||||||
|
refresh(c)
|
||||||
|
assert c.flags()["p6"] is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_first_probe_failing_once_is_unknown():
|
||||||
|
net = Net()
|
||||||
|
net.rtt["ipv4"] = None
|
||||||
|
c = make(net)
|
||||||
|
refresh(c)
|
||||||
|
assert "p4" not in c.flags() and c.flags()["p6"] is True
|
||||||
|
|
||||||
|
|
||||||
|
def test_recovers_on_first_success():
|
||||||
|
net = Net()
|
||||||
|
net.rtt["ipv6"] = None
|
||||||
|
c = make(net, retry_interval=10.0)
|
||||||
|
refresh(c)
|
||||||
|
net.now += 10
|
||||||
|
refresh(c)
|
||||||
|
assert c.flags()["p6"] is False
|
||||||
|
net.rtt["ipv6"] = 25.0
|
||||||
|
net.now += 10
|
||||||
|
refresh(c)
|
||||||
|
assert c.flags()["p6"] is True
|
||||||
|
|
||||||
|
|
||||||
|
def test_address_change_invalidates_cache():
|
||||||
|
net = Net()
|
||||||
|
c = make(net)
|
||||||
|
refresh(c)
|
||||||
|
net.addr["ipv4"] = "198.51.100.7"
|
||||||
|
refresh(c)
|
||||||
|
assert net.pings.count("ipv4") == 2
|
||||||
|
assert c.snapshot()["ipv4"].addr == "198.51.100.7"
|
||||||
|
|
||||||
|
|
||||||
|
def test_gateway_change_invalidates_cache():
|
||||||
|
net = Net()
|
||||||
|
c = make(net)
|
||||||
|
refresh(c)
|
||||||
|
net.gw["ipv4"] = "192.0.2.254"
|
||||||
|
refresh(c)
|
||||||
|
assert net.pings.count("ipv4") == 2
|
||||||
|
|
||||||
|
|
||||||
|
def test_route_loss_then_return_pings_immediately():
|
||||||
|
net = Net()
|
||||||
|
c = make(net)
|
||||||
|
refresh(c)
|
||||||
|
net.addr["ipv6"] = None
|
||||||
|
refresh(c)
|
||||||
|
net.addr["ipv6"] = "2001:db8::5"
|
||||||
|
refresh(c)
|
||||||
|
assert net.pings.count("ipv6") == 2 and c.flags()["p6"] is True
|
||||||
|
|
||||||
|
|
||||||
|
def test_missing_ping_binary_leaves_presence_unknown_and_stops_retrying():
|
||||||
|
net = Net()
|
||||||
|
net.rtt["ipv4"] = "unavailable"
|
||||||
|
c = make(net)
|
||||||
|
refresh(c)
|
||||||
|
net.now += 600
|
||||||
|
refresh(c)
|
||||||
|
assert "p4" not in c.flags()
|
||||||
|
assert net.pings.count("ipv4") == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_ping_less_family_stays_unknown_after_route_returns():
|
||||||
|
net = Net()
|
||||||
|
net.rtt["ipv4"] = "unavailable"
|
||||||
|
c = make(net)
|
||||||
|
refresh(c)
|
||||||
|
assert "p4" not in c.flags()
|
||||||
|
net.addr["ipv4"] = None
|
||||||
|
refresh(c)
|
||||||
|
assert c.flags()["p4"] is False # no route is definitive
|
||||||
|
net.addr["ipv4"] = "192.0.2.5"
|
||||||
|
refresh(c)
|
||||||
|
assert "p4" not in c.flags() and net.pings.count("ipv4") == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_unsupported_family_is_unknown():
|
||||||
|
net = Net()
|
||||||
|
net.addr["ipv6"] = "unavailable"
|
||||||
|
c = make(net)
|
||||||
|
refresh(c)
|
||||||
|
assert "p6" not in c.flags() and c.flags()["p4"] is True
|
||||||
|
|
||||||
|
|
||||||
|
def test_from_config():
|
||||||
|
c = mini._Connectivity.from_config({"probe_v4": "9.9.9.9", "probe_interval": 30})
|
||||||
|
assert c.probes["ipv4"] == "9.9.9.9"
|
||||||
|
assert c.probes["ipv6"] == "2606:4700:4700::1111"
|
||||||
|
assert c.probe_interval == 30.0
|
||||||
|
|
||||||
|
|
||||||
|
def test_local_address_on_this_host_never_raises():
|
||||||
|
# Real UDP connect(): either an address string, None (no route) or _ProbeUnavailable.
|
||||||
|
for fam, target in (("ipv4", "1.1.1.1"), ("ipv6", "2606:4700:4700::1111")):
|
||||||
|
try:
|
||||||
|
r = mini._local_address(fam, target)
|
||||||
|
except mini._ProbeUnavailable:
|
||||||
|
continue
|
||||||
|
assert r is None or isinstance(r, str)
|
||||||
|
|
||||||
|
|
||||||
|
# ---- heartbeat message and plugin wiring ---------------------------------------------
|
||||||
|
|
||||||
|
class _Conn:
|
||||||
|
ackcount = 3
|
||||||
|
rtts = [0.0, 12.5]
|
||||||
|
|
||||||
|
|
||||||
|
class _FlagsProbe:
|
||||||
|
def __init__(self, flags):
|
||||||
|
self._flags = flags
|
||||||
|
|
||||||
|
def flags(self):
|
||||||
|
return self._flags
|
||||||
|
|
||||||
|
|
||||||
|
def test_heartbeat_without_probe_is_unchanged():
|
||||||
|
assert mini._build_heartbeat_msg(_Conn(), 60) == {"acks": 3, "rtt": 12.5, "interval": 60}
|
||||||
|
|
||||||
|
|
||||||
|
def test_heartbeat_carries_presence_flags():
|
||||||
|
msg = mini._build_heartbeat_msg(_Conn(), 60, _FlagsProbe({"p4": True, "p6": False}))
|
||||||
|
assert msg == {"acks": 3, "rtt": 12.5, "interval": 60, "p4": True, "p6": False}
|
||||||
|
|
||||||
|
|
||||||
|
def test_heartbeat_omits_unknown_flags():
|
||||||
|
msg = mini._build_heartbeat_msg(_Conn(), 60, _FlagsProbe({}))
|
||||||
|
assert "p4" not in msg and "p6" not in msg
|
||||||
|
|
||||||
|
|
||||||
|
def test_flags_survive_the_wire_encoding():
|
||||||
|
msg = mini._build_heartbeat_msg(_Conn(), 60, _FlagsProbe({"p4": True, "p6": False}))
|
||||||
|
out = mini._stodict(mini._dicttos("HTB", dict(msg, name="h", id=1)))
|
||||||
|
assert out["p4"] == 1 and out["p6"] == 0
|
||||||
|
|
||||||
|
|
||||||
|
class _FakeProbe:
|
||||||
|
def as_metric(self):
|
||||||
|
return {"ipv4": {"present": True, "addr": "192.0.2.5", "gateway": "192.0.2.1", "rtt": 9.0},
|
||||||
|
"ipv6": {"present": False, "addr": None, "gateway": None, "rtt": None}}
|
||||||
|
|
||||||
|
|
||||||
|
def test_plugin_reports_connectivity_alongside_interfaces(monkeypatch):
|
||||||
|
monkeypatch.setattr(mini, "_probe", _FakeProbe())
|
||||||
|
monkeypatch.setattr(mini.platform, "system", lambda: "Linux")
|
||||||
|
plugin = mini.NetworkMonitorPlugin({})
|
||||||
|
assert asyncio.run(plugin.initialize()) is True
|
||||||
|
data = asyncio.run(plugin._collect_metrics())
|
||||||
|
assert data["connectivity"]["ipv4"]["present"] is True
|
||||||
|
assert data["connectivity"]["ipv6"]["present"] is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_plugin_probe_only_mode_on_non_linux(monkeypatch):
|
||||||
|
monkeypatch.setattr(mini, "_probe", _FakeProbe())
|
||||||
|
monkeypatch.setattr(mini.platform, "system", lambda: "FreeBSD")
|
||||||
|
plugin = mini.NetworkMonitorPlugin({})
|
||||||
|
assert asyncio.run(plugin.initialize()) is True
|
||||||
|
assert list(asyncio.run(plugin._collect_metrics())) == ["connectivity"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_plugin_still_skips_on_non_linux_without_probe(monkeypatch):
|
||||||
|
monkeypatch.setattr(mini, "_probe", None)
|
||||||
|
monkeypatch.setattr(mini.platform, "system", lambda: "FreeBSD")
|
||||||
|
plugin = mini.NetworkMonitorPlugin({})
|
||||||
|
assert asyncio.run(plugin.initialize()) is False
|
||||||
|
assert plugin.skip_reason
|
||||||
|
|
||||||
|
|
||||||
|
def test_plugin_omits_connectivity_without_probe(monkeypatch):
|
||||||
|
monkeypatch.setattr(mini, "_probe", None)
|
||||||
|
monkeypatch.setattr(mini.platform, "system", lambda: "Linux")
|
||||||
|
plugin = mini.NetworkMonitorPlugin({})
|
||||||
|
assert asyncio.run(plugin.initialize()) is True
|
||||||
|
assert "connectivity" not in asyncio.run(plugin._collect_metrics())
|
||||||
Reference in New Issue
Block a user