ap: watchdog checks link ground-truth, not just hostapd state

After a USB re-enumeration the rtw89 radio can come back as a fresh DOWN
netdev, dropped from br0, while the still-running hostapd never notices and
never exits — so Restart=always never fires. hostapd_cli keeps answering
state=ENABLED off stale in-memory state, so the watchdog considered the AP
healthy and left it dark (observed 2026-07-05: AP down ~16:41 until manual
hostapd restart).

Add link_healthy(): require the netdev to be operstate=up and still a port of
its bridge, alongside the existing state=ENABLED check. That catches the wedged
case the control socket can't see.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Andreas Wrede
2026-07-05 17:56:12 -04:00
co-authored by Claude Opus 4.8
parent 877fa14d0c
commit 4117802f69
+41 -8
View File
@@ -8,9 +8,14 @@ exits and systemd restarts it (Restart=always, no start limit) until the interfa
returns. This daemon is the backstop for the case systemd *can't* see: hostapd stays returns. This daemon is the backstop for the case systemd *can't* see: hostapd stays
running but the radio has wedged and stopped serving (dmesg "timed out to flush queues"). running but the radio has wedged and stopped serving (dmesg "timed out to flush queues").
Every `interval` seconds it asks hostapd for its real state via the control socket Every `interval` seconds it checks two things: hostapd's self-reported state via the
(hostapd_cli status -> state=ENABLED). If the interface is present but the AP is not control socket (hostapd_cli status -> state=ENABLED), AND the kernel's ground truth for
ENABLED for `fail_threshold` checks in a row, it clears any failed state and restarts the netdev (operstate up + still a port of the bridge). Both matter because they fail
independently: hostapd_cli keeps answering state=ENABLED off stale in-memory state after
the USB radio is torn down and re-enumerated underneath a still-running hostapd — the
netdev is recreated DOWN and dropped from the bridge, but hostapd never noticed and never
exited, so Restart=always never fired. The link check catches exactly that. If the AP is
unhealthy for `fail_threshold` checks in a row, it clears any failed state and restarts
hostapd. If the interface is simply gone (mid re-enumeration) it waits — there is hostapd. If the interface is simply gone (mid re-enumeration) it waits — there is
nothing to restart onto, and Restart=always reclaims it when it reappears. Stdlib only. nothing to restart onto, and Restart=always reclaims it when it reappears. Stdlib only.
""" """
@@ -33,11 +38,11 @@ def log(msg, level="info"):
print(pri + msg, flush=True) print(pri + msg, flush=True)
def ap_ifname(): def _conf_value(key):
"""The AP interface name, read from hostapd.conf so there's one source of truth.""" """Read a `key=value` from hostapd.conf so there's one source of truth."""
try: try:
for line in Path(HOSTAPD_CONF).read_text().splitlines(): for line in Path(HOSTAPD_CONF).read_text().splitlines():
m = re.match(r"\s*interface=(\S+)", line) m = re.match(rf"\s*{key}=(\S+)", line)
if m: if m:
return m.group(1) return m.group(1)
except OSError as e: except OSError as e:
@@ -45,10 +50,36 @@ def ap_ifname():
return None return None
def ap_ifname():
"""The AP interface name."""
return _conf_value("interface")
def ap_bridge():
"""The bridge the AP netdev is enslaved to, or None if hostapd isn't bridging."""
return _conf_value("bridge")
def iface_present(ifname): def iface_present(ifname):
return Path(f"/sys/class/net/{ifname}").exists() return Path(f"/sys/class/net/{ifname}").exists()
def link_healthy(ifname, bridge):
"""True when the netdev is actually carrying traffic: operationally up and, if
hostapd bridges the AP, still a port of that bridge. This is the ground truth
hostapd_cli can't see — after a USB re-enumeration the radio comes back as a fresh
DOWN netdev outside the bridge while a stale hostapd still reports state=ENABLED."""
try:
operstate = Path(f"/sys/class/net/{ifname}/operstate").read_text().strip()
except OSError:
return False
if operstate != "up":
return False
if bridge and not Path(f"/sys/class/net/{bridge}/brif/{ifname}").exists():
return False
return True
def ap_enabled(ifname): def ap_enabled(ifname):
"""True if hostapd reports the AP as beaconing (state=ENABLED). False if it's """True if hostapd reports the AP as beaconing (state=ENABLED). False if it's
running but not enabled; None if the control socket is unreachable (hostapd down).""" running but not enabled; None if the control socket is unreachable (hostapd down)."""
@@ -83,7 +114,9 @@ def main():
if not ifname: if not ifname:
log("no interface= in hostapd.conf; nothing to watch", "crit") log("no interface= in hostapd.conf; nothing to watch", "crit")
sys.exit(1) sys.exit(1)
log(f"van-ap-watchdog up: watching {ifname} every {INTERVAL}s " bridge = ap_bridge()
log(f"van-ap-watchdog up: watching {ifname}"
f"{f' on {bridge}' if bridge else ''} every {INTERVAL}s "
f"(restart after {FAIL_THRESHOLD} bad checks)") f"(restart after {FAIL_THRESHOLD} bad checks)")
bad = 0 bad = 0
@@ -97,7 +130,7 @@ def main():
bad = 0 bad = 0
else: else:
waiting = False waiting = False
if ap_enabled(ifname): if ap_enabled(ifname) and link_healthy(ifname, bridge):
if bad: if bad:
log(f"AP {ifname} beaconing again") log(f"AP {ifname} beaconing again")
bad = 0 bad = 0