ap: watchdog checks link ground-truth, not just hostapd state
After a USB re-enumeration the rtw89 radio can come back as a fresh DOWN netdev, dropped from br0, while the still-running hostapd never notices and never exits — so Restart=always never fires. hostapd_cli keeps answering state=ENABLED off stale in-memory state, so the watchdog considered the AP healthy and left it dark (observed 2026-07-05: AP down ~16:41 until manual hostapd restart). Add link_healthy(): require the netdev to be operstate=up and still a port of its bridge, alongside the existing state=ENABLED check. That catches the wedged case the control socket can't see. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
877fa14d0c
commit
4117802f69
+41
-8
@@ -8,9 +8,14 @@ exits and systemd restarts it (Restart=always, no start limit) until the interfa
|
|||||||
returns. This daemon is the backstop for the case systemd *can't* see: hostapd stays
|
returns. This daemon is the backstop for the case systemd *can't* see: hostapd stays
|
||||||
running but the radio has wedged and stopped serving (dmesg "timed out to flush queues").
|
running but the radio has wedged and stopped serving (dmesg "timed out to flush queues").
|
||||||
|
|
||||||
Every `interval` seconds it asks hostapd for its real state via the control socket
|
Every `interval` seconds it checks two things: hostapd's self-reported state via the
|
||||||
(hostapd_cli status -> state=ENABLED). If the interface is present but the AP is not
|
control socket (hostapd_cli status -> state=ENABLED), AND the kernel's ground truth for
|
||||||
ENABLED for `fail_threshold` checks in a row, it clears any failed state and restarts
|
the netdev (operstate up + still a port of the bridge). Both matter because they fail
|
||||||
|
independently: hostapd_cli keeps answering state=ENABLED off stale in-memory state after
|
||||||
|
the USB radio is torn down and re-enumerated underneath a still-running hostapd — the
|
||||||
|
netdev is recreated DOWN and dropped from the bridge, but hostapd never noticed and never
|
||||||
|
exited, so Restart=always never fired. The link check catches exactly that. If the AP is
|
||||||
|
unhealthy for `fail_threshold` checks in a row, it clears any failed state and restarts
|
||||||
hostapd. If the interface is simply gone (mid re-enumeration) it waits — there is
|
hostapd. If the interface is simply gone (mid re-enumeration) it waits — there is
|
||||||
nothing to restart onto, and Restart=always reclaims it when it reappears. Stdlib only.
|
nothing to restart onto, and Restart=always reclaims it when it reappears. Stdlib only.
|
||||||
"""
|
"""
|
||||||
@@ -33,11 +38,11 @@ def log(msg, level="info"):
|
|||||||
print(pri + msg, flush=True)
|
print(pri + msg, flush=True)
|
||||||
|
|
||||||
|
|
||||||
def ap_ifname():
|
def _conf_value(key):
|
||||||
"""The AP interface name, read from hostapd.conf so there's one source of truth."""
|
"""Read a `key=value` from hostapd.conf so there's one source of truth."""
|
||||||
try:
|
try:
|
||||||
for line in Path(HOSTAPD_CONF).read_text().splitlines():
|
for line in Path(HOSTAPD_CONF).read_text().splitlines():
|
||||||
m = re.match(r"\s*interface=(\S+)", line)
|
m = re.match(rf"\s*{key}=(\S+)", line)
|
||||||
if m:
|
if m:
|
||||||
return m.group(1)
|
return m.group(1)
|
||||||
except OSError as e:
|
except OSError as e:
|
||||||
@@ -45,10 +50,36 @@ def ap_ifname():
|
|||||||
return None
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def ap_ifname():
|
||||||
|
"""The AP interface name."""
|
||||||
|
return _conf_value("interface")
|
||||||
|
|
||||||
|
|
||||||
|
def ap_bridge():
|
||||||
|
"""The bridge the AP netdev is enslaved to, or None if hostapd isn't bridging."""
|
||||||
|
return _conf_value("bridge")
|
||||||
|
|
||||||
|
|
||||||
def iface_present(ifname):
|
def iface_present(ifname):
|
||||||
return Path(f"/sys/class/net/{ifname}").exists()
|
return Path(f"/sys/class/net/{ifname}").exists()
|
||||||
|
|
||||||
|
|
||||||
|
def link_healthy(ifname, bridge):
|
||||||
|
"""True when the netdev is actually carrying traffic: operationally up and, if
|
||||||
|
hostapd bridges the AP, still a port of that bridge. This is the ground truth
|
||||||
|
hostapd_cli can't see — after a USB re-enumeration the radio comes back as a fresh
|
||||||
|
DOWN netdev outside the bridge while a stale hostapd still reports state=ENABLED."""
|
||||||
|
try:
|
||||||
|
operstate = Path(f"/sys/class/net/{ifname}/operstate").read_text().strip()
|
||||||
|
except OSError:
|
||||||
|
return False
|
||||||
|
if operstate != "up":
|
||||||
|
return False
|
||||||
|
if bridge and not Path(f"/sys/class/net/{bridge}/brif/{ifname}").exists():
|
||||||
|
return False
|
||||||
|
return True
|
||||||
|
|
||||||
|
|
||||||
def ap_enabled(ifname):
|
def ap_enabled(ifname):
|
||||||
"""True if hostapd reports the AP as beaconing (state=ENABLED). False if it's
|
"""True if hostapd reports the AP as beaconing (state=ENABLED). False if it's
|
||||||
running but not enabled; None if the control socket is unreachable (hostapd down)."""
|
running but not enabled; None if the control socket is unreachable (hostapd down)."""
|
||||||
@@ -83,7 +114,9 @@ def main():
|
|||||||
if not ifname:
|
if not ifname:
|
||||||
log("no interface= in hostapd.conf; nothing to watch", "crit")
|
log("no interface= in hostapd.conf; nothing to watch", "crit")
|
||||||
sys.exit(1)
|
sys.exit(1)
|
||||||
log(f"van-ap-watchdog up: watching {ifname} every {INTERVAL}s "
|
bridge = ap_bridge()
|
||||||
|
log(f"van-ap-watchdog up: watching {ifname}"
|
||||||
|
f"{f' on {bridge}' if bridge else ''} every {INTERVAL}s "
|
||||||
f"(restart after {FAIL_THRESHOLD} bad checks)")
|
f"(restart after {FAIL_THRESHOLD} bad checks)")
|
||||||
|
|
||||||
bad = 0
|
bad = 0
|
||||||
@@ -97,7 +130,7 @@ def main():
|
|||||||
bad = 0
|
bad = 0
|
||||||
else:
|
else:
|
||||||
waiting = False
|
waiting = False
|
||||||
if ap_enabled(ifname):
|
if ap_enabled(ifname) and link_healthy(ifname, bridge):
|
||||||
if bad:
|
if bad:
|
||||||
log(f"AP {ifname} beaconing again")
|
log(f"AP {ifname} beaconing again")
|
||||||
bad = 0
|
bad = 0
|
||||||
|
|||||||
Reference in New Issue
Block a user