From 877fa14d0c922d91374f84eb52db4480085ef2f8 Mon Sep 17 00:00:00 2001 From: Andreas Wrede Date: Wed, 1 Jul 2026 14:36:14 -0400 Subject: [PATCH] ap: keep AP beaconing across USB re-enumeration Starlink's USB ethernet shares a hub with the rtw89 Wi-Fi adapter; when Starlink flaps, the hub re-enumerates and tears the radio down, so hostapd loses its interface and exits. Two stock defaults left the AP dark: Restart=on-failure misses clean exits, and the 5-in-10s start limit makes systemd give up during a re-enumeration storm. - hostapd-restart.conf drop-in: Restart=always, RestartSec=5, StartLimitIntervalSec=0 so hostapd retries forever until the interface returns (verified: SIGKILL -> back to state=ENABLED in ~7s). - van-ap-watchdog daemon: backstop for the case systemd can't see (hostapd running but radio wedged). Polls hostapd_cli status; restarts hostapd if the iface is present but not ENABLED, waits it out if the iface is mid-re-enumeration. - hostapd.conf: add ctrl_interface so the watchdog can read the real AP state. Co-Authored-By: Claude Opus 4.8 --- README.md | 6 +- ap/hostapd-restart.conf | 20 +++++++ ap/hostapd.conf | 4 ++ ap/van-ap-watchdog | 116 +++++++++++++++++++++++++++++++++++++ ap/van-ap-watchdog.service | 13 +++++ deploy.sh | 7 ++- 6 files changed, 164 insertions(+), 2 deletions(-) create mode 100644 ap/hostapd-restart.conf create mode 100644 ap/van-ap-watchdog create mode 100644 ap/van-ap-watchdog.service diff --git a/README.md b/README.md index a47ccb8..d4bc19b 100644 --- a/README.md +++ b/README.md @@ -37,6 +37,7 @@ This directory is the source of truth. The live system files live under `/etc`, | WAN interfaces (eth / wifi / gsm), DHCP-client, metrics | **NetworkManager** | | LAN bridge `br0` + IP (10.42.0.1) + wired LAN member | **systemd-networkd** (`2x-van-br0/lan`) | | AP beaconing / WPA + adding the wlan to `br0` | **hostapd** (`bridge=br0`; AP iface + wired port are NM-*unmanaged*) | +| Keep the AP beaconing across USB re-enumeration (Starlink flap) | **hostapd** `Restart=always`/no start-limit drop-in + **van-ap-watchdog** daemon (recovers a wedged radio) | | AP DHCP + DNS | **dnsmasq** (dedicated instance, bound to AP only) | | NAT + forwarding | **nftables** + sysctl | | WAN health + failover | **van-failover** daemon | @@ -64,7 +65,10 @@ This directory is the source of truth. The live system files live under `/etc`, ### `ap/` — access point | file | → installs to | purpose | |---|---|---| -| `hostapd.conf` | `/etc/hostapd/hostapd.conf` | AP: SSID `VanLink`, ch149, VHT80/HE, WPA2-PSK, country CA. **WPA passphrase lives here.** | +| `hostapd.conf` | `/etc/hostapd/hostapd.conf` | AP: SSID `VanLink`, ch149, VHT80/HE, WPA2-PSK, country CA, `ctrl_interface` for the watchdog. **WPA passphrase lives here.** | +| `hostapd-restart.conf` | `/etc/systemd/system/hostapd.service.d/restart.conf` | `Restart=always` + `StartLimitIntervalSec=0` so hostapd never gives up after a USB re-enumeration bounces the radio | +| `van-ap-watchdog` | `/usr/local/sbin/van-ap-watchdog` | watches `hostapd_cli status`; restarts hostapd if the radio wedges (running but not `ENABLED`) | +| `van-ap-watchdog.service` | `/etc/systemd/system/van-ap-watchdog.service` | runs the watchdog | | `default-hostapd` | `/etc/default/hostapd` | `DAEMON_CONF=...` | | `van-ap-dnsmasq.conf` | `/etc/van-ap/dnsmasq.conf` | DHCP/DNS bound to AP iface (`bind-dynamic`, so it does not clash with systemd-resolved) | | `van-ap-dnsmasq.service` | `/etc/systemd/system/van-ap-dnsmasq.service` | dedicated dnsmasq unit (uses the `dnsmasq-base` binary; the distro dnsmasq service is NOT used) | diff --git a/ap/hostapd-restart.conf b/ap/hostapd-restart.conf new file mode 100644 index 0000000..afc948d --- /dev/null +++ b/ap/hostapd-restart.conf @@ -0,0 +1,20 @@ +# Drop-in: /etc/systemd/system/hostapd.service.d/restart.conf +# +# The AP wlan (rtw89, USB) shares a USB hub with the Starlink ethernet (r8152, USB). +# When Starlink flaps, the hub re-enumerates and the Wi-Fi radio is torn down with it: +# hostapd's interface vanishes ("No such device") and it exits. Two stock defaults then +# leave the AP dark until someone intervenes — this drop-in closes both: +# +# 1. Restart=on-failure ignores a *clean* exit, but hostapd exits 0 in some of these +# cases. Restart=always covers every exit. +# 2. The default start limit (5 starts / 10s) is exhausted by a re-enumeration storm, +# after which systemd gives up permanently. StartLimitIntervalSec=0 disables the +# limit so hostapd keeps retrying until the interface comes back. +[Unit] +StartLimitIntervalSec=0 + +[Service] +Restart=always +# A few seconds' backoff so the USB device has a chance to re-enumerate before we retry +# (retrying instantly just fails on "No such device" and spins). +RestartSec=5 diff --git a/ap/hostapd.conf b/ap/hostapd.conf index 04e27e5..a2f9837 100644 --- a/ap/hostapd.conf +++ b/ap/hostapd.conf @@ -3,6 +3,10 @@ interface=wlxc83a35a4ee55 # hostapd adds the wlan to the bridge after setting AP mode; the bridge itself + its # wired member + the gateway IP are defined under /etc/systemd/network (2x-van-br0/lan). bridge=br0 +# Control socket so van-ap-watchdog (and hostapd_cli) can read the AP's real state +# (state=ENABLED means it's actually beaconing) — used to detect a wedged radio that +# stays "running" but stops serving after a USB glitch. +ctrl_interface=/var/run/hostapd driver=nl80211 ssid=VanLink country_code=CA diff --git a/ap/van-ap-watchdog b/ap/van-ap-watchdog new file mode 100644 index 0000000..66a6be8 --- /dev/null +++ b/ap/van-ap-watchdog @@ -0,0 +1,116 @@ +#!/usr/bin/env python3 +"""van-ap-watchdog — keep the VanLink AP beaconing. + +The AP wlan (rtw89, USB) shares a USB hub with the Starlink ethernet (r8152, USB). +When Starlink's link flaps the hub can re-enumerate and take the Wi-Fi radio down +with it. The hostapd.service drop-in (restart.conf) handles the common case — hostapd +exits and systemd restarts it (Restart=always, no start limit) until the interface +returns. This daemon is the backstop for the case systemd *can't* see: hostapd stays +running but the radio has wedged and stopped serving (dmesg "timed out to flush queues"). + +Every `interval` seconds it asks hostapd for its real state via the control socket +(hostapd_cli status -> state=ENABLED). If the interface is present but the AP is not +ENABLED for `fail_threshold` checks in a row, it clears any failed state and restarts +hostapd. If the interface is simply gone (mid re-enumeration) it waits — there is +nothing to restart onto, and Restart=always reclaims it when it reappears. Stdlib only. +""" + +import re +import subprocess +import sys +import time +from pathlib import Path + +HOSTAPD_CONF = "/etc/hostapd/hostapd.conf" +CTRL_DIR = "/var/run/hostapd" +INTERVAL = 15 # seconds between health checks (backstop cadence, not first response) +FAIL_THRESHOLD = 2 # consecutive bad checks before restarting (debounces re-enum blips) + + +def log(msg, level="info"): + # systemd journal severity prefixes (sd-daemon), same convention as van-thermal. + pri = {"info": "<6>", "warn": "<4>", "crit": "<2>"}.get(level, "<6>") + print(pri + msg, flush=True) + + +def ap_ifname(): + """The AP interface name, read from hostapd.conf so there's one source of truth.""" + try: + for line in Path(HOSTAPD_CONF).read_text().splitlines(): + m = re.match(r"\s*interface=(\S+)", line) + if m: + return m.group(1) + except OSError as e: + log(f"cannot read {HOSTAPD_CONF} ({e})", "crit") + return None + + +def iface_present(ifname): + return Path(f"/sys/class/net/{ifname}").exists() + + +def ap_enabled(ifname): + """True if hostapd reports the AP as beaconing (state=ENABLED). False if it's + running but not enabled; None if the control socket is unreachable (hostapd down).""" + try: + out = subprocess.run( + ["hostapd_cli", "-p", CTRL_DIR, "-i", ifname, "status"], + capture_output=True, text=True, timeout=10, + ).stdout + except (OSError, subprocess.SubprocessError) as e: + log(f"hostapd_cli failed: {e}", "warn") + return None + for line in out.splitlines(): + if line.startswith("state="): + return line.strip() == "state=ENABLED" + return None # no state line -> couldn't talk to hostapd + + +def recover(ifname): + log(f"AP {ifname} not beaconing — clearing failed state and restarting hostapd", "warn") + # reset-failed first in case some other path parked the unit in a failed state; + # harmless when it isn't. + subprocess.run(["systemctl", "reset-failed", "hostapd"], + capture_output=True, text=True) + r = subprocess.run(["systemctl", "restart", "hostapd"], + capture_output=True, text=True) + if r.returncode != 0: + log(f"hostapd restart returned {r.returncode}: {r.stderr.strip()}", "warn") + + +def main(): + ifname = ap_ifname() + if not ifname: + log("no interface= in hostapd.conf; nothing to watch", "crit") + sys.exit(1) + log(f"van-ap-watchdog up: watching {ifname} every {INTERVAL}s " + f"(restart after {FAIL_THRESHOLD} bad checks)") + + bad = 0 + waiting = False # latch so "interface absent" logs once, not every tick + while True: + if not iface_present(ifname): + if not waiting: + log(f"{ifname} absent — USB re-enumeration in progress; " + f"waiting for it to return (hostapd Restart=always will reclaim it)", "warn") + waiting = True + bad = 0 + else: + waiting = False + if ap_enabled(ifname): + if bad: + log(f"AP {ifname} beaconing again") + bad = 0 + else: + bad += 1 + if bad >= FAIL_THRESHOLD: + recover(ifname) + bad = 0 + time.sleep(INTERVAL) + + +if __name__ == "__main__": + try: + main() + except KeyboardInterrupt: + sys.exit(0) diff --git a/ap/van-ap-watchdog.service b/ap/van-ap-watchdog.service new file mode 100644 index 0000000..d98c401 --- /dev/null +++ b/ap/van-ap-watchdog.service @@ -0,0 +1,13 @@ +[Unit] +Description=VanLink AP watchdog — recover hostapd if the radio wedges after a USB glitch +After=hostapd.service +Wants=hostapd.service + +[Service] +Type=simple +ExecStart=/usr/local/sbin/van-ap-watchdog +Restart=always +RestartSec=5 + +[Install] +WantedBy=multi-user.target diff --git a/deploy.sh b/deploy.sh index 62da8f8..fa5e66a 100755 --- a/deploy.sh +++ b/deploy.sh @@ -9,7 +9,10 @@ cd "$(dirname "$(readlink -f "$0")")" echo "== access point ==" install -D -m0644 ap/hostapd.conf /etc/hostapd/hostapd.conf +install -D -m0644 ap/hostapd-restart.conf /etc/systemd/system/hostapd.service.d/restart.conf install -D -m0644 ap/default-hostapd /etc/default/hostapd +install -D -m0755 ap/van-ap-watchdog /usr/local/sbin/van-ap-watchdog +install -D -m0644 ap/van-ap-watchdog.service /etc/systemd/system/van-ap-watchdog.service install -D -m0644 ap/van-ap-dnsmasq.conf /etc/van-ap/dnsmasq.conf install -D -m0644 ap/van-ap-dnsmasq.service /etc/systemd/system/van-ap-dnsmasq.service install -D -m0644 ap/10-van-ap.network /etc/systemd/network/10-van-ap.network @@ -82,7 +85,7 @@ systemctl mask systemd-networkd-wait-online.service >/dev/null 2>&1 || true # pick up the lid drop-in (re-execs logind; does NOT drop the network) systemctl restart systemd-logind >/dev/null 2>&1 || true systemctl unmask hostapd >/dev/null 2>&1 || true -systemctl enable regdomain.service hostapd van-ap-dnsmasq nftables systemd-networkd van-failover van-thermal van-battery >/dev/null 2>&1 || true +systemctl enable regdomain.service hostapd van-ap-dnsmasq nftables systemd-networkd van-failover van-thermal van-battery van-ap-watchdog >/dev/null 2>&1 || true systemctl restart van-thermal van-battery # Heartbeat: only enable/start once the client binary is installed (README §4). if [ -x /home/andreas/bin/hbc ]; then @@ -96,6 +99,8 @@ nmcli general reload 2>/dev/null || systemctl reload NetworkManager 2>/dev/null # the wlan to br0, then dnsmasq binds br0, then NAT/failover systemctl restart systemd-networkd systemctl restart hostapd van-ap-dnsmasq nftables van-failover +# AP watchdog last, after hostapd is back up (it only ever restarts a wedged hostapd) +systemctl restart van-ap-watchdog networkctl reload 2>/dev/null || true echo