ap: keep AP beaconing across USB re-enumeration

Starlink's USB ethernet shares a hub with the rtw89 Wi-Fi adapter; when Starlink
flaps, the hub re-enumerates and tears the radio down, so hostapd loses its
interface and exits. Two stock defaults left the AP dark: Restart=on-failure
misses clean exits, and the 5-in-10s start limit makes systemd give up during a
re-enumeration storm.

- hostapd-restart.conf drop-in: Restart=always, RestartSec=5, StartLimitIntervalSec=0
  so hostapd retries forever until the interface returns (verified: SIGKILL -> back
  to state=ENABLED in ~7s).
- van-ap-watchdog daemon: backstop for the case systemd can't see (hostapd running
  but radio wedged). Polls hostapd_cli status; restarts hostapd if the iface is
  present but not ENABLED, waits it out if the iface is mid-re-enumeration.
- hostapd.conf: add ctrl_interface so the watchdog can read the real AP state.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

README.md

@@ -37,6 +37,7 @@ This directory is the source of truth. The live system files live under `/etc`,
 | WAN interfaces (eth / wifi / gsm), DHCP-client, metrics | **NetworkManager** |
 | LAN bridge `br0` + IP (10.42.0.1) + wired LAN member | **systemd-networkd** (`2x-van-br0/lan`) |
 | AP beaconing / WPA + adding the wlan to `br0` | **hostapd** (`bridge=br0`; AP iface + wired port are NM-*unmanaged*) |
+| Keep the AP beaconing across USB re-enumeration (Starlink flap) | **hostapd** `Restart=always`/no start-limit drop-in + **van-ap-watchdog** daemon (recovers a wedged radio) |
 | AP DHCP + DNS | **dnsmasq** (dedicated instance, bound to AP only) |
 | NAT + forwarding | **nftables** + sysctl |
 | WAN health + failover | **van-failover** daemon |
@@ -64,7 +65,10 @@ This directory is the source of truth. The live system files live under `/etc`,
 ### `ap/` — access point
 | file | → installs to | purpose |
 |---|---|---|
-| `hostapd.conf` | `/etc/hostapd/hostapd.conf` | AP: SSID `VanLink`, ch149, VHT80/HE, WPA2-PSK, country CA. **WPA passphrase lives here.** |
+| `hostapd.conf` | `/etc/hostapd/hostapd.conf` | AP: SSID `VanLink`, ch149, VHT80/HE, WPA2-PSK, country CA, `ctrl_interface` for the watchdog. **WPA passphrase lives here.** |
+| `hostapd-restart.conf` | `/etc/systemd/system/hostapd.service.d/restart.conf` | `Restart=always` + `StartLimitIntervalSec=0` so hostapd never gives up after a USB re-enumeration bounces the radio |
+| `van-ap-watchdog` | `/usr/local/sbin/van-ap-watchdog` | watches `hostapd_cli status`; restarts hostapd if the radio wedges (running but not `ENABLED`) |
+| `van-ap-watchdog.service` | `/etc/systemd/system/van-ap-watchdog.service` | runs the watchdog |
 | `default-hostapd` | `/etc/default/hostapd` | `DAEMON_CONF=...` |
 | `van-ap-dnsmasq.conf` | `/etc/van-ap/dnsmasq.conf` | DHCP/DNS bound to AP iface (`bind-dynamic`, so it does not clash with systemd-resolved) |
 | `van-ap-dnsmasq.service` | `/etc/systemd/system/van-ap-dnsmasq.service` | dedicated dnsmasq unit (uses the `dnsmasq-base` binary; the distro dnsmasq service is NOT used) |

ap/hostapd-restart.conf

@@ -0,0 +1,20 @@
+# Drop-in: /etc/systemd/system/hostapd.service.d/restart.conf
+#
+# The AP wlan (rtw89, USB) shares a USB hub with the Starlink ethernet (r8152, USB).
+# When Starlink flaps, the hub re-enumerates and the Wi-Fi radio is torn down with it:
+# hostapd's interface vanishes ("No such device") and it exits. Two stock defaults then
+# leave the AP dark until someone intervenes — this drop-in closes both:
+#
+#   1. Restart=on-failure ignores a *clean* exit, but hostapd exits 0 in some of these
+#      cases. Restart=always covers every exit.
+#   2. The default start limit (5 starts / 10s) is exhausted by a re-enumeration storm,
+#      after which systemd gives up permanently. StartLimitIntervalSec=0 disables the
+#      limit so hostapd keeps retrying until the interface comes back.
+[Unit]
+StartLimitIntervalSec=0
+
+[Service]
+Restart=always
+# A few seconds' backoff so the USB device has a chance to re-enumerate before we retry
+# (retrying instantly just fails on "No such device" and spins).
+RestartSec=5

ap/hostapd.conf

@@ -3,6 +3,10 @@ interface=wlxc83a35a4ee55
 # hostapd adds the wlan to the bridge after setting AP mode; the bridge itself + its
 # wired member + the gateway IP are defined under /etc/systemd/network (2x-van-br0/lan).
 bridge=br0
+# Control socket so van-ap-watchdog (and hostapd_cli) can read the AP's real state
+# (state=ENABLED means it's actually beaconing) — used to detect a wedged radio that
+# stays "running" but stops serving after a USB glitch.
+ctrl_interface=/var/run/hostapd
 driver=nl80211
 ssid=VanLink
 country_code=CA

ap/van-ap-watchdog

@@ -0,0 +1,116 @@
+#!/usr/bin/env python3
+"""van-ap-watchdog — keep the VanLink AP beaconing.
+
+The AP wlan (rtw89, USB) shares a USB hub with the Starlink ethernet (r8152, USB).
+When Starlink's link flaps the hub can re-enumerate and take the Wi-Fi radio down
+with it. The hostapd.service drop-in (restart.conf) handles the common case — hostapd
+exits and systemd restarts it (Restart=always, no start limit) until the interface
+returns. This daemon is the backstop for the case systemd *can't* see: hostapd stays
+running but the radio has wedged and stopped serving (dmesg "timed out to flush queues").
+
+Every `interval` seconds it asks hostapd for its real state via the control socket
+(hostapd_cli status -> state=ENABLED). If the interface is present but the AP is not
+ENABLED for `fail_threshold` checks in a row, it clears any failed state and restarts
+hostapd. If the interface is simply gone (mid re-enumeration) it waits — there is
+nothing to restart onto, and Restart=always reclaims it when it reappears. Stdlib only.
+"""
+
+import re
+import subprocess
+import sys
+import time
+from pathlib import Path
+
+HOSTAPD_CONF = "/etc/hostapd/hostapd.conf"
+CTRL_DIR = "/var/run/hostapd"
+INTERVAL = 15          # seconds between health checks (backstop cadence, not first response)
+FAIL_THRESHOLD = 2     # consecutive bad checks before restarting (debounces re-enum blips)
+
+
+def log(msg, level="info"):
+    # systemd journal severity prefixes (sd-daemon), same convention as van-thermal.
+    pri = {"info": "<6>", "warn": "<4>", "crit": "<2>"}.get(level, "<6>")
+    print(pri + msg, flush=True)
+
+
+def ap_ifname():
+    """The AP interface name, read from hostapd.conf so there's one source of truth."""
+    try:
+        for line in Path(HOSTAPD_CONF).read_text().splitlines():
+            m = re.match(r"\s*interface=(\S+)", line)
+            if m:
+                return m.group(1)
+    except OSError as e:
+        log(f"cannot read {HOSTAPD_CONF} ({e})", "crit")
+    return None
+
+
+def iface_present(ifname):
+    return Path(f"/sys/class/net/{ifname}").exists()
+
+
+def ap_enabled(ifname):
+    """True if hostapd reports the AP as beaconing (state=ENABLED). False if it's
+    running but not enabled; None if the control socket is unreachable (hostapd down)."""
+    try:
+        out = subprocess.run(
+            ["hostapd_cli", "-p", CTRL_DIR, "-i", ifname, "status"],
+            capture_output=True, text=True, timeout=10,
+        ).stdout
+    except (OSError, subprocess.SubprocessError) as e:
+        log(f"hostapd_cli failed: {e}", "warn")
+        return None
+    for line in out.splitlines():
+        if line.startswith("state="):
+            return line.strip() == "state=ENABLED"
+    return None   # no state line -> couldn't talk to hostapd
+
+
+def recover(ifname):
+    log(f"AP {ifname} not beaconing — clearing failed state and restarting hostapd", "warn")
+    # reset-failed first in case some other path parked the unit in a failed state;
+    # harmless when it isn't.
+    subprocess.run(["systemctl", "reset-failed", "hostapd"],
+                   capture_output=True, text=True)
+    r = subprocess.run(["systemctl", "restart", "hostapd"],
+                       capture_output=True, text=True)
+    if r.returncode != 0:
+        log(f"hostapd restart returned {r.returncode}: {r.stderr.strip()}", "warn")
+
+
+def main():
+    ifname = ap_ifname()
+    if not ifname:
+        log("no interface= in hostapd.conf; nothing to watch", "crit")
+        sys.exit(1)
+    log(f"van-ap-watchdog up: watching {ifname} every {INTERVAL}s "
+        f"(restart after {FAIL_THRESHOLD} bad checks)")
+
+    bad = 0
+    waiting = False   # latch so "interface absent" logs once, not every tick
+    while True:
+        if not iface_present(ifname):
+            if not waiting:
+                log(f"{ifname} absent — USB re-enumeration in progress; "
+                    f"waiting for it to return (hostapd Restart=always will reclaim it)", "warn")
+                waiting = True
+            bad = 0
+        else:
+            waiting = False
+            if ap_enabled(ifname):
+                if bad:
+                    log(f"AP {ifname} beaconing again")
+                bad = 0
+            else:
+                bad += 1
+                if bad >= FAIL_THRESHOLD:
+                    recover(ifname)
+                    bad = 0
+        time.sleep(INTERVAL)
+
+
+if __name__ == "__main__":
+    try:
+        main()
+    except KeyboardInterrupt:
+        sys.exit(0)

ap/van-ap-watchdog.service

@@ -0,0 +1,13 @@
+[Unit]
+Description=VanLink AP watchdog — recover hostapd if the radio wedges after a USB glitch
+After=hostapd.service
+Wants=hostapd.service
+
+[Service]
+Type=simple
+ExecStart=/usr/local/sbin/van-ap-watchdog
+Restart=always
+RestartSec=5
+
+[Install]
+WantedBy=multi-user.target

deploy.sh

@@ -9,7 +9,10 @@ cd "$(dirname "$(readlink -f "$0")")"
 
 echo "== access point =="
 install -D -m0644 ap/hostapd.conf           /etc/hostapd/hostapd.conf
+install -D -m0644 ap/hostapd-restart.conf   /etc/systemd/system/hostapd.service.d/restart.conf
 install -D -m0644 ap/default-hostapd        /etc/default/hostapd
+install -D -m0755 ap/van-ap-watchdog        /usr/local/sbin/van-ap-watchdog
+install -D -m0644 ap/van-ap-watchdog.service /etc/systemd/system/van-ap-watchdog.service
 install -D -m0644 ap/van-ap-dnsmasq.conf    /etc/van-ap/dnsmasq.conf
 install -D -m0644 ap/van-ap-dnsmasq.service /etc/systemd/system/van-ap-dnsmasq.service
 install -D -m0644 ap/10-van-ap.network      /etc/systemd/network/10-van-ap.network
@@ -82,7 +85,7 @@ systemctl mask systemd-networkd-wait-online.service >/dev/null 2>&1 || true
 # pick up the lid drop-in (re-execs logind; does NOT drop the network)
 systemctl restart systemd-logind >/dev/null 2>&1 || true
 systemctl unmask hostapd >/dev/null 2>&1 || true
-systemctl enable regdomain.service hostapd van-ap-dnsmasq nftables systemd-networkd van-failover van-thermal van-battery >/dev/null 2>&1 || true
+systemctl enable regdomain.service hostapd van-ap-dnsmasq nftables systemd-networkd van-failover van-thermal van-battery van-ap-watchdog >/dev/null 2>&1 || true
 systemctl restart van-thermal van-battery
 # Heartbeat: only enable/start once the client binary is installed (README §4).
 if [ -x /home/andreas/bin/hbc ]; then
@@ -96,6 +99,8 @@ nmcli general reload 2>/dev/null || systemctl reload NetworkManager 2>/dev/null
 # the wlan to br0, then dnsmasq binds br0, then NAT/failover
 systemctl restart systemd-networkd
 systemctl restart hostapd van-ap-dnsmasq nftables van-failover
+# AP watchdog last, after hostapd is back up (it only ever restarts a wedged hostapd)
+systemctl restart van-ap-watchdog
 networkctl reload 2>/dev/null || true
 
 echo