Fix watchdog timeout and MQTT connectivity issues

Problem: R5C spontaneously rebooted due to hardware watchdog timeout Cause 1: van-gps-owntracks in infinite restart loop (MQTT connect failed) Cause 2: mosquitto only listening on localhost, so clients couldn't connect Solution: - Add restart limits to van-gps-owntracks and van-li3-battery services (max 3 restarts per 5 minutes prevents infinite restart loops) - Configure mosquitto to listen on all interfaces (0.0.0.0:1883) - Deploy mosquitto listener config via deploy.sh - Restart mosquitto during deploy to apply config changes This prevents the system from hitting the 11-second hardware watchdog timeout when services crash and systemd auto-restarts them indefinitely. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
2026-09-18 18:20:11 -04:00
parent bb2e9c4618
commit 565e8f31ab
29 changed files with 761 additions and 39 deletions
@@ -25,7 +25,7 @@ This directory is the source of truth. The live system files live under `/etc`,
     WiFi clients (VanLink, 5GHz ch149, 80MHz WPA2)   wired LAN clients
        wlxc83a35a4ee55 (RTL8852BU, rtw89, hostapd)   enx00e04c331140
                          └───────────┬───────────────────┘
-                                br0   10.42.0.1/24   ← LAN bridge
+                                br0   10.42.1.1/24   ← LAN bridge
                          │  dnsmasq DHCP .10–.254 + DNS
                          │  nftables masquerade (oifname != br0)
                          │  ip_forward=1
@@ -41,7 +41,7 @@ This directory is the source of truth. The live system files live under `/etc`,
 | Concern | Owner |
 |---|---|
 | WAN interfaces (eth / wifi / gsm), DHCP-client, metrics | **NetworkManager** |
-| LAN bridge `br0` + IP (10.42.0.1) + wired LAN member | **systemd-networkd** (`2x-van-br0/lan`) |
+| LAN bridge `br0` + IP (10.42.1.1) + wired LAN member | **systemd-networkd** (`2x-van-br0/lan`) |
 | AP beaconing / WPA + adding the wlan to `br0` | **hostapd** (`bridge=br0`; AP iface + wired port are NM-*unmanaged*) |
 | Keep the AP beaconing across USB re-enumeration (Starlink flap) | **hostapd** `Restart=always`/no start-limit drop-in + **van-ap-watchdog** daemon (recovers a wedged radio) |
 | AP DHCP + DNS | **dnsmasq** (dedicated instance, bound to AP only) |
@@ -56,13 +56,13 @@ This directory is the source of truth. The live system files live under `/etc`,
 | Web UI | **Cockpit** + `vanrouter` plugin |
 ### Key network facts
- LAN: `10.42.0.0/24`, gateway `10.42.0.1`, DHCP `.10–.254` — shared by Wi-Fi (`VanLink`, dual-band: 5GHz `wlxc83a35a4ee55` + 2.4GHz `wlxd8ec5e2faa8c`) and the wired port `enx00e04c331140`, all bridged into `br0`.
+- LAN: `10.42.1.0/24`, gateway `10.42.0.1`, DHCP `.10–.254` — shared by Wi-Fi (`VanLink`, dual-band: 5GHz `wlxc83a35a4ee55` + 2.4GHz `wlxd8ec5e2faa8c`) and the wired port `enx00e04c331140`, all bridged into `br0`.
 - WAN priority (metrics): **wifi 100 → eth/Starlink 200 → 4G 300** (lower = preferred).
 - Management / recovery: ethernet `192.168.10.251`, wifi `192.168.10.27`, ZeroTier `192.168.196.22`.
 - Cockpit: `https://192.168.10.251:9090` (or `.27`, or ZeroTier). Log in with a Unix account; enable *Administrative access* for action buttons.
 - Regulatory domain **CA** (unlocks 5GHz ch149–161 @30dBm, no DFS).
 - ZeroTier network `d3ecf5726d041b2a`, pushed DNS domain `wrede.pvt` via `192.168.196.115` + `192.168.10.5`.
- Home Assistant runs **natively on this host** (Podman Quadlet, host network) at **`http://10.42.0.1:8123`** (name `homeassistant`). The old HAOS VM `ha_van` (`10.42.0.50`) was retired 2026-07-07.
+- Home Assistant runs **natively on this host** (Podman Quadlet, host network) at **`http://10.42.1.1:8123`** (name `homeassistant`). The old HAOS VM `ha_van` (`10.42.0.50`) was retired 2026-07-07.
 ---
@@ -86,10 +86,10 @@ This directory is the source of truth. The live system files live under `/etc`,
 | `10-van-ap.network` | `/etc/systemd/network/10-van-ap.network` | brings the AP wlan up with **no** IP (it's a `br0` member; hostapd enslaves it) |
 | `11-van-ap-2g.network` | `/etc/systemd/network/11-van-ap-2g.network` | same, for the 2.4GHz wlan |
 | `20-van-br0.netdev` | `/etc/systemd/network/20-van-br0.netdev` | creates the LAN bridge `br0` (STP off) |
-| `21-van-br0.network` | `/etc/systemd/network/21-van-br0.network` | static `10.42.0.1/24` on `br0` (`ConfigureWithoutCarrier`) |
+| `21-van-br0.network` | `/etc/systemd/network/21-van-br0.network` | static `10.42.1.1/24` on `br0` (`ConfigureWithoutCarrier`) |
 | `22-van-lan.network` | `/etc/systemd/network/22-van-lan.network` | enslaves the wired LAN port `enx00e04c331140` to `br0` |
 | `van-ap-unmanaged.conf` | `/etc/NetworkManager/conf.d/van-ap-unmanaged.conf` | tells NM to leave the AP wlan **and** the wired LAN port alone |
-| `nftables.conf` | `/etc/nftables.conf` | NAT: `masquerade ip saddr 10.42.0.0/24 oifname != br0` → follows whatever WAN is active |
+| `nftables.conf` | `/etc/nftables.conf` | NAT: `masquerade ip saddr 10.42.1.0/24 oifname != br0` → follows whatever WAN is active |
 | `99-van-router.conf` | `/etc/sysctl.d/99-van-router.conf` | `net.ipv4.ip_forward=1` |
 | `regdomain.service` | `/etc/systemd/system/regdomain.service` | `iw reg set CA` at boot, before NetworkManager |
 | `rtw89.conf` | `/etc/modprobe.d/rtw89.conf` | 5GHz dongle: `disable_ps_mode=Y` (else AP drops beacon when idle) + blacklists in-kernel rtw89 so the out-of-tree morrownr/rtw89 dkms driver (installed separately, not by deploy.sh) loads instead — see gotchas below |
@@ -169,7 +169,7 @@ supply (laptop UPS-style monitor), unrelated hardware and purpose.
 | `homeassistant.container` | `/etc/containers/systemd/homeassistant.container` | Podman Quadlet: HA Container on the host network (`:8123`), config in `/srv/homeassistant`, host D-Bus mounted for onboard Bluetooth. |
 The `homeassistant` LAN name comes from `ap/van-ap-dnsmasq.conf` (`host-record` →
-`10.42.0.1`). The retired HAOS VM's domain XML (`ha_van.xml`) lives in git history.
+`10.42.1.1`). The retired HAOS VM's domain XML (`ha_van.xml`) lives in git history.
 `custom_components/` and `dashboards/` are **not deployed by `deploy.sh`** — HA's
 `/srv/homeassistant` is hand-managed live config (it has other integrations installed
@@ -348,7 +348,7 @@ script's docstring for the full field layout.
 ### Home Assistant (native Podman container)
 - HA Container runs on the host via a **Podman Quadlet** (`ha/homeassistant.container` →
  `/etc/containers/systemd/`; systemd generates `homeassistant.service`). **Host network**:
-  HA binds `:8123` directly, so clients use `http://10.42.0.1:8123` (or
+  HA binds `:8123` directly, so clients use `http://10.42.1.1:8123` (or
  `http://homeassistant:8123`); mDNS/SSDP discovery sees the LAN because there's no bridge
  or NAT in the way. Config lives in `/srv/homeassistant`.
 - **Bluetooth**: host BlueZ (`bluetooth.service`) serves the Pi's onboard `hci0` to HA over
@@ -359,9 +359,9 @@ script's docstring for the full field layout.
  (`podman pull ghcr.io/home-assistant/home-assistant:stable`) and restart. After editing
  the `.container` file: `./deploy.sh` (or install + `systemctl daemon-reload`), then
  `systemctl restart homeassistant`. Verify:
-  `curl -s -o /dev/null -w '%{http_code}' http://10.42.0.1:8123/` → `200`.
+  `curl -s -o /dev/null -w '%{http_code}' http://10.42.1.1:8123/` → `200`.
 - History: HA started as a HAOS VM on wayback (libvirt NAT, then bridged onto `br0` at
-  `10.42.0.50` with a legacy-URL DNAT), moved to this Pi as an aarch64 VM 2026-07-06, and
+  `10.42.1.50` with a legacy-URL DNAT), moved to this Pi as an aarch64 VM 2026-07-06, and
  went native 2026-07-07 — the fixed 2 GiB VM allocation starved the 4 GB Pi. The domain
  XML (`ha/ha_van.xml`) and the DNAT/hairpin nftables rules are in git history.
@@ -0,0 +1,429 @@
 # vanlink — campervan router on `wayback`
 > **This clone = the Pi 4 port (host `wan`, branch `wan`).** Onboard `eth0`+`wlan0` are
 > NM-managed WANs (netplan `ap/50-van-wan.yaml`); the AP radios + wired LAN port arrive
 > with the USB hub from wayback (same MAC-derived names, so all configs port verbatim).
 > Dropped here: HA VM, battery/lid (no hardware), heartbeat + ZeroTier (not installed yet).
 > Watchdog is 10s (bcm2835 max 15s); thermal watches `cpu_thermal`.
 Turns **wayback** (Asus ZenBook UX391U, Ubuntu 24.04, zabbly kernel) into a self-contained
 campervan hub/router/AP:
 - **WiFi-6 access point** for client devices (hand-rolled hostapd, not NetworkManager).
 - **Multi-WAN failover** across WiFi (neighbour AP) → Ethernet (Starlink) → 4G/5G cellular.
 - **ZeroTier-managed DNS** so `*.wrede.pvt` resolves over the overlay when away from home.
 - **Cockpit dashboard** ("Van Router") for status + manual control.
 This directory is the source of truth. The live system files live under `/etc`, `/usr/...`;
 `deploy.sh` copies from here to there. Edit here, run `sudo ./deploy.sh`, done.
 ---
 ## 1. Architecture at a glance
 ```
     WiFi clients (VanLink, 5GHz ch149, 80MHz WPA2)   wired LAN clients
        wlxc83a35a4ee55 (RTL8852BU, rtw89, hostapd)   enx00e04c331140
                          └───────────┬───────────────────┘
                                br0   10.42.0.1/24   ← LAN bridge
                          │  dnsmasq DHCP .10–.254 + DNS
                          │  nftables masquerade (oifname != br0)
                          │  ip_forward=1
        ┌─────────────────┼──────────────────────────────────────┐
   wlp1s0 (m100)     enxd8ec5eeb3512 (m200)        Koodo gsm (m300)
   neighbour WiFi    USB ethernet → Starlink       4G/5G modem (when present)
        └──────── van-failover picks lowest-metric HEALTHY WAN ────┘
   NetworkManager owns the WAN side.  systemd-networkd owns the AP IP + the ZeroTier DNS link.
 ```
 ### Roles / responsibilities
 | Concern | Owner |
 |---|---|
 | WAN interfaces (eth / wifi / gsm), DHCP-client, metrics | **NetworkManager** |
 | LAN bridge `br0` + IP (10.42.0.1) + wired LAN member | **systemd-networkd** (`2x-van-br0/lan`) |
 | AP beaconing / WPA + adding the wlan to `br0` | **hostapd** (`bridge=br0`; AP iface + wired port are NM-*unmanaged*) |
 | Keep the AP beaconing across USB re-enumeration (Starlink flap) | **hostapd** `Restart=always`/no start-limit drop-in + **van-ap-watchdog** daemon (recovers a wedged radio) |
 | AP DHCP + DNS | **dnsmasq** (dedicated instance, bound to AP only) |
 | NAT + forwarding | **nftables** + sysctl |
 | WAN health + failover | **van-failover** daemon |
 | Temperature monitor / alert / log | **van-thermal** daemon |
 | NVMe I/O-timeout/reset watchdog + alert | **van-nvme-watch** daemon |
 | Battery monitor / low-charge alert + shutdown | **van-battery** daemon |
 | Auto-reboot on hang | **systemd hardware watchdog** (`intel_oc_wdt`) |
 | Liveness / dead-man's switch + metrics | **hbc** heartbeat client → hbd.wrede.pvt |
 | ZeroTier DNS → resolved | **zerotier-systemd-manager** + systemd-networkd |
 | Web UI | **Cockpit** + `vanrouter` plugin |
 ### Key network facts
 - LAN: `10.42.0.0/24`, gateway `10.42.0.1`, DHCP `.10–.254` — shared by Wi-Fi (`VanLink`, dual-band: 5GHz `wlxc83a35a4ee55` + 2.4GHz `wlxd8ec5e2faa8c`) and the wired port `enx00e04c331140`, all bridged into `br0`.
 - WAN priority (metrics): **wifi 100 → eth/Starlink 200 → 4G 300** (lower = preferred).
 - Management / recovery: ethernet `192.168.10.251`, wifi `192.168.10.27`, ZeroTier `192.168.196.22`.
 - Cockpit: `https://192.168.10.251:9090` (or `.27`, or ZeroTier). Log in with a Unix account; enable *Administrative access* for action buttons.
 - Regulatory domain **CA** (unlocks 5GHz ch149–161 @30dBm, no DFS).
 - ZeroTier network `d3ecf5726d041b2a`, pushed DNS domain `wrede.pvt` via `192.168.196.115` + `192.168.10.5`.
 - Home Assistant runs **natively on this host** (Podman Quadlet, host network) at **`http://10.42.0.1:8123`** (name `homeassistant`). The old HAOS VM `ha_van` (`10.42.0.50`) was retired 2026-07-07.
 ---
 ## 2. Directory layout → system destinations
 `deploy.sh` performs exactly this mapping.
 ### `ap/` — access point
 | file | → installs to | purpose |
 |---|---|---|
 | `hostapd.conf` | `/etc/hostapd/hostapd.conf` | AP: SSID `VanLink`, ch149, VHT80/HE, WPA2-PSK, country CA, `ctrl_interface` for the watchdog. **WPA passphrase lives here.** |
 | `hostapd-restart.conf` | `/etc/systemd/system/hostapd.service.d/restart.conf` | `Restart=always` + `StartLimitIntervalSec=0` so hostapd never gives up after a USB re-enumeration bounces the radio |
 | `van-ap-watchdog` | `/usr/local/sbin/van-ap-watchdog` | watches `hostapd_cli status`; restarts hostapd if the radio wedges (running but not `ENABLED`) |
 | `van-ap-watchdog.service` | `/etc/systemd/system/van-ap-watchdog.service` | runs the watchdog |
 | `default-hostapd` | `/etc/default/hostapd` | `DAEMON_CONF=...` |
 | `van-ap-dnsmasq.conf` | `/etc/van-ap/dnsmasq.conf` | DHCP/DNS bound to AP iface (`bind-dynamic`, so it does not clash with systemd-resolved) |
 | `van-ap-dnsmasq.service` | `/etc/systemd/system/van-ap-dnsmasq.service` | dedicated dnsmasq unit (uses the `dnsmasq-base` binary; the distro dnsmasq service is NOT used) |
 | `hostapd-2g.conf` | `/etc/hostapd/hostapd-2g.conf` | 2.4GHz AP (Linksys WUSB6300v2 / RTL8822BU): same SSID+PSK, ch6, HT20. Own hostapd instance so one radio wedging never kills the other. |
 | `hostapd-2g.service` | `/etc/systemd/system/hostapd-2g.service` | dedicated unit for the 2.4GHz hostapd (same Restart=always / no-start-limit semantics as the 5GHz drop-in) |
 | `van-ap-watchdog-2g.service` | `/etc/systemd/system/van-ap-watchdog-2g.service` | second watchdog instance: `van-ap-watchdog /etc/hostapd/hostapd-2g.conf hostapd-2g` |
 | `10-van-ap.network` | `/etc/systemd/network/10-van-ap.network` | brings the AP wlan up with **no** IP (it's a `br0` member; hostapd enslaves it) |
 | `11-van-ap-2g.network` | `/etc/systemd/network/11-van-ap-2g.network` | same, for the 2.4GHz wlan |
 | `20-van-br0.netdev` | `/etc/systemd/network/20-van-br0.netdev` | creates the LAN bridge `br0` (STP off) |
 | `21-van-br0.network` | `/etc/systemd/network/21-van-br0.network` | static `10.42.0.1/24` on `br0` (`ConfigureWithoutCarrier`) |
 | `22-van-lan.network` | `/etc/systemd/network/22-van-lan.network` | enslaves the wired LAN port `enx00e04c331140` to `br0` |
 | `van-ap-unmanaged.conf` | `/etc/NetworkManager/conf.d/van-ap-unmanaged.conf` | tells NM to leave the AP wlan **and** the wired LAN port alone |
 | `nftables.conf` | `/etc/nftables.conf` | NAT: `masquerade ip saddr 10.42.0.0/24 oifname != br0` → follows whatever WAN is active |
 | `99-van-router.conf` | `/etc/sysctl.d/99-van-router.conf` | `net.ipv4.ip_forward=1` |
 | `regdomain.service` | `/etc/systemd/system/regdomain.service` | `iw reg set CA` at boot, before NetworkManager |
 | `rtw89.conf` | `/etc/modprobe.d/rtw89.conf` | 5GHz dongle: `disable_ps_mode=Y` (else AP drops beacon when idle) + blacklists in-kernel rtw89 so the out-of-tree morrownr/rtw89 dkms driver (installed separately, not by deploy.sh) loads instead — see gotchas below |
 | `rtw88.conf` | `/etc/modprobe.d/rtw88.conf` | 2.4GHz dongle: `disable_lps_deep=Y` (same PS reasoning) + `switch_usb_mode=N` (don't self-upgrade to USB3 — it radiates into 2.4GHz) |
 ### `failover/` — multi-WAN
 | file | → installs to | purpose |
 |---|---|---|
 | `van-failover` | `/usr/local/sbin/van-failover` | the daemon (Python) |
 | `config.json` | `/etc/van-failover/config.json` | WAN list + priorities + probe settings |
 | `van-failover.service` | `/etc/systemd/system/van-failover.service` | `Restart=always` |
 | `50-disable-eee` | `/etc/NetworkManager/dispatcher.d/50-disable-eee` | disables EEE on `en*`/`eth*` at connect (r8152 idle parking) |
 | `99-van-arp.conf` | `/etc/sysctl.d/99-van-arp.conf` | `arp_ignore=1` / `arp_announce=2` (multi-NIC ARP hygiene) |
 ### `dns/` — ZeroTier managed DNS
 | file | → installs to | purpose |
 |---|---|---|
 | `zt-search.conf` | `/etc/systemd/network/99-ztuga7c2kh.network.d/search.conf` | makes `wrede.pvt` a search domain (bare-hostname completion) |
 | `zt-network.local.conf` | `/var/lib/zerotier-one/networks.d/d3ecf5726d041b2a.local.conf` | contains `allowDNS=1` (prerequisite) |
 | `99-ztuga7c2kh.network.generated` | *(reference only — generated by the manager)* | what zerotier-systemd-manager writes |
 | `zerotier-systemd-manager.{service,timer}` | `/usr/lib/systemd/system/` | the manager (binary installed separately, see §4) |
 ### `cockpit/vanrouter/` — web UI
 `manifest.json`, `index.html`, `vanrouter.css`, `vanrouter.js` → `/usr/share/cockpit/vanrouter/`.
 Being superseded by `router-status/` below (Cockpit's PAM+TLS login was unreliable
 — rejected Firefox sometimes, failed outright from Android — and it was a resource
 hog); left in place until the Home Assistant dashboard fully covers its actions.
 ### `router-status/` — same status/actions, via MQTT -> Home Assistant
 | file | → installs to | purpose |
 |---|---|---|
 | `van-router-mqtt` | `/usr/local/sbin/van-router-mqtt` | status+control daemon (Python, paho-mqtt): AP/clients/Starlink/WAN/modem, HA MQTT discovery; republishes van-failover/van-thermal/van-battery's own state.json verbatim rather than recomputing them |
 | `config.json` | `/etc/van-router-mqtt/config.json` | broker + interface identifiers (`@TOKEN@` templated like `failover/config.json` — no secrets, rendered fresh every deploy) |
 | `van-router-mqtt.service` | `/etc/systemd/system/van-router-mqtt.service` | `Restart=always` |
 ### `power/` — never sleep + thermal monitor
 | file | → installs to | purpose |
 |---|---|---|
 | `10-vanlink-nolid.conf` | `/etc/systemd/logind.conf.d/10-vanlink-nolid.conf` | logind ignores the lid in all states (closed / on AC / docked) |
 | `van-thermal` | `/usr/local/sbin/van-thermal` | temperature daemon (Python): publishes state, alerts, logs history |
 | `thermal-config.json` | `/etc/van-thermal/config.json` | sensors + warn/crit thresholds + sample/log intervals |
 | `van-thermal.service` | `/etc/systemd/system/van-thermal.service` | `Restart=always` |
 | `van-nvme-watch` | `/usr/local/sbin/van-nvme-watch` | NVMe watchdog (Python): tails `journalctl -k` for I/O-timeout/reset events, Pushover alerts |
 | `nvme-watch-config.json` | `/etc/van-nvme-watch/config.json` | grace period, cooldown, device paths |
 | `van-nvme-watch.service` | `/etc/systemd/system/van-nvme-watch.service` | `Restart=always` |
 | `van-battery` | `/usr/local/sbin/van-battery` | battery daemon (Python): Pushover low-charge alerts + safe shutdown |
 | `battery-config.json` | `/etc/van-battery/config.json` | warn levels, shutdown level, poll interval, paths |
 | `van-battery.service` | `/etc/systemd/system/van-battery.service` | `Restart=always` |
 | `pushover.json.example` | → `/etc/van-battery/pushover.json` (seeded if absent) | Pushover token+user **template**; real file is 0600, **not** in the repo |
 | `10-vanlink-watchdog.conf` | `/etc/systemd/system.conf.d/10-vanlink-watchdog.conf` | hardware watchdog (`RuntimeWatchdogSec=20s`); applied via `daemon-reexec` |
 `deploy.sh` additionally **masks** `sleep.target suspend.target hibernate.target hybrid-sleep.target`
 (no repo file — symlinks to `/dev/null` under `/etc/systemd/system/`) so nothing else can suspend either.
 ### `heartbeat/` — dead-man's switch
 | file | → installs to | purpose |
 |---|---|---|
 | `hbc.yaml` | `/etc/hbc.yaml` | heartbeat client config (interval + metric plugins) |
 | `hbc.service` | `/etc/systemd/system/hbc.service` | runs `~/bin/hbc` as `andreas` → hbd.wrede.pvt:50003 |
 The `hbc` binary itself (venv at `~/venvs/hbd`, symlink `~/bin/hbc`) is installed once via the
 heartbeat project's own installer — see §4. `deploy.sh` only starts the service once it exists.
 ### `li3/` — RV house battery (BLE BMS → MQTT/HA)
 | file | → installs to | purpose |
 |---|---|---|
 | `van-li3-battery` | `/usr/local/sbin/van-li3-battery` | BLE→MQTT daemon (Python, bleak + paho-mqtt): reads the Lithionics Li3 BMS, publishes HA MQTT discovery + state |
 | `config.json.example` | → `/etc/van-li3/config.json` (seeded if absent) | BLE address, broker, MQTT topic/discovery **template**; real file is 0600, **not** in the repo |
 | `van-li3-battery.service` | `/etc/systemd/system/van-li3-battery.service` | `Restart=always` |
 Not to be confused with `power/van-battery` — that's the host's own AC/battery power
 supply (laptop UPS-style monitor), unrelated hardware and purpose.
 ### `ha/` — Home Assistant (native container)
 | file | → installs to | purpose |
 |---|---|---|
 | `homeassistant.container` | `/etc/containers/systemd/homeassistant.container` | Podman Quadlet: HA Container on the host network (`:8123`), config in `/srv/homeassistant`, host D-Bus mounted for onboard Bluetooth. |
 The `homeassistant` LAN name comes from `ap/van-ap-dnsmasq.conf` (`host-record` →
 `10.42.0.1`). The retired HAOS VM's domain XML (`ha_van.xml`) lives in git history.
 `custom_components/` and `dashboards/` are **not deployed by `deploy.sh`** — HA's
 `/srv/homeassistant` is hand-managed live config (it has other integrations installed
 via HACS that were never in this repo, e.g. `mbapi2020`); these two are kept here purely
 as a reference/backup copy of the hand-authored pieces. `dashboards/van-router.yaml` is
 the Lovelace dashboard for `van-router-mqtt`'s entities — the live copy is
 `/srv/homeassistant/dashboards/van-router.yaml`, wired in via a `lovelace: dashboards:`
 block manually added to `/srv/homeassistant/configuration.yaml` (also not repo-tracked;
 see that file's own comment). After editing the dashboard, copy both places and
 `systemctl restart homeassistant` (or use HA's own "reload" for lovelace-only changes).
 ---
 ## 3. Components & how to operate
 ### Access Point
 - **Dual band, one SSID**: 5GHz (`hostapd.conf` → unit `hostapd`, ch149 80MHz VHT+HE) and 2.4GHz (`hostapd-2g.conf` → unit `hostapd-2g`, ch6 HT20, for range + 2.4-only IoT). Same SSID/PSK — clients pick their band. Deliberately two hostapd processes: a USB wedge on one radio never takes the other down, and each has its own watchdog (`van-ap-watchdog`, `van-ap-watchdog-2g`).
 - Change SSID / channel / password: edit `ap/hostapd.conf` and/or `ap/hostapd-2g.conf`, then `sudo cp` to `/etc/hostapd/` (or `./deploy.sh`) and restart the matching unit.
 - DHCP range / DNS options: `ap/van-ap-dnsmasq.conf` + restart `van-ap-dnsmasq`.
 - Services: `hostapd`, `hostapd-2g`, `van-ap-dnsmasq`, `systemd-networkd`, `nftables`, `regdomain`.
 - **Do not** add `noscan` to hostapd.conf — Ubuntu's hostapd rejects it and fails to start.
 - Verify each is actually beaconing with `iw dev wlxc83a35a4ee55 info` / `iw dev wlxd8ec5e2faa8c info` (ssid+channel+width present), not just `systemctl is-active`.
 ### Multi-WAN failover (`van-failover`)
 - Probes each WAN's real internet every few seconds (HTTP-204 check, captive-portal-aware), demotes a failed WAN by raising its route-metric, fails back on recovery (hysteresis).
 - Status: `systemctl status van-failover`, `journalctl -u van-failover -f`, or the Cockpit "WAN Failover" card. Live state JSON at `/run/van-failover/state.json`.
 - Tune priorities/timing in `failover/config.json` (`wans[].metric`, `probe_interval`, `probe_timeout`, `fail_threshold`, `ok_threshold`, `probe_urls`) then restart.
 - WAN identity: eth/wifi keyed by `device`, cellular by NM `connection` name (`Koodo`).
 - The daemon changes metrics with **`ip route`**, never `nmcli device reapply` — see Gotchas.
 #### Manual preference (Cockpit "Prefer" button)
 - The daemon is the **single writer** of route-metrics. To force a specific WAN to the top it reads an optional file `/run/van-failover/prefer` containing one **device name** (e.g. `enxd8ec5eeb3512`). That WAN gets base metric **50** (below every config metric) so it wins while healthy; the health PENALTY still stacks on top, so failover/fail-back are unchanged. Empty/absent file = follow config priorities.
 - The Cockpit "Prefer" button just writes that file (root via polkit) — it does **not** set metrics itself. Doing so directly would fight the daemon (reverted within one probe loop) and `reapply`-flap the USB carrier. The daemon applies the change on its next loop (~`probe_interval`); `state.json` carries a `preferred` flag the UI shows as "(preferred)".
 - Single-WAN, sticky override with no explicit "clear": to return to automatic priority, prefer your top WAN (wifi) or `rm /run/van-failover/prefer`. **Ephemeral by design** — `/run` is wiped on reboot, so a reboot returns to config priorities. (To persist it, point `PREFER` at `/etc/van-failover/prefer` in the daemon and the matching path in `vanrouter.js`.)
 - Swapping the preference between two WANs is collision-safe: a second default route can't take the new WAN's target metric while the old preferred WAN still holds it, so `enforce_route` keeps the existing route and retries next loop (settles in ~2 loops) instead of stranding the interface. `enforce_route` also restores a default route that went missing while the carrier is up (gateway from NM via `device_gateway`), not just rebases an existing one.
 ### Thermal monitor (`van-thermal`)
 - One daemon off a single sysfs sample loop does all three jobs: live Cockpit readout, threshold alerting, history.
 - **Live state**: `/run/van-thermal/state.json` (atomic-swapped each sample). The Cockpit "Temperatures" card reads this — no shelling out to `sensors` per refresh. Sensors are resolved by hwmon **name** + **label** (`coretemp`/`Package id 0`, `nvme`/`Composite`), never by `hwmonN` index (not stable across boots).
 - **Alerts**: level changes (ok ↔ warn ↔ crit) are logged to the journal with hysteresis (`clear_margin`, default 5 °C) so a sensor sitting on the line doesn't spam. Watch with `journalctl -u van-thermal -f`; warn/crit carry sd-daemon severity (`-p warning`/`-p err`).
 - **History**: throttled CSV at `/var/log/van-thermal.csv` (one row per `log_interval`, default 60 s), self-rotating to `.csv.1` past `log_max_bytes` (5 MB).
 - **Tuning**: edit `/etc/van-thermal/config.json` (thresholds, intervals, sensor list), then `systemctl restart van-thermal`. Defaults: CPU warn 80 / crit 95 °C (silicon crit is 100), NVMe warn 65 / crit 70 °C (drive crit ~71).
 - Status: `systemctl status van-thermal` or `cat /run/van-thermal/state.json`.
 ### NVMe watchdog (`van-nvme-watch`)
 - Event-driven, not polled: tails `journalctl -kf` for the `nvme nvmeN: I/O tag ... timeout, reset controller` signature that crashed and corrupted the root fs on 2026-08-02 (recurred 2026-08-04, self-healed — see the memory notes for that investigation). There's no sensor to sample, only a log line to catch.
 - On a match it watches a `grace_period` (default 20 s) for either a clean re-init (self-healed) or a second timeout / an ext4 error following it (escalated) before alerting, so the Pushover message already says which outcome happened — no need to SSH in during a scare.
 - The alert body includes a live `smartctl`/`tune2fs` snapshot (SMART health, critical-warning flag, media error count, filesystem state).
 - **Live state**: `/run/van-nvme-watch/state.json` (last event, same convention as van-failover/van-thermal).
 - Shares Pushover credentials with van-thermal/van-battery (`/etc/van-battery/pushover.json`).
 - **Tuning**: edit `/etc/van-nvme-watch/config.json` (`grace_period`, `cooldown`, device paths), then `systemctl restart van-nvme-watch`.
 - Status: `systemctl status van-nvme-watch`, `journalctl -u van-nvme-watch -f`, or `cat /run/van-nvme-watch/state.json`.
 ### Battery monitor (`van-battery`)
 - Watches mains vs battery via `/sys/class/power_supply/AC0/online` (0 = on battery) and charge via `BAT0/capacity`. Both resolve by `type` (Mains/Battery) if those names ever differ.
 - **Only while on battery**, it sends escalating **Pushover** alerts at **25 / 20 / 15 %**, and at **10 %** sends a final alert and runs `systemctl poweroff` (after `shutdown_grace`, default 8 s, so the alert flushes first).
 - Edge-triggered per discharge episode: each severity fires once; the sequence **re-arms when mains returns**. Unplugging already below a warn level fires a single alert for the current severity (no burst), then shutdown at 10 %.
 - **Credentials** live in `/etc/van-battery/pushover.json` (mode 0600), seeded from `pushover.json.example` on first deploy and **never committed**. Missing/placeholder creds disable *sending* but **not** the shutdown — running flat must always power down safely (the skip is logged to the journal).
 - Tune in `/etc/van-battery/config.json` (`warn_levels`, `shutdown_level`, `poll_interval`), then `systemctl restart van-battery`.
 - Status: `systemctl status van-battery`, `journalctl -u van-battery -f`, or `cat /run/van-battery/state.json`.
 - **Caveat — the 10 % shutdown is one-way.** A laptop won't power itself back on when mains returns: "restore on AC loss" is a BIOS/firmware feature (not OS-controllable — `/proc/acpi/wakeup` only covers wake-from-suspend). If your BIOS exposes an "AC power-on / restore on AC loss" option, enable it so the router reboots itself once shore/solar power is back; this ZenBook likely doesn't have it, in which case a low-battery shutdown needs a manual power-on.
 ### Adding the 4G/5G modem
 1. Plug the USB modem in. ModemManager + the `Koodo` gsm NM connection (autoconnect) bring it up.
   On a fresh host create the profile once:
   `nmcli con add type gsm ifname "*" con-name Koodo apn sp.koodo.com connection.autoconnect yes`
 2. It auto-joins as the `cellular` WAN at metric 300 (last resort). Nothing else to configure.
 3. Confirm with `mmcli -L` and the Cockpit failover card (cellular flips from `absent` to `up`).
 4. Gotcha (handled in `van-failover` since the Quectel EC25-AF landed): for MBIM/QMI modems
   NM's device is the control port (`cdc-wdm0`) while IP + routes live on the wwan netdev
   (e.g. `wwu1u2i4`). The daemon maps via `GENERAL.IP-IFACE` — probing or `ip route`
   against `cdc-wdm0` silently sees nothing. The Cockpit WAN table does the same mapping
   (shows the netdev + its IP; nmcli actions still target `cdc-wdm0`).
 5. The Cockpit WAN row shows signal % / tech / operator (from `mmcli`) and has a
   **Restart** button: `mmcli --disable && --enable` (the EC25 MBIM plugin doesn't
   support `--reset`; autoconnect reconnects), falling back to a USB unbind/bind of the
   Quectel device (vendor `2c7c`) if ModemManager can't reach the modem.
 ### GPS (gpsd from the cellular modem)
 The EC25-AF has a GNSS engine that streams NMEA on its USB interface 01 (`ttyUSB1`,
 `gps` port in `mmcli`). Needs its own GPS antenna on the modem's GNSS connector for a fix.
 - **One-time modem config** (persists in modem NV; survives reboot/replug/Restart):
  GNSS auto-start via `AT+QGPSCFG="autogps",1` + `AT+QGPS=1`. ModemManager holds both
  AT ports and swallows replies, so stop it first:
  `systemctl stop ModemManager`, send the ATs on `/dev/ttyUSB3`, `systemctl start ModemManager`.
 - **Host side** (deployed by deploy.sh, needs `apt install gpsd gpsd-clients`):
  `gps/77-modem-gps.rules` udev rule matches the NMEA tty (2c7c:0125 if01), symlinks it
  to `/dev/modem-gps`, and hot-adds it to gpsd via `gpsdctl@%k` — the same mechanism as
  gpsd's own 60-gpsd.rules, so plug/unplug/renumbering just works. `gps/gpsd.default`
  (`/etc/default/gpsd`) keeps `DEVICES` empty (hotplug does it) and runs gpsd with `-n`.
 - Verify: `gpspipe -r -n 10` (raw NMEA), `cgps` (fix view), `systemctl status gpsdctl@ttyUSB1`.
 - Don't enable MM location APIs (`mmcli --location-enable-gps-*`) at the same time —
  MM would open the NMEA port and fight gpsd; `gps-unmanaged` is the only safe one.
 ### OwnTracks publisher (gpsd -> MQTT)
 `gps/van-gps-owntracks` (service `van-gps-owntracks`) streams TPV fixes from local
 gpsd and publishes OwnTracks location JSON to `owntracks/rv/gps` on home.wrede.ca
 (tid `rv`) — at most every 10 min when parked, immediately after >250 m of movement.
 Port of the wayback-era `gps_to_owntracks.py`, adapted to apt-only deps
 (`python3-gps`, `python3-paho-mqtt` — the old `gpsdclient` is pip-only) and the
 paho 2.x callback API.
 - Broker credentials live only in `/etc/van-gps/config.json` (0600, seeded from
  `gps/config.json.example` — edit after first deploy). All knobs (broker, topic,
  intervals, gpsd host) live there too.
 - **client_id must stay `vanq-wan` (or anything unique)**: some legacy client still
  holds `vanq` on the broker (old copy on wayback?) and the broker kicks whoever
  shares its ID — the symptom is connect/disconnect every few seconds.
 - On MQTT disconnect the script exits and systemd restarts it (RestartSec=15);
  that *is* the reconnect logic, so a red blip after a WAN failover is normal.
 - Debug tracing: `kill -HUP` the process toggles per-fix logging to the journal.
 - Verify: `journalctl -u van-gps-owntracks` shows `mqtt connect` and stays quiet;
  a restart publishes the current fix immediately (first-message path), so
  subscribing to `owntracks/rv/gps` while restarting shows a live message.
 - Note: the broker holds an ancient *retained* message on this topic from a 2023
  OwnTracks device (tid `RV`, Winegard SSID); our publishes are not retained.
 ### Li3 battery monitor (`van-li3-battery`)
 Publishes the RV's 12V LiFePO4 house battery (Lithionics Li3, BLE HM-10 UART module —
 service `ffe0`/char `ffe1`, no pairing) to Home Assistant as 10 sensor entities (pack
 voltage, 4 cell voltages, current, SOC, BMS/battery temperature, status) via MQTT
 discovery. Protocol (CSV telemetry lines after sending `$traceon`+`$info`) reverse-
 engineered from the `com.lithionics.bms` Android app's own BLE/parsing code — see the
 script's docstring for the full field layout.
 - Not the same battery as `power/van-battery` (host's own AC/UPS power supply) —
  different hardware, different concern, deliberately different naming.
 - Broker credentials live only in `/etc/van-li3/config.json` (0600, seeded from
  `li3/config.json.example` — edit after first deploy, same pattern as `van-gps`).
 - **Self-healing BLE**: scan/connect retries happen *inside* the running process (MQTT
  session and HA entities stay up across them, no flapping) rather than relying on
  systemd restarts. This Pi's onboard Bluetooth adapter (Cypress/CYW43) occasionally
  wedges bluetoothd's discovery state after a run of failed connects to this specific
  device (`Discovering` stays `yes` forever, every subsequent connect fails with
  `le-connection-abort-by-local`) — as soon as a scan fails the daemon restarts
  `bluetooth.service` itself to clear it, rate-limited to once per 5 min so it
  doesn't repeatedly disrupt the AP's other BLE gear (motion sensors, IR remote).
 - This BLE module accepts only **one central connection at a time** — while
  `van-li3-battery` holds it, the Li3 phone app can't connect simultaneously.
  `systemctl stop van-li3-battery` to free it up for the app.
 - Verify: `journalctl -u van-li3-battery -f` (look for `connected` / `published: {...}`),
  or `mosquitto_sub -h localhost -u homeassistant -P <pw> -t van/li3_battery/state`.
 ### ZeroTier managed DNS
 - `wrede.pvt` resolves over ZeroTier when off the home LAN. Mechanism: `allowDNS=1` (prereq) + `zerotier-systemd-manager` writes `99-ztuga7c2kh.network`, networkd applies it to resolved.
 - Verify: `resolvectl status ztuga7c2kh` shows `DNS` scope + the two servers + `wrede.pvt`.
 - NOTE: ZeroTier's *native* DNS push is a no-op on Linux (`setDns ... not implemented`), which is why the manager is required.
 - `systemd-networkd-wait-online` is **masked** so the ZT link does **not** gate `network-online.target` — see Gotchas.
 ### Cockpit dashboard
 - `https://<wayback>:9090` → "Van Router". Reload the browser after deploying plugin changes (Cockpit caches packages per session).
 - Read-only status works as any user; Prefer/Up/Down/Restart need *Administrative access* (polkit).
 - The Access Points card shows both bands (5GHz `hostapd`, 2.4GHz `hostapd-2g`) with per-band client lists and **Restart** buttons; the radio list lives in `vanrouter.js` (`const APS`).
 - **Prefer** sets the manual WAN preference (see *Manual preference* above); **Up/Down** connect/disconnect the NM device.
 ### Router status via MQTT (`van-router-mqtt`) — replacing Cockpit above
 - Publishes the same status Cockpit's plugin showed, and accepts the same actions, over
  MQTT with Home Assistant discovery — no web server/login of its own; HA (already running
  on this Pi) owns the UI. Broker is the local Mosquitto on `127.0.0.1:1883` (already used by
  HA itself and by `van-gps-owntracks`/`li3`); it currently allows anonymous connections, so
  `config.json` carries no credentials.
 - State topics (`van/router/...`): `ap/5g`, `ap/2g`, `clients`, `starlink`, `wan`,
  `deploywarn`, plus verbatim republishes of `van-failover`'s and `van-thermal`'s own
  `/run/van-*/state.json` (those two daemons were **not** modified — `van-thermal` is
  deliberately stdlib-only for boot resilience, and `van-failover`'s control loop wasn't
  touched). `van/router/battery` only appears if `power/van-battery` (documented in this
  README, not actually deployed on this host) ever starts writing its state file.
 - Command topics (`vanlink/cmd/...`): `ap/{5g,2g}/restart`, `modem/restart`,
  `wan/{wifi,starlink,cellular}/set` (`connect`/`disconnect`), `wan/prefer`
  (`auto`/`wifi`/`starlink`/`cellular` — writes the same `/run/van-failover/prefer` file
  Cockpit's "Prefer" button did). Wi-Fi scan/connect (wlan0 as a WAN) was **not** ported —
  it's an interactive password-prompt flow that doesn't map to MQTT/HA entities well and is
  only used during initial setup; use `nmcli` directly for that.
 - Verify: `journalctl -u van-router-mqtt -f`, `mosquitto_sub -h localhost -t 'van/router/#' -v`.
  Entities should appear in HA under the "Van Router" device once the integration picks up
  the retained discovery messages (Settings → Devices & Services → MQTT).
 ### Home Assistant (native Podman container)
 - HA Container runs on the host via a **Podman Quadlet** (`ha/homeassistant.container` →
  `/etc/containers/systemd/`; systemd generates `homeassistant.service`). **Host network**:
  HA binds `:8123` directly, so clients use `http://10.42.0.1:8123` (or
  `http://homeassistant:8123`); mDNS/SSDP discovery sees the LAN because there's no bridge
  or NAT in the way. Config lives in `/srv/homeassistant`.
 - **Bluetooth**: host BlueZ (`bluetooth.service`) serves the Pi's onboard `hci0` to HA over
  the mounted `/run/dbus` socket — add the Bluetooth integration in HA and it appears.
 - No add-on store (that was HAOS's Supervisor): Mosquitto/Zigbee2MQTT-style add-ons become
  their own containers/services if ever needed.
 - Operate: `systemctl {status,restart} homeassistant`. Update: bump/pull the image
  (`podman pull ghcr.io/home-assistant/home-assistant:stable`) and restart. After editing
  the `.container` file: `./deploy.sh` (or install + `systemctl daemon-reload`), then
  `systemctl restart homeassistant`. Verify:
  `curl -s -o /dev/null -w '%{http_code}' http://10.42.0.1:8123/` → `200`.
 - History: HA started as a HAOS VM on wayback (libvirt NAT, then bridged onto `br0` at
  `10.42.0.50` with a legacy-URL DNAT), moved to this Pi as an aarch64 VM 2026-07-06, and
  went native 2026-07-07 — the fixed 2 GiB VM allocation starved the 4 GB Pi. The domain
  XML (`ha/ha_van.xml`) and the DNAT/hairpin nftables rules are in git history.
 ### Never sleep (lid-closed operation)
 - wayback lives lid-closed in the van and must stay up. Stock logind `HandleLidSwitch=suspend` would sleep it on lid close (even on AC). The `power/10-vanlink-nolid.conf` drop-in sets all three lid actions to `ignore`; `deploy.sh` also masks every sleep target so idle / GUI / a stray `systemctl suspend` can't suspend it.
 - Verify: `busctl get-property org.freedesktop.login1 /org/freedesktop/login1 org.freedesktop.login1.Manager HandleLidSwitch` → `"ignore"`, and `systemctl is-enabled suspend.target` → `masked`. `busctl call ... CanSuspend` should return `"no"`.
 - **To re-enable sleep** (e.g. if wayback is ever a laptop again):
  ```bash
  sudo rm /etc/systemd/logind.conf.d/10-vanlink-nolid.conf
  sudo systemctl unmask sleep.target suspend.target hibernate.target hybrid-sleep.target
  sudo systemctl restart systemd-logind
  ```
  (then remove the `power/` block from `deploy.sh`, or the next deploy re-applies it).
 ---
 ## 4. Deploy / rebuild
 ```bash
 cd ~/vanlink
 sudo ./deploy.sh          # copies all files to their system locations, reloads + enables services
 ```
 Four things `deploy.sh` does **not** do (one-time, manual):
 1. **zerotier-systemd-manager binary** (v0.4.0, hand-installed — not in the ZeroTier apt repo):
   ```bash
   # copy /usr/bin/zerotier-systemd-manager from a host that has it (e.g. wertvoll), then:
   sudo systemctl enable --now zerotier-systemd-manager.timer
   sudo zerotier-cli set d3ecf5726d041b2a allowDNS=1
   ```
 2. **hostapd unmask** (Ubuntu ships it masked): `sudo systemctl unmask hostapd`.
 3. **Pushover credentials** for `van-battery` — deploy seeds `/etc/van-battery/pushover.json` (0600) with placeholders; fill in your app token + user key, then `sudo systemctl restart van-battery`. Until then low-battery alerts are skipped (logged), but the 10 % auto-shutdown still works.
 4. **Heartbeat client `hbc`** (dead-man's switch) — install the binary once into andreas' venv:
   ```bash
   git clone https://git.wrede.ca/andreas/heartbeat.git ~/git/heartbeat   # if not already cloned
   sh ~/git/heartbeat/scripts/hb_install.sh client                        # -> ~/venvs/hbd + ~/bin/hbc
   ```
   `deploy.sh` installs `/etc/hbc.yaml` + the `hbc.service` unit and starts it once `~/bin/hbc` exists. The service reports to **hbd.wrede.pvt** (UDP 50003) as user `andreas`. Upgrade later with `~/bin/hb_install.sh client`.
 ---
 ## 5. Key design decisions & hard-won gotchas
 - **AP on hostapd, not NetworkManager.** NM's hotspot caps the rtw89 radio at HT20/20MHz; hostapd gives the full VHT80/HE (WiFi-6). The AP iface is therefore NM-*unmanaged*; networkd gives it its static IP.
 - **`rtw89` power-save must be off** (`disable_ps_mode=Y`) or the AP stops beaconing when idle and the SSID vanishes.
 - **RTL8852BU is USB-2.0 and hangs under load if it shares a USB hub.** Keep the AP dongle on its **own** USB controller, separate from the WAN ethernet. Symptom of a shared bus: `c2h reg timeout` + `Polling beacon packet empty fail` under throughput, SSID drops. (`timed out to flush queues` alone is benign.) True regardless of which driver (in-kernel or out-of-tree) is bound — it's a physical bus-bandwidth issue, not a driver bug.
 - **5GHz dongle runs the out-of-tree `morrownr/rtw89` driver (since 2026-08-23), not the in-kernel one.** This Ubuntu kernel base (`7.0.0-1017-raspi`) predates mainline's rtw89 USB2→3 auto-switch, so the in-kernel driver permanently caps the 8852BU at USB2/480M. The out-of-tree driver has that switch (`switch_usb_mode=y` in `ap/rtw89.conf`) and needs a real USB3-capable port to actually benefit from it — a USB2-only port still caps it regardless of driver (confirmed live: 480M on a USB2-only controller, 5000M/SuperSpeed after moving to a dedicated USB3 controller). Driver itself logs `"2.4 GHz performance may be better in a USB 2 port"` on load — watch the 2.4GHz radio for new interference now that 5GHz runs SuperSpeed signalling nearby. Installed/updated via `sudo ./ap/install-rtw89-driver.sh` (pins a specific upstream commit, `dkms install`, `make install_fw`) — **not** run automatically by `deploy.sh`, since a dkms rebuild is slow and shouldn't fire on every routine deploy; run it manually after a fresh Pi provisioning or to bump the pin. DKMS itself auto-rebuilds across kernel upgrades. Module names get a `_git` suffix (`rtw89_8852bu_git` etc.) — `ap/rtw89.conf`'s blacklist stops the in-kernel modules from claiming the device instead. **Gotcha hit switching a live system over**: the in-kernel `rtw89_core` won't unload while its own in-kernel dependents (`rtw89_8852b`, `rtw89_8852b_common`) are still loaded — remove all of them first or the out-of-tree module fails to load (`exports duplicate symbol`). The install script's own output has the exact recovery steps.
 - **van-failover changes metrics via `ip route`, never `nmcli device reapply`.** `reapply` **resets the r8152 USB-ethernet carrier**, which caused a ~10s-ping-drop flapping feedback loop. Pure `ip route` changes are carrier-safe.
 - **Disable EEE on USB ethernet** (`50-disable-eee` dispatcher) — its idle power-save parks the *backup* WAN link and breaks health probes.
 - **Per-WAN probing needs `curl --interface if!<dev>`** (forces `SO_BINDTODEVICE`); plain `--interface <name>` only sets the source IP and still routes via the default WAN. `rp_filter` is loose (`2`), required for this.
 - **`arp_ignore`/`arp_announce`** matter only when two WANs share a subnet (a home-LAN test artifact; Starlink + neighbour-wifi will be on different subnets in the van). Harmless to keep.
 - **dnsmasq uses `bind-dynamic` bound to the AP iface** so it coexists with systemd-resolved (no port-53 fight) — resolved stays intact for the host.
 - **`systemd-networkd-wait-online` must be masked** or it deadlocks boot. networkd here manages only two links — the AP (`wlxc…`, local-only) and the ZT overlay `ztuga7c2kh` — and **neither is a real uplink**. The ZT link only appears once `zerotier-one` starts, but `zerotier-one` is ordered `After=network-online.target`, so wait-online would block on the not-yet-existing link until its 120s timeout. Marking the links `RequiredForOnline=no` does **not** help: with *zero* candidate links, wait-online can never satisfy "online" and times out anyway (systemd 255 — verified: `Timeout occurred while waiting for network connectivity` at exactly +120s). Both add ~2min to every boot and delay ZeroTier (the recovery path). The fix is to mask the service entirely (`deploy.sh` does this); real uplink readiness is covered by `NetworkManager-wait-online`, which owns the actual WANs.
 ---
 ## 6. Recovery / safety
 - Three independent management paths: ethernet `.251`, wifi `.27`, ZeroTier `192.168.196.22`. Changing default-route metrics does **not** affect the local-subnet (SSH) routes.
 - The AP fan/EC, hostapd, networkd, nftables, sysctl, failover, regdomain are all enabled and reboot-persistent.
 ## 7. Pending / future ideas
 - Real-world soak once the actual Starlink terminal + USB 4G/5G modem are installed (current eth is the home LAN; subnets/probe targets may want tuning).
 - Cockpit plugin: add an AP-config form (SSID/channel/PSK editing `hostapd.conf`). *(Manual failover override via the "Prefer" button is done — see §3.)*
 - Consider pinning a fixed interface name for the WAN ethernet (USB MAC-derived names change if the adapter is swapped).
@@ -4,7 +4,7 @@ Name=@WIFI_5G_IFACE@
 [Link]
 RequiredForOnline=no
-# No IP here: the AP wlan is a member of br0, which carries the 10.42.0.1 gateway.
+# No IP here: the AP wlan is a member of br0, which carries the 10.42.1.1 gateway.
 # hostapd adds this interface to br0 (bridge=br0) after it puts the radio into AP
 # mode — networkd must NOT enslave it (a station-mode wlan can't be bridged), so
 # this file only brings the link up.
@@ -0,0 +1,12 @@
 [Match]
 Name=@WIFI_5G_IFACE@
 [Link]
 RequiredForOnline=no
 # No IP here: the AP wlan is a member of br0, which carries the 10.42.0.1 gateway.
 # hostapd adds this interface to br0 (bridge=br0) after it puts the radio into AP
 # mode — networkd must NOT enslave it (a station-mode wlan can't be bridged), so
 # this file only brings the link up.
 [Network]
 ConfigureWithoutCarrier=yes
@@ -1,6 +1,6 @@
 # LAN bridge: joins the AP Wi-Fi radios (added by hostapd, once in AP mode) and the
 # wired LAN ports (enx00e04c331140 + onboard eth0, added by networkd) into one
-# 10.42.0.0/24 segment. STP off: all members are leaf ports, no loops (don't cable
+# 10.42.1.0/24 segment. STP off: all members are leaf ports, no loops (don't cable
 # the two wired ports to the same switch), and it avoids the forwarding delay that
 # would otherwise stall the first DHCP handshake on a freshly-plugged client.
 [NetDev]
@@ -0,0 +1,11 @@
 # LAN bridge: joins the AP Wi-Fi radios (added by hostapd, once in AP mode) and the
 # wired LAN ports (enx00e04c331140 + onboard eth0, added by networkd) into one
 # 10.42.0.0/24 segment. STP off: all members are leaf ports, no loops (don't cable
 # the two wired ports to the same switch), and it avoids the forwarding delay that
 # would otherwise stall the first DHCP handshake on a freshly-plugged client.
 [NetDev]
 Name=br0
 Kind=bridge
 [Bridge]
 STP=no
@@ -0,0 +1,16 @@
 # The LAN gateway IP now lives on the bridge, not on the AP wlan (which is just a
 # bridge member). ConfigureWithoutCarrier keeps 10.42.0.1 up even before any wired
 # device is plugged, exactly as it was on the wlan-only setup.
 [Match]
 Name=br0
 [Link]
 RequiredForOnline=no
 [Network]
 Address=10.42.0.1/24
 ConfigureWithoutCarrier=yes
 # Resolve .local (mDNS) for devices on the van's own LAN (ESPHome nodes,
 # etc.) — scoped to br0 only, not the WAN links, since mDNS is link-local
 # and that's the only interface those devices are actually on.
 MulticastDNS=yes
@@ -1,4 +1,4 @@
-# Wired LAN port: enslaved to br0 so anything plugged in lands on 10.42.0.0/24 and
+# Wired LAN port: enslaved to br0 so anything plugged in lands on 10.42.1.0/24 and
 # gets DHCP/DNS from the same dnsmasq as Wi-Fi clients. NM must leave this device
 # alone (see van-ap-unmanaged.conf) for networkd to own it here.
 [Match]
@@ -0,0 +1,11 @@
 # Wired LAN port: enslaved to br0 so anything plugged in lands on 10.42.0.0/24 and
 # gets DHCP/DNS from the same dnsmasq as Wi-Fi clients. NM must leave this device
 # alone (see van-ap-unmanaged.conf) for networkd to own it here.
 [Match]
 Name=@LAN_USB_IFACE@
 [Link]
 RequiredForOnline=no
 [Network]
 Bridge=br0
@@ -1,5 +1,5 @@
 # Onboard GbE (RP1 eth0) as the internal LAN port: enslaved to br0 so anything
-# plugged in lands on 10.42.0.0/24 with DHCP/DNS from the same dnsmasq as Wi-Fi
+# plugged in lands on 10.42.1.0/24 with DHCP/DNS from the same dnsmasq as Wi-Fi
 # clients. Was the metric-150 ethernet WAN until 2026-07-12 — with wifi +
 # Starlink + cellular as uplinks, the wired port earns its keep as LAN instead.
 # NM must leave it alone (see van-ap-unmanaged.conf) for networkd to own it here.
@@ -0,0 +1,13 @@
 # Onboard GbE (RP1 eth0) as the internal LAN port: enslaved to br0 so anything
 # plugged in lands on 10.42.0.0/24 with DHCP/DNS from the same dnsmasq as Wi-Fi
 # clients. Was the metric-150 ethernet WAN until 2026-07-12 — with wifi +
 # Starlink + cellular as uplinks, the wired port earns its keep as LAN instead.
 # NM must leave it alone (see van-ap-unmanaged.conf) for networkd to own it here.
 [Match]
 Name=eth0
 [Link]
 RequiredForOnline=no
 [Network]
 Bridge=br0
@@ -8,7 +8,7 @@ ctrl_interface=/var/run/hostapd
 driver=nl80211
 # Same SSID + PSK as 5GHz: dual-band clients pick a band themselves, 2.4-only IoT
 # devices just join. Rename here (e.g. VanLink-2G) if band pinning is ever needed.
-ssid=VanLink
+ssid=VanLink2-2G
 country_code=CA
 ieee80211d=1
 hw_mode=g
@@ -1,5 +1,5 @@
 interface=@WIFI_5G_IFACE@
-# Put the AP into br0 so Wi-Fi and the wired LAN port share one 10.42.0.0/24 segment.
+# Put the AP into br0 so Wi-Fi and the wired LAN port share one 10.42.1.0/24 segment.
 # hostapd adds the wlan to the bridge after setting AP mode; the bridge itself + its
 # wired member + the gateway IP are defined under /etc/systemd/network (2x-van-br0/lan).
 bridge=br0
@@ -8,7 +8,7 @@ bridge=br0
 # stays "running" but stops serving after a USB glitch.
 ctrl_interface=/var/run/hostapd
 driver=nl80211
-ssid=VanLink
+ssid=VanLink2
 country_code=CA
 ieee80211d=1
 ieee80211h=0
@@ -0,0 +1,31 @@
 interface=@WIFI_5G_IFACE@
 # Put the AP into br0 so Wi-Fi and the wired LAN port share one 10.42.0.0/24 segment.
 # hostapd adds the wlan to the bridge after setting AP mode; the bridge itself + its
 # wired member + the gateway IP are defined under /etc/systemd/network (2x-van-br0/lan).
 bridge=br0
 # Control socket so van-ap-watchdog (and hostapd_cli) can read the AP's real state
 # (state=ENABLED means it's actually beaconing) — used to detect a wedged radio that
 # stays "running" but stops serving after a USB glitch.
 ctrl_interface=/var/run/hostapd
 driver=nl80211
 ssid=VanLink2
 country_code=CA
 ieee80211d=1
 ieee80211h=0
 hw_mode=a
 channel=149
 wmm_enabled=1
 ieee80211n=1
 ht_capab=[HT40+][SHORT-GI-20][SHORT-GI-40]
 ieee80211ac=1
 vht_oper_chwidth=1
 vht_oper_centr_freq_seg0_idx=155
 vht_capab=[RXLDPC][SHORT-GI-80][SU-BEAMFORMEE]
 ieee80211ax=1
 he_oper_chwidth=1
 he_oper_centr_freq_seg0_idx=155
 auth_algs=1
 wpa=2
 wpa_passphrase=1foot11foot11
 wpa_key_mgmt=WPA-PSK
 rsn_pairwise=CCMP
@@ -0,0 +1,11 @@
 #!/usr/sbin/nft -f
 # van-router NAT — masquerade LAN out whatever the WAN of the moment is
 # (anything that is NOT the LAN bridge br0: ethernet/Starlink, wifi, future 4G)
 table ip van_router_nat
 delete table ip van_router_nat
 table ip van_router_nat {
    chain postrouting {
        type nat hook postrouting priority srcnat; policy accept;
        ip saddr 10.42.0.0/24 oifname != "br0" masquerade
    }
 }
@@ -0,0 +1,12 @@
 interface=br0
 bind-dynamic
 except-interface=lo
 domain-needed
 bogus-priv
 dhcp-authoritative
 dhcp-range=10.42.0.10,10.42.0.254,255.255.255.0,12h
 # Home Assistant runs natively on this host (Podman, host network, :8123) —
 # "homeassistant" is just another name for the router address.
 host-record=homeassistant,10.42.0.1
 dhcp-option=option:router,10.42.0.1
 dhcp-option=option:dns-server,10.42.0.1
@@ -227,6 +227,11 @@ install_rendered router-status/config.json              /etc/van-router-mqtt/con
 install -D -m0644 router-status/van-router-mqtt.service /etc/systemd/system/van-router-mqtt.service
 python3 -c 'import paho.mqtt' 2>/dev/null \
    || warn "python3-paho-mqtt missing (apt install python3-paho-mqtt) — van-router-mqtt won't start"
 # MQTT broker must listen on all interfaces (not just localhost) for clients on br0
 install -D -m0644 router-status/mosquitto-listener.conf /etc/mosquitto/conf.d/van-listener.conf
 # Restart limit drop-ins: prevent infinite restart loops that trigger hardware watchdog
 install -D -m0644 router-status/van-gps-owntracks-restart-limit.conf /etc/systemd/system/van-gps-owntracks.service.d/restart-limit.conf
 install -D -m0644 li3/van-li3-battery-restart-limit.conf /etc/systemd/system/van-li3-battery.service.d/restart-limit.conf
 echo "== gps (cellular modem GNSS -> gpsd) =="
 if dpkg -s gpsd >/dev/null 2>&1; then
@@ -308,6 +313,7 @@ systemctl daemon-reexec
 # an already-up WAN keeps its currently-applied DNS until it reconnects (or
 # reboot), deliberately not forced here (reconnecting a WAN flaps it).
 systemctl restart systemd-resolved
 systemctl restart mosquitto
 # networkd here owns only the AP radios + bridge + wired LAN port (no real uplink);
 # its wait-online would just stall network-online.target. NM-wait-online covers WANs.
 systemctl mask systemd-networkd-wait-online.service >/dev/null 2>&1 || true
@@ -15,6 +15,11 @@ set -euo pipefail
 USB_ROOT_PART=/dev/sda2
 MOUNT_POINT=/mnt/usb-backup-root
 MOUNT_POINT=10.42.1.44:/media/andreas/writable-nvme
 [[ "$MOUNT_POINT" == *":"* ]]
 LOCAL_MOUNT=$?
 root_src=$(findmnt -no SOURCE /)
 if [ "$root_src" = "$USB_ROOT_PART" ]; then
@@ -22,6 +27,7 @@ if [ "$root_src" = "$USB_ROOT_PART" ]; then
    exit 1
 fi
 if [ $LOCAL_MOUNT -eq 1 ]; then
 	mkdir -p "$MOUNT_POINT"
 	mounted_here=0
 	if ! mountpoint -q "$MOUNT_POINT"; then
@@ -37,6 +43,7 @@ cleanup() {
 	    fi
 	}
 	trap cleanup EXIT
 fi
 # rsync exit 24 ("partial transfer due to vanished source files") is expected on a
 # live root — files like container shm sockets or in-progress logs routinely disappear
@@ -0,0 +1,68 @@
 #!/usr/bin/env bash
 # Refresh the USB disk (sda) as a live boot-fallback clone of the
 # running root filesystem. Run manually after a push:
 #   sudo ./failover/sync-usb-backup.sh
 #
 # EEPROM BOOT_ORDER falls back to the SD card, then this USB disk, if the
 # primary boot disk (NVMe) doesn't come up. /etc/fstab, /etc/machine-id,
 # and /var/log are excluded from the sync so the clone keeps its own
 # partition identity (PARTUUIDs differ per disk) and boot/journal history
 # instead of inheriting the source disk's — see failover/README or repo
 # memory "sd-fallback-fstab-clobber" for what breaks if those leak across.
 set -euo pipefail
 [ "$(id -u)" -eq 0 ] || { echo "Run as root (sudo)." >&2; exit 1; }
 USB_ROOT_PART=/dev/sda2
 MOUNT_POINT=/mnt/usb-backup-root
 MOUNT_POINT=10.42.0.44:/media/andreas/writable-nvme
 [[ "$MOUNT_POINT" == *":"* ]]
 LOCAL_MOUNT=$?
 root_src=$(findmnt -no SOURCE /)
 if [ "$root_src" = "$USB_ROOT_PART" ]; then
    echo "Currently booted from $USB_ROOT_PART — refusing to sync USB onto itself." >&2
    exit 1
 fi
 if [ $LOCAL_MOUNT -eq 1 ]; then
 	mkdir -p "$MOUNT_POINT"
 	mounted_here=0
 	if ! mountpoint -q "$MOUNT_POINT"; then
 	    mount "$USB_ROOT_PART" "$MOUNT_POINT"
 	    mounted_here=1
 	else
 	    mount -o remount,rw "$MOUNT_POINT"
 	fi
 	cleanup() {
 	    if [ "$mounted_here" -eq 1 ]; then
 	        umount "$MOUNT_POINT"
 	    fi
 	}
 	trap cleanup EXIT
 fi
 # rsync exit 24 ("partial transfer due to vanished source files") is expected on a
 # live root — files like container shm sockets or in-progress logs routinely disappear
 # mid-sync. Tolerate that one code; anything else is a real failure and still aborts.
 rc=0
 rsync -aHAX --numeric-ids --delete \
    --info=progress2 \
    --exclude=/proc/* --exclude=/sys/* --exclude=/dev/* --exclude=/run/* \
    --exclude=/tmp/* --exclude=/mnt/* --exclude=/media/* --exclude=/lost+found \
    --exclude=/boot/firmware/* --exclude=/swapfile \
    --exclude=/etc/fstab --exclude=/etc/machine-id --exclude=/var/log/* \
    --stats / "$MOUNT_POINT/" || rc=$?
 if [ "$rc" -ne 0 ] && [ "$rc" -ne 24 ]; then
    echo "rsync failed (exit $rc)." >&2
    exit "$rc"
 fi
 if [ "$rc" -eq 24 ]; then
    echo "Note: some files vanished mid-sync (rsync exit 24) — normal on a live root." >&2
 fi
 echo "USB backup ($USB_ROOT_PART) synced from $root_src."
@@ -3,7 +3,7 @@
 # container is the equivalent). Installed by deploy.sh to /etc/containers/systemd/;
 # systemd generates esphome.service from it.
 #
-# Host networking: the dashboard binds :6052 directly (http://10.42.0.1:6052),
+# Host networking: the dashboard binds :6052 directly (http://10.42.1.1:6052),
 # and ESPHome's mDNS-based OTA discovery/flashing of already-provisioned nodes
 # needs to see the LAN as the host does — a bridged network would need explicit
 # port/mDNS forwarding for the same result.
@@ -0,0 +1,35 @@
 # ESPHome dashboard as a Podman Quadlet, same pattern as homeassistant.container
 # (this isn't Home Assistant Supervised, so there's no add-on store — a sibling
 # container is the equivalent). Installed by deploy.sh to /etc/containers/systemd/;
 # systemd generates esphome.service from it.
 #
 # Host networking: the dashboard binds :6052 directly (http://10.42.0.1:6052),
 # and ESPHome's mDNS-based OTA discovery/flashing of already-provisioned nodes
 # needs to see the LAN as the host does — a bridged network would need explicit
 # port/mDNS forwarding for the same result.
 [Unit]
 Description=ESPHome dashboard (Podman container)
 Wants=network-online.target
 After=network-online.target
 [Container]
 Image=ghcr.io/esphome/esphome:stable
 ContainerName=esphome
 Network=host
 Volume=/srv/esphome:/config
 Environment=TZ=America/Toronto
 # Initial flash of a new device needs USB serial access; OTA re-flashes of an
 # already-provisioned node don't. Uncomment and set the real path to flash
 # over USB (find it with `ls /dev/ttyUSB* /dev/ttyACM*` after plugging the
 # device in):
 #Volume=/dev/ttyUSB0:/dev/ttyUSB0
 #AddCapability=SYS_ADMIN
 [Service]
 Restart=always
 # First start pulls the image; PlatformIO also downloads toolchains on a
 # node's first compile — both want a working WAN, not just a fast one.
 TimeoutStartSec=900
 [Install]
 WantedBy=multi-user.target
@@ -4,7 +4,7 @@
 # /etc/containers/systemd/; systemd generates homeassistant.service from it.
 #
 # Host networking: HA binds :8123 on the host directly, so the LAN URL is
-# http://10.42.0.1:8123 (name: homeassistant) — no bridge port, no DNAT.
+# http://10.42.1.1:8123 (name: homeassistant) — no bridge port, no DNAT.
 # /run/dbus gives HA the host BlueZ stack = the Pi's onboard Bluetooth (hci0).
 [Unit]
 Description=Home Assistant (Podman container)
@@ -0,0 +1,37 @@
 # Home Assistant Container as a Podman Quadlet (replaced the ha_van HAOS VM
 # 2026-07-07 — the fixed 2 GiB VM allocation starved the 4 GB Pi; VM domain XML
 # is in git history under ha/ha_van.xml). Installed by deploy.sh to
 # /etc/containers/systemd/; systemd generates homeassistant.service from it.
 #
 # Host networking: HA binds :8123 on the host directly, so the LAN URL is
 # http://10.42.0.1:8123 (name: homeassistant) — no bridge port, no DNAT.
 # /run/dbus gives HA the host BlueZ stack = the Pi's onboard Bluetooth (hci0).
 [Unit]
 Description=Home Assistant (Podman container)
 Wants=network-online.target bluetooth.service
 After=network-online.target bluetooth.service
 [Container]
 Image=ghcr.io/home-assistant/home-assistant:stable
 ContainerName=homeassistant
 Network=host
 Volume=/srv/homeassistant:/config
 Volume=/run/dbus:/run/dbus:ro
 Environment=TZ=America/Toronto
 # Bluetooth needs two escapes: Ubuntu's dbus-daemon mediates D-Bus per
 # AppArmor label and the default containers profile can't send to BlueZ
 # (AddMatch denied), and habluetooth wants NET_ADMIN+NET_RAW for adapter
 # recovery. HA is rootful + host-net anyway.
 PodmanArgs=--security-opt apparmor=unconfined
 AddCapability=NET_ADMIN NET_RAW
 # Podman's default 10s stop window SIGKILLs HA mid-flush and the recorder
 # complains about an unclean sqlite shutdown on the next start.
 StopTimeout=120
 [Service]
 Restart=always
 # First start pulls the image (~600 MB) — allow for a slow uplink.
 TimeoutStartSec=900
 [Install]
 WantedBy=multi-user.target
@@ -12,10 +12,17 @@
 # attempts (Discovering stays "yes" forever, and every subsequent connect
 # fails with le-connection-abort-by-local) — observed happening from our own
 # repeated scan/connect cycling, not anything external. We restart
-# bluetooth.service ourselves as soon as a scan fails to clear that, rate-
+# bluetooth.service ourselves once scan misses run past STUCK_DISCOVERY_THRESHOLD
-# limited (BLUETOOTH_RESTART_COOLDOWN_S) so we don't do it so often it
+# consecutive attempts, rate-limited (BLUETOOTH_RESTART_COOLDOWN_S) so we don't do
-# disrupts the other BLE gear on this hub (motion sensors, IR remote, etc.
+# it so often it disrupts the other BLE gear on this hub (motion sensors, IR
-# also served by bluetoothd here).
+# remote, etc. also served by bluetoothd here).
 #
 # The battery itself sits at the edge of the onboard adapter's range (around
 # -71 to -73 dBm from its fixed position, confirmed 2026-08-24 against a phone
 # that connects to the same device instantly from up close) — an ordinary scan
 # miss from weak RSSI looks identical to a wedged discovery state, so
 # STUCK_DISCOVERY_THRESHOLD is set well above 1 to avoid restarting bluetoothd
 # (which doesn't fix an RF problem) on every routine miss.
 #
 # Only exits (letting systemd Restart=always give us a fresh process) on
 # MQTT-level failure, which shouldn't happen in normal operation.
@@ -30,7 +37,7 @@ import paho.mqtt.client as mqtt
 from bleak import BleakClient, BleakScanner
 BLE_RETRY_DELAY_S = 8
-STUCK_DISCOVERY_THRESHOLD = 1       # consecutive scan failures before we intervene
+STUCK_DISCOVERY_THRESHOLD = 10      # consecutive scan failures before we intervene
 BLUETOOTH_RESTART_COOLDOWN_S = 300  # don't restart bluetooth.service more than this often
 config_path = sys.argv[1] if len(sys.argv) > 1 else "/etc/van-li3/config.json"
@@ -338,7 +345,7 @@ async def stream_from_device(dev, mqc):
                if reading:
                    latest_cs.update(reading)
-    async with BleakClient(dev, timeout=15) as client:
+    async with BleakClient(dev, timeout=30) as client:
        print("connected")
        await client.start_notify(FFE1, notify_handler)
        await client.write_gatt_char(FFE1, b"$traceon\r\n", response=False)
@@ -0,0 +1,6 @@
 # Restart limit for van-li3-battery to prevent watchdog timeout
 # Allows max 3 restarts per 5 minutes; after that, the service stays failed
 # (systemd will not keep restarting it indefinitely)
 [Service]
 StartLimitIntervalSec=300
 StartLimitBurst=3
@@ -0,0 +1,4 @@
 # Van Router MQTT broker configuration
 # Listen on all interfaces for local network access (clients on br0, ZeroTier, etc.)
 listener 1883
 protocol mqtt
@@ -0,0 +1,6 @@
 # Restart limit for van-gps-owntracks to prevent watchdog timeout
 # Allows max 3 restarts per 5 minutes; after that, the service stays failed
 # (systemd will not keep restarting it indefinitely)
 [Service]
 StartLimitIntervalSec=300
 StartLimitBurst=3