first commit

README.md

@@ -0,0 +1,174 @@
+# vanlink — campervan router on `wayback`
+
+Turns **wayback** (Asus ZenBook UX391U, Ubuntu 24.04, zabbly kernel) into a self-contained
+campervan hub/router/AP:
+
+- **WiFi-6 access point** for client devices (hand-rolled hostapd, not NetworkManager).
+- **Multi-WAN failover** across WiFi (neighbour AP) → Ethernet (Starlink) → 4G/5G cellular.
+- **ZeroTier-managed DNS** so `*.wrede.pvt` resolves over the overlay when away from home.
+- **Cockpit dashboard** ("Van Router") for status + manual control.
+
+This directory is the source of truth. The live system files live under `/etc`, `/usr/...`;
+`deploy.sh` copies from here to there. Edit here, run `sudo ./deploy.sh`, done.
+
+---
+
+## 1. Architecture at a glance
+
+```
+            clients ──WiFi(VanLink, 5GHz ch149, 80MHz WPA2)
+                          │
+        wlxc83a35a4ee55  (Realtek RTL8852BU, rtw89)  10.42.0.1/24   ← AP, hostapd
+                          │  dnsmasq DHCP .10–.254 + DNS
+                          │  nftables masquerade (oifname != AP)
+                          │  ip_forward=1
+        ┌─────────────────┼──────────────────────────────────────┐
+   wlp1s0 (m100)     enxd8ec5eeb3512 (m200)        Koodo gsm (m300)
+   neighbour WiFi    USB ethernet → Starlink       4G/5G modem (when present)
+        └──────── van-failover picks lowest-metric HEALTHY WAN ────┘
+
+   NetworkManager owns the WAN side.  systemd-networkd owns the AP IP + the ZeroTier DNS link.
+```
+
+### Roles / responsibilities
+| Concern | Owner |
+|---|---|
+| WAN interfaces (eth / wifi / gsm), DHCP-client, metrics | **NetworkManager** |
+| AP interface IP (10.42.0.1) | **systemd-networkd** (`10-van-ap.network`) |
+| AP beaconing / WPA | **hostapd** (AP iface is NM-*unmanaged*) |
+| AP DHCP + DNS | **dnsmasq** (dedicated instance, bound to AP only) |
+| NAT + forwarding | **nftables** + sysctl |
+| WAN health + failover | **van-failover** daemon |
+| ZeroTier DNS → resolved | **zerotier-systemd-manager** + systemd-networkd |
+| Web UI | **Cockpit** + `vanrouter` plugin |
+
+### Key network facts
+- AP LAN: `10.42.0.0/24`, gateway/AP `10.42.0.1`, DHCP `.10–.254`.
+- WAN priority (metrics): **wifi 100 → eth/Starlink 200 → 4G 300** (lower = preferred).
+- Management / recovery: ethernet `192.168.10.251`, wifi `192.168.10.27`, ZeroTier `192.168.196.22`.
+- Cockpit: `https://192.168.10.251:9090` (or `.27`, or ZeroTier). Log in with a Unix account; enable *Administrative access* for action buttons.
+- Regulatory domain **CA** (unlocks 5GHz ch149–161 @30dBm, no DFS).
+- ZeroTier network `d3ecf5726d041b2a`, pushed DNS domain `wrede.pvt` via `192.168.196.115` + `192.168.10.5`.
+
+---
+
+## 2. Directory layout → system destinations
+
+`deploy.sh` performs exactly this mapping.
+
+### `ap/` — access point
+| file | → installs to | purpose |
+|---|---|---|
+| `hostapd.conf` | `/etc/hostapd/hostapd.conf` | AP: SSID `VanLink`, ch149, VHT80/HE, WPA2-PSK, country CA. **WPA passphrase lives here.** |
+| `default-hostapd` | `/etc/default/hostapd` | `DAEMON_CONF=...` |
+| `van-ap-dnsmasq.conf` | `/etc/van-ap/dnsmasq.conf` | DHCP/DNS bound to AP iface (`bind-dynamic`, so it does not clash with systemd-resolved) |
+| `van-ap-dnsmasq.service` | `/etc/systemd/system/van-ap-dnsmasq.service` | dedicated dnsmasq unit (uses the `dnsmasq-base` binary; the distro dnsmasq service is NOT used) |
+| `10-van-ap.network` | `/etc/systemd/network/10-van-ap.network` | static `10.42.0.1/24` on the AP iface (`ConfigureWithoutCarrier`) |
+| `van-ap-unmanaged.conf` | `/etc/NetworkManager/conf.d/van-ap-unmanaged.conf` | tells NM to leave the AP iface alone |
+| `nftables.conf` | `/etc/nftables.conf` | NAT: `masquerade ip saddr 10.42.0.0/24 oifname != <AP>` → follows whatever WAN is active |
+| `99-van-router.conf` | `/etc/sysctl.d/99-van-router.conf` | `net.ipv4.ip_forward=1` |
+| `regdomain.service` | `/etc/systemd/system/regdomain.service` | `iw reg set CA` at boot, before NetworkManager |
+| `rtw89.conf` | `/etc/modprobe.d/rtw89.conf` | `options rtw89_core disable_ps_mode=Y` (else AP drops beacon when idle) |
+
+### `failover/` — multi-WAN
+| file | → installs to | purpose |
+|---|---|---|
+| `van-failover` | `/usr/local/sbin/van-failover` | the daemon (Python) |
+| `config.json` | `/etc/van-failover/config.json` | WAN list + priorities + probe settings |
+| `van-failover.service` | `/etc/systemd/system/van-failover.service` | `Restart=always` |
+| `50-disable-eee` | `/etc/NetworkManager/dispatcher.d/50-disable-eee` | disables EEE on `en*`/`eth*` at connect (r8152 idle parking) |
+| `99-van-arp.conf` | `/etc/sysctl.d/99-van-arp.conf` | `arp_ignore=1` / `arp_announce=2` (multi-NIC ARP hygiene) |
+
+### `dns/` — ZeroTier managed DNS
+| file | → installs to | purpose |
+|---|---|---|
+| `zt-search.conf` | `/etc/systemd/network/99-ztuga7c2kh.network.d/search.conf` | makes `wrede.pvt` a search domain (bare-hostname completion) |
+| `zt-network.local.conf` | `/var/lib/zerotier-one/networks.d/d3ecf5726d041b2a.local.conf` | contains `allowDNS=1` (prerequisite) |
+| `99-ztuga7c2kh.network.generated` | *(reference only — generated by the manager)* | what zerotier-systemd-manager writes |
+| `zerotier-systemd-manager.{service,timer}` | `/usr/lib/systemd/system/` | the manager (binary installed separately, see §4) |
+
+### `cockpit/vanrouter/` — web UI
+`manifest.json`, `index.html`, `vanrouter.css`, `vanrouter.js` → `/usr/share/cockpit/vanrouter/`.
+
+---
+
+## 3. Components & how to operate
+
+### Access Point
+- Change SSID / channel / password: edit `ap/hostapd.conf`, then `sudo cp` it to `/etc/hostapd/hostapd.conf` (or `./deploy.sh`) and `sudo systemctl restart hostapd`.
+- Width is 80MHz VHT + HE (WiFi-6). Channel 149 (UNII-3, non-DFS).
+- DHCP range / DNS options: `ap/van-ap-dnsmasq.conf` + restart `van-ap-dnsmasq`.
+- Services: `hostapd`, `van-ap-dnsmasq`, `systemd-networkd`, `nftables`, `regdomain`.
+- **Do not** add `noscan` to hostapd.conf — Ubuntu's hostapd rejects it and fails to start.
+- Verify it's actually beaconing with `iw dev wlxc83a35a4ee55 info` (ssid+channel+width present), not just `systemctl is-active hostapd`.
+
+### Multi-WAN failover (`van-failover`)
+- Probes each WAN's real internet every few seconds (HTTP-204 check, captive-portal-aware), demotes a failed WAN by raising its route-metric, fails back on recovery (hysteresis).
+- Status: `systemctl status van-failover`, `journalctl -u van-failover -f`, or the Cockpit "WAN Failover" card. Live state JSON at `/run/van-failover/state.json`.
+- Tune priorities/timing in `failover/config.json` (`wans[].metric`, `probe_interval`, `probe_timeout`, `fail_threshold`, `ok_threshold`, `probe_urls`) then restart.
+- WAN identity: eth/wifi keyed by `device`, cellular by NM `connection` name (`Koodo`).
+- The daemon changes metrics with **`ip route`**, never `nmcli device reapply` — see Gotchas.
+
+#### Manual preference (Cockpit "Prefer" button)
+- The daemon is the **single writer** of route-metrics. To force a specific WAN to the top it reads an optional file `/run/van-failover/prefer` containing one **device name** (e.g. `enxd8ec5eeb3512`). That WAN gets base metric **50** (below every config metric) so it wins while healthy; the health PENALTY still stacks on top, so failover/fail-back are unchanged. Empty/absent file = follow config priorities.
+- The Cockpit "Prefer" button just writes that file (root via polkit) — it does **not** set metrics itself. Doing so directly would fight the daemon (reverted within one probe loop) and `reapply`-flap the USB carrier. The daemon applies the change on its next loop (~`probe_interval`); `state.json` carries a `preferred` flag the UI shows as "(preferred)".
+- Single-WAN, sticky override with no explicit "clear": to return to automatic priority, prefer your top WAN (wifi) or `rm /run/van-failover/prefer`. **Ephemeral by design** — `/run` is wiped on reboot, so a reboot returns to config priorities. (To persist it, point `PREFER` at `/etc/van-failover/prefer` in the daemon and the matching path in `vanrouter.js`.)
+- Swapping the preference between two WANs is collision-safe: a second default route can't take the new WAN's target metric while the old preferred WAN still holds it, so `enforce_route` keeps the existing route and retries next loop (settles in ~2 loops) instead of stranding the interface. `enforce_route` also restores a default route that went missing while the carrier is up (gateway from NM via `device_gateway`), not just rebases an existing one.
+
+### Adding the 4G/5G modem
+1. Plug the USB modem in. ModemManager + the existing `Koodo` gsm NM connection (autoconnect) bring it up.
+2. It auto-joins as the `cellular` WAN at metric 300 (last resort). Nothing else to configure.
+3. Confirm with `mmcli -L` and the Cockpit failover card (cellular flips from `absent` to `up`).
+
+### ZeroTier managed DNS
+- `wrede.pvt` resolves over ZeroTier when off the home LAN. Mechanism: `allowDNS=1` (prereq) + `zerotier-systemd-manager` writes `99-ztuga7c2kh.network`, networkd applies it to resolved.
+- Verify: `resolvectl status ztuga7c2kh` shows `DNS` scope + the two servers + `wrede.pvt`.
+- NOTE: ZeroTier's *native* DNS push is a no-op on Linux (`setDns ... not implemented`), which is why the manager is required.
+
+### Cockpit dashboard
+- `https://<wayback>:9090` → "Van Router". Reload the browser after deploying plugin changes (Cockpit caches packages per session).
+- Read-only status works as any user; Prefer/Up/Down/Restart need *Administrative access* (polkit).
+- **Prefer** sets the manual WAN preference (see *Manual preference* above); **Up/Down** connect/disconnect the NM device; **Restart** restarts hostapd.
+
+---
+
+## 4. Deploy / rebuild
+
+```bash
+cd ~/vanlink
+sudo ./deploy.sh          # copies all files to their system locations, reloads + enables services
+```
+
+Two things `deploy.sh` does **not** do (one-time, manual):
+
+1. **zerotier-systemd-manager binary** (v0.4.0, hand-installed — not in the ZeroTier apt repo):
+   ```bash
+   # copy /usr/bin/zerotier-systemd-manager from a host that has it (e.g. wertvoll), then:
+   sudo systemctl enable --now zerotier-systemd-manager.timer
+   sudo zerotier-cli set d3ecf5726d041b2a allowDNS=1
+   ```
+2. **hostapd unmask** (Ubuntu ships it masked): `sudo systemctl unmask hostapd`.
+
+---
+
+## 5. Key design decisions & hard-won gotchas
+
+- **AP on hostapd, not NetworkManager.** NM's hotspot caps the rtw89 radio at HT20/20MHz; hostapd gives the full VHT80/HE (WiFi-6). The AP iface is therefore NM-*unmanaged*; networkd gives it its static IP.
+- **`rtw89` power-save must be off** (`disable_ps_mode=Y`) or the AP stops beaconing when idle and the SSID vanishes.
+- **RTL8852BU is USB-2.0 and hangs under load if it shares a USB hub.** Keep the AP dongle on its **own** USB controller, separate from the WAN ethernet. Symptom of a shared bus: `c2h reg timeout` + `Polling beacon packet empty fail` under throughput, SSID drops. (`timed out to flush queues` alone is benign.)
+- **van-failover changes metrics via `ip route`, never `nmcli device reapply`.** `reapply` **resets the r8152 USB-ethernet carrier**, which caused a ~10s-ping-drop flapping feedback loop. Pure `ip route` changes are carrier-safe.
+- **Disable EEE on USB ethernet** (`50-disable-eee` dispatcher) — its idle power-save parks the *backup* WAN link and breaks health probes.
+- **Per-WAN probing needs `curl --interface if!<dev>`** (forces `SO_BINDTODEVICE`); plain `--interface <name>` only sets the source IP and still routes via the default WAN. `rp_filter` is loose (`2`), required for this.
+- **`arp_ignore`/`arp_announce`** matter only when two WANs share a subnet (a home-LAN test artifact; Starlink + neighbour-wifi will be on different subnets in the van). Harmless to keep.
+- **dnsmasq uses `bind-dynamic` bound to the AP iface** so it coexists with systemd-resolved (no port-53 fight) — resolved stays intact for the host.
+
+---
+
+## 6. Recovery / safety
+- Three independent management paths: ethernet `.251`, wifi `.27`, ZeroTier `192.168.196.22`. Changing default-route metrics does **not** affect the local-subnet (SSH) routes.
+- The AP fan/EC, hostapd, networkd, nftables, sysctl, failover, regdomain are all enabled and reboot-persistent.
+
+## 7. Pending / future ideas
+- Real-world soak once the actual Starlink terminal + USB 4G/5G modem are installed (current eth is the home LAN; subnets/probe targets may want tuning).
+- Cockpit plugin: add an AP-config form (SSID/channel/PSK editing `hostapd.conf`). *(Manual failover override via the "Prefer" button is done — see §3.)*
+- Consider pinning a fixed interface name for the WAN ethernet (USB MAC-derived names change if the adapter is swapped).