From bd865831d66b183fff70fb138da0c09b96fa7a5e Mon Sep 17 00:00:00 2001 From: Andreas Wrede Date: Sun, 28 Jun 2026 15:08:04 -0400 Subject: [PATCH] first commit --- README.md | 174 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 174 insertions(+) create mode 100644 README.md diff --git a/README.md b/README.md new file mode 100644 index 0000000..36ab9a2 --- /dev/null +++ b/README.md @@ -0,0 +1,174 @@ +# vanlink — campervan router on `wayback` + +Turns **wayback** (Asus ZenBook UX391U, Ubuntu 24.04, zabbly kernel) into a self-contained +campervan hub/router/AP: + +- **WiFi-6 access point** for client devices (hand-rolled hostapd, not NetworkManager). +- **Multi-WAN failover** across WiFi (neighbour AP) → Ethernet (Starlink) → 4G/5G cellular. +- **ZeroTier-managed DNS** so `*.wrede.pvt` resolves over the overlay when away from home. +- **Cockpit dashboard** ("Van Router") for status + manual control. + +This directory is the source of truth. The live system files live under `/etc`, `/usr/...`; +`deploy.sh` copies from here to there. Edit here, run `sudo ./deploy.sh`, done. + +--- + +## 1. Architecture at a glance + +``` + clients ──WiFi(VanLink, 5GHz ch149, 80MHz WPA2) + │ + wlxc83a35a4ee55 (Realtek RTL8852BU, rtw89) 10.42.0.1/24 ← AP, hostapd + │ dnsmasq DHCP .10–.254 + DNS + │ nftables masquerade (oifname != AP) + │ ip_forward=1 + ┌─────────────────┼──────────────────────────────────────┐ + wlp1s0 (m100) enxd8ec5eeb3512 (m200) Koodo gsm (m300) + neighbour WiFi USB ethernet → Starlink 4G/5G modem (when present) + └──────── van-failover picks lowest-metric HEALTHY WAN ────┘ + + NetworkManager owns the WAN side. systemd-networkd owns the AP IP + the ZeroTier DNS link. +``` + +### Roles / responsibilities +| Concern | Owner | +|---|---| +| WAN interfaces (eth / wifi / gsm), DHCP-client, metrics | **NetworkManager** | +| AP interface IP (10.42.0.1) | **systemd-networkd** (`10-van-ap.network`) | +| AP beaconing / WPA | **hostapd** (AP iface is NM-*unmanaged*) | +| AP DHCP + DNS | **dnsmasq** (dedicated instance, bound to AP only) | +| NAT + forwarding | **nftables** + sysctl | +| WAN health + failover | **van-failover** daemon | +| ZeroTier DNS → resolved | **zerotier-systemd-manager** + systemd-networkd | +| Web UI | **Cockpit** + `vanrouter` plugin | + +### Key network facts +- AP LAN: `10.42.0.0/24`, gateway/AP `10.42.0.1`, DHCP `.10–.254`. +- WAN priority (metrics): **wifi 100 → eth/Starlink 200 → 4G 300** (lower = preferred). +- Management / recovery: ethernet `192.168.10.251`, wifi `192.168.10.27`, ZeroTier `192.168.196.22`. +- Cockpit: `https://192.168.10.251:9090` (or `.27`, or ZeroTier). Log in with a Unix account; enable *Administrative access* for action buttons. +- Regulatory domain **CA** (unlocks 5GHz ch149–161 @30dBm, no DFS). +- ZeroTier network `d3ecf5726d041b2a`, pushed DNS domain `wrede.pvt` via `192.168.196.115` + `192.168.10.5`. + +--- + +## 2. Directory layout → system destinations + +`deploy.sh` performs exactly this mapping. + +### `ap/` — access point +| file | → installs to | purpose | +|---|---|---| +| `hostapd.conf` | `/etc/hostapd/hostapd.conf` | AP: SSID `VanLink`, ch149, VHT80/HE, WPA2-PSK, country CA. **WPA passphrase lives here.** | +| `default-hostapd` | `/etc/default/hostapd` | `DAEMON_CONF=...` | +| `van-ap-dnsmasq.conf` | `/etc/van-ap/dnsmasq.conf` | DHCP/DNS bound to AP iface (`bind-dynamic`, so it does not clash with systemd-resolved) | +| `van-ap-dnsmasq.service` | `/etc/systemd/system/van-ap-dnsmasq.service` | dedicated dnsmasq unit (uses the `dnsmasq-base` binary; the distro dnsmasq service is NOT used) | +| `10-van-ap.network` | `/etc/systemd/network/10-van-ap.network` | static `10.42.0.1/24` on the AP iface (`ConfigureWithoutCarrier`) | +| `van-ap-unmanaged.conf` | `/etc/NetworkManager/conf.d/van-ap-unmanaged.conf` | tells NM to leave the AP iface alone | +| `nftables.conf` | `/etc/nftables.conf` | NAT: `masquerade ip saddr 10.42.0.0/24 oifname != ` → follows whatever WAN is active | +| `99-van-router.conf` | `/etc/sysctl.d/99-van-router.conf` | `net.ipv4.ip_forward=1` | +| `regdomain.service` | `/etc/systemd/system/regdomain.service` | `iw reg set CA` at boot, before NetworkManager | +| `rtw89.conf` | `/etc/modprobe.d/rtw89.conf` | `options rtw89_core disable_ps_mode=Y` (else AP drops beacon when idle) | + +### `failover/` — multi-WAN +| file | → installs to | purpose | +|---|---|---| +| `van-failover` | `/usr/local/sbin/van-failover` | the daemon (Python) | +| `config.json` | `/etc/van-failover/config.json` | WAN list + priorities + probe settings | +| `van-failover.service` | `/etc/systemd/system/van-failover.service` | `Restart=always` | +| `50-disable-eee` | `/etc/NetworkManager/dispatcher.d/50-disable-eee` | disables EEE on `en*`/`eth*` at connect (r8152 idle parking) | +| `99-van-arp.conf` | `/etc/sysctl.d/99-van-arp.conf` | `arp_ignore=1` / `arp_announce=2` (multi-NIC ARP hygiene) | + +### `dns/` — ZeroTier managed DNS +| file | → installs to | purpose | +|---|---|---| +| `zt-search.conf` | `/etc/systemd/network/99-ztuga7c2kh.network.d/search.conf` | makes `wrede.pvt` a search domain (bare-hostname completion) | +| `zt-network.local.conf` | `/var/lib/zerotier-one/networks.d/d3ecf5726d041b2a.local.conf` | contains `allowDNS=1` (prerequisite) | +| `99-ztuga7c2kh.network.generated` | *(reference only — generated by the manager)* | what zerotier-systemd-manager writes | +| `zerotier-systemd-manager.{service,timer}` | `/usr/lib/systemd/system/` | the manager (binary installed separately, see §4) | + +### `cockpit/vanrouter/` — web UI +`manifest.json`, `index.html`, `vanrouter.css`, `vanrouter.js` → `/usr/share/cockpit/vanrouter/`. + +--- + +## 3. Components & how to operate + +### Access Point +- Change SSID / channel / password: edit `ap/hostapd.conf`, then `sudo cp` it to `/etc/hostapd/hostapd.conf` (or `./deploy.sh`) and `sudo systemctl restart hostapd`. +- Width is 80MHz VHT + HE (WiFi-6). Channel 149 (UNII-3, non-DFS). +- DHCP range / DNS options: `ap/van-ap-dnsmasq.conf` + restart `van-ap-dnsmasq`. +- Services: `hostapd`, `van-ap-dnsmasq`, `systemd-networkd`, `nftables`, `regdomain`. +- **Do not** add `noscan` to hostapd.conf — Ubuntu's hostapd rejects it and fails to start. +- Verify it's actually beaconing with `iw dev wlxc83a35a4ee55 info` (ssid+channel+width present), not just `systemctl is-active hostapd`. + +### Multi-WAN failover (`van-failover`) +- Probes each WAN's real internet every few seconds (HTTP-204 check, captive-portal-aware), demotes a failed WAN by raising its route-metric, fails back on recovery (hysteresis). +- Status: `systemctl status van-failover`, `journalctl -u van-failover -f`, or the Cockpit "WAN Failover" card. Live state JSON at `/run/van-failover/state.json`. +- Tune priorities/timing in `failover/config.json` (`wans[].metric`, `probe_interval`, `probe_timeout`, `fail_threshold`, `ok_threshold`, `probe_urls`) then restart. +- WAN identity: eth/wifi keyed by `device`, cellular by NM `connection` name (`Koodo`). +- The daemon changes metrics with **`ip route`**, never `nmcli device reapply` — see Gotchas. + +#### Manual preference (Cockpit "Prefer" button) +- The daemon is the **single writer** of route-metrics. To force a specific WAN to the top it reads an optional file `/run/van-failover/prefer` containing one **device name** (e.g. `enxd8ec5eeb3512`). That WAN gets base metric **50** (below every config metric) so it wins while healthy; the health PENALTY still stacks on top, so failover/fail-back are unchanged. Empty/absent file = follow config priorities. +- The Cockpit "Prefer" button just writes that file (root via polkit) — it does **not** set metrics itself. Doing so directly would fight the daemon (reverted within one probe loop) and `reapply`-flap the USB carrier. The daemon applies the change on its next loop (~`probe_interval`); `state.json` carries a `preferred` flag the UI shows as "(preferred)". +- Single-WAN, sticky override with no explicit "clear": to return to automatic priority, prefer your top WAN (wifi) or `rm /run/van-failover/prefer`. **Ephemeral by design** — `/run` is wiped on reboot, so a reboot returns to config priorities. (To persist it, point `PREFER` at `/etc/van-failover/prefer` in the daemon and the matching path in `vanrouter.js`.) +- Swapping the preference between two WANs is collision-safe: a second default route can't take the new WAN's target metric while the old preferred WAN still holds it, so `enforce_route` keeps the existing route and retries next loop (settles in ~2 loops) instead of stranding the interface. `enforce_route` also restores a default route that went missing while the carrier is up (gateway from NM via `device_gateway`), not just rebases an existing one. + +### Adding the 4G/5G modem +1. Plug the USB modem in. ModemManager + the existing `Koodo` gsm NM connection (autoconnect) bring it up. +2. It auto-joins as the `cellular` WAN at metric 300 (last resort). Nothing else to configure. +3. Confirm with `mmcli -L` and the Cockpit failover card (cellular flips from `absent` to `up`). + +### ZeroTier managed DNS +- `wrede.pvt` resolves over ZeroTier when off the home LAN. Mechanism: `allowDNS=1` (prereq) + `zerotier-systemd-manager` writes `99-ztuga7c2kh.network`, networkd applies it to resolved. +- Verify: `resolvectl status ztuga7c2kh` shows `DNS` scope + the two servers + `wrede.pvt`. +- NOTE: ZeroTier's *native* DNS push is a no-op on Linux (`setDns ... not implemented`), which is why the manager is required. + +### Cockpit dashboard +- `https://:9090` → "Van Router". Reload the browser after deploying plugin changes (Cockpit caches packages per session). +- Read-only status works as any user; Prefer/Up/Down/Restart need *Administrative access* (polkit). +- **Prefer** sets the manual WAN preference (see *Manual preference* above); **Up/Down** connect/disconnect the NM device; **Restart** restarts hostapd. + +--- + +## 4. Deploy / rebuild + +```bash +cd ~/vanlink +sudo ./deploy.sh # copies all files to their system locations, reloads + enables services +``` + +Two things `deploy.sh` does **not** do (one-time, manual): + +1. **zerotier-systemd-manager binary** (v0.4.0, hand-installed — not in the ZeroTier apt repo): + ```bash + # copy /usr/bin/zerotier-systemd-manager from a host that has it (e.g. wertvoll), then: + sudo systemctl enable --now zerotier-systemd-manager.timer + sudo zerotier-cli set d3ecf5726d041b2a allowDNS=1 + ``` +2. **hostapd unmask** (Ubuntu ships it masked): `sudo systemctl unmask hostapd`. + +--- + +## 5. Key design decisions & hard-won gotchas + +- **AP on hostapd, not NetworkManager.** NM's hotspot caps the rtw89 radio at HT20/20MHz; hostapd gives the full VHT80/HE (WiFi-6). The AP iface is therefore NM-*unmanaged*; networkd gives it its static IP. +- **`rtw89` power-save must be off** (`disable_ps_mode=Y`) or the AP stops beaconing when idle and the SSID vanishes. +- **RTL8852BU is USB-2.0 and hangs under load if it shares a USB hub.** Keep the AP dongle on its **own** USB controller, separate from the WAN ethernet. Symptom of a shared bus: `c2h reg timeout` + `Polling beacon packet empty fail` under throughput, SSID drops. (`timed out to flush queues` alone is benign.) +- **van-failover changes metrics via `ip route`, never `nmcli device reapply`.** `reapply` **resets the r8152 USB-ethernet carrier**, which caused a ~10s-ping-drop flapping feedback loop. Pure `ip route` changes are carrier-safe. +- **Disable EEE on USB ethernet** (`50-disable-eee` dispatcher) — its idle power-save parks the *backup* WAN link and breaks health probes. +- **Per-WAN probing needs `curl --interface if!`** (forces `SO_BINDTODEVICE`); plain `--interface ` only sets the source IP and still routes via the default WAN. `rp_filter` is loose (`2`), required for this. +- **`arp_ignore`/`arp_announce`** matter only when two WANs share a subnet (a home-LAN test artifact; Starlink + neighbour-wifi will be on different subnets in the van). Harmless to keep. +- **dnsmasq uses `bind-dynamic` bound to the AP iface** so it coexists with systemd-resolved (no port-53 fight) — resolved stays intact for the host. + +--- + +## 6. Recovery / safety +- Three independent management paths: ethernet `.251`, wifi `.27`, ZeroTier `192.168.196.22`. Changing default-route metrics does **not** affect the local-subnet (SSH) routes. +- The AP fan/EC, hostapd, networkd, nftables, sysctl, failover, regdomain are all enabled and reboot-persistent. + +## 7. Pending / future ideas +- Real-world soak once the actual Starlink terminal + USB 4G/5G modem are installed (current eth is the home LAN; subnets/probe targets may want tuning). +- Cockpit plugin: add an AP-config form (SSID/channel/PSK editing `hostapd.conf`). *(Manual failover override via the "Prefer" button is done — see §3.)* +- Consider pinning a fixed interface name for the WAN ethernet (USB MAC-derived names change if the adapter is swapped).