vanlink/README.md

# vanlink — campervan router on `wayback`

Turns **wayback** (Asus ZenBook UX391U, Ubuntu 24.04, zabbly kernel) into a self-contained
campervan hub/router/AP:

- **WiFi-6 access point** for client devices (hand-rolled hostapd, not NetworkManager).
- **Multi-WAN failover** across WiFi (neighbour AP) → Ethernet (Starlink) → 4G/5G cellular.
- **ZeroTier-managed DNS** so `*.wrede.pvt` resolves over the overlay when away from home.
- **Cockpit dashboard** ("Van Router") for status + manual control.

This directory is the source of truth. The live system files live under `/etc`, `/usr/...`;
`deploy.sh` copies from here to there. Edit here, run `sudo ./deploy.sh`, done.

---

## 1. Architecture at a glance

```
     WiFi clients (VanLink, 5GHz ch149, 80MHz WPA2)   wired LAN clients
        wlxc83a35a4ee55 (RTL8852BU, rtw89, hostapd)   enx00e04c331140
                          └───────────┬───────────────────┘
                                br0   10.42.0.1/24   ← LAN bridge
                          │  dnsmasq DHCP .10–.254 + DNS
                          │  nftables masquerade (oifname != br0)
                          │  ip_forward=1
        ┌─────────────────┼──────────────────────────────────────┐
   wlp1s0 (m100)     enxd8ec5eeb3512 (m200)        Koodo gsm (m300)
   neighbour WiFi    USB ethernet → Starlink       4G/5G modem (when present)
        └──────── van-failover picks lowest-metric HEALTHY WAN ────┘

   NetworkManager owns the WAN side.  systemd-networkd owns the AP IP + the ZeroTier DNS link.
```

### Roles / responsibilities
| Concern | Owner |
|---|---|
| WAN interfaces (eth / wifi / gsm), DHCP-client, metrics | **NetworkManager** |
| LAN bridge `br0` + IP (10.42.0.1) + wired LAN member | **systemd-networkd** (`2x-van-br0/lan`) |
| AP beaconing / WPA + adding the wlan to `br0` | **hostapd** (`bridge=br0`; AP iface + wired port are NM-*unmanaged*) |
| Keep the AP beaconing across USB re-enumeration (Starlink flap) | **hostapd** `Restart=always`/no start-limit drop-in + **van-ap-watchdog** daemon (recovers a wedged radio) |
| AP DHCP + DNS | **dnsmasq** (dedicated instance, bound to AP only) |
| NAT + forwarding | **nftables** + sysctl |
| WAN health + failover | **van-failover** daemon |
| Temperature monitor / alert / log | **van-thermal** daemon |
| Battery monitor / low-charge alert + shutdown | **van-battery** daemon |
| Auto-reboot on hang | **systemd hardware watchdog** (`intel_oc_wdt`) |
| Liveness / dead-man's switch + metrics | **hbc** heartbeat client → hbd.wrede.pvt |
| ZeroTier DNS → resolved | **zerotier-systemd-manager** + systemd-networkd |
| Web UI | **Cockpit** + `vanrouter` plugin |

### Key network facts
- LAN: `10.42.0.0/24`, gateway `10.42.0.1`, DHCP `.10–.254` — shared by Wi-Fi (`VanLink`) and the wired port `enx00e04c331140`, both bridged into `br0`.
- WAN priority (metrics): **wifi 100 → eth/Starlink 200 → 4G 300** (lower = preferred).
- Management / recovery: ethernet `192.168.10.251`, wifi `192.168.10.27`, ZeroTier `192.168.196.22`.
- Cockpit: `https://192.168.10.251:9090` (or `.27`, or ZeroTier). Log in with a Unix account; enable *Administrative access* for action buttons.
- Regulatory domain **CA** (unlocks 5GHz ch149–161 @30dBm, no DFS).
- ZeroTier network `d3ecf5726d041b2a`, pushed DNS domain `wrede.pvt` via `192.168.196.115` + `192.168.10.5`.

---

## 2. Directory layout → system destinations

`deploy.sh` performs exactly this mapping.

### `ap/` — access point
| file | → installs to | purpose |
|---|---|---|
| `hostapd.conf` | `/etc/hostapd/hostapd.conf` | AP: SSID `VanLink`, ch149, VHT80/HE, WPA2-PSK, country CA, `ctrl_interface` for the watchdog. **WPA passphrase lives here.** |
| `hostapd-restart.conf` | `/etc/systemd/system/hostapd.service.d/restart.conf` | `Restart=always` + `StartLimitIntervalSec=0` so hostapd never gives up after a USB re-enumeration bounces the radio |
| `van-ap-watchdog` | `/usr/local/sbin/van-ap-watchdog` | watches `hostapd_cli status`; restarts hostapd if the radio wedges (running but not `ENABLED`) |
| `van-ap-watchdog.service` | `/etc/systemd/system/van-ap-watchdog.service` | runs the watchdog |
| `default-hostapd` | `/etc/default/hostapd` | `DAEMON_CONF=...` |
| `van-ap-dnsmasq.conf` | `/etc/van-ap/dnsmasq.conf` | DHCP/DNS bound to AP iface (`bind-dynamic`, so it does not clash with systemd-resolved) |
| `van-ap-dnsmasq.service` | `/etc/systemd/system/van-ap-dnsmasq.service` | dedicated dnsmasq unit (uses the `dnsmasq-base` binary; the distro dnsmasq service is NOT used) |
| `10-van-ap.network` | `/etc/systemd/network/10-van-ap.network` | brings the AP wlan up with **no** IP (it's a `br0` member; hostapd enslaves it) |
| `20-van-br0.netdev` | `/etc/systemd/network/20-van-br0.netdev` | creates the LAN bridge `br0` (STP off) |
| `21-van-br0.network` | `/etc/systemd/network/21-van-br0.network` | static `10.42.0.1/24` on `br0` (`ConfigureWithoutCarrier`) |
| `22-van-lan.network` | `/etc/systemd/network/22-van-lan.network` | enslaves the wired LAN port `enx00e04c331140` to `br0` |
| `van-ap-unmanaged.conf` | `/etc/NetworkManager/conf.d/van-ap-unmanaged.conf` | tells NM to leave the AP wlan **and** the wired LAN port alone |
| `nftables.conf` | `/etc/nftables.conf` | NAT: `masquerade ip saddr 10.42.0.0/24 oifname != br0` → follows whatever WAN is active |
| `99-van-router.conf` | `/etc/sysctl.d/99-van-router.conf` | `net.ipv4.ip_forward=1` |
| `regdomain.service` | `/etc/systemd/system/regdomain.service` | `iw reg set CA` at boot, before NetworkManager |
| `rtw89.conf` | `/etc/modprobe.d/rtw89.conf` | `options rtw89_core disable_ps_mode=Y` (else AP drops beacon when idle) |

### `failover/` — multi-WAN
| file | → installs to | purpose |
|---|---|---|
| `van-failover` | `/usr/local/sbin/van-failover` | the daemon (Python) |
| `config.json` | `/etc/van-failover/config.json` | WAN list + priorities + probe settings |
| `van-failover.service` | `/etc/systemd/system/van-failover.service` | `Restart=always` |
| `50-disable-eee` | `/etc/NetworkManager/dispatcher.d/50-disable-eee` | disables EEE on `en*`/`eth*` at connect (r8152 idle parking) |
| `99-van-arp.conf` | `/etc/sysctl.d/99-van-arp.conf` | `arp_ignore=1` / `arp_announce=2` (multi-NIC ARP hygiene) |

### `dns/` — ZeroTier managed DNS
| file | → installs to | purpose |
|---|---|---|
| `zt-search.conf` | `/etc/systemd/network/99-ztuga7c2kh.network.d/search.conf` | makes `wrede.pvt` a search domain (bare-hostname completion) |
| `zt-network.local.conf` | `/var/lib/zerotier-one/networks.d/d3ecf5726d041b2a.local.conf` | contains `allowDNS=1` (prerequisite) |
| `99-ztuga7c2kh.network.generated` | *(reference only — generated by the manager)* | what zerotier-systemd-manager writes |
| `zerotier-systemd-manager.{service,timer}` | `/usr/lib/systemd/system/` | the manager (binary installed separately, see §4) |

### `cockpit/vanrouter/` — web UI
`manifest.json`, `index.html`, `vanrouter.css`, `vanrouter.js` → `/usr/share/cockpit/vanrouter/`.

### `power/` — never sleep + thermal monitor
| file | → installs to | purpose |
|---|---|---|
| `10-vanlink-nolid.conf` | `/etc/systemd/logind.conf.d/10-vanlink-nolid.conf` | logind ignores the lid in all states (closed / on AC / docked) |
| `van-thermal` | `/usr/local/sbin/van-thermal` | temperature daemon (Python): publishes state, alerts, logs history |
| `thermal-config.json` | `/etc/van-thermal/config.json` | sensors + warn/crit thresholds + sample/log intervals |
| `van-thermal.service` | `/etc/systemd/system/van-thermal.service` | `Restart=always` |
| `van-battery` | `/usr/local/sbin/van-battery` | battery daemon (Python): Pushover low-charge alerts + safe shutdown |
| `battery-config.json` | `/etc/van-battery/config.json` | warn levels, shutdown level, poll interval, paths |
| `van-battery.service` | `/etc/systemd/system/van-battery.service` | `Restart=always` |
| `pushover.json.example` | → `/etc/van-battery/pushover.json` (seeded if absent) | Pushover token+user **template**; real file is 0600, **not** in the repo |
| `10-vanlink-watchdog.conf` | `/etc/systemd/system.conf.d/10-vanlink-watchdog.conf` | hardware watchdog (`RuntimeWatchdogSec=20s`); applied via `daemon-reexec` |

`deploy.sh` additionally **masks** `sleep.target suspend.target hibernate.target hybrid-sleep.target`
(no repo file — symlinks to `/dev/null` under `/etc/systemd/system/`) so nothing else can suspend either.

### `heartbeat/` — dead-man's switch
| file | → installs to | purpose |
|---|---|---|
| `hbc.yaml` | `/etc/hbc.yaml` | heartbeat client config (interval + metric plugins) |
| `hbc.service` | `/etc/systemd/system/hbc.service` | runs `~/bin/hbc` as `andreas` → hbd.wrede.pvt:50003 |

The `hbc` binary itself (venv at `~/venvs/hbd`, symlink `~/bin/hbc`) is installed once via the
heartbeat project's own installer — see §4. `deploy.sh` only starts the service once it exists.

---

## 3. Components & how to operate

### Access Point
- Change SSID / channel / password: edit `ap/hostapd.conf`, then `sudo cp` it to `/etc/hostapd/hostapd.conf` (or `./deploy.sh`) and `sudo systemctl restart hostapd`.
- Width is 80MHz VHT + HE (WiFi-6). Channel 149 (UNII-3, non-DFS).
- DHCP range / DNS options: `ap/van-ap-dnsmasq.conf` + restart `van-ap-dnsmasq`.
- Services: `hostapd`, `van-ap-dnsmasq`, `systemd-networkd`, `nftables`, `regdomain`.
- **Do not** add `noscan` to hostapd.conf — Ubuntu's hostapd rejects it and fails to start.
- Verify it's actually beaconing with `iw dev wlxc83a35a4ee55 info` (ssid+channel+width present), not just `systemctl is-active hostapd`.

### Multi-WAN failover (`van-failover`)
- Probes each WAN's real internet every few seconds (HTTP-204 check, captive-portal-aware), demotes a failed WAN by raising its route-metric, fails back on recovery (hysteresis).
- Status: `systemctl status van-failover`, `journalctl -u van-failover -f`, or the Cockpit "WAN Failover" card. Live state JSON at `/run/van-failover/state.json`.
- Tune priorities/timing in `failover/config.json` (`wans[].metric`, `probe_interval`, `probe_timeout`, `fail_threshold`, `ok_threshold`, `probe_urls`) then restart.
- WAN identity: eth/wifi keyed by `device`, cellular by NM `connection` name (`Koodo`).
- The daemon changes metrics with **`ip route`**, never `nmcli device reapply` — see Gotchas.

#### Manual preference (Cockpit "Prefer" button)
- The daemon is the **single writer** of route-metrics. To force a specific WAN to the top it reads an optional file `/run/van-failover/prefer` containing one **device name** (e.g. `enxd8ec5eeb3512`). That WAN gets base metric **50** (below every config metric) so it wins while healthy; the health PENALTY still stacks on top, so failover/fail-back are unchanged. Empty/absent file = follow config priorities.
- The Cockpit "Prefer" button just writes that file (root via polkit) — it does **not** set metrics itself. Doing so directly would fight the daemon (reverted within one probe loop) and `reapply`-flap the USB carrier. The daemon applies the change on its next loop (~`probe_interval`); `state.json` carries a `preferred` flag the UI shows as "(preferred)".
- Single-WAN, sticky override with no explicit "clear": to return to automatic priority, prefer your top WAN (wifi) or `rm /run/van-failover/prefer`. **Ephemeral by design** — `/run` is wiped on reboot, so a reboot returns to config priorities. (To persist it, point `PREFER` at `/etc/van-failover/prefer` in the daemon and the matching path in `vanrouter.js`.)
- Swapping the preference between two WANs is collision-safe: a second default route can't take the new WAN's target metric while the old preferred WAN still holds it, so `enforce_route` keeps the existing route and retries next loop (settles in ~2 loops) instead of stranding the interface. `enforce_route` also restores a default route that went missing while the carrier is up (gateway from NM via `device_gateway`), not just rebases an existing one.

### Thermal monitor (`van-thermal`)
- One daemon off a single sysfs sample loop does all three jobs: live Cockpit readout, threshold alerting, history.
- **Live state**: `/run/van-thermal/state.json` (atomic-swapped each sample). The Cockpit "Temperatures" card reads this — no shelling out to `sensors` per refresh. Sensors are resolved by hwmon **name** + **label** (`coretemp`/`Package id 0`, `nvme`/`Composite`), never by `hwmonN` index (not stable across boots).
- **Alerts**: level changes (ok ↔ warn ↔ crit) are logged to the journal with hysteresis (`clear_margin`, default 5 °C) so a sensor sitting on the line doesn't spam. Watch with `journalctl -u van-thermal -f`; warn/crit carry sd-daemon severity (`-p warning`/`-p err`).
- **History**: throttled CSV at `/var/log/van-thermal.csv` (one row per `log_interval`, default 60 s), self-rotating to `.csv.1` past `log_max_bytes` (5 MB).
- **Tuning**: edit `/etc/van-thermal/config.json` (thresholds, intervals, sensor list), then `systemctl restart van-thermal`. Defaults: CPU warn 80 / crit 95 °C (silicon crit is 100), NVMe warn 65 / crit 70 °C (drive crit ~71).
- Status: `systemctl status van-thermal` or `cat /run/van-thermal/state.json`.

### Battery monitor (`van-battery`)
- Watches mains vs battery via `/sys/class/power_supply/AC0/online` (0 = on battery) and charge via `BAT0/capacity`. Both resolve by `type` (Mains/Battery) if those names ever differ.
- **Only while on battery**, it sends escalating **Pushover** alerts at **25 / 20 / 15 %**, and at **10 %** sends a final alert and runs `systemctl poweroff` (after `shutdown_grace`, default 8 s, so the alert flushes first).
- Edge-triggered per discharge episode: each severity fires once; the sequence **re-arms when mains returns**. Unplugging already below a warn level fires a single alert for the current severity (no burst), then shutdown at 10 %.
- **Credentials** live in `/etc/van-battery/pushover.json` (mode 0600), seeded from `pushover.json.example` on first deploy and **never committed**. Missing/placeholder creds disable *sending* but **not** the shutdown — running flat must always power down safely (the skip is logged to the journal).
- Tune in `/etc/van-battery/config.json` (`warn_levels`, `shutdown_level`, `poll_interval`), then `systemctl restart van-battery`.
- Status: `systemctl status van-battery`, `journalctl -u van-battery -f`, or `cat /run/van-battery/state.json`.
- **Caveat — the 10 % shutdown is one-way.** A laptop won't power itself back on when mains returns: "restore on AC loss" is a BIOS/firmware feature (not OS-controllable — `/proc/acpi/wakeup` only covers wake-from-suspend). If your BIOS exposes an "AC power-on / restore on AC loss" option, enable it so the router reboots itself once shore/solar power is back; this ZenBook likely doesn't have it, in which case a low-battery shutdown needs a manual power-on.

### Adding the 4G/5G modem
1. Plug the USB modem in. ModemManager + the existing `Koodo` gsm NM connection (autoconnect) bring it up.
2. It auto-joins as the `cellular` WAN at metric 300 (last resort). Nothing else to configure.
3. Confirm with `mmcli -L` and the Cockpit failover card (cellular flips from `absent` to `up`).

### ZeroTier managed DNS
- `wrede.pvt` resolves over ZeroTier when off the home LAN. Mechanism: `allowDNS=1` (prereq) + `zerotier-systemd-manager` writes `99-ztuga7c2kh.network`, networkd applies it to resolved.
- Verify: `resolvectl status ztuga7c2kh` shows `DNS` scope + the two servers + `wrede.pvt`.
- NOTE: ZeroTier's *native* DNS push is a no-op on Linux (`setDns ... not implemented`), which is why the manager is required.
- `systemd-networkd-wait-online` is **masked** so the ZT link does **not** gate `network-online.target` — see Gotchas.

### Cockpit dashboard
- `https://<wayback>:9090` → "Van Router". Reload the browser after deploying plugin changes (Cockpit caches packages per session).
- Read-only status works as any user; Prefer/Up/Down/Restart need *Administrative access* (polkit).
- **Prefer** sets the manual WAN preference (see *Manual preference* above); **Up/Down** connect/disconnect the NM device; **Restart** restarts hostapd.

### Never sleep (lid-closed operation)
- wayback lives lid-closed in the van and must stay up. Stock logind `HandleLidSwitch=suspend` would sleep it on lid close (even on AC). The `power/10-vanlink-nolid.conf` drop-in sets all three lid actions to `ignore`; `deploy.sh` also masks every sleep target so idle / GUI / a stray `systemctl suspend` can't suspend it.
- Verify: `busctl get-property org.freedesktop.login1 /org/freedesktop/login1 org.freedesktop.login1.Manager HandleLidSwitch` → `"ignore"`, and `systemctl is-enabled suspend.target` → `masked`. `busctl call ... CanSuspend` should return `"no"`.
- **To re-enable sleep** (e.g. if wayback is ever a laptop again):
  ```bash
  sudo rm /etc/systemd/logind.conf.d/10-vanlink-nolid.conf
  sudo systemctl unmask sleep.target suspend.target hibernate.target hybrid-sleep.target
  sudo systemctl restart systemd-logind
  ```
  (then remove the `power/` block from `deploy.sh`, or the next deploy re-applies it).

---

## 4. Deploy / rebuild

```bash
cd ~/vanlink
sudo ./deploy.sh          # copies all files to their system locations, reloads + enables services
```

Four things `deploy.sh` does **not** do (one-time, manual):

1. **zerotier-systemd-manager binary** (v0.4.0, hand-installed — not in the ZeroTier apt repo):
   ```bash
   # copy /usr/bin/zerotier-systemd-manager from a host that has it (e.g. wertvoll), then:
   sudo systemctl enable --now zerotier-systemd-manager.timer
   sudo zerotier-cli set d3ecf5726d041b2a allowDNS=1
   ```
2. **hostapd unmask** (Ubuntu ships it masked): `sudo systemctl unmask hostapd`.
3. **Pushover credentials** for `van-battery` — deploy seeds `/etc/van-battery/pushover.json` (0600) with placeholders; fill in your app token + user key, then `sudo systemctl restart van-battery`. Until then low-battery alerts are skipped (logged), but the 10 % auto-shutdown still works.
4. **Heartbeat client `hbc`** (dead-man's switch) — install the binary once into andreas' venv:
   ```bash
   git clone https://git.wrede.ca/andreas/heartbeat.git ~/git/heartbeat   # if not already cloned
   sh ~/git/heartbeat/scripts/hb_install.sh client                        # -> ~/venvs/hbd + ~/bin/hbc
   ```
   `deploy.sh` installs `/etc/hbc.yaml` + the `hbc.service` unit and starts it once `~/bin/hbc` exists. The service reports to **hbd.wrede.pvt** (UDP 50003) as user `andreas`. Upgrade later with `~/bin/hb_install.sh client`.

---

## 5. Key design decisions & hard-won gotchas

- **AP on hostapd, not NetworkManager.** NM's hotspot caps the rtw89 radio at HT20/20MHz; hostapd gives the full VHT80/HE (WiFi-6). The AP iface is therefore NM-*unmanaged*; networkd gives it its static IP.
- **`rtw89` power-save must be off** (`disable_ps_mode=Y`) or the AP stops beaconing when idle and the SSID vanishes.
- **RTL8852BU is USB-2.0 and hangs under load if it shares a USB hub.** Keep the AP dongle on its **own** USB controller, separate from the WAN ethernet. Symptom of a shared bus: `c2h reg timeout` + `Polling beacon packet empty fail` under throughput, SSID drops. (`timed out to flush queues` alone is benign.)
- **van-failover changes metrics via `ip route`, never `nmcli device reapply`.** `reapply` **resets the r8152 USB-ethernet carrier**, which caused a ~10s-ping-drop flapping feedback loop. Pure `ip route` changes are carrier-safe.
- **Disable EEE on USB ethernet** (`50-disable-eee` dispatcher) — its idle power-save parks the *backup* WAN link and breaks health probes.
- **Per-WAN probing needs `curl --interface if!<dev>`** (forces `SO_BINDTODEVICE`); plain `--interface <name>` only sets the source IP and still routes via the default WAN. `rp_filter` is loose (`2`), required for this.
- **`arp_ignore`/`arp_announce`** matter only when two WANs share a subnet (a home-LAN test artifact; Starlink + neighbour-wifi will be on different subnets in the van). Harmless to keep.
- **dnsmasq uses `bind-dynamic` bound to the AP iface** so it coexists with systemd-resolved (no port-53 fight) — resolved stays intact for the host.
- **`systemd-networkd-wait-online` must be masked** or it deadlocks boot. networkd here manages only two links — the AP (`wlxc…`, local-only) and the ZT overlay `ztuga7c2kh` — and **neither is a real uplink**. The ZT link only appears once `zerotier-one` starts, but `zerotier-one` is ordered `After=network-online.target`, so wait-online would block on the not-yet-existing link until its 120s timeout. Marking the links `RequiredForOnline=no` does **not** help: with *zero* candidate links, wait-online can never satisfy "online" and times out anyway (systemd 255 — verified: `Timeout occurred while waiting for network connectivity` at exactly +120s). Both add ~2min to every boot and delay ZeroTier (the recovery path). The fix is to mask the service entirely (`deploy.sh` does this); real uplink readiness is covered by `NetworkManager-wait-online`, which owns the actual WANs.

---

## 6. Recovery / safety
- Three independent management paths: ethernet `.251`, wifi `.27`, ZeroTier `192.168.196.22`. Changing default-route metrics does **not** affect the local-subnet (SSH) routes.
- The AP fan/EC, hostapd, networkd, nftables, sysctl, failover, regdomain are all enabled and reboot-persistent.

## 7. Pending / future ideas
- Real-world soak once the actual Starlink terminal + USB 4G/5G modem are installed (current eth is the home LAN; subnets/probe targets may want tuning).
- Cockpit plugin: add an AP-config form (SSID/channel/PSK editing `hostapd.conf`). *(Manual failover override via the "Prefer" button is done — see §3.)*
- Consider pinning a fixed interface name for the WAN ethernet (USB MAC-derived names change if the adapter is swapped).