andreas/vanlink
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
wan
vanlink/README.md
T
Andreas WredeandClaude Sonnet 5 5c793186ef ap: switch 5GHz driver to out-of-tree morrownr/rtw89 for USB3 support 
The in-kernel rtw89_8852bu on this Ubuntu kernel base predates mainline's
USB2->3 auto-switch for this chip, permanently capping the AP dongle at
USB2/480M. morrownr/rtw89 (dkms) has that switch. ap/rtw89.conf now
blacklists the in-kernel rtw89 modules and tunes the replacement
(disable_ps_mode + switch_usb_mode); ap/install-rtw89-driver.sh builds
and installs it, pinned to a specific upstream commit, kept separate
from deploy.sh since a dkms rebuild is too slow to run on every deploy.

Verified live: negotiates USB3/5000M on a dedicated USB3 controller
(480M on a USB2-only one, as expected), AP recovered via hostapd's
Restart=always + van-ap-watchdog with no manual intervention. Hit and
documented one real gotcha along the way: in-kernel rtw89_core refused
to unload while its own dependents (rtw89_8852b, rtw89_8852b_common)
were still loaded, which blocked the new module with a duplicate-symbol
error until all in-kernel modules were removed first.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YJfEELeh3ercpRBp8yYrYS
2026-08-23 15:42:23 -04:00

388 lines
35 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# vanlink — campervan router on `wayback`
> **This clone = the Pi 4 port (host `wan`, branch `wan`).** Onboard `eth0`+`wlan0` are
> NM-managed WANs (netplan `ap/50-van-wan.yaml`); the AP radios + wired LAN port arrive
> with the USB hub from wayback (same MAC-derived names, so all configs port verbatim).
> Dropped here: HA VM, battery/lid (no hardware), heartbeat + ZeroTier (not installed yet).
> Watchdog is 10s (bcm2835 max 15s); thermal watches `cpu_thermal`.
Turns **wayback** (Asus ZenBook UX391U, Ubuntu 24.04, zabbly kernel) into a self-contained
campervan hub/router/AP:
- **WiFi-6 access point** for client devices (hand-rolled hostapd, not NetworkManager).
- **Multi-WAN failover** across WiFi (neighbour AP) → Ethernet (Starlink) → 4G/5G cellular.
- **ZeroTier-managed DNS** so `*.wrede.pvt` resolves over the overlay when away from home.
- **Cockpit dashboard** ("Van Router") for status + manual control.
This directory is the source of truth. The live system files live under `/etc`, `/usr/...`;
`deploy.sh` copies from here to there. Edit here, run `sudo ./deploy.sh`, done.
---
## 1. Architecture at a glance
```
WiFi clients (VanLink, 5GHz ch149, 80MHz WPA2) wired LAN clients
wlxc83a35a4ee55 (RTL8852BU, rtw89, hostapd) enx00e04c331140
└───────────┬───────────────────┘
br0 10.42.0.1/24 ← LAN bridge
│ dnsmasq DHCP .10–.254 + DNS
│ nftables masquerade (oifname != br0)
│ ip_forward=1
┌─────────────────┼──────────────────────────────────────┐
wlp1s0 (m100) enxd8ec5eeb3512 (m200) Koodo gsm (m300)
neighbour WiFi USB ethernet → Starlink 4G/5G modem (when present)
└──────── van-failover picks lowest-metric HEALTHY WAN ────┘
NetworkManager owns the WAN side. systemd-networkd owns the AP IP + the ZeroTier DNS link.
```
### Roles / responsibilities
| Concern | Owner |
|---|---|
| WAN interfaces (eth / wifi / gsm), DHCP-client, metrics | **NetworkManager** |
| LAN bridge `br0` + IP (10.42.0.1) + wired LAN member | **systemd-networkd** (`2x-van-br0/lan`) |
| AP beaconing / WPA + adding the wlan to `br0` | **hostapd** (`bridge=br0`; AP iface + wired port are NM-*unmanaged*) |
| Keep the AP beaconing across USB re-enumeration (Starlink flap) | **hostapd** `Restart=always`/no start-limit drop-in + **van-ap-watchdog** daemon (recovers a wedged radio) |
| AP DHCP + DNS | **dnsmasq** (dedicated instance, bound to AP only) |
| NAT + forwarding | **nftables** + sysctl |
| WAN health + failover | **van-failover** daemon |
| Temperature monitor / alert / log | **van-thermal** daemon |
| NVMe I/O-timeout/reset watchdog + alert | **van-nvme-watch** daemon |
| Battery monitor / low-charge alert + shutdown | **van-battery** daemon |
| Auto-reboot on hang | **systemd hardware watchdog** (`intel_oc_wdt`) |
| Liveness / dead-man's switch + metrics | **hbc** heartbeat client → hbd.wrede.pvt |
| ZeroTier DNS → resolved | **zerotier-systemd-manager** + systemd-networkd |
| Web UI | **Cockpit** + `vanrouter` plugin |
### Key network facts
- LAN: `10.42.0.0/24`, gateway `10.42.0.1`, DHCP `.10–.254` — shared by Wi-Fi (`VanLink`, dual-band: 5GHz `wlxc83a35a4ee55` + 2.4GHz `wlxd8ec5e2faa8c`) and the wired port `enx00e04c331140`, all bridged into `br0`.
- WAN priority (metrics): **wifi 100 → eth/Starlink 200 → 4G 300** (lower = preferred).
- Management / recovery: ethernet `192.168.10.251`, wifi `192.168.10.27`, ZeroTier `192.168.196.22`.
- Cockpit: `https://192.168.10.251:9090` (or `.27`, or ZeroTier). Log in with a Unix account; enable *Administrative access* for action buttons.
- Regulatory domain **CA** (unlocks 5GHz ch149–161 @30dBm, no DFS).
- ZeroTier network `d3ecf5726d041b2a`, pushed DNS domain `wrede.pvt` via `192.168.196.115` + `192.168.10.5`.
- Home Assistant runs **natively on this host** (Podman Quadlet, host network) at **`http://10.42.0.1:8123`** (name `homeassistant`). The old HAOS VM `ha_van` (`10.42.0.50`) was retired 2026-07-07.
---
## 2. Directory layout → system destinations
`deploy.sh` performs exactly this mapping.
### `ap/` — access point
| file | → installs to | purpose |
|---|---|---|
| `hostapd.conf` | `/etc/hostapd/hostapd.conf` | AP: SSID `VanLink`, ch149, VHT80/HE, WPA2-PSK, country CA, `ctrl_interface` for the watchdog. **WPA passphrase lives here.** |
| `hostapd-restart.conf` | `/etc/systemd/system/hostapd.service.d/restart.conf` | `Restart=always` + `StartLimitIntervalSec=0` so hostapd never gives up after a USB re-enumeration bounces the radio |
| `van-ap-watchdog` | `/usr/local/sbin/van-ap-watchdog` | watches `hostapd_cli status`; restarts hostapd if the radio wedges (running but not `ENABLED`) |
| `van-ap-watchdog.service` | `/etc/systemd/system/van-ap-watchdog.service` | runs the watchdog |
| `default-hostapd` | `/etc/default/hostapd` | `DAEMON_CONF=...` |
| `van-ap-dnsmasq.conf` | `/etc/van-ap/dnsmasq.conf` | DHCP/DNS bound to AP iface (`bind-dynamic`, so it does not clash with systemd-resolved) |
| `van-ap-dnsmasq.service` | `/etc/systemd/system/van-ap-dnsmasq.service` | dedicated dnsmasq unit (uses the `dnsmasq-base` binary; the distro dnsmasq service is NOT used) |
| `hostapd-2g.conf` | `/etc/hostapd/hostapd-2g.conf` | 2.4GHz AP (Linksys WUSB6300v2 / RTL8822BU): same SSID+PSK, ch6, HT20. Own hostapd instance so one radio wedging never kills the other. |
| `hostapd-2g.service` | `/etc/systemd/system/hostapd-2g.service` | dedicated unit for the 2.4GHz hostapd (same Restart=always / no-start-limit semantics as the 5GHz drop-in) |
| `van-ap-watchdog-2g.service` | `/etc/systemd/system/van-ap-watchdog-2g.service` | second watchdog instance: `van-ap-watchdog /etc/hostapd/hostapd-2g.conf hostapd-2g` |
| `10-van-ap.network` | `/etc/systemd/network/10-van-ap.network` | brings the AP wlan up with **no** IP (it's a `br0` member; hostapd enslaves it) |
| `11-van-ap-2g.network` | `/etc/systemd/network/11-van-ap-2g.network` | same, for the 2.4GHz wlan |
| `20-van-br0.netdev` | `/etc/systemd/network/20-van-br0.netdev` | creates the LAN bridge `br0` (STP off) |
| `21-van-br0.network` | `/etc/systemd/network/21-van-br0.network` | static `10.42.0.1/24` on `br0` (`ConfigureWithoutCarrier`) |
| `22-van-lan.network` | `/etc/systemd/network/22-van-lan.network` | enslaves the wired LAN port `enx00e04c331140` to `br0` |
| `van-ap-unmanaged.conf` | `/etc/NetworkManager/conf.d/van-ap-unmanaged.conf` | tells NM to leave the AP wlan **and** the wired LAN port alone |
| `nftables.conf` | `/etc/nftables.conf` | NAT: `masquerade ip saddr 10.42.0.0/24 oifname != br0` → follows whatever WAN is active |
| `99-van-router.conf` | `/etc/sysctl.d/99-van-router.conf` | `net.ipv4.ip_forward=1` |
| `regdomain.service` | `/etc/systemd/system/regdomain.service` | `iw reg set CA` at boot, before NetworkManager |
| `rtw89.conf` | `/etc/modprobe.d/rtw89.conf` | 5GHz dongle: `disable_ps_mode=Y` (else AP drops beacon when idle) + blacklists in-kernel rtw89 so the out-of-tree morrownr/rtw89 dkms driver (installed separately, not by deploy.sh) loads instead — see gotchas below |
| `rtw88.conf` | `/etc/modprobe.d/rtw88.conf` | 2.4GHz dongle: `disable_lps_deep=Y` (same PS reasoning) + `switch_usb_mode=N` (don't self-upgrade to USB3 — it radiates into 2.4GHz) |
### `failover/` — multi-WAN
| file | → installs to | purpose |
|---|---|---|
| `van-failover` | `/usr/local/sbin/van-failover` | the daemon (Python) |
| `config.json` | `/etc/van-failover/config.json` | WAN list + priorities + probe settings |
| `van-failover.service` | `/etc/systemd/system/van-failover.service` | `Restart=always` |
| `50-disable-eee` | `/etc/NetworkManager/dispatcher.d/50-disable-eee` | disables EEE on `en*`/`eth*` at connect (r8152 idle parking) |
| `99-van-arp.conf` | `/etc/sysctl.d/99-van-arp.conf` | `arp_ignore=1` / `arp_announce=2` (multi-NIC ARP hygiene) |
### `dns/` — ZeroTier managed DNS
| file | → installs to | purpose |
|---|---|---|
| `zt-search.conf` | `/etc/systemd/network/99-ztuga7c2kh.network.d/search.conf` | makes `wrede.pvt` a search domain (bare-hostname completion) |
| `zt-network.local.conf` | `/var/lib/zerotier-one/networks.d/d3ecf5726d041b2a.local.conf` | contains `allowDNS=1` (prerequisite) |
| `99-ztuga7c2kh.network.generated` | *(reference only — generated by the manager)* | what zerotier-systemd-manager writes |
| `zerotier-systemd-manager.{service,timer}` | `/usr/lib/systemd/system/` | the manager (binary installed separately, see §4) |
### `cockpit/vanrouter/` — web UI
`manifest.json`, `index.html`, `vanrouter.css`, `vanrouter.js` → `/usr/share/cockpit/vanrouter/`.
### `power/` — never sleep + thermal monitor
| file | → installs to | purpose |
|---|---|---|
| `10-vanlink-nolid.conf` | `/etc/systemd/logind.conf.d/10-vanlink-nolid.conf` | logind ignores the lid in all states (closed / on AC / docked) |
| `van-thermal` | `/usr/local/sbin/van-thermal` | temperature daemon (Python): publishes state, alerts, logs history |
| `thermal-config.json` | `/etc/van-thermal/config.json` | sensors + warn/crit thresholds + sample/log intervals |
| `van-thermal.service` | `/etc/systemd/system/van-thermal.service` | `Restart=always` |
| `van-nvme-watch` | `/usr/local/sbin/van-nvme-watch` | NVMe watchdog (Python): tails `journalctl -k` for I/O-timeout/reset events, Pushover alerts |
| `nvme-watch-config.json` | `/etc/van-nvme-watch/config.json` | grace period, cooldown, device paths |
| `van-nvme-watch.service` | `/etc/systemd/system/van-nvme-watch.service` | `Restart=always` |
| `van-battery` | `/usr/local/sbin/van-battery` | battery daemon (Python): Pushover low-charge alerts + safe shutdown |
| `battery-config.json` | `/etc/van-battery/config.json` | warn levels, shutdown level, poll interval, paths |
| `van-battery.service` | `/etc/systemd/system/van-battery.service` | `Restart=always` |
| `pushover.json.example` | → `/etc/van-battery/pushover.json` (seeded if absent) | Pushover token+user **template**; real file is 0600, **not** in the repo |
| `10-vanlink-watchdog.conf` | `/etc/systemd/system.conf.d/10-vanlink-watchdog.conf` | hardware watchdog (`RuntimeWatchdogSec=20s`); applied via `daemon-reexec` |
`deploy.sh` additionally **masks** `sleep.target suspend.target hibernate.target hybrid-sleep.target`
(no repo file — symlinks to `/dev/null` under `/etc/systemd/system/`) so nothing else can suspend either.
### `heartbeat/` — dead-man's switch
| file | → installs to | purpose |
|---|---|---|
| `hbc.yaml` | `/etc/hbc.yaml` | heartbeat client config (interval + metric plugins) |
| `hbc.service` | `/etc/systemd/system/hbc.service` | runs `~/bin/hbc` as `andreas` → hbd.wrede.pvt:50003 |
The `hbc` binary itself (venv at `~/venvs/hbd`, symlink `~/bin/hbc`) is installed once via the
heartbeat project's own installer — see §4. `deploy.sh` only starts the service once it exists.
### `li3/` — RV house battery (BLE BMS → MQTT/HA)
| file | → installs to | purpose |
|---|---|---|
| `van-li3-battery` | `/usr/local/sbin/van-li3-battery` | BLE→MQTT daemon (Python, bleak + paho-mqtt): reads the Lithionics Li3 BMS, publishes HA MQTT discovery + state |
| `config.json.example` | → `/etc/van-li3/config.json` (seeded if absent) | BLE address, broker, MQTT topic/discovery **template**; real file is 0600, **not** in the repo |
| `van-li3-battery.service` | `/etc/systemd/system/van-li3-battery.service` | `Restart=always` |
Not to be confused with `power/van-battery` — that's the host's own AC/battery power
supply (laptop UPS-style monitor), unrelated hardware and purpose.
### `ha/` — Home Assistant (native container)
| file | → installs to | purpose |
|---|---|---|
| `homeassistant.container` | `/etc/containers/systemd/homeassistant.container` | Podman Quadlet: HA Container on the host network (`:8123`), config in `/srv/homeassistant`, host D-Bus mounted for onboard Bluetooth. |
The `homeassistant` LAN name comes from `ap/van-ap-dnsmasq.conf` (`host-record` →
`10.42.0.1`). The retired HAOS VM's domain XML (`ha_van.xml`) lives in git history.
---
## 3. Components & how to operate
### Access Point
- **Dual band, one SSID**: 5GHz (`hostapd.conf` → unit `hostapd`, ch149 80MHz VHT+HE) and 2.4GHz (`hostapd-2g.conf` → unit `hostapd-2g`, ch6 HT20, for range + 2.4-only IoT). Same SSID/PSK — clients pick their band. Deliberately two hostapd processes: a USB wedge on one radio never takes the other down, and each has its own watchdog (`van-ap-watchdog`, `van-ap-watchdog-2g`).
- Change SSID / channel / password: edit `ap/hostapd.conf` and/or `ap/hostapd-2g.conf`, then `sudo cp` to `/etc/hostapd/` (or `./deploy.sh`) and restart the matching unit.
- DHCP range / DNS options: `ap/van-ap-dnsmasq.conf` + restart `van-ap-dnsmasq`.
- Services: `hostapd`, `hostapd-2g`, `van-ap-dnsmasq`, `systemd-networkd`, `nftables`, `regdomain`.
- **Do not** add `noscan` to hostapd.conf — Ubuntu's hostapd rejects it and fails to start.
- Verify each is actually beaconing with `iw dev wlxc83a35a4ee55 info` / `iw dev wlxd8ec5e2faa8c info` (ssid+channel+width present), not just `systemctl is-active`.
### Multi-WAN failover (`van-failover`)
- Probes each WAN's real internet every few seconds (HTTP-204 check, captive-portal-aware), demotes a failed WAN by raising its route-metric, fails back on recovery (hysteresis).
- Status: `systemctl status van-failover`, `journalctl -u van-failover -f`, or the Cockpit "WAN Failover" card. Live state JSON at `/run/van-failover/state.json`.
- Tune priorities/timing in `failover/config.json` (`wans[].metric`, `probe_interval`, `probe_timeout`, `fail_threshold`, `ok_threshold`, `probe_urls`) then restart.
- WAN identity: eth/wifi keyed by `device`, cellular by NM `connection` name (`Koodo`).
- The daemon changes metrics with **`ip route`**, never `nmcli device reapply` — see Gotchas.
#### Manual preference (Cockpit "Prefer" button)
- The daemon is the **single writer** of route-metrics. To force a specific WAN to the top it reads an optional file `/run/van-failover/prefer` containing one **device name** (e.g. `enxd8ec5eeb3512`). That WAN gets base metric **50** (below every config metric) so it wins while healthy; the health PENALTY still stacks on top, so failover/fail-back are unchanged. Empty/absent file = follow config priorities.
- The Cockpit "Prefer" button just writes that file (root via polkit) — it does **not** set metrics itself. Doing so directly would fight the daemon (reverted within one probe loop) and `reapply`-flap the USB carrier. The daemon applies the change on its next loop (~`probe_interval`); `state.json` carries a `preferred` flag the UI shows as "(preferred)".
- Single-WAN, sticky override with no explicit "clear": to return to automatic priority, prefer your top WAN (wifi) or `rm /run/van-failover/prefer`. **Ephemeral by design** — `/run` is wiped on reboot, so a reboot returns to config priorities. (To persist it, point `PREFER` at `/etc/van-failover/prefer` in the daemon and the matching path in `vanrouter.js`.)
- Swapping the preference between two WANs is collision-safe: a second default route can't take the new WAN's target metric while the old preferred WAN still holds it, so `enforce_route` keeps the existing route and retries next loop (settles in ~2 loops) instead of stranding the interface. `enforce_route` also restores a default route that went missing while the carrier is up (gateway from NM via `device_gateway`), not just rebases an existing one.
### Thermal monitor (`van-thermal`)
- One daemon off a single sysfs sample loop does all three jobs: live Cockpit readout, threshold alerting, history.
- **Live state**: `/run/van-thermal/state.json` (atomic-swapped each sample). The Cockpit "Temperatures" card reads this — no shelling out to `sensors` per refresh. Sensors are resolved by hwmon **name** + **label** (`coretemp`/`Package id 0`, `nvme`/`Composite`), never by `hwmonN` index (not stable across boots).
- **Alerts**: level changes (ok ↔ warn ↔ crit) are logged to the journal with hysteresis (`clear_margin`, default 5 °C) so a sensor sitting on the line doesn't spam. Watch with `journalctl -u van-thermal -f`; warn/crit carry sd-daemon severity (`-p warning`/`-p err`).
- **History**: throttled CSV at `/var/log/van-thermal.csv` (one row per `log_interval`, default 60 s), self-rotating to `.csv.1` past `log_max_bytes` (5 MB).
- **Tuning**: edit `/etc/van-thermal/config.json` (thresholds, intervals, sensor list), then `systemctl restart van-thermal`. Defaults: CPU warn 80 / crit 95 °C (silicon crit is 100), NVMe warn 65 / crit 70 °C (drive crit ~71).
- Status: `systemctl status van-thermal` or `cat /run/van-thermal/state.json`.
### NVMe watchdog (`van-nvme-watch`)
- Event-driven, not polled: tails `journalctl -kf` for the `nvme nvmeN: I/O tag ... timeout, reset controller` signature that crashed and corrupted the root fs on 2026-08-02 (recurred 2026-08-04, self-healed — see the memory notes for that investigation). There's no sensor to sample, only a log line to catch.
- On a match it watches a `grace_period` (default 20 s) for either a clean re-init (self-healed) or a second timeout / an ext4 error following it (escalated) before alerting, so the Pushover message already says which outcome happened — no need to SSH in during a scare.
- The alert body includes a live `smartctl`/`tune2fs` snapshot (SMART health, critical-warning flag, media error count, filesystem state).
- **Live state**: `/run/van-nvme-watch/state.json` (last event, same convention as van-failover/van-thermal).
- Shares Pushover credentials with van-thermal/van-battery (`/etc/van-battery/pushover.json`).
- **Tuning**: edit `/etc/van-nvme-watch/config.json` (`grace_period`, `cooldown`, device paths), then `systemctl restart van-nvme-watch`.
- Status: `systemctl status van-nvme-watch`, `journalctl -u van-nvme-watch -f`, or `cat /run/van-nvme-watch/state.json`.
### Battery monitor (`van-battery`)
- Watches mains vs battery via `/sys/class/power_supply/AC0/online` (0 = on battery) and charge via `BAT0/capacity`. Both resolve by `type` (Mains/Battery) if those names ever differ.
- **Only while on battery**, it sends escalating **Pushover** alerts at **25 / 20 / 15 %**, and at **10 %** sends a final alert and runs `systemctl poweroff` (after `shutdown_grace`, default 8 s, so the alert flushes first).
- Edge-triggered per discharge episode: each severity fires once; the sequence **re-arms when mains returns**. Unplugging already below a warn level fires a single alert for the current severity (no burst), then shutdown at 10 %.
- **Credentials** live in `/etc/van-battery/pushover.json` (mode 0600), seeded from `pushover.json.example` on first deploy and **never committed**. Missing/placeholder creds disable *sending* but **not** the shutdown — running flat must always power down safely (the skip is logged to the journal).
- Tune in `/etc/van-battery/config.json` (`warn_levels`, `shutdown_level`, `poll_interval`), then `systemctl restart van-battery`.
- Status: `systemctl status van-battery`, `journalctl -u van-battery -f`, or `cat /run/van-battery/state.json`.
- **Caveat — the 10 % shutdown is one-way.** A laptop won't power itself back on when mains returns: "restore on AC loss" is a BIOS/firmware feature (not OS-controllable — `/proc/acpi/wakeup` only covers wake-from-suspend). If your BIOS exposes an "AC power-on / restore on AC loss" option, enable it so the router reboots itself once shore/solar power is back; this ZenBook likely doesn't have it, in which case a low-battery shutdown needs a manual power-on.
### Adding the 4G/5G modem
1. Plug the USB modem in. ModemManager + the `Koodo` gsm NM connection (autoconnect) bring it up.
On a fresh host create the profile once:
`nmcli con add type gsm ifname "*" con-name Koodo apn sp.koodo.com connection.autoconnect yes`
2. It auto-joins as the `cellular` WAN at metric 300 (last resort). Nothing else to configure.
3. Confirm with `mmcli -L` and the Cockpit failover card (cellular flips from `absent` to `up`).
4. Gotcha (handled in `van-failover` since the Quectel EC25-AF landed): for MBIM/QMI modems
NM's device is the control port (`cdc-wdm0`) while IP + routes live on the wwan netdev
(e.g. `wwu1u2i4`). The daemon maps via `GENERAL.IP-IFACE` — probing or `ip route`
against `cdc-wdm0` silently sees nothing. The Cockpit WAN table does the same mapping
(shows the netdev + its IP; nmcli actions still target `cdc-wdm0`).
5. The Cockpit WAN row shows signal % / tech / operator (from `mmcli`) and has a
**Restart** button: `mmcli --disable && --enable` (the EC25 MBIM plugin doesn't
support `--reset`; autoconnect reconnects), falling back to a USB unbind/bind of the
Quectel device (vendor `2c7c`) if ModemManager can't reach the modem.
### GPS (gpsd from the cellular modem)
The EC25-AF has a GNSS engine that streams NMEA on its USB interface 01 (`ttyUSB1`,
`gps` port in `mmcli`). Needs its own GPS antenna on the modem's GNSS connector for a fix.
- **One-time modem config** (persists in modem NV; survives reboot/replug/Restart):
GNSS auto-start via `AT+QGPSCFG="autogps",1` + `AT+QGPS=1`. ModemManager holds both
AT ports and swallows replies, so stop it first:
`systemctl stop ModemManager`, send the ATs on `/dev/ttyUSB3`, `systemctl start ModemManager`.
- **Host side** (deployed by deploy.sh, needs `apt install gpsd gpsd-clients`):
`gps/77-modem-gps.rules` udev rule matches the NMEA tty (2c7c:0125 if01), symlinks it
to `/dev/modem-gps`, and hot-adds it to gpsd via `gpsdctl@%k` — the same mechanism as
gpsd's own 60-gpsd.rules, so plug/unplug/renumbering just works. `gps/gpsd.default`
(`/etc/default/gpsd`) keeps `DEVICES` empty (hotplug does it) and runs gpsd with `-n`.
- Verify: `gpspipe -r -n 10` (raw NMEA), `cgps` (fix view), `systemctl status gpsdctl@ttyUSB1`.
- Don't enable MM location APIs (`mmcli --location-enable-gps-*`) at the same time —
MM would open the NMEA port and fight gpsd; `gps-unmanaged` is the only safe one.
### OwnTracks publisher (gpsd -> MQTT)
`gps/van-gps-owntracks` (service `van-gps-owntracks`) streams TPV fixes from local
gpsd and publishes OwnTracks location JSON to `owntracks/rv/gps` on home.wrede.ca
(tid `rv`) — at most every 10 min when parked, immediately after >250 m of movement.
Port of the wayback-era `gps_to_owntracks.py`, adapted to apt-only deps
(`python3-gps`, `python3-paho-mqtt` — the old `gpsdclient` is pip-only) and the
paho 2.x callback API.
- Broker credentials live only in `/etc/van-gps/config.json` (0600, seeded from
`gps/config.json.example` — edit after first deploy). All knobs (broker, topic,
intervals, gpsd host) live there too.
- **client_id must stay `vanq-wan` (or anything unique)**: some legacy client still
holds `vanq` on the broker (old copy on wayback?) and the broker kicks whoever
shares its ID — the symptom is connect/disconnect every few seconds.
- On MQTT disconnect the script exits and systemd restarts it (RestartSec=15);
that *is* the reconnect logic, so a red blip after a WAN failover is normal.
- Debug tracing: `kill -HUP` the process toggles per-fix logging to the journal.
- Verify: `journalctl -u van-gps-owntracks` shows `mqtt connect` and stays quiet;
a restart publishes the current fix immediately (first-message path), so
subscribing to `owntracks/rv/gps` while restarting shows a live message.
- Note: the broker holds an ancient *retained* message on this topic from a 2023
OwnTracks device (tid `RV`, Winegard SSID); our publishes are not retained.
### Li3 battery monitor (`van-li3-battery`)
Publishes the RV's 12V LiFePO4 house battery (Lithionics Li3, BLE HM-10 UART module —
service `ffe0`/char `ffe1`, no pairing) to Home Assistant as 10 sensor entities (pack
voltage, 4 cell voltages, current, SOC, BMS/battery temperature, status) via MQTT
discovery. Protocol (CSV telemetry lines after sending `$traceon`+`$info`) reverse-
engineered from the `com.lithionics.bms` Android app's own BLE/parsing code — see the
script's docstring for the full field layout.
- Not the same battery as `power/van-battery` (host's own AC/UPS power supply) —
different hardware, different concern, deliberately different naming.
- Broker credentials live only in `/etc/van-li3/config.json` (0600, seeded from
`li3/config.json.example` — edit after first deploy, same pattern as `van-gps`).
- **Self-healing BLE**: scan/connect retries happen *inside* the running process (MQTT
session and HA entities stay up across them, no flapping) rather than relying on
systemd restarts. This Pi's onboard Bluetooth adapter (Cypress/CYW43) occasionally
wedges bluetoothd's discovery state after a run of failed connects to this specific
device (`Discovering` stays `yes` forever, every subsequent connect fails with
`le-connection-abort-by-local`) — as soon as a scan fails the daemon restarts
`bluetooth.service` itself to clear it, rate-limited to once per 5 min so it
doesn't repeatedly disrupt the AP's other BLE gear (motion sensors, IR remote).
- This BLE module accepts only **one central connection at a time** — while
`van-li3-battery` holds it, the Li3 phone app can't connect simultaneously.
`systemctl stop van-li3-battery` to free it up for the app.
- Verify: `journalctl -u van-li3-battery -f` (look for `connected` / `published: {...}`),
or `mosquitto_sub -h localhost -u homeassistant -P <pw> -t van/li3_battery/state`.
### ZeroTier managed DNS
- `wrede.pvt` resolves over ZeroTier when off the home LAN. Mechanism: `allowDNS=1` (prereq) + `zerotier-systemd-manager` writes `99-ztuga7c2kh.network`, networkd applies it to resolved.
- Verify: `resolvectl status ztuga7c2kh` shows `DNS` scope + the two servers + `wrede.pvt`.
- NOTE: ZeroTier's *native* DNS push is a no-op on Linux (`setDns ... not implemented`), which is why the manager is required.
- `systemd-networkd-wait-online` is **masked** so the ZT link does **not** gate `network-online.target` — see Gotchas.
### Cockpit dashboard
- `https://<wayback>:9090` → "Van Router". Reload the browser after deploying plugin changes (Cockpit caches packages per session).
- Read-only status works as any user; Prefer/Up/Down/Restart need *Administrative access* (polkit).
- The Access Points card shows both bands (5GHz `hostapd`, 2.4GHz `hostapd-2g`) with per-band client lists and **Restart** buttons; the radio list lives in `vanrouter.js` (`const APS`).
- **Prefer** sets the manual WAN preference (see *Manual preference* above); **Up/Down** connect/disconnect the NM device.
### Home Assistant (native Podman container)
- HA Container runs on the host via a **Podman Quadlet** (`ha/homeassistant.container` →
`/etc/containers/systemd/`; systemd generates `homeassistant.service`). **Host network**:
HA binds `:8123` directly, so clients use `http://10.42.0.1:8123` (or
`http://homeassistant:8123`); mDNS/SSDP discovery sees the LAN because there's no bridge
or NAT in the way. Config lives in `/srv/homeassistant`.
- **Bluetooth**: host BlueZ (`bluetooth.service`) serves the Pi's onboard `hci0` to HA over
the mounted `/run/dbus` socket — add the Bluetooth integration in HA and it appears.
- No add-on store (that was HAOS's Supervisor): Mosquitto/Zigbee2MQTT-style add-ons become
their own containers/services if ever needed.
- Operate: `systemctl {status,restart} homeassistant`. Update: bump/pull the image
(`podman pull ghcr.io/home-assistant/home-assistant:stable`) and restart. After editing
the `.container` file: `./deploy.sh` (or install + `systemctl daemon-reload`), then
`systemctl restart homeassistant`. Verify:
`curl -s -o /dev/null -w '%{http_code}' http://10.42.0.1:8123/` → `200`.
- History: HA started as a HAOS VM on wayback (libvirt NAT, then bridged onto `br0` at
`10.42.0.50` with a legacy-URL DNAT), moved to this Pi as an aarch64 VM 2026-07-06, and
went native 2026-07-07 — the fixed 2 GiB VM allocation starved the 4 GB Pi. The domain
XML (`ha/ha_van.xml`) and the DNAT/hairpin nftables rules are in git history.
### Never sleep (lid-closed operation)
- wayback lives lid-closed in the van and must stay up. Stock logind `HandleLidSwitch=suspend` would sleep it on lid close (even on AC). The `power/10-vanlink-nolid.conf` drop-in sets all three lid actions to `ignore`; `deploy.sh` also masks every sleep target so idle / GUI / a stray `systemctl suspend` can't suspend it.
- Verify: `busctl get-property org.freedesktop.login1 /org/freedesktop/login1 org.freedesktop.login1.Manager HandleLidSwitch` → `"ignore"`, and `systemctl is-enabled suspend.target` → `masked`. `busctl call ... CanSuspend` should return `"no"`.
- **To re-enable sleep** (e.g. if wayback is ever a laptop again):
```bash
sudo rm /etc/systemd/logind.conf.d/10-vanlink-nolid.conf
sudo systemctl unmask sleep.target suspend.target hibernate.target hybrid-sleep.target
sudo systemctl restart systemd-logind
```
(then remove the `power/` block from `deploy.sh`, or the next deploy re-applies it).
---
## 4. Deploy / rebuild
```bash
cd ~/vanlink
sudo ./deploy.sh # copies all files to their system locations, reloads + enables services
```
Four things `deploy.sh` does **not** do (one-time, manual):
1. **zerotier-systemd-manager binary** (v0.4.0, hand-installed — not in the ZeroTier apt repo):
```bash
# copy /usr/bin/zerotier-systemd-manager from a host that has it (e.g. wertvoll), then:
sudo systemctl enable --now zerotier-systemd-manager.timer
sudo zerotier-cli set d3ecf5726d041b2a allowDNS=1
```
2. **hostapd unmask** (Ubuntu ships it masked): `sudo systemctl unmask hostapd`.
3. **Pushover credentials** for `van-battery` — deploy seeds `/etc/van-battery/pushover.json` (0600) with placeholders; fill in your app token + user key, then `sudo systemctl restart van-battery`. Until then low-battery alerts are skipped (logged), but the 10 % auto-shutdown still works.
4. **Heartbeat client `hbc`** (dead-man's switch) — install the binary once into andreas' venv:
```bash
git clone https://git.wrede.ca/andreas/heartbeat.git ~/git/heartbeat # if not already cloned
sh ~/git/heartbeat/scripts/hb_install.sh client # -> ~/venvs/hbd + ~/bin/hbc
```
`deploy.sh` installs `/etc/hbc.yaml` + the `hbc.service` unit and starts it once `~/bin/hbc` exists. The service reports to **hbd.wrede.pvt** (UDP 50003) as user `andreas`. Upgrade later with `~/bin/hb_install.sh client`.
---
## 5. Key design decisions & hard-won gotchas
- **AP on hostapd, not NetworkManager.** NM's hotspot caps the rtw89 radio at HT20/20MHz; hostapd gives the full VHT80/HE (WiFi-6). The AP iface is therefore NM-*unmanaged*; networkd gives it its static IP.
- **`rtw89` power-save must be off** (`disable_ps_mode=Y`) or the AP stops beaconing when idle and the SSID vanishes.
- **RTL8852BU is USB-2.0 and hangs under load if it shares a USB hub.** Keep the AP dongle on its **own** USB controller, separate from the WAN ethernet. Symptom of a shared bus: `c2h reg timeout` + `Polling beacon packet empty fail` under throughput, SSID drops. (`timed out to flush queues` alone is benign.) True regardless of which driver (in-kernel or out-of-tree) is bound — it's a physical bus-bandwidth issue, not a driver bug.
- **5GHz dongle runs the out-of-tree `morrownr/rtw89` driver (since 2026-08-23), not the in-kernel one.** This Ubuntu kernel base (`7.0.0-1017-raspi`) predates mainline's rtw89 USB2→3 auto-switch, so the in-kernel driver permanently caps the 8852BU at USB2/480M. The out-of-tree driver has that switch (`switch_usb_mode=y` in `ap/rtw89.conf`) and needs a real USB3-capable port to actually benefit from it — a USB2-only port still caps it regardless of driver (confirmed live: 480M on a USB2-only controller, 5000M/SuperSpeed after moving to a dedicated USB3 controller). Driver itself logs `"2.4 GHz performance may be better in a USB 2 port"` on load — watch the 2.4GHz radio for new interference now that 5GHz runs SuperSpeed signalling nearby. Installed/updated via `sudo ./ap/install-rtw89-driver.sh` (pins a specific upstream commit, `dkms install`, `make install_fw`) — **not** run automatically by `deploy.sh`, since a dkms rebuild is slow and shouldn't fire on every routine deploy; run it manually after a fresh Pi provisioning or to bump the pin. DKMS itself auto-rebuilds across kernel upgrades. Module names get a `_git` suffix (`rtw89_8852bu_git` etc.) — `ap/rtw89.conf`'s blacklist stops the in-kernel modules from claiming the device instead. **Gotcha hit switching a live system over**: the in-kernel `rtw89_core` won't unload while its own in-kernel dependents (`rtw89_8852b`, `rtw89_8852b_common`) are still loaded — remove all of them first or the out-of-tree module fails to load (`exports duplicate symbol`). The install script's own output has the exact recovery steps.
- **van-failover changes metrics via `ip route`, never `nmcli device reapply`.** `reapply` **resets the r8152 USB-ethernet carrier**, which caused a ~10s-ping-drop flapping feedback loop. Pure `ip route` changes are carrier-safe.
- **Disable EEE on USB ethernet** (`50-disable-eee` dispatcher) — its idle power-save parks the *backup* WAN link and breaks health probes.
- **Per-WAN probing needs `curl --interface if!<dev>`** (forces `SO_BINDTODEVICE`); plain `--interface <name>` only sets the source IP and still routes via the default WAN. `rp_filter` is loose (`2`), required for this.
- **`arp_ignore`/`arp_announce`** matter only when two WANs share a subnet (a home-LAN test artifact; Starlink + neighbour-wifi will be on different subnets in the van). Harmless to keep.
- **dnsmasq uses `bind-dynamic` bound to the AP iface** so it coexists with systemd-resolved (no port-53 fight) — resolved stays intact for the host.
- **`systemd-networkd-wait-online` must be masked** or it deadlocks boot. networkd here manages only two links — the AP (`wlxc…`, local-only) and the ZT overlay `ztuga7c2kh` — and **neither is a real uplink**. The ZT link only appears once `zerotier-one` starts, but `zerotier-one` is ordered `After=network-online.target`, so wait-online would block on the not-yet-existing link until its 120s timeout. Marking the links `RequiredForOnline=no` does **not** help: with *zero* candidate links, wait-online can never satisfy "online" and times out anyway (systemd 255 — verified: `Timeout occurred while waiting for network connectivity` at exactly +120s). Both add ~2min to every boot and delay ZeroTier (the recovery path). The fix is to mask the service entirely (`deploy.sh` does this); real uplink readiness is covered by `NetworkManager-wait-online`, which owns the actual WANs.
---
## 6. Recovery / safety
- Three independent management paths: ethernet `.251`, wifi `.27`, ZeroTier `192.168.196.22`. Changing default-route metrics does **not** affect the local-subnet (SSH) routes.
- The AP fan/EC, hostapd, networkd, nftables, sysctl, failover, regdomain are all enabled and reboot-persistent.
## 7. Pending / future ideas
- Real-world soak once the actual Starlink terminal + USB 4G/5G modem are installed (current eth is the home LAN; subnets/probe targets may want tuning).
- Cockpit plugin: add an AP-config form (SSID/channel/PSK editing `hostapd.conf`). *(Manual failover override via the "Prefer" button is done — see §3.)*
- Consider pinning a fixed interface name for the WAN ethernet (USB MAC-derived names change if the adapter is swapped).
Reference in New Issue View Git Blame Copy Permalink