vanlink — campervan router on wayback

This clone = the Pi 4 port (host wan, branch wan). Onboard eth0+wlan0 are NM-managed WANs (netplan ap/50-van-wan.yaml); the AP radios + wired LAN port arrive with the USB hub from wayback (same MAC-derived names, so all configs port verbatim). Dropped here: HA VM, battery/lid (no hardware), heartbeat + ZeroTier (not installed yet). Watchdog is 10s (bcm2835 max 15s); thermal watches cpu_thermal.

Turns wayback (Asus ZenBook UX391U, Ubuntu 24.04, zabbly kernel) into a self-contained campervan hub/router/AP:

• WiFi-6 access point for client devices (hand-rolled hostapd, not NetworkManager).
• Multi-WAN failover across WiFi (neighbour AP) → Ethernet (Starlink) → 4G/5G cellular.
• ZeroTier-managed DNS so *.wrede.pvt resolves over the overlay when away from home.
• Cockpit dashboard ("Van Router") for status + manual control.

This directory is the source of truth. The live system files live under /etc, /usr/...; deploy.sh copies from here to there. Edit here, run sudo ./deploy.sh, done.

---

1. Architecture at a glance

     WiFi clients (VanLink, 5GHz ch149, 80MHz WPA2)   wired LAN clients
        wlxc83a35a4ee55 (RTL8852BU, rtw89, hostapd)   enx00e04c331140
                          └───────────┬───────────────────┘
                                br0   10.42.0.1/24   ← LAN bridge
                          │  dnsmasq DHCP .10–.254 + DNS
                          │  nftables masquerade (oifname != br0)
                          │  ip_forward=1
        ┌─────────────────┼──────────────────────────────────────┐
   wlp1s0 (m100)     enxd8ec5eeb3512 (m200)        Koodo gsm (m300)
   neighbour WiFi    USB ethernet → Starlink       4G/5G modem (when present)
        └──────── van-failover picks lowest-metric HEALTHY WAN ────┘

   NetworkManager owns the WAN side.  systemd-networkd owns the AP IP + the ZeroTier DNS link.

Roles / responsibilities

Concern	Owner
WAN interfaces (eth / wifi / gsm), DHCP-client, metrics	NetworkManager
LAN bridge br0 + IP (10.42.0.1) + wired LAN member	systemd-networkd (2x-van-br0/lan)
AP beaconing / WPA + adding the wlan to br0	hostapd (bridge=br0; AP iface + wired port are NM-unmanaged)
Keep the AP beaconing across USB re-enumeration (Starlink flap)	hostapd Restart=always/no start-limit drop-in + van-ap-watchdog daemon (recovers a wedged radio)
AP DHCP + DNS	dnsmasq (dedicated instance, bound to AP only)
NAT + forwarding	nftables + sysctl
WAN health + failover	van-failover daemon
Temperature monitor / alert / log	van-thermal daemon
NVMe I/O-timeout/reset watchdog + alert	van-nvme-watch daemon
Battery monitor / low-charge alert + shutdown	van-battery daemon
Auto-reboot on hang	systemd hardware watchdog (intel_oc_wdt)
Liveness / dead-man's switch + metrics	hbc heartbeat client → hbd.wrede.pvt
ZeroTier DNS → resolved	zerotier-systemd-manager + systemd-networkd
Web UI	Cockpit + vanrouter plugin

Key network facts

• LAN: 10.42.0.0/24, gateway 10.42.0.1, DHCP .10–.254 — shared by Wi-Fi (VanLink, dual-band: 5GHz wlxc83a35a4ee55 + 2.4GHz wlxd8ec5e2faa8c) and the wired port enx00e04c331140, all bridged into br0.
• WAN priority (metrics): wifi 100 → eth/Starlink 200 → 4G 300 (lower = preferred).
• Management / recovery: ethernet 192.168.10.251, wifi 192.168.10.27, ZeroTier 192.168.196.22.
• Cockpit: https://192.168.10.251:9090 (or .27, or ZeroTier). Log in with a Unix account; enable Administrative access for action buttons.
• Regulatory domain CA (unlocks 5GHz ch149–161 @30dBm, no DFS).
• ZeroTier network d3ecf5726d041b2a, pushed DNS domain wrede.pvt via 192.168.196.115 + 192.168.10.5.
• Home Assistant runs natively on this host (Podman Quadlet, host network) at http://10.42.0.1:8123 (name homeassistant). The old HAOS VM ha_van (10.42.0.50) was retired 2026-07-07.

---

2. Directory layout → system destinations

deploy.sh performs exactly this mapping.

ap/ — access point

file	→ installs to	purpose
hostapd.conf	/etc/hostapd/hostapd.conf	AP: SSID VanLink, ch149, VHT80/HE, WPA2-PSK, country CA, ctrl_interface for the watchdog. WPA passphrase lives here.
hostapd-restart.conf	/etc/systemd/system/hostapd.service.d/restart.conf	Restart=always + StartLimitIntervalSec=0 so hostapd never gives up after a USB re-enumeration bounces the radio
van-ap-watchdog	/usr/local/sbin/van-ap-watchdog	watches hostapd_cli status; restarts hostapd if the radio wedges (running but not ENABLED)
van-ap-watchdog.service	/etc/systemd/system/van-ap-watchdog.service	runs the watchdog
default-hostapd	/etc/default/hostapd	DAEMON_CONF=...
van-ap-dnsmasq.conf	/etc/van-ap/dnsmasq.conf	DHCP/DNS bound to AP iface (bind-dynamic, so it does not clash with systemd-resolved)
van-ap-dnsmasq.service	/etc/systemd/system/van-ap-dnsmasq.service	dedicated dnsmasq unit (uses the dnsmasq-base binary; the distro dnsmasq service is NOT used)
hostapd-2g.conf	/etc/hostapd/hostapd-2g.conf	2.4GHz AP (Linksys WUSB6300v2 / RTL8822BU): same SSID+PSK, ch6, HT20. Own hostapd instance so one radio wedging never kills the other.
hostapd-2g.service	/etc/systemd/system/hostapd-2g.service	dedicated unit for the 2.4GHz hostapd (same Restart=always / no-start-limit semantics as the 5GHz drop-in)
van-ap-watchdog-2g.service	/etc/systemd/system/van-ap-watchdog-2g.service	second watchdog instance: van-ap-watchdog /etc/hostapd/hostapd-2g.conf hostapd-2g
10-van-ap.network	/etc/systemd/network/10-van-ap.network	brings the AP wlan up with no IP (it's a br0 member; hostapd enslaves it)
11-van-ap-2g.network	/etc/systemd/network/11-van-ap-2g.network	same, for the 2.4GHz wlan
20-van-br0.netdev	/etc/systemd/network/20-van-br0.netdev	creates the LAN bridge br0 (STP off)
21-van-br0.network	/etc/systemd/network/21-van-br0.network	static 10.42.0.1/24 on br0 (ConfigureWithoutCarrier)
22-van-lan.network	/etc/systemd/network/22-van-lan.network	enslaves the wired LAN port enx00e04c331140 to br0
van-ap-unmanaged.conf	/etc/NetworkManager/conf.d/van-ap-unmanaged.conf	tells NM to leave the AP wlan and the wired LAN port alone
nftables.conf	/etc/nftables.conf	NAT: masquerade ip saddr 10.42.0.0/24 oifname != br0 → follows whatever WAN is active
99-van-router.conf	/etc/sysctl.d/99-van-router.conf	net.ipv4.ip_forward=1
regdomain.service	/etc/systemd/system/regdomain.service	iw reg set CA at boot, before NetworkManager
rtw89.conf	/etc/modprobe.d/rtw89.conf	5GHz dongle: disable_ps_mode=Y (else AP drops beacon when idle) + blacklists in-kernel rtw89 so the out-of-tree morrownr/rtw89 dkms driver (installed separately, not by deploy.sh) loads instead — see gotchas below
rtw88.conf	/etc/modprobe.d/rtw88.conf	2.4GHz dongle: disable_lps_deep=Y (same PS reasoning) + switch_usb_mode=N (don't self-upgrade to USB3 — it radiates into 2.4GHz)

failover/ — multi-WAN

file	→ installs to	purpose
van-failover	/usr/local/sbin/van-failover	the daemon (Python)
config.json	/etc/van-failover/config.json	WAN list + priorities + probe settings
van-failover.service	/etc/systemd/system/van-failover.service	Restart=always
50-disable-eee	/etc/NetworkManager/dispatcher.d/50-disable-eee	disables EEE on en*/eth* at connect (r8152 idle parking)
99-van-arp.conf	/etc/sysctl.d/99-van-arp.conf	arp_ignore=1 / arp_announce=2 (multi-NIC ARP hygiene)

dns/ — ZeroTier managed DNS

file	→ installs to	purpose
zt-search.conf	/etc/systemd/network/99-ztuga7c2kh.network.d/search.conf	makes wrede.pvt a search domain (bare-hostname completion)
zt-network.local.conf	/var/lib/zerotier-one/networks.d/d3ecf5726d041b2a.local.conf	contains allowDNS=1 (prerequisite)
99-ztuga7c2kh.network.generated	(reference only — generated by the manager)	what zerotier-systemd-manager writes
zerotier-systemd-manager.{service,timer}	/usr/lib/systemd/system/	the manager (binary installed separately, see §4)

cockpit/vanrouter/ — web UI

manifest.json, index.html, vanrouter.css, vanrouter.js → /usr/share/cockpit/vanrouter/.

power/ — never sleep + thermal monitor

file	→ installs to	purpose
10-vanlink-nolid.conf	/etc/systemd/logind.conf.d/10-vanlink-nolid.conf	logind ignores the lid in all states (closed / on AC / docked)
van-thermal	/usr/local/sbin/van-thermal	temperature daemon (Python): publishes state, alerts, logs history
thermal-config.json	/etc/van-thermal/config.json	sensors + warn/crit thresholds + sample/log intervals
van-thermal.service	/etc/systemd/system/van-thermal.service	Restart=always
van-nvme-watch	/usr/local/sbin/van-nvme-watch	NVMe watchdog (Python): tails journalctl -k for I/O-timeout/reset events, Pushover alerts
nvme-watch-config.json	/etc/van-nvme-watch/config.json	grace period, cooldown, device paths
van-nvme-watch.service	/etc/systemd/system/van-nvme-watch.service	Restart=always
van-battery	/usr/local/sbin/van-battery	battery daemon (Python): Pushover low-charge alerts + safe shutdown
battery-config.json	/etc/van-battery/config.json	warn levels, shutdown level, poll interval, paths
van-battery.service	/etc/systemd/system/van-battery.service	Restart=always
pushover.json.example	→ /etc/van-battery/pushover.json (seeded if absent)	Pushover token+user template; real file is 0600, not in the repo
10-vanlink-watchdog.conf	/etc/systemd/system.conf.d/10-vanlink-watchdog.conf	hardware watchdog (RuntimeWatchdogSec=20s); applied via daemon-reexec

deploy.sh additionally masks sleep.target suspend.target hibernate.target hybrid-sleep.target (no repo file — symlinks to /dev/null under /etc/systemd/system/) so nothing else can suspend either.

heartbeat/ — dead-man's switch

file	→ installs to	purpose
hbc.yaml	/etc/hbc.yaml	heartbeat client config (interval + metric plugins)
hbc.service	/etc/systemd/system/hbc.service	runs ~/bin/hbc as andreas → hbd.wrede.pvt:50003

The hbc binary itself (venv at ~/venvs/hbd, symlink ~/bin/hbc) is installed once via the heartbeat project's own installer — see §4. deploy.sh only starts the service once it exists.

li3/ — RV house battery (BLE BMS → MQTT/HA)

file	→ installs to	purpose
van-li3-battery	/usr/local/sbin/van-li3-battery	BLE→MQTT daemon (Python, bleak + paho-mqtt): reads the Lithionics Li3 BMS, publishes HA MQTT discovery + state
config.json.example	→ /etc/van-li3/config.json (seeded if absent)	BLE address, broker, MQTT topic/discovery template; real file is 0600, not in the repo
van-li3-battery.service	/etc/systemd/system/van-li3-battery.service	Restart=always

Not to be confused with power/van-battery — that's the host's own AC/battery power supply (laptop UPS-style monitor), unrelated hardware and purpose.

ha/ — Home Assistant (native container)

file	→ installs to	purpose
homeassistant.container	/etc/containers/systemd/homeassistant.container	Podman Quadlet: HA Container on the host network (:8123), config in /srv/homeassistant, host D-Bus mounted for onboard Bluetooth.

The homeassistant LAN name comes from ap/van-ap-dnsmasq.conf (host-record → 10.42.0.1). The retired HAOS VM's domain XML (ha_van.xml) lives in git history.

---

3. Components & how to operate

Access Point

• Dual band, one SSID: 5GHz (hostapd.conf → unit hostapd, ch149 80MHz VHT+HE) and 2.4GHz (hostapd-2g.conf → unit hostapd-2g, ch6 HT20, for range + 2.4-only IoT). Same SSID/PSK — clients pick their band. Deliberately two hostapd processes: a USB wedge on one radio never takes the other down, and each has its own watchdog (van-ap-watchdog, van-ap-watchdog-2g).
• Change SSID / channel / password: edit ap/hostapd.conf and/or ap/hostapd-2g.conf, then sudo cp to /etc/hostapd/ (or ./deploy.sh) and restart the matching unit.
• DHCP range / DNS options: ap/van-ap-dnsmasq.conf + restart van-ap-dnsmasq.
• Services: hostapd, hostapd-2g, van-ap-dnsmasq, systemd-networkd, nftables, regdomain.
• Do not add noscan to hostapd.conf — Ubuntu's hostapd rejects it and fails to start.
• Verify each is actually beaconing with iw dev wlxc83a35a4ee55 info / iw dev wlxd8ec5e2faa8c info (ssid+channel+width present), not just systemctl is-active.

Multi-WAN failover (van-failover)

• Probes each WAN's real internet every few seconds (HTTP-204 check, captive-portal-aware), demotes a failed WAN by raising its route-metric, fails back on recovery (hysteresis).
• Status: systemctl status van-failover, journalctl -u van-failover -f, or the Cockpit "WAN Failover" card. Live state JSON at /run/van-failover/state.json.
• Tune priorities/timing in failover/config.json (wans[].metric, probe_interval, probe_timeout, fail_threshold, ok_threshold, probe_urls) then restart.
• WAN identity: eth/wifi keyed by device, cellular by NM connection name (Koodo).
• The daemon changes metrics with ip route, never nmcli device reapply — see Gotchas.

Manual preference (Cockpit "Prefer" button)

• The daemon is the single writer of route-metrics. To force a specific WAN to the top it reads an optional file /run/van-failover/prefer containing one device name (e.g. enxd8ec5eeb3512). That WAN gets base metric 50 (below every config metric) so it wins while healthy; the health PENALTY still stacks on top, so failover/fail-back are unchanged. Empty/absent file = follow config priorities.
• The Cockpit "Prefer" button just writes that file (root via polkit) — it does not set metrics itself. Doing so directly would fight the daemon (reverted within one probe loop) and reapply-flap the USB carrier. The daemon applies the change on its next loop (~probe_interval); state.json carries a preferred flag the UI shows as "(preferred)".
• Single-WAN, sticky override with no explicit "clear": to return to automatic priority, prefer your top WAN (wifi) or rm /run/van-failover/prefer. Ephemeral by design — /run is wiped on reboot, so a reboot returns to config priorities. (To persist it, point PREFER at /etc/van-failover/prefer in the daemon and the matching path in vanrouter.js.)
• Swapping the preference between two WANs is collision-safe: a second default route can't take the new WAN's target metric while the old preferred WAN still holds it, so enforce_route keeps the existing route and retries next loop (settles in ~2 loops) instead of stranding the interface. enforce_route also restores a default route that went missing while the carrier is up (gateway from NM via device_gateway), not just rebases an existing one.

Thermal monitor (van-thermal)

• One daemon off a single sysfs sample loop does all three jobs: live Cockpit readout, threshold alerting, history.
• Live state: /run/van-thermal/state.json (atomic-swapped each sample). The Cockpit "Temperatures" card reads this — no shelling out to sensors per refresh. Sensors are resolved by hwmon name + label (coretemp/Package id 0, nvme/Composite), never by hwmonN index (not stable across boots).
• Alerts: level changes (ok ↔ warn ↔ crit) are logged to the journal with hysteresis (clear_margin, default 5 °C) so a sensor sitting on the line doesn't spam. Watch with journalctl -u van-thermal -f; warn/crit carry sd-daemon severity (-p warning/-p err).
• History: throttled CSV at /var/log/van-thermal.csv (one row per log_interval, default 60 s), self-rotating to .csv.1 past log_max_bytes (5 MB).
• Tuning: edit /etc/van-thermal/config.json (thresholds, intervals, sensor list), then systemctl restart van-thermal. Defaults: CPU warn 80 / crit 95 °C (silicon crit is 100), NVMe warn 65 / crit 70 °C (drive crit ~71).
• Status: systemctl status van-thermal or cat /run/van-thermal/state.json.

NVMe watchdog (van-nvme-watch)

• Event-driven, not polled: tails journalctl -kf for the nvme nvmeN: I/O tag ... timeout, reset controller signature that crashed and corrupted the root fs on 2026-08-02 (recurred 2026-08-04, self-healed — see the memory notes for that investigation). There's no sensor to sample, only a log line to catch.
• On a match it watches a grace_period (default 20 s) for either a clean re-init (self-healed) or a second timeout / an ext4 error following it (escalated) before alerting, so the Pushover message already says which outcome happened — no need to SSH in during a scare.
• The alert body includes a live smartctl/tune2fs snapshot (SMART health, critical-warning flag, media error count, filesystem state).
• Live state: /run/van-nvme-watch/state.json (last event, same convention as van-failover/van-thermal).
• Shares Pushover credentials with van-thermal/van-battery (/etc/van-battery/pushover.json).
• Tuning: edit /etc/van-nvme-watch/config.json (grace_period, cooldown, device paths), then systemctl restart van-nvme-watch.
• Status: systemctl status van-nvme-watch, journalctl -u van-nvme-watch -f, or cat /run/van-nvme-watch/state.json.

Battery monitor (van-battery)

• Watches mains vs battery via /sys/class/power_supply/AC0/online (0 = on battery) and charge via BAT0/capacity. Both resolve by type (Mains/Battery) if those names ever differ.
• Only while on battery, it sends escalating Pushover alerts at 25 / 20 / 15 %, and at 10 % sends a final alert and runs systemctl poweroff (after shutdown_grace, default 8 s, so the alert flushes first).
• Edge-triggered per discharge episode: each severity fires once; the sequence re-arms when mains returns. Unplugging already below a warn level fires a single alert for the current severity (no burst), then shutdown at 10 %.
• Credentials live in /etc/van-battery/pushover.json (mode 0600), seeded from pushover.json.example on first deploy and never committed. Missing/placeholder creds disable sending but not the shutdown — running flat must always power down safely (the skip is logged to the journal).
• Tune in /etc/van-battery/config.json (warn_levels, shutdown_level, poll_interval), then systemctl restart van-battery.
• Status: systemctl status van-battery, journalctl -u van-battery -f, or cat /run/van-battery/state.json.
• Caveat — the 10 % shutdown is one-way. A laptop won't power itself back on when mains returns: "restore on AC loss" is a BIOS/firmware feature (not OS-controllable — /proc/acpi/wakeup only covers wake-from-suspend). If your BIOS exposes an "AC power-on / restore on AC loss" option, enable it so the router reboots itself once shore/solar power is back; this ZenBook likely doesn't have it, in which case a low-battery shutdown needs a manual power-on.

Adding the 4G/5G modem

1. Plug the USB modem in. ModemManager + the Koodo gsm NM connection (autoconnect) bring it up. On a fresh host create the profile once: nmcli con add type gsm ifname "*" con-name Koodo apn sp.koodo.com connection.autoconnect yes
2. It auto-joins as the cellular WAN at metric 300 (last resort). Nothing else to configure.
3. Confirm with mmcli -L and the Cockpit failover card (cellular flips from absent to up).
4. Gotcha (handled in van-failover since the Quectel EC25-AF landed): for MBIM/QMI modems NM's device is the control port (cdc-wdm0) while IP + routes live on the wwan netdev (e.g. wwu1u2i4). The daemon maps via GENERAL.IP-IFACE — probing or ip route against cdc-wdm0 silently sees nothing. The Cockpit WAN table does the same mapping (shows the netdev + its IP; nmcli actions still target cdc-wdm0).
5. The Cockpit WAN row shows signal % / tech / operator (from mmcli) and has a Restart button: mmcli --disable && --enable (the EC25 MBIM plugin doesn't support --reset; autoconnect reconnects), falling back to a USB unbind/bind of the Quectel device (vendor 2c7c) if ModemManager can't reach the modem.

GPS (gpsd from the cellular modem)

The EC25-AF has a GNSS engine that streams NMEA on its USB interface 01 (ttyUSB1, gps port in mmcli). Needs its own GPS antenna on the modem's GNSS connector for a fix.

• One-time modem config (persists in modem NV; survives reboot/replug/Restart): GNSS auto-start via AT+QGPSCFG="autogps",1 + AT+QGPS=1. ModemManager holds both AT ports and swallows replies, so stop it first: systemctl stop ModemManager, send the ATs on /dev/ttyUSB3, systemctl start ModemManager.
• Host side (deployed by deploy.sh, needs apt install gpsd gpsd-clients): gps/77-modem-gps.rules udev rule matches the NMEA tty (2c7c:0125 if01), symlinks it to /dev/modem-gps, and hot-adds it to gpsd via gpsdctl@%k — the same mechanism as gpsd's own 60-gpsd.rules, so plug/unplug/renumbering just works. gps/gpsd.default (/etc/default/gpsd) keeps DEVICES empty (hotplug does it) and runs gpsd with -n.
• Verify: gpspipe -r -n 10 (raw NMEA), cgps (fix view), systemctl status gpsdctl@ttyUSB1.
• Don't enable MM location APIs (mmcli --location-enable-gps-*) at the same time — MM would open the NMEA port and fight gpsd; gps-unmanaged is the only safe one.

OwnTracks publisher (gpsd -> MQTT)

gps/van-gps-owntracks (service van-gps-owntracks) streams TPV fixes from local gpsd and publishes OwnTracks location JSON to owntracks/rv/gps on home.wrede.ca (tid rv) — at most every 10 min when parked, immediately after >250 m of movement. Port of the wayback-era gps_to_owntracks.py, adapted to apt-only deps (python3-gps, python3-paho-mqtt — the old gpsdclient is pip-only) and the paho 2.x callback API.

• Broker credentials live only in /etc/van-gps/config.json (0600, seeded from gps/config.json.example — edit after first deploy). All knobs (broker, topic, intervals, gpsd host) live there too.
• client_id must stay vanq-wan (or anything unique): some legacy client still holds vanq on the broker (old copy on wayback?) and the broker kicks whoever shares its ID — the symptom is connect/disconnect every few seconds.
• On MQTT disconnect the script exits and systemd restarts it (RestartSec=15); that is the reconnect logic, so a red blip after a WAN failover is normal.
• Debug tracing: kill -HUP the process toggles per-fix logging to the journal.
• Verify: journalctl -u van-gps-owntracks shows mqtt connect and stays quiet; a restart publishes the current fix immediately (first-message path), so subscribing to owntracks/rv/gps while restarting shows a live message.
• Note: the broker holds an ancient retained message on this topic from a 2023 OwnTracks device (tid RV, Winegard SSID); our publishes are not retained.

Li3 battery monitor (van-li3-battery)

Publishes the RV's 12V LiFePO4 house battery (Lithionics Li3, BLE HM-10 UART module — service ffe0/char ffe1, no pairing) to Home Assistant as 10 sensor entities (pack voltage, 4 cell voltages, current, SOC, BMS/battery temperature, status) via MQTT discovery. Protocol (CSV telemetry lines after sending $traceon+$info) reverse- engineered from the com.lithionics.bms Android app's own BLE/parsing code — see the script's docstring for the full field layout.

• Not the same battery as power/van-battery (host's own AC/UPS power supply) — different hardware, different concern, deliberately different naming.
• Broker credentials live only in /etc/van-li3/config.json (0600, seeded from li3/config.json.example — edit after first deploy, same pattern as van-gps).
• Self-healing BLE: scan/connect retries happen inside the running process (MQTT session and HA entities stay up across them, no flapping) rather than relying on systemd restarts. This Pi's onboard Bluetooth adapter (Cypress/CYW43) occasionally wedges bluetoothd's discovery state after a run of failed connects to this specific device (Discovering stays yes forever, every subsequent connect fails with le-connection-abort-by-local) — as soon as a scan fails the daemon restarts bluetooth.service itself to clear it, rate-limited to once per 5 min so it doesn't repeatedly disrupt the AP's other BLE gear (motion sensors, IR remote).
• This BLE module accepts only one central connection at a time — while van-li3-battery holds it, the Li3 phone app can't connect simultaneously. systemctl stop van-li3-battery to free it up for the app.
• Verify: journalctl -u van-li3-battery -f (look for connected / published: {...}), or mosquitto_sub -h localhost -u homeassistant -P <pw> -t van/li3_battery/state.

ZeroTier managed DNS

• wrede.pvt resolves over ZeroTier when off the home LAN. Mechanism: allowDNS=1 (prereq) + zerotier-systemd-manager writes 99-ztuga7c2kh.network, networkd applies it to resolved.
• Verify: resolvectl status ztuga7c2kh shows DNS scope + the two servers + wrede.pvt.
• NOTE: ZeroTier's native DNS push is a no-op on Linux (setDns ... not implemented), which is why the manager is required.
• systemd-networkd-wait-online is masked so the ZT link does not gate network-online.target — see Gotchas.

Cockpit dashboard

• https://<wayback>:9090 → "Van Router". Reload the browser after deploying plugin changes (Cockpit caches packages per session).
• Read-only status works as any user; Prefer/Up/Down/Restart need Administrative access (polkit).
• The Access Points card shows both bands (5GHz hostapd, 2.4GHz hostapd-2g) with per-band client lists and Restart buttons; the radio list lives in vanrouter.js (const APS).
• Prefer sets the manual WAN preference (see Manual preference above); Up/Down connect/disconnect the NM device.

Home Assistant (native Podman container)

• HA Container runs on the host via a Podman Quadlet (ha/homeassistant.container → /etc/containers/systemd/; systemd generates homeassistant.service). Host network: HA binds :8123 directly, so clients use http://10.42.0.1:8123 (or http://homeassistant:8123); mDNS/SSDP discovery sees the LAN because there's no bridge or NAT in the way. Config lives in /srv/homeassistant.
• Bluetooth: host BlueZ (bluetooth.service) serves the Pi's onboard hci0 to HA over the mounted /run/dbus socket — add the Bluetooth integration in HA and it appears.
• No add-on store (that was HAOS's Supervisor): Mosquitto/Zigbee2MQTT-style add-ons become their own containers/services if ever needed.
• Operate: systemctl {status,restart} homeassistant. Update: bump/pull the image (podman pull ghcr.io/home-assistant/home-assistant:stable) and restart. After editing the .container file: ./deploy.sh (or install + systemctl daemon-reload), then systemctl restart homeassistant. Verify: curl -s -o /dev/null -w '%{http_code}' http://10.42.0.1:8123/ → 200.
• History: HA started as a HAOS VM on wayback (libvirt NAT, then bridged onto br0 at 10.42.0.50 with a legacy-URL DNAT), moved to this Pi as an aarch64 VM 2026-07-06, and went native 2026-07-07 — the fixed 2 GiB VM allocation starved the 4 GB Pi. The domain XML (ha/ha_van.xml) and the DNAT/hairpin nftables rules are in git history.

Never sleep (lid-closed operation)

• wayback lives lid-closed in the van and must stay up. Stock logind HandleLidSwitch=suspend would sleep it on lid close (even on AC). The power/10-vanlink-nolid.conf drop-in sets all three lid actions to ignore; deploy.sh also masks every sleep target so idle / GUI / a stray systemctl suspend can't suspend it.
• Verify: busctl get-property org.freedesktop.login1 /org/freedesktop/login1 org.freedesktop.login1.Manager HandleLidSwitch → "ignore", and systemctl is-enabled suspend.target → masked. busctl call ... CanSuspend should return "no".
• To re-enable sleep (e.g. if wayback is ever a laptop again):

  sudo rm /etc/systemd/logind.conf.d/10-vanlink-nolid.conf
  sudo systemctl unmask sleep.target suspend.target hibernate.target hybrid-sleep.target
  sudo systemctl restart systemd-logind
  

  (then remove the power/ block from deploy.sh, or the next deploy re-applies it).

---

4. Deploy / rebuild

cd ~/vanlink
sudo ./deploy.sh          # copies all files to their system locations, reloads + enables services

Four things deploy.sh does not do (one-time, manual):

1. zerotier-systemd-manager binary (v0.4.0, hand-installed — not in the ZeroTier apt repo):

   # copy /usr/bin/zerotier-systemd-manager from a host that has it (e.g. wertvoll), then:
   sudo systemctl enable --now zerotier-systemd-manager.timer
   sudo zerotier-cli set d3ecf5726d041b2a allowDNS=1
   

2. hostapd unmask (Ubuntu ships it masked): sudo systemctl unmask hostapd.
3. Pushover credentials for van-battery — deploy seeds /etc/van-battery/pushover.json (0600) with placeholders; fill in your app token + user key, then sudo systemctl restart van-battery. Until then low-battery alerts are skipped (logged), but the 10 % auto-shutdown still works.
4. Heartbeat client hbc (dead-man's switch) — install the binary once into andreas' venv:

   git clone https://git.wrede.ca/andreas/heartbeat.git ~/git/heartbeat   # if not already cloned
   sh ~/git/heartbeat/scripts/hb_install.sh client                        # -> ~/venvs/hbd + ~/bin/hbc
   

   deploy.sh installs /etc/hbc.yaml + the hbc.service unit and starts it once ~/bin/hbc exists. The service reports to hbd.wrede.pvt (UDP 50003) as user andreas. Upgrade later with ~/bin/hb_install.sh client.

---

5. Key design decisions & hard-won gotchas

• AP on hostapd, not NetworkManager. NM's hotspot caps the rtw89 radio at HT20/20MHz; hostapd gives the full VHT80/HE (WiFi-6). The AP iface is therefore NM-unmanaged; networkd gives it its static IP.
• rtw89 power-save must be off (disable_ps_mode=Y) or the AP stops beaconing when idle and the SSID vanishes.
• RTL8852BU is USB-2.0 and hangs under load if it shares a USB hub. Keep the AP dongle on its own USB controller, separate from the WAN ethernet. Symptom of a shared bus: c2h reg timeout + Polling beacon packet empty fail under throughput, SSID drops. (timed out to flush queues alone is benign.) True regardless of which driver (in-kernel or out-of-tree) is bound — it's a physical bus-bandwidth issue, not a driver bug.
• 5GHz dongle runs the out-of-tree morrownr/rtw89 driver (since 2026-08-23), not the in-kernel one. This Ubuntu kernel base (7.0.0-1017-raspi) predates mainline's rtw89 USB2→3 auto-switch, so the in-kernel driver permanently caps the 8852BU at USB2/480M. The out-of-tree driver has that switch (switch_usb_mode=y in ap/rtw89.conf) and needs a real USB3-capable port to actually benefit from it — a USB2-only port still caps it regardless of driver (confirmed live: 480M on a USB2-only controller, 5000M/SuperSpeed after moving to a dedicated USB3 controller). Driver itself logs "2.4 GHz performance may be better in a USB 2 port" on load — watch the 2.4GHz radio for new interference now that 5GHz runs SuperSpeed signalling nearby. Installed/updated via sudo ./ap/install-rtw89-driver.sh (pins a specific upstream commit, dkms install, make install_fw) — not run automatically by deploy.sh, since a dkms rebuild is slow and shouldn't fire on every routine deploy; run it manually after a fresh Pi provisioning or to bump the pin. DKMS itself auto-rebuilds across kernel upgrades. Module names get a _git suffix (rtw89_8852bu_git etc.) — ap/rtw89.conf's blacklist stops the in-kernel modules from claiming the device instead. Gotcha hit switching a live system over: the in-kernel rtw89_core won't unload while its own in-kernel dependents (rtw89_8852b, rtw89_8852b_common) are still loaded — remove all of them first or the out-of-tree module fails to load (exports duplicate symbol). The install script's own output has the exact recovery steps.
• van-failover changes metrics via ip route, never nmcli device reapply. reapply resets the r8152 USB-ethernet carrier, which caused a ~10s-ping-drop flapping feedback loop. Pure ip route changes are carrier-safe.
• Disable EEE on USB ethernet (50-disable-eee dispatcher) — its idle power-save parks the backup WAN link and breaks health probes.
• Per-WAN probing needs curl --interface if!<dev> (forces SO_BINDTODEVICE); plain --interface <name> only sets the source IP and still routes via the default WAN. rp_filter is loose (2), required for this.
• arp_ignore/arp_announce matter only when two WANs share a subnet (a home-LAN test artifact; Starlink + neighbour-wifi will be on different subnets in the van). Harmless to keep.
• dnsmasq uses bind-dynamic bound to the AP iface so it coexists with systemd-resolved (no port-53 fight) — resolved stays intact for the host.
• systemd-networkd-wait-online must be masked or it deadlocks boot. networkd here manages only two links — the AP (wlxc…, local-only) and the ZT overlay ztuga7c2kh — and neither is a real uplink. The ZT link only appears once zerotier-one starts, but zerotier-one is ordered After=network-online.target, so wait-online would block on the not-yet-existing link until its 120s timeout. Marking the links RequiredForOnline=no does not help: with zero candidate links, wait-online can never satisfy "online" and times out anyway (systemd 255 — verified: Timeout occurred while waiting for network connectivity at exactly +120s). Both add ~2min to every boot and delay ZeroTier (the recovery path). The fix is to mask the service entirely (deploy.sh does this); real uplink readiness is covered by NetworkManager-wait-online, which owns the actual WANs.

---

6. Recovery / safety

• Three independent management paths: ethernet .251, wifi .27, ZeroTier 192.168.196.22. Changing default-route metrics does not affect the local-subnet (SSH) routes.
• The AP fan/EC, hostapd, networkd, nftables, sysctl, failover, regdomain are all enabled and reboot-persistent.

7. Pending / future ideas

• Real-world soak once the actual Starlink terminal + USB 4G/5G modem are installed (current eth is the home LAN; subnets/probe targets may want tuning).
• Cockpit plugin: add an AP-config form (SSID/channel/PSK editing hostapd.conf). (Manual failover override via the "Prefer" button is done — see §3.)
• Consider pinning a fixed interface name for the WAN ethernet (USB MAC-derived names change if the adapter is swapped).