From 565e8f31abc60e919122cc45aa677e7d3e8c276b Mon Sep 17 00:00:00 2001 From: Andreas Wrede Date: Fri, 18 Sep 2026 18:20:11 -0400 Subject: [PATCH] Fix watchdog timeout and MQTT connectivity issues Problem: R5C spontaneously rebooted due to hardware watchdog timeout Cause 1: van-gps-owntracks in infinite restart loop (MQTT connect failed) Cause 2: mosquitto only listening on localhost, so clients couldn't connect Solution: - Add restart limits to van-gps-owntracks and van-li3-battery services (max 3 restarts per 5 minutes prevents infinite restart loops) - Configure mosquitto to listen on all interfaces (0.0.0.0:1883) - Deploy mosquitto listener config via deploy.sh - Restart mosquitto during deploy to apply config changes This prevents the system from hitting the 11-second hardware watchdog timeout when services crash and systemd auto-restarts them indefinitely. Co-Authored-By: Claude Haiku 4.5 --- .deploy.conf.swp | Bin 0 -> 12288 bytes README.md | 20 +- README.md.bak | 429 ++++++++++++++++++ ap/.van-ap-dnsmasq.conf.swp | Bin 0 -> 12288 bytes ap/10-van-ap.network | 2 +- ap/10-van-ap.network.bak | 12 + ap/20-van-br0.netdev | 2 +- ap/20-van-br0.netdev.bak | 11 + ap/21-van-br0.network.bak | 16 + ap/22-van-lan.network | 2 +- ap/22-van-lan.network.bak | 11 + ap/23-van-lan-eth0.network | 2 +- ap/23-van-lan-eth0.network.bak | 13 + ap/hostapd-2g.conf | 2 +- ap/hostapd.conf | 4 +- ap/hostapd.conf.bak | 31 ++ ap/nftables.conf.bak | 11 + ap/van-ap-dnsmasq.conf.bak | 12 + deploy.sh | 6 + failover/sync-usb-backup.sh | 35 +- failover/sync-usb-backup.sh.bak | 68 +++ ha/esphome.container | 2 +- ha/esphome.container.bak | 35 ++ ha/homeassistant.container | 2 +- ha/homeassistant.container.bak | 37 ++ li3/van-li3-battery | 19 +- li3/van-li3-battery-restart-limit.conf | 6 + router-status/mosquitto-listener.conf | 4 + .../van-gps-owntracks-restart-limit.conf | 6 + 29 files changed, 761 insertions(+), 39 deletions(-) create mode 100644 .deploy.conf.swp create mode 100644 README.md.bak create mode 100644 ap/.van-ap-dnsmasq.conf.swp create mode 100644 ap/10-van-ap.network.bak create mode 100644 ap/20-van-br0.netdev.bak create mode 100644 ap/21-van-br0.network.bak create mode 100644 ap/22-van-lan.network.bak create mode 100644 ap/23-van-lan-eth0.network.bak create mode 100644 ap/hostapd.conf.bak create mode 100755 ap/nftables.conf.bak create mode 100644 ap/van-ap-dnsmasq.conf.bak create mode 100755 failover/sync-usb-backup.sh.bak create mode 100644 ha/esphome.container.bak create mode 100644 ha/homeassistant.container.bak create mode 100644 li3/van-li3-battery-restart-limit.conf create mode 100644 router-status/mosquitto-listener.conf create mode 100644 router-status/van-gps-owntracks-restart-limit.conf diff --git a/.deploy.conf.swp b/.deploy.conf.swp new file mode 100644 index 0000000000000000000000000000000000000000..cc12da5e2c38a50b39f19255dfff767feeee6186 GIT binary patch literal 12288 zcmeI2Pj4GV7{;ePAPN*vTdwG(mZ$@^6Wev;6asZ@r!`5OU?+8N_IP*f9kQNX?aszA z5-MK-i31XR0FZ$1zzsO?9q18moZx`qnYA0W0aWO%Vr)IOJUjFL`OW)|6lI1hw_7{3 zRwy#OUS#asPrrJvbKxTU{VB$xAkb#`QLiJe9yKL>zU&1PvCNel2_7$Fl`qxdvMWMA zm=&BrjsBGtJINBT1RjsTc~&kJSIpYX)MmVhN-30MM_fF)oFSOS*7 z6G=daXV_gtes&T4tBdb5r@j|I?UyBB30MM_fF)oFSOS)SC143y0+xU!U zC+IWi14u$1l!ab`EDZvXs%}Fr;w}S^j;>tZbsME^5#Jl zBw9o#&z1_bB}5p*PB2x}xK-a^7P^{e$jC|8}$8*zK&BoK**I>I(ui30yHDOrRs7T(D_1DEn?A94&lm z)=QOqZF7mS?tZPa-D+?3Tbs3db6se!NGx-DR2^S)ilfo0i+gswLX@F{n~@NTs`)U8 zGecUu^bTo2`E7=7QeT=c>dqrw5PAPPtqzt*n$*u3dxA zQsL^&TklhCk0S2MfU-^!MJQK)cAXmeZ4@&#dbNCUwU{H5MYU9{ALPi5q$z<&Ppj3w z8FUh4l{%|D7(sCn!2^TGPPNR-6@FESN(B*A{?UI&a9@kVx!+1@<6*yR@LMSjjookZHH=x4(ThiFm z`4W25taxhxYXcclei)J2>H%e^W}ncN4;3^F`Ee2t2d3XQrE=k0Ap#eHI06+Z6-r^m z9YKmua0rY{WrTh*9wQ7wwMYdUGF_268ObG_BP;`Z=IXW_lNOWE=NgS@@|brc9dSRw z$WV+-i%S)2?juml_Qb>l^Opn%WO^JP<760Xsd3uGbYp*atJ%JRh!CQObHf}dsC!re zkT5`_XDIY$aeC;@v&x~(G&vquW1#iWgJ$;p*I_X{4{;}3_%)BWnn{H9zq zb1v<*MU$E{J7xXAZTXaw$0K(NpEyk5Rbl_;Xa7hVLV%8>aa!(4?D68p{i3`k=%WJr E6JU2kb^rhX literal 0 HcmV?d00001 diff --git a/README.md b/README.md index f3a1688..b188e8c 100644 --- a/README.md +++ b/README.md @@ -25,7 +25,7 @@ This directory is the source of truth. The live system files live under `/etc`, WiFi clients (VanLink, 5GHz ch149, 80MHz WPA2) wired LAN clients wlxc83a35a4ee55 (RTL8852BU, rtw89, hostapd) enx00e04c331140 └───────────┬───────────────────┘ - br0 10.42.0.1/24 ← LAN bridge + br0 10.42.1.1/24 ← LAN bridge │ dnsmasq DHCP .10–.254 + DNS │ nftables masquerade (oifname != br0) │ ip_forward=1 @@ -41,7 +41,7 @@ This directory is the source of truth. The live system files live under `/etc`, | Concern | Owner | |---|---| | WAN interfaces (eth / wifi / gsm), DHCP-client, metrics | **NetworkManager** | -| LAN bridge `br0` + IP (10.42.0.1) + wired LAN member | **systemd-networkd** (`2x-van-br0/lan`) | +| LAN bridge `br0` + IP (10.42.1.1) + wired LAN member | **systemd-networkd** (`2x-van-br0/lan`) | | AP beaconing / WPA + adding the wlan to `br0` | **hostapd** (`bridge=br0`; AP iface + wired port are NM-*unmanaged*) | | Keep the AP beaconing across USB re-enumeration (Starlink flap) | **hostapd** `Restart=always`/no start-limit drop-in + **van-ap-watchdog** daemon (recovers a wedged radio) | | AP DHCP + DNS | **dnsmasq** (dedicated instance, bound to AP only) | @@ -56,13 +56,13 @@ This directory is the source of truth. The live system files live under `/etc`, | Web UI | **Cockpit** + `vanrouter` plugin | ### Key network facts -- LAN: `10.42.0.0/24`, gateway `10.42.0.1`, DHCP `.10–.254` — shared by Wi-Fi (`VanLink`, dual-band: 5GHz `wlxc83a35a4ee55` + 2.4GHz `wlxd8ec5e2faa8c`) and the wired port `enx00e04c331140`, all bridged into `br0`. +- LAN: `10.42.1.0/24`, gateway `10.42.0.1`, DHCP `.10–.254` — shared by Wi-Fi (`VanLink`, dual-band: 5GHz `wlxc83a35a4ee55` + 2.4GHz `wlxd8ec5e2faa8c`) and the wired port `enx00e04c331140`, all bridged into `br0`. - WAN priority (metrics): **wifi 100 → eth/Starlink 200 → 4G 300** (lower = preferred). - Management / recovery: ethernet `192.168.10.251`, wifi `192.168.10.27`, ZeroTier `192.168.196.22`. - Cockpit: `https://192.168.10.251:9090` (or `.27`, or ZeroTier). Log in with a Unix account; enable *Administrative access* for action buttons. - Regulatory domain **CA** (unlocks 5GHz ch149–161 @30dBm, no DFS). - ZeroTier network `d3ecf5726d041b2a`, pushed DNS domain `wrede.pvt` via `192.168.196.115` + `192.168.10.5`. -- Home Assistant runs **natively on this host** (Podman Quadlet, host network) at **`http://10.42.0.1:8123`** (name `homeassistant`). The old HAOS VM `ha_van` (`10.42.0.50`) was retired 2026-07-07. +- Home Assistant runs **natively on this host** (Podman Quadlet, host network) at **`http://10.42.1.1:8123`** (name `homeassistant`). The old HAOS VM `ha_van` (`10.42.0.50`) was retired 2026-07-07. --- @@ -86,10 +86,10 @@ This directory is the source of truth. The live system files live under `/etc`, | `10-van-ap.network` | `/etc/systemd/network/10-van-ap.network` | brings the AP wlan up with **no** IP (it's a `br0` member; hostapd enslaves it) | | `11-van-ap-2g.network` | `/etc/systemd/network/11-van-ap-2g.network` | same, for the 2.4GHz wlan | | `20-van-br0.netdev` | `/etc/systemd/network/20-van-br0.netdev` | creates the LAN bridge `br0` (STP off) | -| `21-van-br0.network` | `/etc/systemd/network/21-van-br0.network` | static `10.42.0.1/24` on `br0` (`ConfigureWithoutCarrier`) | +| `21-van-br0.network` | `/etc/systemd/network/21-van-br0.network` | static `10.42.1.1/24` on `br0` (`ConfigureWithoutCarrier`) | | `22-van-lan.network` | `/etc/systemd/network/22-van-lan.network` | enslaves the wired LAN port `enx00e04c331140` to `br0` | | `van-ap-unmanaged.conf` | `/etc/NetworkManager/conf.d/van-ap-unmanaged.conf` | tells NM to leave the AP wlan **and** the wired LAN port alone | -| `nftables.conf` | `/etc/nftables.conf` | NAT: `masquerade ip saddr 10.42.0.0/24 oifname != br0` → follows whatever WAN is active | +| `nftables.conf` | `/etc/nftables.conf` | NAT: `masquerade ip saddr 10.42.1.0/24 oifname != br0` → follows whatever WAN is active | | `99-van-router.conf` | `/etc/sysctl.d/99-van-router.conf` | `net.ipv4.ip_forward=1` | | `regdomain.service` | `/etc/systemd/system/regdomain.service` | `iw reg set CA` at boot, before NetworkManager | | `rtw89.conf` | `/etc/modprobe.d/rtw89.conf` | 5GHz dongle: `disable_ps_mode=Y` (else AP drops beacon when idle) + blacklists in-kernel rtw89 so the out-of-tree morrownr/rtw89 dkms driver (installed separately, not by deploy.sh) loads instead — see gotchas below | @@ -169,7 +169,7 @@ supply (laptop UPS-style monitor), unrelated hardware and purpose. | `homeassistant.container` | `/etc/containers/systemd/homeassistant.container` | Podman Quadlet: HA Container on the host network (`:8123`), config in `/srv/homeassistant`, host D-Bus mounted for onboard Bluetooth. | The `homeassistant` LAN name comes from `ap/van-ap-dnsmasq.conf` (`host-record` → -`10.42.0.1`). The retired HAOS VM's domain XML (`ha_van.xml`) lives in git history. +`10.42.1.1`). The retired HAOS VM's domain XML (`ha_van.xml`) lives in git history. `custom_components/` and `dashboards/` are **not deployed by `deploy.sh`** — HA's `/srv/homeassistant` is hand-managed live config (it has other integrations installed @@ -348,7 +348,7 @@ script's docstring for the full field layout. ### Home Assistant (native Podman container) - HA Container runs on the host via a **Podman Quadlet** (`ha/homeassistant.container` → `/etc/containers/systemd/`; systemd generates `homeassistant.service`). **Host network**: - HA binds `:8123` directly, so clients use `http://10.42.0.1:8123` (or + HA binds `:8123` directly, so clients use `http://10.42.1.1:8123` (or `http://homeassistant:8123`); mDNS/SSDP discovery sees the LAN because there's no bridge or NAT in the way. Config lives in `/srv/homeassistant`. - **Bluetooth**: host BlueZ (`bluetooth.service`) serves the Pi's onboard `hci0` to HA over @@ -359,9 +359,9 @@ script's docstring for the full field layout. (`podman pull ghcr.io/home-assistant/home-assistant:stable`) and restart. After editing the `.container` file: `./deploy.sh` (or install + `systemctl daemon-reload`), then `systemctl restart homeassistant`. Verify: - `curl -s -o /dev/null -w '%{http_code}' http://10.42.0.1:8123/` → `200`. + `curl -s -o /dev/null -w '%{http_code}' http://10.42.1.1:8123/` → `200`. - History: HA started as a HAOS VM on wayback (libvirt NAT, then bridged onto `br0` at - `10.42.0.50` with a legacy-URL DNAT), moved to this Pi as an aarch64 VM 2026-07-06, and + `10.42.1.50` with a legacy-URL DNAT), moved to this Pi as an aarch64 VM 2026-07-06, and went native 2026-07-07 — the fixed 2 GiB VM allocation starved the 4 GB Pi. The domain XML (`ha/ha_van.xml`) and the DNAT/hairpin nftables rules are in git history. diff --git a/README.md.bak b/README.md.bak new file mode 100644 index 0000000..f3a1688 --- /dev/null +++ b/README.md.bak @@ -0,0 +1,429 @@ +# vanlink — campervan router on `wayback` + +> **This clone = the Pi 4 port (host `wan`, branch `wan`).** Onboard `eth0`+`wlan0` are +> NM-managed WANs (netplan `ap/50-van-wan.yaml`); the AP radios + wired LAN port arrive +> with the USB hub from wayback (same MAC-derived names, so all configs port verbatim). +> Dropped here: HA VM, battery/lid (no hardware), heartbeat + ZeroTier (not installed yet). +> Watchdog is 10s (bcm2835 max 15s); thermal watches `cpu_thermal`. + +Turns **wayback** (Asus ZenBook UX391U, Ubuntu 24.04, zabbly kernel) into a self-contained +campervan hub/router/AP: + +- **WiFi-6 access point** for client devices (hand-rolled hostapd, not NetworkManager). +- **Multi-WAN failover** across WiFi (neighbour AP) → Ethernet (Starlink) → 4G/5G cellular. +- **ZeroTier-managed DNS** so `*.wrede.pvt` resolves over the overlay when away from home. +- **Cockpit dashboard** ("Van Router") for status + manual control. + +This directory is the source of truth. The live system files live under `/etc`, `/usr/...`; +`deploy.sh` copies from here to there. Edit here, run `sudo ./deploy.sh`, done. + +--- + +## 1. Architecture at a glance + +``` + WiFi clients (VanLink, 5GHz ch149, 80MHz WPA2) wired LAN clients + wlxc83a35a4ee55 (RTL8852BU, rtw89, hostapd) enx00e04c331140 + └───────────┬───────────────────┘ + br0 10.42.0.1/24 ← LAN bridge + │ dnsmasq DHCP .10–.254 + DNS + │ nftables masquerade (oifname != br0) + │ ip_forward=1 + ┌─────────────────┼──────────────────────────────────────┐ + wlp1s0 (m100) enxd8ec5eeb3512 (m200) Koodo gsm (m300) + neighbour WiFi USB ethernet → Starlink 4G/5G modem (when present) + └──────── van-failover picks lowest-metric HEALTHY WAN ────┘ + + NetworkManager owns the WAN side. systemd-networkd owns the AP IP + the ZeroTier DNS link. +``` + +### Roles / responsibilities +| Concern | Owner | +|---|---| +| WAN interfaces (eth / wifi / gsm), DHCP-client, metrics | **NetworkManager** | +| LAN bridge `br0` + IP (10.42.0.1) + wired LAN member | **systemd-networkd** (`2x-van-br0/lan`) | +| AP beaconing / WPA + adding the wlan to `br0` | **hostapd** (`bridge=br0`; AP iface + wired port are NM-*unmanaged*) | +| Keep the AP beaconing across USB re-enumeration (Starlink flap) | **hostapd** `Restart=always`/no start-limit drop-in + **van-ap-watchdog** daemon (recovers a wedged radio) | +| AP DHCP + DNS | **dnsmasq** (dedicated instance, bound to AP only) | +| NAT + forwarding | **nftables** + sysctl | +| WAN health + failover | **van-failover** daemon | +| Temperature monitor / alert / log | **van-thermal** daemon | +| NVMe I/O-timeout/reset watchdog + alert | **van-nvme-watch** daemon | +| Battery monitor / low-charge alert + shutdown | **van-battery** daemon | +| Auto-reboot on hang | **systemd hardware watchdog** (`intel_oc_wdt`) | +| Liveness / dead-man's switch + metrics | **hbc** heartbeat client → hbd.wrede.pvt | +| ZeroTier DNS → resolved | **zerotier-systemd-manager** + systemd-networkd | +| Web UI | **Cockpit** + `vanrouter` plugin | + +### Key network facts +- LAN: `10.42.0.0/24`, gateway `10.42.0.1`, DHCP `.10–.254` — shared by Wi-Fi (`VanLink`, dual-band: 5GHz `wlxc83a35a4ee55` + 2.4GHz `wlxd8ec5e2faa8c`) and the wired port `enx00e04c331140`, all bridged into `br0`. +- WAN priority (metrics): **wifi 100 → eth/Starlink 200 → 4G 300** (lower = preferred). +- Management / recovery: ethernet `192.168.10.251`, wifi `192.168.10.27`, ZeroTier `192.168.196.22`. +- Cockpit: `https://192.168.10.251:9090` (or `.27`, or ZeroTier). Log in with a Unix account; enable *Administrative access* for action buttons. +- Regulatory domain **CA** (unlocks 5GHz ch149–161 @30dBm, no DFS). +- ZeroTier network `d3ecf5726d041b2a`, pushed DNS domain `wrede.pvt` via `192.168.196.115` + `192.168.10.5`. +- Home Assistant runs **natively on this host** (Podman Quadlet, host network) at **`http://10.42.0.1:8123`** (name `homeassistant`). The old HAOS VM `ha_van` (`10.42.0.50`) was retired 2026-07-07. + +--- + +## 2. Directory layout → system destinations + +`deploy.sh` performs exactly this mapping. + +### `ap/` — access point +| file | → installs to | purpose | +|---|---|---| +| `hostapd.conf` | `/etc/hostapd/hostapd.conf` | AP: SSID `VanLink`, ch149, VHT80/HE, WPA2-PSK, country CA, `ctrl_interface` for the watchdog. **WPA passphrase lives here.** | +| `hostapd-restart.conf` | `/etc/systemd/system/hostapd.service.d/restart.conf` | `Restart=always` + `StartLimitIntervalSec=0` so hostapd never gives up after a USB re-enumeration bounces the radio | +| `van-ap-watchdog` | `/usr/local/sbin/van-ap-watchdog` | watches `hostapd_cli status`; restarts hostapd if the radio wedges (running but not `ENABLED`) | +| `van-ap-watchdog.service` | `/etc/systemd/system/van-ap-watchdog.service` | runs the watchdog | +| `default-hostapd` | `/etc/default/hostapd` | `DAEMON_CONF=...` | +| `van-ap-dnsmasq.conf` | `/etc/van-ap/dnsmasq.conf` | DHCP/DNS bound to AP iface (`bind-dynamic`, so it does not clash with systemd-resolved) | +| `van-ap-dnsmasq.service` | `/etc/systemd/system/van-ap-dnsmasq.service` | dedicated dnsmasq unit (uses the `dnsmasq-base` binary; the distro dnsmasq service is NOT used) | +| `hostapd-2g.conf` | `/etc/hostapd/hostapd-2g.conf` | 2.4GHz AP (Linksys WUSB6300v2 / RTL8822BU): same SSID+PSK, ch6, HT20. Own hostapd instance so one radio wedging never kills the other. | +| `hostapd-2g.service` | `/etc/systemd/system/hostapd-2g.service` | dedicated unit for the 2.4GHz hostapd (same Restart=always / no-start-limit semantics as the 5GHz drop-in) | +| `van-ap-watchdog-2g.service` | `/etc/systemd/system/van-ap-watchdog-2g.service` | second watchdog instance: `van-ap-watchdog /etc/hostapd/hostapd-2g.conf hostapd-2g` | +| `10-van-ap.network` | `/etc/systemd/network/10-van-ap.network` | brings the AP wlan up with **no** IP (it's a `br0` member; hostapd enslaves it) | +| `11-van-ap-2g.network` | `/etc/systemd/network/11-van-ap-2g.network` | same, for the 2.4GHz wlan | +| `20-van-br0.netdev` | `/etc/systemd/network/20-van-br0.netdev` | creates the LAN bridge `br0` (STP off) | +| `21-van-br0.network` | `/etc/systemd/network/21-van-br0.network` | static `10.42.0.1/24` on `br0` (`ConfigureWithoutCarrier`) | +| `22-van-lan.network` | `/etc/systemd/network/22-van-lan.network` | enslaves the wired LAN port `enx00e04c331140` to `br0` | +| `van-ap-unmanaged.conf` | `/etc/NetworkManager/conf.d/van-ap-unmanaged.conf` | tells NM to leave the AP wlan **and** the wired LAN port alone | +| `nftables.conf` | `/etc/nftables.conf` | NAT: `masquerade ip saddr 10.42.0.0/24 oifname != br0` → follows whatever WAN is active | +| `99-van-router.conf` | `/etc/sysctl.d/99-van-router.conf` | `net.ipv4.ip_forward=1` | +| `regdomain.service` | `/etc/systemd/system/regdomain.service` | `iw reg set CA` at boot, before NetworkManager | +| `rtw89.conf` | `/etc/modprobe.d/rtw89.conf` | 5GHz dongle: `disable_ps_mode=Y` (else AP drops beacon when idle) + blacklists in-kernel rtw89 so the out-of-tree morrownr/rtw89 dkms driver (installed separately, not by deploy.sh) loads instead — see gotchas below | +| `rtw88.conf` | `/etc/modprobe.d/rtw88.conf` | 2.4GHz dongle: `disable_lps_deep=Y` (same PS reasoning) + `switch_usb_mode=N` (don't self-upgrade to USB3 — it radiates into 2.4GHz) | + +### `failover/` — multi-WAN +| file | → installs to | purpose | +|---|---|---| +| `van-failover` | `/usr/local/sbin/van-failover` | the daemon (Python) | +| `config.json` | `/etc/van-failover/config.json` | WAN list + priorities + probe settings | +| `van-failover.service` | `/etc/systemd/system/van-failover.service` | `Restart=always` | +| `50-disable-eee` | `/etc/NetworkManager/dispatcher.d/50-disable-eee` | disables EEE on `en*`/`eth*` at connect (r8152 idle parking) | +| `99-van-arp.conf` | `/etc/sysctl.d/99-van-arp.conf` | `arp_ignore=1` / `arp_announce=2` (multi-NIC ARP hygiene) | + +### `dns/` — ZeroTier managed DNS +| file | → installs to | purpose | +|---|---|---| +| `zt-search.conf` | `/etc/systemd/network/99-ztuga7c2kh.network.d/search.conf` | makes `wrede.pvt` a search domain (bare-hostname completion) | +| `zt-network.local.conf` | `/var/lib/zerotier-one/networks.d/d3ecf5726d041b2a.local.conf` | contains `allowDNS=1` (prerequisite) | +| `99-ztuga7c2kh.network.generated` | *(reference only — generated by the manager)* | what zerotier-systemd-manager writes | +| `zerotier-systemd-manager.{service,timer}` | `/usr/lib/systemd/system/` | the manager (binary installed separately, see §4) | + +### `cockpit/vanrouter/` — web UI +`manifest.json`, `index.html`, `vanrouter.css`, `vanrouter.js` → `/usr/share/cockpit/vanrouter/`. +Being superseded by `router-status/` below (Cockpit's PAM+TLS login was unreliable +— rejected Firefox sometimes, failed outright from Android — and it was a resource +hog); left in place until the Home Assistant dashboard fully covers its actions. + +### `router-status/` — same status/actions, via MQTT -> Home Assistant +| file | → installs to | purpose | +|---|---|---| +| `van-router-mqtt` | `/usr/local/sbin/van-router-mqtt` | status+control daemon (Python, paho-mqtt): AP/clients/Starlink/WAN/modem, HA MQTT discovery; republishes van-failover/van-thermal/van-battery's own state.json verbatim rather than recomputing them | +| `config.json` | `/etc/van-router-mqtt/config.json` | broker + interface identifiers (`@TOKEN@` templated like `failover/config.json` — no secrets, rendered fresh every deploy) | +| `van-router-mqtt.service` | `/etc/systemd/system/van-router-mqtt.service` | `Restart=always` | + +### `power/` — never sleep + thermal monitor +| file | → installs to | purpose | +|---|---|---| +| `10-vanlink-nolid.conf` | `/etc/systemd/logind.conf.d/10-vanlink-nolid.conf` | logind ignores the lid in all states (closed / on AC / docked) | +| `van-thermal` | `/usr/local/sbin/van-thermal` | temperature daemon (Python): publishes state, alerts, logs history | +| `thermal-config.json` | `/etc/van-thermal/config.json` | sensors + warn/crit thresholds + sample/log intervals | +| `van-thermal.service` | `/etc/systemd/system/van-thermal.service` | `Restart=always` | +| `van-nvme-watch` | `/usr/local/sbin/van-nvme-watch` | NVMe watchdog (Python): tails `journalctl -k` for I/O-timeout/reset events, Pushover alerts | +| `nvme-watch-config.json` | `/etc/van-nvme-watch/config.json` | grace period, cooldown, device paths | +| `van-nvme-watch.service` | `/etc/systemd/system/van-nvme-watch.service` | `Restart=always` | +| `van-battery` | `/usr/local/sbin/van-battery` | battery daemon (Python): Pushover low-charge alerts + safe shutdown | +| `battery-config.json` | `/etc/van-battery/config.json` | warn levels, shutdown level, poll interval, paths | +| `van-battery.service` | `/etc/systemd/system/van-battery.service` | `Restart=always` | +| `pushover.json.example` | → `/etc/van-battery/pushover.json` (seeded if absent) | Pushover token+user **template**; real file is 0600, **not** in the repo | +| `10-vanlink-watchdog.conf` | `/etc/systemd/system.conf.d/10-vanlink-watchdog.conf` | hardware watchdog (`RuntimeWatchdogSec=20s`); applied via `daemon-reexec` | + +`deploy.sh` additionally **masks** `sleep.target suspend.target hibernate.target hybrid-sleep.target` +(no repo file — symlinks to `/dev/null` under `/etc/systemd/system/`) so nothing else can suspend either. + +### `heartbeat/` — dead-man's switch +| file | → installs to | purpose | +|---|---|---| +| `hbc.yaml` | `/etc/hbc.yaml` | heartbeat client config (interval + metric plugins) | +| `hbc.service` | `/etc/systemd/system/hbc.service` | runs `~/bin/hbc` as `andreas` → hbd.wrede.pvt:50003 | + +The `hbc` binary itself (venv at `~/venvs/hbd`, symlink `~/bin/hbc`) is installed once via the +heartbeat project's own installer — see §4. `deploy.sh` only starts the service once it exists. + +### `li3/` — RV house battery (BLE BMS → MQTT/HA) +| file | → installs to | purpose | +|---|---|---| +| `van-li3-battery` | `/usr/local/sbin/van-li3-battery` | BLE→MQTT daemon (Python, bleak + paho-mqtt): reads the Lithionics Li3 BMS, publishes HA MQTT discovery + state | +| `config.json.example` | → `/etc/van-li3/config.json` (seeded if absent) | BLE address, broker, MQTT topic/discovery **template**; real file is 0600, **not** in the repo | +| `van-li3-battery.service` | `/etc/systemd/system/van-li3-battery.service` | `Restart=always` | + +Not to be confused with `power/van-battery` — that's the host's own AC/battery power +supply (laptop UPS-style monitor), unrelated hardware and purpose. + +### `ha/` — Home Assistant (native container) +| file | → installs to | purpose | +|---|---|---| +| `homeassistant.container` | `/etc/containers/systemd/homeassistant.container` | Podman Quadlet: HA Container on the host network (`:8123`), config in `/srv/homeassistant`, host D-Bus mounted for onboard Bluetooth. | + +The `homeassistant` LAN name comes from `ap/van-ap-dnsmasq.conf` (`host-record` → +`10.42.0.1`). The retired HAOS VM's domain XML (`ha_van.xml`) lives in git history. + +`custom_components/` and `dashboards/` are **not deployed by `deploy.sh`** — HA's +`/srv/homeassistant` is hand-managed live config (it has other integrations installed +via HACS that were never in this repo, e.g. `mbapi2020`); these two are kept here purely +as a reference/backup copy of the hand-authored pieces. `dashboards/van-router.yaml` is +the Lovelace dashboard for `van-router-mqtt`'s entities — the live copy is +`/srv/homeassistant/dashboards/van-router.yaml`, wired in via a `lovelace: dashboards:` +block manually added to `/srv/homeassistant/configuration.yaml` (also not repo-tracked; +see that file's own comment). After editing the dashboard, copy both places and +`systemctl restart homeassistant` (or use HA's own "reload" for lovelace-only changes). + +--- + +## 3. Components & how to operate + +### Access Point +- **Dual band, one SSID**: 5GHz (`hostapd.conf` → unit `hostapd`, ch149 80MHz VHT+HE) and 2.4GHz (`hostapd-2g.conf` → unit `hostapd-2g`, ch6 HT20, for range + 2.4-only IoT). Same SSID/PSK — clients pick their band. Deliberately two hostapd processes: a USB wedge on one radio never takes the other down, and each has its own watchdog (`van-ap-watchdog`, `van-ap-watchdog-2g`). +- Change SSID / channel / password: edit `ap/hostapd.conf` and/or `ap/hostapd-2g.conf`, then `sudo cp` to `/etc/hostapd/` (or `./deploy.sh`) and restart the matching unit. +- DHCP range / DNS options: `ap/van-ap-dnsmasq.conf` + restart `van-ap-dnsmasq`. +- Services: `hostapd`, `hostapd-2g`, `van-ap-dnsmasq`, `systemd-networkd`, `nftables`, `regdomain`. +- **Do not** add `noscan` to hostapd.conf — Ubuntu's hostapd rejects it and fails to start. +- Verify each is actually beaconing with `iw dev wlxc83a35a4ee55 info` / `iw dev wlxd8ec5e2faa8c info` (ssid+channel+width present), not just `systemctl is-active`. + +### Multi-WAN failover (`van-failover`) +- Probes each WAN's real internet every few seconds (HTTP-204 check, captive-portal-aware), demotes a failed WAN by raising its route-metric, fails back on recovery (hysteresis). +- Status: `systemctl status van-failover`, `journalctl -u van-failover -f`, or the Cockpit "WAN Failover" card. Live state JSON at `/run/van-failover/state.json`. +- Tune priorities/timing in `failover/config.json` (`wans[].metric`, `probe_interval`, `probe_timeout`, `fail_threshold`, `ok_threshold`, `probe_urls`) then restart. +- WAN identity: eth/wifi keyed by `device`, cellular by NM `connection` name (`Koodo`). +- The daemon changes metrics with **`ip route`**, never `nmcli device reapply` — see Gotchas. + +#### Manual preference (Cockpit "Prefer" button) +- The daemon is the **single writer** of route-metrics. To force a specific WAN to the top it reads an optional file `/run/van-failover/prefer` containing one **device name** (e.g. `enxd8ec5eeb3512`). That WAN gets base metric **50** (below every config metric) so it wins while healthy; the health PENALTY still stacks on top, so failover/fail-back are unchanged. Empty/absent file = follow config priorities. +- The Cockpit "Prefer" button just writes that file (root via polkit) — it does **not** set metrics itself. Doing so directly would fight the daemon (reverted within one probe loop) and `reapply`-flap the USB carrier. The daemon applies the change on its next loop (~`probe_interval`); `state.json` carries a `preferred` flag the UI shows as "(preferred)". +- Single-WAN, sticky override with no explicit "clear": to return to automatic priority, prefer your top WAN (wifi) or `rm /run/van-failover/prefer`. **Ephemeral by design** — `/run` is wiped on reboot, so a reboot returns to config priorities. (To persist it, point `PREFER` at `/etc/van-failover/prefer` in the daemon and the matching path in `vanrouter.js`.) +- Swapping the preference between two WANs is collision-safe: a second default route can't take the new WAN's target metric while the old preferred WAN still holds it, so `enforce_route` keeps the existing route and retries next loop (settles in ~2 loops) instead of stranding the interface. `enforce_route` also restores a default route that went missing while the carrier is up (gateway from NM via `device_gateway`), not just rebases an existing one. + +### Thermal monitor (`van-thermal`) +- One daemon off a single sysfs sample loop does all three jobs: live Cockpit readout, threshold alerting, history. +- **Live state**: `/run/van-thermal/state.json` (atomic-swapped each sample). The Cockpit "Temperatures" card reads this — no shelling out to `sensors` per refresh. Sensors are resolved by hwmon **name** + **label** (`coretemp`/`Package id 0`, `nvme`/`Composite`), never by `hwmonN` index (not stable across boots). +- **Alerts**: level changes (ok ↔ warn ↔ crit) are logged to the journal with hysteresis (`clear_margin`, default 5 °C) so a sensor sitting on the line doesn't spam. Watch with `journalctl -u van-thermal -f`; warn/crit carry sd-daemon severity (`-p warning`/`-p err`). +- **History**: throttled CSV at `/var/log/van-thermal.csv` (one row per `log_interval`, default 60 s), self-rotating to `.csv.1` past `log_max_bytes` (5 MB). +- **Tuning**: edit `/etc/van-thermal/config.json` (thresholds, intervals, sensor list), then `systemctl restart van-thermal`. Defaults: CPU warn 80 / crit 95 °C (silicon crit is 100), NVMe warn 65 / crit 70 °C (drive crit ~71). +- Status: `systemctl status van-thermal` or `cat /run/van-thermal/state.json`. + +### NVMe watchdog (`van-nvme-watch`) +- Event-driven, not polled: tails `journalctl -kf` for the `nvme nvmeN: I/O tag ... timeout, reset controller` signature that crashed and corrupted the root fs on 2026-08-02 (recurred 2026-08-04, self-healed — see the memory notes for that investigation). There's no sensor to sample, only a log line to catch. +- On a match it watches a `grace_period` (default 20 s) for either a clean re-init (self-healed) or a second timeout / an ext4 error following it (escalated) before alerting, so the Pushover message already says which outcome happened — no need to SSH in during a scare. +- The alert body includes a live `smartctl`/`tune2fs` snapshot (SMART health, critical-warning flag, media error count, filesystem state). +- **Live state**: `/run/van-nvme-watch/state.json` (last event, same convention as van-failover/van-thermal). +- Shares Pushover credentials with van-thermal/van-battery (`/etc/van-battery/pushover.json`). +- **Tuning**: edit `/etc/van-nvme-watch/config.json` (`grace_period`, `cooldown`, device paths), then `systemctl restart van-nvme-watch`. +- Status: `systemctl status van-nvme-watch`, `journalctl -u van-nvme-watch -f`, or `cat /run/van-nvme-watch/state.json`. + +### Battery monitor (`van-battery`) +- Watches mains vs battery via `/sys/class/power_supply/AC0/online` (0 = on battery) and charge via `BAT0/capacity`. Both resolve by `type` (Mains/Battery) if those names ever differ. +- **Only while on battery**, it sends escalating **Pushover** alerts at **25 / 20 / 15 %**, and at **10 %** sends a final alert and runs `systemctl poweroff` (after `shutdown_grace`, default 8 s, so the alert flushes first). +- Edge-triggered per discharge episode: each severity fires once; the sequence **re-arms when mains returns**. Unplugging already below a warn level fires a single alert for the current severity (no burst), then shutdown at 10 %. +- **Credentials** live in `/etc/van-battery/pushover.json` (mode 0600), seeded from `pushover.json.example` on first deploy and **never committed**. Missing/placeholder creds disable *sending* but **not** the shutdown — running flat must always power down safely (the skip is logged to the journal). +- Tune in `/etc/van-battery/config.json` (`warn_levels`, `shutdown_level`, `poll_interval`), then `systemctl restart van-battery`. +- Status: `systemctl status van-battery`, `journalctl -u van-battery -f`, or `cat /run/van-battery/state.json`. +- **Caveat — the 10 % shutdown is one-way.** A laptop won't power itself back on when mains returns: "restore on AC loss" is a BIOS/firmware feature (not OS-controllable — `/proc/acpi/wakeup` only covers wake-from-suspend). If your BIOS exposes an "AC power-on / restore on AC loss" option, enable it so the router reboots itself once shore/solar power is back; this ZenBook likely doesn't have it, in which case a low-battery shutdown needs a manual power-on. + +### Adding the 4G/5G modem +1. Plug the USB modem in. ModemManager + the `Koodo` gsm NM connection (autoconnect) bring it up. + On a fresh host create the profile once: + `nmcli con add type gsm ifname "*" con-name Koodo apn sp.koodo.com connection.autoconnect yes` +2. It auto-joins as the `cellular` WAN at metric 300 (last resort). Nothing else to configure. +3. Confirm with `mmcli -L` and the Cockpit failover card (cellular flips from `absent` to `up`). +4. Gotcha (handled in `van-failover` since the Quectel EC25-AF landed): for MBIM/QMI modems + NM's device is the control port (`cdc-wdm0`) while IP + routes live on the wwan netdev + (e.g. `wwu1u2i4`). The daemon maps via `GENERAL.IP-IFACE` — probing or `ip route` + against `cdc-wdm0` silently sees nothing. The Cockpit WAN table does the same mapping + (shows the netdev + its IP; nmcli actions still target `cdc-wdm0`). +5. The Cockpit WAN row shows signal % / tech / operator (from `mmcli`) and has a + **Restart** button: `mmcli --disable && --enable` (the EC25 MBIM plugin doesn't + support `--reset`; autoconnect reconnects), falling back to a USB unbind/bind of the + Quectel device (vendor `2c7c`) if ModemManager can't reach the modem. + +### GPS (gpsd from the cellular modem) +The EC25-AF has a GNSS engine that streams NMEA on its USB interface 01 (`ttyUSB1`, +`gps` port in `mmcli`). Needs its own GPS antenna on the modem's GNSS connector for a fix. +- **One-time modem config** (persists in modem NV; survives reboot/replug/Restart): + GNSS auto-start via `AT+QGPSCFG="autogps",1` + `AT+QGPS=1`. ModemManager holds both + AT ports and swallows replies, so stop it first: + `systemctl stop ModemManager`, send the ATs on `/dev/ttyUSB3`, `systemctl start ModemManager`. +- **Host side** (deployed by deploy.sh, needs `apt install gpsd gpsd-clients`): + `gps/77-modem-gps.rules` udev rule matches the NMEA tty (2c7c:0125 if01), symlinks it + to `/dev/modem-gps`, and hot-adds it to gpsd via `gpsdctl@%k` — the same mechanism as + gpsd's own 60-gpsd.rules, so plug/unplug/renumbering just works. `gps/gpsd.default` + (`/etc/default/gpsd`) keeps `DEVICES` empty (hotplug does it) and runs gpsd with `-n`. +- Verify: `gpspipe -r -n 10` (raw NMEA), `cgps` (fix view), `systemctl status gpsdctl@ttyUSB1`. +- Don't enable MM location APIs (`mmcli --location-enable-gps-*`) at the same time — + MM would open the NMEA port and fight gpsd; `gps-unmanaged` is the only safe one. + +### OwnTracks publisher (gpsd -> MQTT) +`gps/van-gps-owntracks` (service `van-gps-owntracks`) streams TPV fixes from local +gpsd and publishes OwnTracks location JSON to `owntracks/rv/gps` on home.wrede.ca +(tid `rv`) — at most every 10 min when parked, immediately after >250 m of movement. +Port of the wayback-era `gps_to_owntracks.py`, adapted to apt-only deps +(`python3-gps`, `python3-paho-mqtt` — the old `gpsdclient` is pip-only) and the +paho 2.x callback API. +- Broker credentials live only in `/etc/van-gps/config.json` (0600, seeded from + `gps/config.json.example` — edit after first deploy). All knobs (broker, topic, + intervals, gpsd host) live there too. +- **client_id must stay `vanq-wan` (or anything unique)**: some legacy client still + holds `vanq` on the broker (old copy on wayback?) and the broker kicks whoever + shares its ID — the symptom is connect/disconnect every few seconds. +- On MQTT disconnect the script exits and systemd restarts it (RestartSec=15); + that *is* the reconnect logic, so a red blip after a WAN failover is normal. +- Debug tracing: `kill -HUP` the process toggles per-fix logging to the journal. +- Verify: `journalctl -u van-gps-owntracks` shows `mqtt connect` and stays quiet; + a restart publishes the current fix immediately (first-message path), so + subscribing to `owntracks/rv/gps` while restarting shows a live message. +- Note: the broker holds an ancient *retained* message on this topic from a 2023 + OwnTracks device (tid `RV`, Winegard SSID); our publishes are not retained. + +### Li3 battery monitor (`van-li3-battery`) +Publishes the RV's 12V LiFePO4 house battery (Lithionics Li3, BLE HM-10 UART module — +service `ffe0`/char `ffe1`, no pairing) to Home Assistant as 10 sensor entities (pack +voltage, 4 cell voltages, current, SOC, BMS/battery temperature, status) via MQTT +discovery. Protocol (CSV telemetry lines after sending `$traceon`+`$info`) reverse- +engineered from the `com.lithionics.bms` Android app's own BLE/parsing code — see the +script's docstring for the full field layout. +- Not the same battery as `power/van-battery` (host's own AC/UPS power supply) — + different hardware, different concern, deliberately different naming. +- Broker credentials live only in `/etc/van-li3/config.json` (0600, seeded from + `li3/config.json.example` — edit after first deploy, same pattern as `van-gps`). +- **Self-healing BLE**: scan/connect retries happen *inside* the running process (MQTT + session and HA entities stay up across them, no flapping) rather than relying on + systemd restarts. This Pi's onboard Bluetooth adapter (Cypress/CYW43) occasionally + wedges bluetoothd's discovery state after a run of failed connects to this specific + device (`Discovering` stays `yes` forever, every subsequent connect fails with + `le-connection-abort-by-local`) — as soon as a scan fails the daemon restarts + `bluetooth.service` itself to clear it, rate-limited to once per 5 min so it + doesn't repeatedly disrupt the AP's other BLE gear (motion sensors, IR remote). +- This BLE module accepts only **one central connection at a time** — while + `van-li3-battery` holds it, the Li3 phone app can't connect simultaneously. + `systemctl stop van-li3-battery` to free it up for the app. +- Verify: `journalctl -u van-li3-battery -f` (look for `connected` / `published: {...}`), + or `mosquitto_sub -h localhost -u homeassistant -P -t van/li3_battery/state`. + +### ZeroTier managed DNS +- `wrede.pvt` resolves over ZeroTier when off the home LAN. Mechanism: `allowDNS=1` (prereq) + `zerotier-systemd-manager` writes `99-ztuga7c2kh.network`, networkd applies it to resolved. +- Verify: `resolvectl status ztuga7c2kh` shows `DNS` scope + the two servers + `wrede.pvt`. +- NOTE: ZeroTier's *native* DNS push is a no-op on Linux (`setDns ... not implemented`), which is why the manager is required. +- `systemd-networkd-wait-online` is **masked** so the ZT link does **not** gate `network-online.target` — see Gotchas. + +### Cockpit dashboard +- `https://:9090` → "Van Router". Reload the browser after deploying plugin changes (Cockpit caches packages per session). +- Read-only status works as any user; Prefer/Up/Down/Restart need *Administrative access* (polkit). +- The Access Points card shows both bands (5GHz `hostapd`, 2.4GHz `hostapd-2g`) with per-band client lists and **Restart** buttons; the radio list lives in `vanrouter.js` (`const APS`). +- **Prefer** sets the manual WAN preference (see *Manual preference* above); **Up/Down** connect/disconnect the NM device. + +### Router status via MQTT (`van-router-mqtt`) — replacing Cockpit above +- Publishes the same status Cockpit's plugin showed, and accepts the same actions, over + MQTT with Home Assistant discovery — no web server/login of its own; HA (already running + on this Pi) owns the UI. Broker is the local Mosquitto on `127.0.0.1:1883` (already used by + HA itself and by `van-gps-owntracks`/`li3`); it currently allows anonymous connections, so + `config.json` carries no credentials. +- State topics (`van/router/...`): `ap/5g`, `ap/2g`, `clients`, `starlink`, `wan`, + `deploywarn`, plus verbatim republishes of `van-failover`'s and `van-thermal`'s own + `/run/van-*/state.json` (those two daemons were **not** modified — `van-thermal` is + deliberately stdlib-only for boot resilience, and `van-failover`'s control loop wasn't + touched). `van/router/battery` only appears if `power/van-battery` (documented in this + README, not actually deployed on this host) ever starts writing its state file. +- Command topics (`vanlink/cmd/...`): `ap/{5g,2g}/restart`, `modem/restart`, + `wan/{wifi,starlink,cellular}/set` (`connect`/`disconnect`), `wan/prefer` + (`auto`/`wifi`/`starlink`/`cellular` — writes the same `/run/van-failover/prefer` file + Cockpit's "Prefer" button did). Wi-Fi scan/connect (wlan0 as a WAN) was **not** ported — + it's an interactive password-prompt flow that doesn't map to MQTT/HA entities well and is + only used during initial setup; use `nmcli` directly for that. +- Verify: `journalctl -u van-router-mqtt -f`, `mosquitto_sub -h localhost -t 'van/router/#' -v`. + Entities should appear in HA under the "Van Router" device once the integration picks up + the retained discovery messages (Settings → Devices & Services → MQTT). + +### Home Assistant (native Podman container) +- HA Container runs on the host via a **Podman Quadlet** (`ha/homeassistant.container` → + `/etc/containers/systemd/`; systemd generates `homeassistant.service`). **Host network**: + HA binds `:8123` directly, so clients use `http://10.42.0.1:8123` (or + `http://homeassistant:8123`); mDNS/SSDP discovery sees the LAN because there's no bridge + or NAT in the way. Config lives in `/srv/homeassistant`. +- **Bluetooth**: host BlueZ (`bluetooth.service`) serves the Pi's onboard `hci0` to HA over + the mounted `/run/dbus` socket — add the Bluetooth integration in HA and it appears. +- No add-on store (that was HAOS's Supervisor): Mosquitto/Zigbee2MQTT-style add-ons become + their own containers/services if ever needed. +- Operate: `systemctl {status,restart} homeassistant`. Update: bump/pull the image + (`podman pull ghcr.io/home-assistant/home-assistant:stable`) and restart. After editing + the `.container` file: `./deploy.sh` (or install + `systemctl daemon-reload`), then + `systemctl restart homeassistant`. Verify: + `curl -s -o /dev/null -w '%{http_code}' http://10.42.0.1:8123/` → `200`. +- History: HA started as a HAOS VM on wayback (libvirt NAT, then bridged onto `br0` at + `10.42.0.50` with a legacy-URL DNAT), moved to this Pi as an aarch64 VM 2026-07-06, and + went native 2026-07-07 — the fixed 2 GiB VM allocation starved the 4 GB Pi. The domain + XML (`ha/ha_van.xml`) and the DNAT/hairpin nftables rules are in git history. + +### Never sleep (lid-closed operation) +- wayback lives lid-closed in the van and must stay up. Stock logind `HandleLidSwitch=suspend` would sleep it on lid close (even on AC). The `power/10-vanlink-nolid.conf` drop-in sets all three lid actions to `ignore`; `deploy.sh` also masks every sleep target so idle / GUI / a stray `systemctl suspend` can't suspend it. +- Verify: `busctl get-property org.freedesktop.login1 /org/freedesktop/login1 org.freedesktop.login1.Manager HandleLidSwitch` → `"ignore"`, and `systemctl is-enabled suspend.target` → `masked`. `busctl call ... CanSuspend` should return `"no"`. +- **To re-enable sleep** (e.g. if wayback is ever a laptop again): + ```bash + sudo rm /etc/systemd/logind.conf.d/10-vanlink-nolid.conf + sudo systemctl unmask sleep.target suspend.target hibernate.target hybrid-sleep.target + sudo systemctl restart systemd-logind + ``` + (then remove the `power/` block from `deploy.sh`, or the next deploy re-applies it). + +--- + +## 4. Deploy / rebuild + +```bash +cd ~/vanlink +sudo ./deploy.sh # copies all files to their system locations, reloads + enables services +``` + +Four things `deploy.sh` does **not** do (one-time, manual): + +1. **zerotier-systemd-manager binary** (v0.4.0, hand-installed — not in the ZeroTier apt repo): + ```bash + # copy /usr/bin/zerotier-systemd-manager from a host that has it (e.g. wertvoll), then: + sudo systemctl enable --now zerotier-systemd-manager.timer + sudo zerotier-cli set d3ecf5726d041b2a allowDNS=1 + ``` +2. **hostapd unmask** (Ubuntu ships it masked): `sudo systemctl unmask hostapd`. +3. **Pushover credentials** for `van-battery` — deploy seeds `/etc/van-battery/pushover.json` (0600) with placeholders; fill in your app token + user key, then `sudo systemctl restart van-battery`. Until then low-battery alerts are skipped (logged), but the 10 % auto-shutdown still works. +4. **Heartbeat client `hbc`** (dead-man's switch) — install the binary once into andreas' venv: + ```bash + git clone https://git.wrede.ca/andreas/heartbeat.git ~/git/heartbeat # if not already cloned + sh ~/git/heartbeat/scripts/hb_install.sh client # -> ~/venvs/hbd + ~/bin/hbc + ``` + `deploy.sh` installs `/etc/hbc.yaml` + the `hbc.service` unit and starts it once `~/bin/hbc` exists. The service reports to **hbd.wrede.pvt** (UDP 50003) as user `andreas`. Upgrade later with `~/bin/hb_install.sh client`. + +--- + +## 5. Key design decisions & hard-won gotchas + +- **AP on hostapd, not NetworkManager.** NM's hotspot caps the rtw89 radio at HT20/20MHz; hostapd gives the full VHT80/HE (WiFi-6). The AP iface is therefore NM-*unmanaged*; networkd gives it its static IP. +- **`rtw89` power-save must be off** (`disable_ps_mode=Y`) or the AP stops beaconing when idle and the SSID vanishes. +- **RTL8852BU is USB-2.0 and hangs under load if it shares a USB hub.** Keep the AP dongle on its **own** USB controller, separate from the WAN ethernet. Symptom of a shared bus: `c2h reg timeout` + `Polling beacon packet empty fail` under throughput, SSID drops. (`timed out to flush queues` alone is benign.) True regardless of which driver (in-kernel or out-of-tree) is bound — it's a physical bus-bandwidth issue, not a driver bug. +- **5GHz dongle runs the out-of-tree `morrownr/rtw89` driver (since 2026-08-23), not the in-kernel one.** This Ubuntu kernel base (`7.0.0-1017-raspi`) predates mainline's rtw89 USB2→3 auto-switch, so the in-kernel driver permanently caps the 8852BU at USB2/480M. The out-of-tree driver has that switch (`switch_usb_mode=y` in `ap/rtw89.conf`) and needs a real USB3-capable port to actually benefit from it — a USB2-only port still caps it regardless of driver (confirmed live: 480M on a USB2-only controller, 5000M/SuperSpeed after moving to a dedicated USB3 controller). Driver itself logs `"2.4 GHz performance may be better in a USB 2 port"` on load — watch the 2.4GHz radio for new interference now that 5GHz runs SuperSpeed signalling nearby. Installed/updated via `sudo ./ap/install-rtw89-driver.sh` (pins a specific upstream commit, `dkms install`, `make install_fw`) — **not** run automatically by `deploy.sh`, since a dkms rebuild is slow and shouldn't fire on every routine deploy; run it manually after a fresh Pi provisioning or to bump the pin. DKMS itself auto-rebuilds across kernel upgrades. Module names get a `_git` suffix (`rtw89_8852bu_git` etc.) — `ap/rtw89.conf`'s blacklist stops the in-kernel modules from claiming the device instead. **Gotcha hit switching a live system over**: the in-kernel `rtw89_core` won't unload while its own in-kernel dependents (`rtw89_8852b`, `rtw89_8852b_common`) are still loaded — remove all of them first or the out-of-tree module fails to load (`exports duplicate symbol`). The install script's own output has the exact recovery steps. +- **van-failover changes metrics via `ip route`, never `nmcli device reapply`.** `reapply` **resets the r8152 USB-ethernet carrier**, which caused a ~10s-ping-drop flapping feedback loop. Pure `ip route` changes are carrier-safe. +- **Disable EEE on USB ethernet** (`50-disable-eee` dispatcher) — its idle power-save parks the *backup* WAN link and breaks health probes. +- **Per-WAN probing needs `curl --interface if!`** (forces `SO_BINDTODEVICE`); plain `--interface ` only sets the source IP and still routes via the default WAN. `rp_filter` is loose (`2`), required for this. +- **`arp_ignore`/`arp_announce`** matter only when two WANs share a subnet (a home-LAN test artifact; Starlink + neighbour-wifi will be on different subnets in the van). Harmless to keep. +- **dnsmasq uses `bind-dynamic` bound to the AP iface** so it coexists with systemd-resolved (no port-53 fight) — resolved stays intact for the host. +- **`systemd-networkd-wait-online` must be masked** or it deadlocks boot. networkd here manages only two links — the AP (`wlxc…`, local-only) and the ZT overlay `ztuga7c2kh` — and **neither is a real uplink**. The ZT link only appears once `zerotier-one` starts, but `zerotier-one` is ordered `After=network-online.target`, so wait-online would block on the not-yet-existing link until its 120s timeout. Marking the links `RequiredForOnline=no` does **not** help: with *zero* candidate links, wait-online can never satisfy "online" and times out anyway (systemd 255 — verified: `Timeout occurred while waiting for network connectivity` at exactly +120s). Both add ~2min to every boot and delay ZeroTier (the recovery path). The fix is to mask the service entirely (`deploy.sh` does this); real uplink readiness is covered by `NetworkManager-wait-online`, which owns the actual WANs. + +--- + +## 6. Recovery / safety +- Three independent management paths: ethernet `.251`, wifi `.27`, ZeroTier `192.168.196.22`. Changing default-route metrics does **not** affect the local-subnet (SSH) routes. +- The AP fan/EC, hostapd, networkd, nftables, sysctl, failover, regdomain are all enabled and reboot-persistent. + +## 7. Pending / future ideas +- Real-world soak once the actual Starlink terminal + USB 4G/5G modem are installed (current eth is the home LAN; subnets/probe targets may want tuning). +- Cockpit plugin: add an AP-config form (SSID/channel/PSK editing `hostapd.conf`). *(Manual failover override via the "Prefer" button is done — see §3.)* +- Consider pinning a fixed interface name for the WAN ethernet (USB MAC-derived names change if the adapter is swapped). diff --git a/ap/.van-ap-dnsmasq.conf.swp b/ap/.van-ap-dnsmasq.conf.swp new file mode 100644 index 0000000000000000000000000000000000000000..23d8bca2fd531b50513a64c3133f57017c50c64a GIT binary patch literal 12288 zcmeI&zi-qq6bEpZnO`dZfVmq0DNd8LCzVP!u)%=DQg&bLYusM!)OHTK6lP{dR@m4X zK`gBN3kd!J{sLZ_OKB16)TPqb(v#16elN~vi_Wv``ID#oVVW}wSVT- z5)%X<009U<00Izz00bZa0SG_<0#{H#D#ntVj75s>|Nr6p|MxA%eo=g<_)77K;yuM7 z#hBtJ?f*>iZdDt)dJG6a00Izz00bZa0SG_<0uX?}839$5jc`pg&QAMjhwguip#4(& zB+t^VBF*VW`hWI$*T!>uDi@;gx^!MmD_7HX^^r7CLJorK?i=2R_h(yy$J zFPuL}_|A4-Y(C&WUcc>M%1gVar>Ae4ogTKBY$e5Hk`6(cqUvj88!P8cJb5mgxxF@s s#+xNmu9n6MtF_XK>6fK$qA-@6=Cah&g<~^gl~70YmZoHXGc%vDKbfYkCIA2c literal 0 HcmV?d00001 diff --git a/ap/10-van-ap.network b/ap/10-van-ap.network index 2fdb8d2..6419efd 100644 --- a/ap/10-van-ap.network +++ b/ap/10-van-ap.network @@ -4,7 +4,7 @@ Name=@WIFI_5G_IFACE@ [Link] RequiredForOnline=no -# No IP here: the AP wlan is a member of br0, which carries the 10.42.0.1 gateway. +# No IP here: the AP wlan is a member of br0, which carries the 10.42.1.1 gateway. # hostapd adds this interface to br0 (bridge=br0) after it puts the radio into AP # mode — networkd must NOT enslave it (a station-mode wlan can't be bridged), so # this file only brings the link up. diff --git a/ap/10-van-ap.network.bak b/ap/10-van-ap.network.bak new file mode 100644 index 0000000..2fdb8d2 --- /dev/null +++ b/ap/10-van-ap.network.bak @@ -0,0 +1,12 @@ +[Match] +Name=@WIFI_5G_IFACE@ + +[Link] +RequiredForOnline=no + +# No IP here: the AP wlan is a member of br0, which carries the 10.42.0.1 gateway. +# hostapd adds this interface to br0 (bridge=br0) after it puts the radio into AP +# mode — networkd must NOT enslave it (a station-mode wlan can't be bridged), so +# this file only brings the link up. +[Network] +ConfigureWithoutCarrier=yes diff --git a/ap/20-van-br0.netdev b/ap/20-van-br0.netdev index 514adf0..432efc7 100644 --- a/ap/20-van-br0.netdev +++ b/ap/20-van-br0.netdev @@ -1,6 +1,6 @@ # LAN bridge: joins the AP Wi-Fi radios (added by hostapd, once in AP mode) and the # wired LAN ports (enx00e04c331140 + onboard eth0, added by networkd) into one -# 10.42.0.0/24 segment. STP off: all members are leaf ports, no loops (don't cable +# 10.42.1.0/24 segment. STP off: all members are leaf ports, no loops (don't cable # the two wired ports to the same switch), and it avoids the forwarding delay that # would otherwise stall the first DHCP handshake on a freshly-plugged client. [NetDev] diff --git a/ap/20-van-br0.netdev.bak b/ap/20-van-br0.netdev.bak new file mode 100644 index 0000000..514adf0 --- /dev/null +++ b/ap/20-van-br0.netdev.bak @@ -0,0 +1,11 @@ +# LAN bridge: joins the AP Wi-Fi radios (added by hostapd, once in AP mode) and the +# wired LAN ports (enx00e04c331140 + onboard eth0, added by networkd) into one +# 10.42.0.0/24 segment. STP off: all members are leaf ports, no loops (don't cable +# the two wired ports to the same switch), and it avoids the forwarding delay that +# would otherwise stall the first DHCP handshake on a freshly-plugged client. +[NetDev] +Name=br0 +Kind=bridge + +[Bridge] +STP=no diff --git a/ap/21-van-br0.network.bak b/ap/21-van-br0.network.bak new file mode 100644 index 0000000..54924d7 --- /dev/null +++ b/ap/21-van-br0.network.bak @@ -0,0 +1,16 @@ +# The LAN gateway IP now lives on the bridge, not on the AP wlan (which is just a +# bridge member). ConfigureWithoutCarrier keeps 10.42.0.1 up even before any wired +# device is plugged, exactly as it was on the wlan-only setup. +[Match] +Name=br0 + +[Link] +RequiredForOnline=no + +[Network] +Address=10.42.0.1/24 +ConfigureWithoutCarrier=yes +# Resolve .local (mDNS) for devices on the van's own LAN (ESPHome nodes, +# etc.) — scoped to br0 only, not the WAN links, since mDNS is link-local +# and that's the only interface those devices are actually on. +MulticastDNS=yes diff --git a/ap/22-van-lan.network b/ap/22-van-lan.network index 357e9b9..dc34fcd 100644 --- a/ap/22-van-lan.network +++ b/ap/22-van-lan.network @@ -1,4 +1,4 @@ -# Wired LAN port: enslaved to br0 so anything plugged in lands on 10.42.0.0/24 and +# Wired LAN port: enslaved to br0 so anything plugged in lands on 10.42.1.0/24 and # gets DHCP/DNS from the same dnsmasq as Wi-Fi clients. NM must leave this device # alone (see van-ap-unmanaged.conf) for networkd to own it here. [Match] diff --git a/ap/22-van-lan.network.bak b/ap/22-van-lan.network.bak new file mode 100644 index 0000000..357e9b9 --- /dev/null +++ b/ap/22-van-lan.network.bak @@ -0,0 +1,11 @@ +# Wired LAN port: enslaved to br0 so anything plugged in lands on 10.42.0.0/24 and +# gets DHCP/DNS from the same dnsmasq as Wi-Fi clients. NM must leave this device +# alone (see van-ap-unmanaged.conf) for networkd to own it here. +[Match] +Name=@LAN_USB_IFACE@ + +[Link] +RequiredForOnline=no + +[Network] +Bridge=br0 diff --git a/ap/23-van-lan-eth0.network b/ap/23-van-lan-eth0.network index 439aad4..67b8f26 100644 --- a/ap/23-van-lan-eth0.network +++ b/ap/23-van-lan-eth0.network @@ -1,5 +1,5 @@ # Onboard GbE (RP1 eth0) as the internal LAN port: enslaved to br0 so anything -# plugged in lands on 10.42.0.0/24 with DHCP/DNS from the same dnsmasq as Wi-Fi +# plugged in lands on 10.42.1.0/24 with DHCP/DNS from the same dnsmasq as Wi-Fi # clients. Was the metric-150 ethernet WAN until 2026-07-12 — with wifi + # Starlink + cellular as uplinks, the wired port earns its keep as LAN instead. # NM must leave it alone (see van-ap-unmanaged.conf) for networkd to own it here. diff --git a/ap/23-van-lan-eth0.network.bak b/ap/23-van-lan-eth0.network.bak new file mode 100644 index 0000000..439aad4 --- /dev/null +++ b/ap/23-van-lan-eth0.network.bak @@ -0,0 +1,13 @@ +# Onboard GbE (RP1 eth0) as the internal LAN port: enslaved to br0 so anything +# plugged in lands on 10.42.0.0/24 with DHCP/DNS from the same dnsmasq as Wi-Fi +# clients. Was the metric-150 ethernet WAN until 2026-07-12 — with wifi + +# Starlink + cellular as uplinks, the wired port earns its keep as LAN instead. +# NM must leave it alone (see van-ap-unmanaged.conf) for networkd to own it here. +[Match] +Name=eth0 + +[Link] +RequiredForOnline=no + +[Network] +Bridge=br0 diff --git a/ap/hostapd-2g.conf b/ap/hostapd-2g.conf index 299e210..535683d 100644 --- a/ap/hostapd-2g.conf +++ b/ap/hostapd-2g.conf @@ -8,7 +8,7 @@ ctrl_interface=/var/run/hostapd driver=nl80211 # Same SSID + PSK as 5GHz: dual-band clients pick a band themselves, 2.4-only IoT # devices just join. Rename here (e.g. VanLink-2G) if band pinning is ever needed. -ssid=VanLink +ssid=VanLink2-2G country_code=CA ieee80211d=1 hw_mode=g diff --git a/ap/hostapd.conf b/ap/hostapd.conf index f7633c7..99ffa78 100644 --- a/ap/hostapd.conf +++ b/ap/hostapd.conf @@ -1,5 +1,5 @@ interface=@WIFI_5G_IFACE@ -# Put the AP into br0 so Wi-Fi and the wired LAN port share one 10.42.0.0/24 segment. +# Put the AP into br0 so Wi-Fi and the wired LAN port share one 10.42.1.0/24 segment. # hostapd adds the wlan to the bridge after setting AP mode; the bridge itself + its # wired member + the gateway IP are defined under /etc/systemd/network (2x-van-br0/lan). bridge=br0 @@ -8,7 +8,7 @@ bridge=br0 # stays "running" but stops serving after a USB glitch. ctrl_interface=/var/run/hostapd driver=nl80211 -ssid=VanLink +ssid=VanLink2 country_code=CA ieee80211d=1 ieee80211h=0 diff --git a/ap/hostapd.conf.bak b/ap/hostapd.conf.bak new file mode 100644 index 0000000..dec8de2 --- /dev/null +++ b/ap/hostapd.conf.bak @@ -0,0 +1,31 @@ +interface=@WIFI_5G_IFACE@ +# Put the AP into br0 so Wi-Fi and the wired LAN port share one 10.42.0.0/24 segment. +# hostapd adds the wlan to the bridge after setting AP mode; the bridge itself + its +# wired member + the gateway IP are defined under /etc/systemd/network (2x-van-br0/lan). +bridge=br0 +# Control socket so van-ap-watchdog (and hostapd_cli) can read the AP's real state +# (state=ENABLED means it's actually beaconing) — used to detect a wedged radio that +# stays "running" but stops serving after a USB glitch. +ctrl_interface=/var/run/hostapd +driver=nl80211 +ssid=VanLink2 +country_code=CA +ieee80211d=1 +ieee80211h=0 +hw_mode=a +channel=149 +wmm_enabled=1 +ieee80211n=1 +ht_capab=[HT40+][SHORT-GI-20][SHORT-GI-40] +ieee80211ac=1 +vht_oper_chwidth=1 +vht_oper_centr_freq_seg0_idx=155 +vht_capab=[RXLDPC][SHORT-GI-80][SU-BEAMFORMEE] +ieee80211ax=1 +he_oper_chwidth=1 +he_oper_centr_freq_seg0_idx=155 +auth_algs=1 +wpa=2 +wpa_passphrase=1foot11foot11 +wpa_key_mgmt=WPA-PSK +rsn_pairwise=CCMP diff --git a/ap/nftables.conf.bak b/ap/nftables.conf.bak new file mode 100755 index 0000000..91ee99f --- /dev/null +++ b/ap/nftables.conf.bak @@ -0,0 +1,11 @@ +#!/usr/sbin/nft -f +# van-router NAT — masquerade LAN out whatever the WAN of the moment is +# (anything that is NOT the LAN bridge br0: ethernet/Starlink, wifi, future 4G) +table ip van_router_nat +delete table ip van_router_nat +table ip van_router_nat { + chain postrouting { + type nat hook postrouting priority srcnat; policy accept; + ip saddr 10.42.0.0/24 oifname != "br0" masquerade + } +} diff --git a/ap/van-ap-dnsmasq.conf.bak b/ap/van-ap-dnsmasq.conf.bak new file mode 100644 index 0000000..abab1b6 --- /dev/null +++ b/ap/van-ap-dnsmasq.conf.bak @@ -0,0 +1,12 @@ +interface=br0 +bind-dynamic +except-interface=lo +domain-needed +bogus-priv +dhcp-authoritative +dhcp-range=10.42.0.10,10.42.0.254,255.255.255.0,12h +# Home Assistant runs natively on this host (Podman, host network, :8123) — +# "homeassistant" is just another name for the router address. +host-record=homeassistant,10.42.0.1 +dhcp-option=option:router,10.42.0.1 +dhcp-option=option:dns-server,10.42.0.1 diff --git a/deploy.sh b/deploy.sh index b590030..c3d676f 100755 --- a/deploy.sh +++ b/deploy.sh @@ -227,6 +227,11 @@ install_rendered router-status/config.json /etc/van-router-mqtt/con install -D -m0644 router-status/van-router-mqtt.service /etc/systemd/system/van-router-mqtt.service python3 -c 'import paho.mqtt' 2>/dev/null \ || warn "python3-paho-mqtt missing (apt install python3-paho-mqtt) — van-router-mqtt won't start" +# MQTT broker must listen on all interfaces (not just localhost) for clients on br0 +install -D -m0644 router-status/mosquitto-listener.conf /etc/mosquitto/conf.d/van-listener.conf +# Restart limit drop-ins: prevent infinite restart loops that trigger hardware watchdog +install -D -m0644 router-status/van-gps-owntracks-restart-limit.conf /etc/systemd/system/van-gps-owntracks.service.d/restart-limit.conf +install -D -m0644 li3/van-li3-battery-restart-limit.conf /etc/systemd/system/van-li3-battery.service.d/restart-limit.conf echo "== gps (cellular modem GNSS -> gpsd) ==" if dpkg -s gpsd >/dev/null 2>&1; then @@ -308,6 +313,7 @@ systemctl daemon-reexec # an already-up WAN keeps its currently-applied DNS until it reconnects (or # reboot), deliberately not forced here (reconnecting a WAN flaps it). systemctl restart systemd-resolved +systemctl restart mosquitto # networkd here owns only the AP radios + bridge + wired LAN port (no real uplink); # its wait-online would just stall network-online.target. NM-wait-online covers WANs. systemctl mask systemd-networkd-wait-online.service >/dev/null 2>&1 || true diff --git a/failover/sync-usb-backup.sh b/failover/sync-usb-backup.sh index 4949b39..64396ca 100755 --- a/failover/sync-usb-backup.sh +++ b/failover/sync-usb-backup.sh @@ -15,6 +15,11 @@ set -euo pipefail USB_ROOT_PART=/dev/sda2 MOUNT_POINT=/mnt/usb-backup-root +MOUNT_POINT=10.42.1.44:/media/andreas/writable-nvme + + +[[ "$MOUNT_POINT" == *":"* ]] +LOCAL_MOUNT=$? root_src=$(findmnt -no SOURCE /) if [ "$root_src" = "$USB_ROOT_PART" ]; then @@ -22,21 +27,23 @@ if [ "$root_src" = "$USB_ROOT_PART" ]; then exit 1 fi -mkdir -p "$MOUNT_POINT" -mounted_here=0 -if ! mountpoint -q "$MOUNT_POINT"; then - mount "$USB_ROOT_PART" "$MOUNT_POINT" - mounted_here=1 -else - mount -o remount,rw "$MOUNT_POINT" -fi +if [ $LOCAL_MOUNT -eq 1 ]; then + mkdir -p "$MOUNT_POINT" + mounted_here=0 + if ! mountpoint -q "$MOUNT_POINT"; then + mount "$USB_ROOT_PART" "$MOUNT_POINT" + mounted_here=1 + else + mount -o remount,rw "$MOUNT_POINT" + fi -cleanup() { - if [ "$mounted_here" -eq 1 ]; then - umount "$MOUNT_POINT" - fi -} -trap cleanup EXIT + cleanup() { + if [ "$mounted_here" -eq 1 ]; then + umount "$MOUNT_POINT" + fi + } + trap cleanup EXIT +fi # rsync exit 24 ("partial transfer due to vanished source files") is expected on a # live root — files like container shm sockets or in-progress logs routinely disappear diff --git a/failover/sync-usb-backup.sh.bak b/failover/sync-usb-backup.sh.bak new file mode 100755 index 0000000..b2c9ca9 --- /dev/null +++ b/failover/sync-usb-backup.sh.bak @@ -0,0 +1,68 @@ +#!/usr/bin/env bash +# Refresh the USB disk (sda) as a live boot-fallback clone of the +# running root filesystem. Run manually after a push: +# sudo ./failover/sync-usb-backup.sh +# +# EEPROM BOOT_ORDER falls back to the SD card, then this USB disk, if the +# primary boot disk (NVMe) doesn't come up. /etc/fstab, /etc/machine-id, +# and /var/log are excluded from the sync so the clone keeps its own +# partition identity (PARTUUIDs differ per disk) and boot/journal history +# instead of inheriting the source disk's — see failover/README or repo +# memory "sd-fallback-fstab-clobber" for what breaks if those leak across. +set -euo pipefail + +[ "$(id -u)" -eq 0 ] || { echo "Run as root (sudo)." >&2; exit 1; } + +USB_ROOT_PART=/dev/sda2 +MOUNT_POINT=/mnt/usb-backup-root +MOUNT_POINT=10.42.0.44:/media/andreas/writable-nvme + + +[[ "$MOUNT_POINT" == *":"* ]] +LOCAL_MOUNT=$? + +root_src=$(findmnt -no SOURCE /) +if [ "$root_src" = "$USB_ROOT_PART" ]; then + echo "Currently booted from $USB_ROOT_PART — refusing to sync USB onto itself." >&2 + exit 1 +fi + +if [ $LOCAL_MOUNT -eq 1 ]; then + mkdir -p "$MOUNT_POINT" + mounted_here=0 + if ! mountpoint -q "$MOUNT_POINT"; then + mount "$USB_ROOT_PART" "$MOUNT_POINT" + mounted_here=1 + else + mount -o remount,rw "$MOUNT_POINT" + fi + + cleanup() { + if [ "$mounted_here" -eq 1 ]; then + umount "$MOUNT_POINT" + fi + } + trap cleanup EXIT +fi + +# rsync exit 24 ("partial transfer due to vanished source files") is expected on a +# live root — files like container shm sockets or in-progress logs routinely disappear +# mid-sync. Tolerate that one code; anything else is a real failure and still aborts. +rc=0 +rsync -aHAX --numeric-ids --delete \ + --info=progress2 \ + --exclude=/proc/* --exclude=/sys/* --exclude=/dev/* --exclude=/run/* \ + --exclude=/tmp/* --exclude=/mnt/* --exclude=/media/* --exclude=/lost+found \ + --exclude=/boot/firmware/* --exclude=/swapfile \ + --exclude=/etc/fstab --exclude=/etc/machine-id --exclude=/var/log/* \ + --stats / "$MOUNT_POINT/" || rc=$? + +if [ "$rc" -ne 0 ] && [ "$rc" -ne 24 ]; then + echo "rsync failed (exit $rc)." >&2 + exit "$rc" +fi +if [ "$rc" -eq 24 ]; then + echo "Note: some files vanished mid-sync (rsync exit 24) — normal on a live root." >&2 +fi + +echo "USB backup ($USB_ROOT_PART) synced from $root_src." diff --git a/ha/esphome.container b/ha/esphome.container index 0b02341..f687304 100644 --- a/ha/esphome.container +++ b/ha/esphome.container @@ -3,7 +3,7 @@ # container is the equivalent). Installed by deploy.sh to /etc/containers/systemd/; # systemd generates esphome.service from it. # -# Host networking: the dashboard binds :6052 directly (http://10.42.0.1:6052), +# Host networking: the dashboard binds :6052 directly (http://10.42.1.1:6052), # and ESPHome's mDNS-based OTA discovery/flashing of already-provisioned nodes # needs to see the LAN as the host does — a bridged network would need explicit # port/mDNS forwarding for the same result. diff --git a/ha/esphome.container.bak b/ha/esphome.container.bak new file mode 100644 index 0000000..0b02341 --- /dev/null +++ b/ha/esphome.container.bak @@ -0,0 +1,35 @@ +# ESPHome dashboard as a Podman Quadlet, same pattern as homeassistant.container +# (this isn't Home Assistant Supervised, so there's no add-on store — a sibling +# container is the equivalent). Installed by deploy.sh to /etc/containers/systemd/; +# systemd generates esphome.service from it. +# +# Host networking: the dashboard binds :6052 directly (http://10.42.0.1:6052), +# and ESPHome's mDNS-based OTA discovery/flashing of already-provisioned nodes +# needs to see the LAN as the host does — a bridged network would need explicit +# port/mDNS forwarding for the same result. +[Unit] +Description=ESPHome dashboard (Podman container) +Wants=network-online.target +After=network-online.target + +[Container] +Image=ghcr.io/esphome/esphome:stable +ContainerName=esphome +Network=host +Volume=/srv/esphome:/config +Environment=TZ=America/Toronto +# Initial flash of a new device needs USB serial access; OTA re-flashes of an +# already-provisioned node don't. Uncomment and set the real path to flash +# over USB (find it with `ls /dev/ttyUSB* /dev/ttyACM*` after plugging the +# device in): +#Volume=/dev/ttyUSB0:/dev/ttyUSB0 +#AddCapability=SYS_ADMIN + +[Service] +Restart=always +# First start pulls the image; PlatformIO also downloads toolchains on a +# node's first compile — both want a working WAN, not just a fast one. +TimeoutStartSec=900 + +[Install] +WantedBy=multi-user.target diff --git a/ha/homeassistant.container b/ha/homeassistant.container index b93f0f2..090f747 100644 --- a/ha/homeassistant.container +++ b/ha/homeassistant.container @@ -4,7 +4,7 @@ # /etc/containers/systemd/; systemd generates homeassistant.service from it. # # Host networking: HA binds :8123 on the host directly, so the LAN URL is -# http://10.42.0.1:8123 (name: homeassistant) — no bridge port, no DNAT. +# http://10.42.1.1:8123 (name: homeassistant) — no bridge port, no DNAT. # /run/dbus gives HA the host BlueZ stack = the Pi's onboard Bluetooth (hci0). [Unit] Description=Home Assistant (Podman container) diff --git a/ha/homeassistant.container.bak b/ha/homeassistant.container.bak new file mode 100644 index 0000000..b93f0f2 --- /dev/null +++ b/ha/homeassistant.container.bak @@ -0,0 +1,37 @@ +# Home Assistant Container as a Podman Quadlet (replaced the ha_van HAOS VM +# 2026-07-07 — the fixed 2 GiB VM allocation starved the 4 GB Pi; VM domain XML +# is in git history under ha/ha_van.xml). Installed by deploy.sh to +# /etc/containers/systemd/; systemd generates homeassistant.service from it. +# +# Host networking: HA binds :8123 on the host directly, so the LAN URL is +# http://10.42.0.1:8123 (name: homeassistant) — no bridge port, no DNAT. +# /run/dbus gives HA the host BlueZ stack = the Pi's onboard Bluetooth (hci0). +[Unit] +Description=Home Assistant (Podman container) +Wants=network-online.target bluetooth.service +After=network-online.target bluetooth.service + +[Container] +Image=ghcr.io/home-assistant/home-assistant:stable +ContainerName=homeassistant +Network=host +Volume=/srv/homeassistant:/config +Volume=/run/dbus:/run/dbus:ro +Environment=TZ=America/Toronto +# Bluetooth needs two escapes: Ubuntu's dbus-daemon mediates D-Bus per +# AppArmor label and the default containers profile can't send to BlueZ +# (AddMatch denied), and habluetooth wants NET_ADMIN+NET_RAW for adapter +# recovery. HA is rootful + host-net anyway. +PodmanArgs=--security-opt apparmor=unconfined +AddCapability=NET_ADMIN NET_RAW +# Podman's default 10s stop window SIGKILLs HA mid-flush and the recorder +# complains about an unclean sqlite shutdown on the next start. +StopTimeout=120 + +[Service] +Restart=always +# First start pulls the image (~600 MB) — allow for a slow uplink. +TimeoutStartSec=900 + +[Install] +WantedBy=multi-user.target diff --git a/li3/van-li3-battery b/li3/van-li3-battery index 5d82a85..1bd33f1 100755 --- a/li3/van-li3-battery +++ b/li3/van-li3-battery @@ -12,10 +12,17 @@ # attempts (Discovering stays "yes" forever, and every subsequent connect # fails with le-connection-abort-by-local) — observed happening from our own # repeated scan/connect cycling, not anything external. We restart -# bluetooth.service ourselves as soon as a scan fails to clear that, rate- -# limited (BLUETOOTH_RESTART_COOLDOWN_S) so we don't do it so often it -# disrupts the other BLE gear on this hub (motion sensors, IR remote, etc. -# also served by bluetoothd here). +# bluetooth.service ourselves once scan misses run past STUCK_DISCOVERY_THRESHOLD +# consecutive attempts, rate-limited (BLUETOOTH_RESTART_COOLDOWN_S) so we don't do +# it so often it disrupts the other BLE gear on this hub (motion sensors, IR +# remote, etc. also served by bluetoothd here). +# +# The battery itself sits at the edge of the onboard adapter's range (around +# -71 to -73 dBm from its fixed position, confirmed 2026-08-24 against a phone +# that connects to the same device instantly from up close) — an ordinary scan +# miss from weak RSSI looks identical to a wedged discovery state, so +# STUCK_DISCOVERY_THRESHOLD is set well above 1 to avoid restarting bluetoothd +# (which doesn't fix an RF problem) on every routine miss. # # Only exits (letting systemd Restart=always give us a fresh process) on # MQTT-level failure, which shouldn't happen in normal operation. @@ -30,7 +37,7 @@ import paho.mqtt.client as mqtt from bleak import BleakClient, BleakScanner BLE_RETRY_DELAY_S = 8 -STUCK_DISCOVERY_THRESHOLD = 1 # consecutive scan failures before we intervene +STUCK_DISCOVERY_THRESHOLD = 10 # consecutive scan failures before we intervene BLUETOOTH_RESTART_COOLDOWN_S = 300 # don't restart bluetooth.service more than this often config_path = sys.argv[1] if len(sys.argv) > 1 else "/etc/van-li3/config.json" @@ -338,7 +345,7 @@ async def stream_from_device(dev, mqc): if reading: latest_cs.update(reading) - async with BleakClient(dev, timeout=15) as client: + async with BleakClient(dev, timeout=30) as client: print("connected") await client.start_notify(FFE1, notify_handler) await client.write_gatt_char(FFE1, b"$traceon\r\n", response=False) diff --git a/li3/van-li3-battery-restart-limit.conf b/li3/van-li3-battery-restart-limit.conf new file mode 100644 index 0000000..ffad07a --- /dev/null +++ b/li3/van-li3-battery-restart-limit.conf @@ -0,0 +1,6 @@ +# Restart limit for van-li3-battery to prevent watchdog timeout +# Allows max 3 restarts per 5 minutes; after that, the service stays failed +# (systemd will not keep restarting it indefinitely) +[Service] +StartLimitIntervalSec=300 +StartLimitBurst=3 diff --git a/router-status/mosquitto-listener.conf b/router-status/mosquitto-listener.conf new file mode 100644 index 0000000..33c5123 --- /dev/null +++ b/router-status/mosquitto-listener.conf @@ -0,0 +1,4 @@ +# Van Router MQTT broker configuration +# Listen on all interfaces for local network access (clients on br0, ZeroTier, etc.) +listener 1883 +protocol mqtt diff --git a/router-status/van-gps-owntracks-restart-limit.conf b/router-status/van-gps-owntracks-restart-limit.conf new file mode 100644 index 0000000..d4b132e --- /dev/null +++ b/router-status/van-gps-owntracks-restart-limit.conf @@ -0,0 +1,6 @@ +# Restart limit for van-gps-owntracks to prevent watchdog timeout +# Allows max 3 restarts per 5 minutes; after that, the service stays failed +# (systemd will not keep restarting it indefinitely) +[Service] +StartLimitIntervalSec=300 +StartLimitBurst=3