diff --git a/docs/superpowers/specs/2026-09-23-nm-wan-provisioning-design.md b/docs/superpowers/specs/2026-09-23-nm-wan-provisioning-design.md new file mode 100644 index 0000000..6cc250a --- /dev/null +++ b/docs/superpowers/specs/2026-09-23-nm-wan-provisioning-design.md @@ -0,0 +1,127 @@ +# NetworkManager-owned WANs, provisioned by deploy.sh + +Date: 2026-09-23. Status: design approved in chat; spec for review. Builds on +`2026-09-23-host-config-design.md` (host files, `ENABLE_*` flags, `--check/--diff/--post-check`). + +## Problem + +`van-failover` steers multi-WAN through NetworkManager: it reads active connections and +gateways with `nmcli`, sets route metrics per NM profile, and the cellular WAN is a NM `gsm` +profile (`Koodo`). The Pi 5 runs that model. The R5C has no NetworkManager (networkd only; +Armbian's `10-dhcp-all-interfaces.yaml` does DHCP on every `e*` port), so on the R5C +`van-failover` sees no connections at all (`present: false`, `connection: null`); it "works" +only because networkd's single DHCP default route happens to be on the Starlink port. A GSM +modem (Quectel EC25-AF, MBIM via ModemManager) will be attached later, and the home LAN on +`enP2p33s0` is a stand-in for the field's Starlink connection, so the R5C needs the same +NM-owned WAN model as the Pi, and it must be provisioned by `deploy.sh` rather than by hand. + +## Decisions (approved) + +- NetworkManager owns the WAN side on every host that sets `ENABLE_NM_WAN=1`; `systemd-networkd` + keeps br0, the AP radios and the wired LAN port(s), as today. One code path on both hosts. +- A routine `deploy.sh` never flaps uplinks. Changing the WAN netplan configuration is an explicit + step, `./deploy.sh --apply-network`, with an automatic rollback. +- `deploy.sh` installs the packages it needs. + +## Design + +### 1. Host variables (add to `REQUIRED_VARS` / `OPTIONAL_VARS`; `hosts/vanlink.conf.example` and both host files) + +| Variable | Kind | Meaning | +|---|---|---| +| `ENABLE_NM_WAN` | required flag 0/1 | NetworkManager owns the WAN ports and the modem. Both `hosts/van.conf` and `hosts/wan.conf` set 1. | +| `CELLULAR_CONN_NAME` | optional | NM connection name of the cellular WAN; `failover/config.json` uses `@CELLULAR_CONN_NAME@` instead of the hardcoded `Koodo`. Default in both host files: `Koodo`. | +| `CELLULAR_APN` | optional | APN for the cellular profile (`sp.koodo.com`). Empty: no profile is created. | +| `WIFI_WAN_IFACE` | now optional | Empty means the host has no Wi-Fi uplink (the R5C: both radios are APs); the `wifis:` stanza of the netplan template is then omitted. | + +`ENABLE_NM_WAN=0` means none of the below applies (current R5C behaviour). + +### 2. Package installation + +- Missing REQUIRED packages are installed by `deploy.sh` itself (`DEBIAN_FRONTEND=noninteractive + apt-get install -y`) instead of aborting with "apt install X". If apt cannot install one, abort + with the apt error (today's outcome). `--check/--diff/--post-check` never install anything. +- `network-manager` is the exception in ordering: it is installed **after** the files that keep + NM off the AP radios and LAN ports (`van-ap-unmanaged.conf`) are in place, because the package + starts NM on install and NM manages Wi-Fi and gsm devices by default. After installing it, + `deploy.sh` checks `nmcli -t -f DEVICE,STATE device` and warns if `WIFI_5G_IFACE`, + `WIFI_2G_IFACE`, `LAN_IFACE` or `ONBOARD_LAN_IFACE` is not `unmanaged`. +- With `ENABLE_NM_WAN=1`: `network-manager` is required; NetworkManager and ModemManager are enabled. + +### 3. Routine deploy behaviour (`ENABLE_NM_WAN=1`) + +- The cellular NM profile is created if absent, matching the Pi's working `Koodo`: type `gsm`, + no interface binding, `apn=$CELLULAR_APN`, no credentials, autoconnect yes, `ipv4.method auto`, + `ipv4.route-metric 300`, `ipv4.never-default yes`, `ipv6.method auto`. An existing profile is + never modified. Creation needs no modem to be present. +- `/etc/netplan/50-van-wan.yaml`, the Armbian netplan file and the netplan state are **not** + touched. If `/etc/netplan/50-van-wan.yaml` is missing or differs from the rendered template, + deploy warns and names `./deploy.sh --apply-network` (this replaces today's "run netplan by + hand" text). The Pi 5's live uplinks are therefore never flapped by a routine deploy. + +### 4. `./deploy.sh --apply-network` (root, explicit) + +1. Refuse unless `ENABLE_NM_WAN=1` and `network-manager` is installed. +2. Back up `/etc/netplan` to `/var/backups/vanlink-netplan-/`. +3. Install the rendered `50-van-wan.yaml`. If `/etc/netplan/10-dhcp-all-interfaces.yaml` exists + (Armbian), move it aside as `10-dhcp-all-interfaces.yaml.vanlink-disabled` so networkd stops + DHCP-ing the WAN port behind NM's back. +4. Apply from a detached transient unit (`systemd-run`), so the sequence survives the SSH session + dropping: `netplan generate && netplan apply`, then wait up to 60 s for: NM reports an active + connection on `STARLINK_IFACE` (or the WAN is otherwise up), a default route exists, and + `ping -c1 1.1.1.1` works. +5. On success: log, keep the backup. On failure: restore the backup (including the Armbian file), + re-apply, and record a warning with the reason. The exit status is non-zero on failure. +6. The detached unit writes its outcome to `/var/log/vanlink-apply-network.log`. + +### 5. Template changes + +- `ap/50-van-wan.yaml` is split: the ethernet (Starlink) part stays; the `wifis:` stanza (with + the Wapana access point) moves to a fragment `ap/50-van-wan-wifi.yaml` that `--apply-network` + appends only when `WIFI_WAN_IFACE` is non-empty. The rendered result for the Pi 5 must be + byte-identical to today's `/etc/netplan/50-van-wan.yaml` there (checked with `--diff` before + the Pi is ever touched). The Wapana PSK stays where it is (known, out of scope). +- `failover/config.json`: cellular `connection` becomes `@CELLULAR_CONN_NAME@`. + +### 6. Post-check additions (only when `ENABLE_NM_WAN=1`) + +NetworkManager active; ModemManager active; the cellular profile exists (if `CELLULAR_APN` set); +AP/LAN devices are `unmanaged` in NM; `van-failover`'s `/run/van-failover/state.json` shows each +WAN whose device exists as `present: true`. + +## Testing + +- `tests/check-templates.sh`: new variables in the example (existing sync check); host files pass + `--check`; the awk flag-guard check gains `ENABLE_NM_WAN` for `network-manager`, `nmcli`, + `50-van-wan` and the netplan-moving code (only inside guards); rendered `failover/config.json` + has `Koodo` for both hosts; the rendered netplan for `wan.conf` contains the wifis stanza and for + `van.conf` does not; a `--wan-plan` sub-command prints (no changes, no root) the actions + `--apply-network` would take, asserted for both hosts; the rollback decision logic is a small + function tested through a seam like `--classify-unit`. +- Not testable without root/live hosts: the apply and rollback themselves. Those are verified on the + R5C (Rollout) with a console or alternate access path available. + +## Rollout + +1. Build, review, push. Sync to the R5C (excluding `hosts/van.conf`). Set `ENABLE_NM_WAN=1`, + `CELLULAR_CONN_NAME`, `CELLULAR_APN` in its `hosts/van.conf` (currently an uncommitted local edit). +2. R5C: routine `sudo ./deploy.sh` (installs `network-manager`, creates the cellular profile, warns + about the pending WAN netplan), then `sudo ./deploy.sh --apply-network` once. Uplink today is the + home LAN on `enP2p33s0`; the port changes from networkd to NM, a short blip. +3. Pi 5: routine deploy only; `--diff` must show no change to `50-van-wan.yaml`, and no apply is run. +4. When the modem is attached: `mmcli -L`, the profile connects, `cat /run/van-failover/state.json` + shows cellular present, route metrics 200 (Starlink) / 300 (cellular). + +## Out of scope + +Moving the Wapana PSK out of git; choosing the modem's USB port; a Wi-Fi WAN on the R5C (needs a +spare radio); replacing NetworkManager on either host; cellular credentials/PIN (the Koodo profile +has none). + +## Risks + +- The cutover changes the manager of the uplink port on a live host: mitigated by the detached apply, + the connectivity gate and automatic rollback; a console or ZeroTier path should exist anyway. +- NM installation must not touch AP/LAN ports: mitigated by the install ordering and the unmanaged check. +- Auto-installing packages changes deploy's contract: only missing required packages are installed and + a failure aborts with the apt error.