vanlink/docs/superpowers/specs/2026-09-23-nm-wan-provisioning-design.md

# NetworkManager-owned WANs, provisioned by deploy.sh

Date: 2026-09-23. Status: design approved in chat; spec for review. Builds on
`2026-09-23-host-config-design.md` (host files, `ENABLE_*` flags, `--check/--diff/--post-check`).

## Problem

`van-failover` steers multi-WAN through NetworkManager: it reads active connections and
gateways with `nmcli`, sets route metrics per NM profile, and the cellular WAN is a NM `gsm`
profile (`Koodo`). The Pi 5 runs that model. The R5C has no NetworkManager (networkd only;
Armbian's `10-dhcp-all-interfaces.yaml` does DHCP on every `e*` port), so on the R5C
`van-failover` sees no connections at all (`present: false`, `connection: null`); it "works"
only because networkd's single DHCP default route happens to be on the Starlink port. A GSM
modem (Quectel EC25-AF, MBIM via ModemManager) will be attached later, and the home LAN on
`enP2p33s0` is a stand-in for the field's Starlink connection, so the R5C needs the same
NM-owned WAN model as the Pi, and it must be provisioned by `deploy.sh` rather than by hand.

## Decisions (approved)

- NetworkManager owns the WAN side on every host that sets `ENABLE_NM_WAN=1`; `systemd-networkd`
  keeps br0, the AP radios and the wired LAN port(s), as today. One code path on both hosts.
- A routine `deploy.sh` never flaps uplinks. Changing the WAN netplan configuration is an explicit
  step, `./deploy.sh --apply-network`, with an automatic rollback.
- `deploy.sh` installs the packages it needs.

## Design

### 1. Host variables (add to `REQUIRED_VARS` / `OPTIONAL_VARS`; `hosts/vanlink.conf.example` and both host files)

| Variable | Kind | Meaning |
|---|---|---|
| `ENABLE_NM_WAN` | required flag 0/1 | NetworkManager owns the WAN ports and the modem. Both `hosts/van.conf` and `hosts/wan.conf` set 1. |
| `CELLULAR_CONN_NAME` | optional | NM connection name of the cellular WAN; `failover/config.json` uses `@CELLULAR_CONN_NAME@` instead of the hardcoded `Koodo`. Default in both host files: `Koodo`. |
| `CELLULAR_APN` | optional | APN for the cellular profile (`sp.koodo.com`). Empty: no profile is created. |
| `WIFI_WAN_IFACE` | now optional | Empty means the host has no Wi-Fi uplink (the R5C: both radios are APs); the `wifis:` stanza of the netplan template is then omitted. |

`ENABLE_NM_WAN=0` means none of the below applies (current R5C behaviour).

### 2. Package installation

- Missing REQUIRED packages are installed by `deploy.sh` itself (`DEBIAN_FRONTEND=noninteractive
  apt-get install -y`) instead of aborting with "apt install X". If apt cannot install one, abort
  with the apt error (today's outcome). `--check/--diff/--post-check` never install anything.
- `network-manager` is the exception in ordering: it is installed **after** the files that keep
  NM off the AP radios and LAN ports (`van-ap-unmanaged.conf`) are in place, because the package
  starts NM on install and NM manages Wi-Fi and gsm devices by default. After installing it,
  `deploy.sh` checks `nmcli -t -f DEVICE,STATE device` and warns if `WIFI_5G_IFACE`,
  `WIFI_2G_IFACE`, `LAN_IFACE` or `ONBOARD_LAN_IFACE` is not `unmanaged`.
- With `ENABLE_NM_WAN=1`: `network-manager` is required; NetworkManager and ModemManager are enabled.

### 3. Routine deploy behaviour (`ENABLE_NM_WAN=1`)

- The cellular NM profile is created if absent, matching the Pi's working `Koodo`: type `gsm`,
  no interface binding, `apn=$CELLULAR_APN`, no credentials, autoconnect yes, `ipv4.method auto`,
  `ipv4.route-metric 300`, `ipv4.never-default yes`, `ipv6.method auto`. An existing profile is
  never modified. Creation needs no modem to be present.
- `/etc/netplan/50-van-wan.yaml`, the Armbian netplan file and the netplan state are **not**
  touched. If `/etc/netplan/50-van-wan.yaml` is missing or differs from the rendered template,
  deploy warns and names `./deploy.sh --apply-network` (this replaces today's "run netplan by
  hand" text). The Pi 5's live uplinks are therefore never flapped by a routine deploy.

### 4. `./deploy.sh --apply-network` (root, explicit)

1. Refuse unless `ENABLE_NM_WAN=1` and `network-manager` is installed.
2. Back up `/etc/netplan` to `/var/backups/vanlink-netplan-<timestamp>/`.
3. Install the rendered `50-van-wan.yaml`. If `/etc/netplan/10-dhcp-all-interfaces.yaml` exists
   (Armbian), move it aside as `10-dhcp-all-interfaces.yaml.vanlink-disabled` so networkd stops
   DHCP-ing the WAN port behind NM's back.
4. Apply from a detached transient unit (`systemd-run`), so the sequence survives the SSH session
   dropping: `netplan generate && netplan apply`, then wait up to 60 s for: NM reports an active
   connection on `STARLINK_IFACE` (or the WAN is otherwise up), a default route exists, and
   `ping -c1 1.1.1.1` works.
5. On success: log, keep the backup. On failure: restore the backup (including the Armbian file),
   re-apply, and record a warning with the reason. The exit status is non-zero on failure.
6. The detached unit writes its outcome to `/var/log/vanlink-apply-network.log`.

### 5. Template changes

- `ap/50-van-wan.yaml` is split: the ethernet (Starlink) part stays; the `wifis:` stanza (with
  the Wapana access point) moves to a fragment `ap/50-van-wan-wifi.yaml` that `--apply-network`
  appends only when `WIFI_WAN_IFACE` is non-empty. The rendered result for the Pi 5 must be
  byte-identical to today's `/etc/netplan/50-van-wan.yaml` there (checked with `--diff` before
  the Pi is ever touched). The Wapana PSK stays where it is (known, out of scope).
- `failover/config.json`: cellular `connection` becomes `@CELLULAR_CONN_NAME@`.

### 6. Post-check additions (only when `ENABLE_NM_WAN=1`)

NetworkManager active; ModemManager active; the cellular profile exists (if `CELLULAR_APN` set);
AP/LAN devices are `unmanaged` in NM; `van-failover`'s `/run/van-failover/state.json` shows each
WAN whose device exists as `present: true`.

## Testing

- `tests/check-templates.sh`: new variables in the example (existing sync check); host files pass
  `--check`; the awk flag-guard check gains `ENABLE_NM_WAN` for `network-manager`, `nmcli`,
  `50-van-wan` and the netplan-moving code (only inside guards); rendered `failover/config.json`
  has `Koodo` for both hosts; the rendered netplan for `wan.conf` contains the wifis stanza and for
  `van.conf` does not; a `--wan-plan` sub-command prints (no changes, no root) the actions
  `--apply-network` would take, asserted for both hosts; the rollback decision logic is a small
  function tested through a seam like `--classify-unit`.
- Not testable without root/live hosts: the apply and rollback themselves. Those are verified on the
  R5C (Rollout) with a console or alternate access path available.

## Rollout

1. Build, review, push. Sync to the R5C (excluding `hosts/van.conf`). Set `ENABLE_NM_WAN=1`,
   `CELLULAR_CONN_NAME`, `CELLULAR_APN` in its `hosts/van.conf` (currently an uncommitted local edit).
2. R5C: routine `sudo ./deploy.sh` (installs `network-manager`, creates the cellular profile, warns
   about the pending WAN netplan), then `sudo ./deploy.sh --apply-network` once. Uplink today is the
   home LAN on `enP2p33s0`; the port changes from networkd to NM, a short blip.
3. Pi 5: routine deploy only; `--diff` must show no change to `50-van-wan.yaml`, and no apply is run.
4. When the modem is attached: `mmcli -L`, the profile connects, `cat /run/van-failover/state.json`
   shows cellular present, route metrics 200 (Starlink) / 300 (cellular).

## Out of scope

Moving the Wapana PSK out of git; choosing the modem's USB port; a Wi-Fi WAN on the R5C (needs a
spare radio); replacing NetworkManager on either host; cellular credentials/PIN (the Koodo profile
has none).

## Risks

- The cutover changes the manager of the uplink port on a live host: mitigated by the detached apply,
  the connectivity gate and automatic rollback; a console or ZeroTier path should exist anyway.
- NM installation must not touch AP/LAN ports: mitigated by the install ordering and the unmanaged check.
- Auto-installing packages changes deploy's contract: only missing required packages are installed and
  a failure aborts with the apt error.

## Amendments made during implementation (review-driven; the code and tests are authoritative)

- **§3/§4 cellular profile** is written as a NetworkManager keyfile from `ap/cellular.nmconnection` (uuid derived in pure bash from the connection name), only if no connection with that id exists (keyfile scan by `id=` plus `nmcli`); `ap/nm-no-auto-default.conf` (`no-auto-default=*`) is installed before the package so NM never auto-creates DHCP "Wired connection" profiles on ports not yet handed to it.
- **§4 `--apply-network`** changes nothing under `/etc/netplan` itself. It validates (NM installed and running, `van-failover` unit present, no apply already running, last status not `running`/`rollback-failed` unless `--force`), no-ops when the netplan already matches and no Armbian file exists (unless `--force`), takes a per-run backup plus a sticky pristine backup `/var/backups/vanlink-netplan-orig`, renders to `/run`, and launches a detached transient unit. The helper installs the file, moves the Armbian file aside, applies, and gates: **stage 1** (≤60 s): apply succeeded, NM device state `100 (connected)` with a `netplan-*` connection, IPv4 address, gateway (`IP4.GATEWAY`, else the DHCP lease's `routers` option, because a never-default profile leaves `IP4.GATEWAY` empty) reachable on-link; **stage 2**: restart `van-failover`, then default route + `ping 1.1.1.1` (≥120 s). Failure → checked rollback (restore, re-apply, connectivity re-check, `ROLLBACK FAILED` if not restored; a last-resort reboot only when the rollback itself failed). Timeouts, a TERM/INT/HUP trap that restores exactly once, budgets derived in one block, log `/var/log/vanlink-apply-network.log`, status file `/var/lib/vanlink/apply-network.status` (`running|ok|rolled-back|rollback-failed <ts> …`). Not built: a boot-time recovery unit (power loss mid-apply boots the new config ungated and leaves status `running`), an arping fallback for gateways that do not answer ICMP.
- **§6 post-check**: the failover-state rule is refined: `present` in `van-failover`'s state means "has an active NM connection", so an unplugged or out-of-range uplink is normal in a van. Post-check warns only when the WAN's device has link (`carrier` 1) but `present` is false; it also reports the last `--apply-network` status (non-`ok` warns) and checks NM/ModemManager, NM-unmanaged AP/LAN ports, the cellular profile and the netplan state.
- Routine deploys also install missing REQUIRED packages via apt (spec §2) and never touch `/etc/netplan`.
- **Uplink stays with networkd until the handover.** While the WAN netplan is *missing*, the routine deploy installs `/etc/NetworkManager/conf.d/zz-van-wan-pending.conf` (`unmanaged-devices+=interface-name:<STARLINK_IFACE>`, from `ap/nm-uplink-pending.conf`) so NM, freshly installed, does not adopt the uplink as "connected (externally)" and van-failover does not start steering it. It is removed when the netplan `matches`, and deliberately left alone when it `differs` or is `unreadable` (a host whose uplink NM already manages, such as the Pi 5 after any template edit, must not lose it). The helper removes it just before `netplan apply` and restores it on rollback. This relies on NM honouring `unmanaged-devices+=` after `nmcli general reload conf` (not live-tested; if ignored the behaviour degrades to "NM adopts the uplink early", not an outage).
- **`--apply-network` preflight** (die unless `--force`): the uplink port exists on this host, and the resolved host file's name matches `hostname -s`; stage 2 additionally requires van-failover's own `proto static` default route on the port. A netplan file the current user cannot read is reported `unreadable` (not `differs`) by the read-only commands.
- **The rendered `ap/50-van-wan.yaml` must stay byte-identical to what live hosts have installed** (measured on the Pi 5): comments in `ap/50-van-wan*.yaml` are part of the rendered bytes, so editing them makes every host read `WAN netplan differs`. Check 14 pins the Pi render against commit 3d117e7.