vanlink/docs/superpowers/specs/2026-09-23-nm-wan-provisioning-design.md

NetworkManager-owned WANs, provisioned by deploy.sh

Date: 2026-09-23. Status: design approved in chat; spec for review. Builds on 2026-09-23-host-config-design.md (host files, ENABLE_* flags, --check/--diff/--post-check).

Problem

van-failover steers multi-WAN through NetworkManager: it reads active connections and gateways with nmcli, sets route metrics per NM profile, and the cellular WAN is a NM gsm profile (Koodo). The Pi 5 runs that model. The R5C has no NetworkManager (networkd only; Armbian's 10-dhcp-all-interfaces.yaml does DHCP on every e* port), so on the R5C van-failover sees no connections at all (present: false, connection: null); it "works" only because networkd's single DHCP default route happens to be on the Starlink port. A GSM modem (Quectel EC25-AF, MBIM via ModemManager) will be attached later, and the home LAN on enP2p33s0 is a stand-in for the field's Starlink connection, so the R5C needs the same NM-owned WAN model as the Pi, and it must be provisioned by deploy.sh rather than by hand.

Decisions (approved)

• NetworkManager owns the WAN side on every host that sets ENABLE_NM_WAN=1; systemd-networkd keeps br0, the AP radios and the wired LAN port(s), as today. One code path on both hosts.
• A routine deploy.sh never flaps uplinks. Changing the WAN netplan configuration is an explicit step, ./deploy.sh --apply-network, with an automatic rollback.
• deploy.sh installs the packages it needs.

Design

1. Host variables (add to REQUIRED_VARS / OPTIONAL_VARS; hosts/vanlink.conf.example and both host files)

Variable	Kind	Meaning
ENABLE_NM_WAN	required flag 0/1	NetworkManager owns the WAN ports and the modem. Both hosts/van.conf and hosts/wan.conf set 1.
CELLULAR_CONN_NAME	optional	NM connection name of the cellular WAN; failover/config.json uses @CELLULAR_CONN_NAME@ instead of the hardcoded Koodo. Default in both host files: Koodo.
CELLULAR_APN	optional	APN for the cellular profile (sp.koodo.com). Empty: no profile is created.
WIFI_WAN_IFACE	now optional	Empty means the host has no Wi-Fi uplink (the R5C: both radios are APs); the wifis: stanza of the netplan template is then omitted.

ENABLE_NM_WAN=0 means none of the below applies (current R5C behaviour).

2. Package installation

• Missing REQUIRED packages are installed by deploy.sh itself (DEBIAN_FRONTEND=noninteractive apt-get install -y) instead of aborting with "apt install X". If apt cannot install one, abort with the apt error (today's outcome). --check/--diff/--post-check never install anything.
• network-manager is the exception in ordering: it is installed after the files that keep NM off the AP radios and LAN ports (van-ap-unmanaged.conf) are in place, because the package starts NM on install and NM manages Wi-Fi and gsm devices by default. After installing it, deploy.sh checks nmcli -t -f DEVICE,STATE device and warns if WIFI_5G_IFACE, WIFI_2G_IFACE, LAN_IFACE or ONBOARD_LAN_IFACE is not unmanaged.
• With ENABLE_NM_WAN=1: network-manager is required; NetworkManager and ModemManager are enabled.

3. Routine deploy behaviour (ENABLE_NM_WAN=1)

• The cellular NM profile is created if absent, matching the Pi's working Koodo: type gsm, no interface binding, apn=$CELLULAR_APN, no credentials, autoconnect yes, ipv4.method auto, ipv4.route-metric 300, ipv4.never-default yes, ipv6.method auto. An existing profile is never modified. Creation needs no modem to be present.
• /etc/netplan/50-van-wan.yaml, the Armbian netplan file and the netplan state are not touched. If /etc/netplan/50-van-wan.yaml is missing or differs from the rendered template, deploy warns and names ./deploy.sh --apply-network (this replaces today's "run netplan by hand" text). The Pi 5's live uplinks are therefore never flapped by a routine deploy.

4. ./deploy.sh --apply-network (root, explicit)

1. Refuse unless ENABLE_NM_WAN=1 and network-manager is installed.
2. Back up /etc/netplan to /var/backups/vanlink-netplan-<timestamp>/.
3. Install the rendered 50-van-wan.yaml. If /etc/netplan/10-dhcp-all-interfaces.yaml exists (Armbian), move it aside as 10-dhcp-all-interfaces.yaml.vanlink-disabled so networkd stops DHCP-ing the WAN port behind NM's back.
4. Apply from a detached transient unit (systemd-run), so the sequence survives the SSH session dropping: netplan generate && netplan apply, then wait up to 60 s for: NM reports an active connection on STARLINK_IFACE (or the WAN is otherwise up), a default route exists, and ping -c1 1.1.1.1 works.
5. On success: log, keep the backup. On failure: restore the backup (including the Armbian file), re-apply, and record a warning with the reason. The exit status is non-zero on failure.
6. The detached unit writes its outcome to /var/log/vanlink-apply-network.log.

5. Template changes

• ap/50-van-wan.yaml is split: the ethernet (Starlink) part stays; the wifis: stanza (with the Wapana access point) moves to a fragment ap/50-van-wan-wifi.yaml that --apply-network appends only when WIFI_WAN_IFACE is non-empty. The rendered result for the Pi 5 must be byte-identical to today's /etc/netplan/50-van-wan.yaml there (checked with --diff before the Pi is ever touched). The Wapana PSK stays where it is (known, out of scope).
• failover/config.json: cellular connection becomes @CELLULAR_CONN_NAME@.

6. Post-check additions (only when ENABLE_NM_WAN=1)

NetworkManager active; ModemManager active; the cellular profile exists (if CELLULAR_APN set); AP/LAN devices are unmanaged in NM; van-failover's /run/van-failover/state.json shows each WAN whose device exists as present: true.

Testing

• tests/check-templates.sh: new variables in the example (existing sync check); host files pass --check; the awk flag-guard check gains ENABLE_NM_WAN for network-manager, nmcli, 50-van-wan and the netplan-moving code (only inside guards); rendered failover/config.json has Koodo for both hosts; the rendered netplan for wan.conf contains the wifis stanza and for van.conf does not; a --wan-plan sub-command prints (no changes, no root) the actions --apply-network would take, asserted for both hosts; the rollback decision logic is a small function tested through a seam like --classify-unit.
• Not testable without root/live hosts: the apply and rollback themselves. Those are verified on the R5C (Rollout) with a console or alternate access path available.

Rollout

1. Build, review, push. Sync to the R5C (excluding hosts/van.conf). Set ENABLE_NM_WAN=1, CELLULAR_CONN_NAME, CELLULAR_APN in its hosts/van.conf (currently an uncommitted local edit).
2. R5C: routine sudo ./deploy.sh (installs network-manager, creates the cellular profile, warns about the pending WAN netplan), then sudo ./deploy.sh --apply-network once. Uplink today is the home LAN on enP2p33s0; the port changes from networkd to NM, a short blip.
3. Pi 5: routine deploy only; --diff must show no change to 50-van-wan.yaml, and no apply is run.
4. When the modem is attached: mmcli -L, the profile connects, cat /run/van-failover/state.json shows cellular present, route metrics 200 (Starlink) / 300 (cellular).

Out of scope

Moving the Wapana PSK out of git; choosing the modem's USB port; a Wi-Fi WAN on the R5C (needs a spare radio); replacing NetworkManager on either host; cellular credentials/PIN (the Koodo profile has none).

Risks

• The cutover changes the manager of the uplink port on a live host: mitigated by the detached apply, the connectivity gate and automatic rollback; a console or ZeroTier path should exist anyway.
• NM installation must not touch AP/LAN ports: mitigated by the install ordering and the unmanaged check.
• Auto-installing packages changes deploy's contract: only missing required packages are installed and a failure aborts with the apt error.

Amendments made during implementation (review-driven; the code and tests are authoritative)

• §3/§4 cellular profile is written as a NetworkManager keyfile from ap/cellular.nmconnection (uuid derived in pure bash from the connection name), only if no connection with that id exists (keyfile scan by id= plus nmcli); ap/nm-no-auto-default.conf (no-auto-default=*) is installed before the package so NM never auto-creates DHCP "Wired connection" profiles on ports not yet handed to it.
• §4 --apply-network changes nothing under /etc/netplan itself. It validates (NM installed and running, van-failover unit present, no apply already running, last status not running/rollback-failed unless --force), no-ops when the netplan already matches and no Armbian file exists (unless --force), takes a per-run backup plus a sticky pristine backup /var/backups/vanlink-netplan-orig, renders to /run, and launches a detached transient unit. The helper installs the file, moves the Armbian file aside, applies, and gates: stage 1 (≤60 s): apply succeeded, NM device state 100 (connected) with a netplan-* connection, IPv4 address, gateway (IP4.GATEWAY, else the DHCP lease's routers option, because a never-default profile leaves IP4.GATEWAY empty) reachable on-link; stage 2: restart van-failover, then default route + ping 1.1.1.1 (≥120 s). Failure → checked rollback (restore, re-apply, connectivity re-check, ROLLBACK FAILED if not restored; a last-resort reboot only when the rollback itself failed). Timeouts, a TERM/INT/HUP trap that restores exactly once, budgets derived in one block, log /var/log/vanlink-apply-network.log, status file /var/lib/vanlink/apply-network.status (running|ok|rolled-back|rollback-failed <ts> …). Not built: a boot-time recovery unit (power loss mid-apply boots the new config ungated and leaves status running), an arping fallback for gateways that do not answer ICMP.
• §6 post-check: the failover-state rule is refined: present in van-failover's state means "has an active NM connection", so an unplugged or out-of-range uplink is normal in a van. Post-check warns only when the WAN's device has link (carrier 1) but present is false; it also reports the last --apply-network status (non-ok warns) and checks NM/ModemManager, NM-unmanaged AP/LAN ports, the cellular profile and the netplan state.
• Routine deploys also install missing REQUIRED packages via apt (spec §2) and never touch /etc/netplan.
• Uplink stays with networkd until the handover. While the WAN netplan is missing, the routine deploy installs /etc/NetworkManager/conf.d/zz-van-wan-pending.conf (unmanaged-devices+=interface-name:<STARLINK_IFACE>, from ap/nm-uplink-pending.conf) so NM, freshly installed, does not adopt the uplink as "connected (externally)" and van-failover does not start steering it. It is removed when the netplan matches, and deliberately left alone when it differs or is unreadable (a host whose uplink NM already manages, such as the Pi 5 after any template edit, must not lose it). The helper removes it just before netplan apply and restores it on rollback. This relies on NM honouring unmanaged-devices+= after nmcli general reload conf (not live-tested; if ignored the behaviour degrades to "NM adopts the uplink early", not an outage).
• --apply-network preflight (die unless --force): the uplink port exists on this host, and the resolved host file's name matches hostname -s; stage 2 additionally requires van-failover's own proto static default route on the port. A netplan file the current user cannot read is reported unreadable (not differs) by the read-only commands.
• The rendered ap/50-van-wan.yaml must stay byte-identical to what live hosts have installed (measured on the Pi 5): comments in ap/50-van-wan*.yaml are part of the rendered bytes, so editing them makes every host read WAN netplan differs. Check 14 pins the Pi render against commit 3d117e7.