ap: switch 5GHz driver to out-of-tree morrownr/rtw89 for USB3 support

The in-kernel rtw89_8852bu on this Ubuntu kernel base predates mainline's
USB2->3 auto-switch for this chip, permanently capping the AP dongle at
USB2/480M. morrownr/rtw89 (dkms) has that switch. ap/rtw89.conf now
blacklists the in-kernel rtw89 modules and tunes the replacement
(disable_ps_mode + switch_usb_mode); ap/install-rtw89-driver.sh builds
and installs it, pinned to a specific upstream commit, kept separate
from deploy.sh since a dkms rebuild is too slow to run on every deploy.

Verified live: negotiates USB3/5000M on a dedicated USB3 controller
(480M on a USB2-only one, as expected), AP recovered via hostapd's
Restart=always + van-ap-watchdog with no manual intervention. Hit and
documented one real gotcha along the way: in-kernel rtw89_core refused
to unload while its own dependents (rtw89_8852b, rtw89_8852b_common)
were still loaded, which blocked the new module with a duplicate-symbol
error until all in-kernel modules were removed first.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YJfEELeh3ercpRBp8yYrYS

ap/install-rtw89-driver.sh

@@ -0,0 +1,67 @@
+#!/usr/bin/env bash
+# ap/install-rtw89-driver.sh — install/update the out-of-tree morrownr/rtw89 driver
+# for the 5GHz AP dongle (RTL8852BU, USB).
+#
+# Why: this Ubuntu kernel base predates mainline rtw89's USB2->3 auto-switch for this
+# chip, so the in-kernel rtw89_8852bu permanently caps the dongle at USB2/480M. The
+# morrownr/rtw89 driver has that switch. ap/rtw89.conf blacklists the in-kernel rtw89
+# modules so udev loads this one instead — that file is deploy.sh's job to install;
+# this script is the (heavier, slower, not idempotent-per-deploy) driver build/install
+# itself, kept separate so a routine `deploy.sh` run doesn't trigger a dkms rebuild.
+#
+# Run manually after a fresh Pi provisioning, or to bump PIN_COMMIT to a newer upstream
+# revision: sudo ./ap/install-rtw89-driver.sh
+#
+# Safe to re-run — no-ops if the pinned commit is already built+installed for the
+# running kernel (e.g. after a kernel upgrade where DKMS's own auto-rebuild already
+# handled it).
+set -euo pipefail
+
+[ "$(id -u)" -eq 0 ] || { echo "Run as root (sudo)." >&2; exit 1; }
+
+REPO_URL=https://github.com/morrownr/rtw89
+# Pinned for reproducibility — bump deliberately, not by tracking a moving branch.
+PIN_COMMIT=8b2b78deb357d01fd5808164046e31239139ed9f
+PKG_NAME=rtw89
+PKG_VERSION=7.3   # from upstream's dkms.conf; independent of PIN_COMMIT
+SRC="/usr/src/${PKG_NAME}-${PKG_VERSION}"
+KVER=$(uname -r)
+
+if [ -d "$SRC/.git" ]; then
+    git -C "$SRC" fetch origin
+else
+    rm -rf "$SRC"
+    git clone "$REPO_URL" "$SRC"
+fi
+git -C "$SRC" checkout "$PIN_COMMIT"
+
+if dkms status "${PKG_NAME}/${PKG_VERSION}" 2>/dev/null | grep -q "${KVER}.*: installed" \
+   && [ "$(git -C "$SRC" rev-parse HEAD)" = "$PIN_COMMIT" ]; then
+    echo "rtw89 driver already installed at pinned commit ${PIN_COMMIT:0:12} for ${KVER}; nothing to do."
+    exit 0
+fi
+
+dkms remove "${PKG_NAME}/${PKG_VERSION}" --all 2>/dev/null || true
+dkms add "$SRC"
+dkms install "${PKG_NAME}/${PKG_VERSION}"
+make -C "$SRC" install_fw
+
+cat <<EOF
+
+rtw89 driver installed (commit ${PIN_COMMIT:0:12}) for kernel ${KVER}.
+
+If in-kernel rtw89 modules are currently loaded, they won't unload themselves —
+loading this driver alongside them fails with "exports duplicate symbol" (hit this
+2026-08-23: rtw89_core stayed resident because rtw89_8852b/rtw89_8852b_common, its own
+in-kernel dependents, were still loaded too). To switch a live system over:
+
+  sudo rmmod rtw89_8852bu rtw89_8852b rtw89_8852b_common rtw89_core rtw89_usb 2>/dev/null
+  # then force the AP dongle to re-enumerate, e.g.:
+  echo 0 | sudo tee /sys/bus/usb/devices/<bus-port>/authorized
+  echo 1 | sudo tee /sys/bus/usb/devices/<bus-port>/authorized
+
+(or just reboot). hostapd's Restart=always + van-ap-watchdog recover the AP
+automatically once the new driver claims the interface — no manual hostapd restart
+needed. ap/rtw89.conf must already be deployed (sudo ./deploy.sh) so the blacklist is
+in place before the device re-enumerates, or the in-kernel driver will just reclaim it.
+EOF

ap/rtw89.conf

@@ -1 +1,48 @@
-options rtw89_core disable_ps_mode=Y
+# 5GHz AP dongle (Realtek RTL8852BU). Since 2026-08-23 this uses the out-of-tree
+# morrownr/rtw89 driver (dkms package "rtw89", modules suffixed _git) instead of the
+# in-kernel rtw89_8852bu — the in-kernel driver on this Ubuntu kernel base predates
+# mainline's USB2->3 auto-switch, so the dongle was permanently capped at USB2/480M.
+# Install: https://github.com/morrownr/rtw89 (`sudo dkms install`, `sudo make
+# install_fw`). This file both tunes the driver and blacklists the in-kernel modules
+# so udev picks the out-of-tree ones on (re)enumeration.
+options rtw89_core_git disable_ps_mode=Y
+options rtw89_usb_git switch_usb_mode=y
+
+# Blacklist the in-kernel rtw89 drivers.
+blacklist rtw89_8851bu
+blacklist rtw89_8851be
+blacklist rtw89_8851b
+blacklist rtw89_8852au
+blacklist rtw89_8852ae
+blacklist rtw89_8852a
+blacklist rtw89_8852b_common
+blacklist rtw89_8852bu
+blacklist rtw89_8852be
+blacklist rtw89_8852b
+blacklist rtw89_8852bte
+blacklist rtw89_8852bt
+blacklist rtw89_8852cu
+blacklist rtw89_8852ce
+blacklist rtw89_8852c
+blacklist rtw89_8922au
+blacklist rtw89_8922ae
+blacklist rtw89_8922a
+blacklist rtw89_8922de
+blacklist rtw89_8922d
+blacklist rtw89_core
+blacklist rtw89_usb
+blacklist rtw89_pci
+
+# Blacklist Larry Finger's out-of-tree rtw89 driver too, in case it's ever installed.
+blacklist rtw89core
+blacklist rtw89pci
+blacklist rtw_8851b
+blacklist rtw_8851be
+blacklist rtw_8852a
+blacklist rtw_8852ae
+blacklist rtw_8852b
+blacklist rtw_8852be
+blacklist rtw_8852c
+blacklist rtw_8852ce
+blacklist rtw_8922a
+blacklist rtw_8922ae