Tails journalctl -kf for the "nvme nvmeN: I/O tag ... timeout, reset
controller" signature that crashed and corrupted the root fs on 2026-08-02
(and recurred 2026-08-04, that time self-healing). Watches a short grace
window to tell a clean self-heal from an escalation (repeated timeout or a
following ext4 error) before paging via the existing Pushover credentials,
with a live SMART/superblock snapshot in the alert body.
van-thermal grows sensor kinds beyond plain temperatures: "fan" alerts when
pwmfan is commanded on (pwm > 0) but reads 0 RPM (warn on first sample, crit
if it persists — 0 RPM with pwm 0 is just the firmware idling a cool SoC),
and "undervolt" goes crit on the live rpi_volt alarm plus a sticky warn off
the firmware's latched since-boot bit, so dips shorter than the 10s sample
interval still surface once. All kinds share the existing hysteresis /
journal / Pushover machinery (fan 🌀, undervolt ⚡).
Config adds nvme Composite (65/70), rp1_adc (80/85 — the RP1 die drives all
USB/eth I/O), fan, and undervolt to the existing cpu sensor. The CSV logger
now derives per-kind columns and self-rotates when the header changes; the
Cockpit Temps card renders RPM + pwm duty and undervoltage state alongside
the temperatures.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
van-thermal fires its first sample seconds after start, and the Pi often
boots hot — order after network-online.target and retry network failures
in the Pushover send path (3 attempts, 15s apart) so that page survives
DNS not being up yet. Non-200 responses still don't retry.
The Quadlet gets StopTimeout=120: podman's default 10s window SIGKILLed
HA mid-flush and the recorder complained about an unclean sqlite shutdown
on every start. Verified clean after a full stop/start cycle.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The USB hub (5GHz + 2.4GHz AP dongles, Starlink + LAN RTL8153s) moves over
from wayback; MAC-derived wlx*/enx* names travel with it, so hostapd/
networkd/cockpit configs are unchanged. Pi diffs only: failover WAN list
(wlan0 wifi 100, eth0 150, starlink USB 200, Koodo 300), cpu_thermal
sensor, bcm2835 watchdog 10s, no HA DNAT/lease, and deploy.sh drops
battery/lid/heartbeat/ZT-dns. Netplan reference in ap/50-van-wan.yaml.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
van-thermal already logged falling crossings to the journal but only paged
Pushover on rising ones. announce() now returns the crossing direction, and a
falling crossing pages an "all-clear" when leaving a level we'd have alerted on,
so the phone that got the rising alert also gets the recovery. thermal-config
gains the pushover_level + shared credentials_path keys. Ignore __pycache__.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two TODO Tier-1 items:
- Hardware watchdog: power/10-vanlink-watchdog.conf drop-in sets
RuntimeWatchdogSec=20s + RebootWatchdogSec=5min; deploy.sh applies it via
daemon-reexec. PID1 pets intel_oc_wdt; a >20s systemd hang hard-resets the box.
- Heartbeat / dead-man's switch: vendored hbc.yaml + hbc.service report to
hbd.wrede.pvt (UDP 50003) every 15s as user andreas, shipping cpu/mem/disk/
network/zfs metrics. The hbc binary is installed once via the heartbeat
project's installer (README §4); deploy.sh starts the service when present.
Auto-power-on-when-mains-returns: documented as a BIOS-only option (not
OS-controllable on this ZenBook) in README §3 + TODO, per decision.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
New "Battery / Power" card shows mains vs battery, charge % with a severity-
colored bar (only bites while on battery), the last alert level, and the
configured warn/shutdown thresholds. van-battery now includes warn_levels +
shutdown_level in /run/van-battery/state.json so the card stays self-describing
instead of hardcoding the numbers.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Pushover notification when the router loses mains (now on battery) and when
mains is restored. One alert per real transition; first sample is skipped so a
restart while already on battery doesn't false-fire. Independent of the
low-charge 25/20/15/10% alerts.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
While on battery (AC0/online=0), sends escalating Pushover alerts at 25/20/15%
and at 10% sends a final alert then `systemctl poweroff` (after shutdown_grace
so the alert flushes first). Edge-triggered per discharge episode; re-arms when
mains returns; plug-out already-low collapses to a single alert (no burst).
Pushover token+user live only in /etc/van-battery/pushover.json (0600), seeded
from pushover.json.example on first deploy and never committed. Missing/
placeholder creds disable sending but NOT the shutdown — running flat must
always power down safely. Publishes /run/van-battery/state.json. Stdlib only.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Single sysfs sample loop drives three jobs off one daemon (modeled on
van-failover):
- live state at /run/van-thermal/state.json; new Cockpit "Temperatures"
card reads it in the existing 5s loop (no per-refresh `sensors` shell-out)
- journal alerts on ok<->warn<->crit crossings, with clear_margin hysteresis
- throttled, self-rotating CSV history at /var/log/van-thermal.csv
Sensors resolved by hwmon name+label (coretemp/Package id 0, nvme/Composite),
never by hwmonN index (not stable across boots). Defaults: CPU warn 80 /
crit 95, NVMe warn 65 / crit 70. deploy.sh installs + enables the service.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
wayback runs as an always-on router living lid-closed in the van, but
stock logind HandleLidSwitch=suspend would sleep it on lid close (even
on AC). Add a logind drop-in that ignores the lid in all states and
mask every sleep target so idle/GUI/stray `systemctl suspend` can't
suspend it either.
- power/10-vanlink-nolid.conf -> /etc/systemd/logind.conf.d/
- deploy.sh: install drop-in, mask sleep/suspend/hibernate/hybrid-sleep,
restart systemd-logind
- README: power/ layout row, never-sleep operation notes + reversal recipe
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Andreas Wrede
