power: add van-nvme-watch — pages on NVMe I/O-timeout/reset recurrence

Tails journalctl -kf for the "nvme nvmeN: I/O tag ... timeout, reset
controller" signature that crashed and corrupted the root fs on 2026-08-02
(and recurred 2026-08-04, that time self-healing). Watches a short grace
window to tell a clean self-heal from an escalation (repeated timeout or a
following ext4 error) before paging via the existing Pushover credentials,
with a live SMART/superblock snapshot in the alert body.
This commit is contained in:
Andreas Wrede
2026-08-04 08:44:53 -04:00
parent 86fa9509df
commit e7a82126f2
5 changed files with 302 additions and 1 deletions
+13
View File
@@ -48,6 +48,7 @@ This directory is the source of truth. The live system files live under `/etc`,
| NAT + forwarding | **nftables** + sysctl |
| WAN health + failover | **van-failover** daemon |
| Temperature monitor / alert / log | **van-thermal** daemon |
| NVMe I/O-timeout/reset watchdog + alert | **van-nvme-watch** daemon |
| Battery monitor / low-charge alert + shutdown | **van-battery** daemon |
| Auto-reboot on hang | **systemd hardware watchdog** (`intel_oc_wdt`) |
| Liveness / dead-man's switch + metrics | **hbc** heartbeat client → hbd.wrede.pvt |
@@ -121,6 +122,9 @@ This directory is the source of truth. The live system files live under `/etc`,
| `van-thermal` | `/usr/local/sbin/van-thermal` | temperature daemon (Python): publishes state, alerts, logs history |
| `thermal-config.json` | `/etc/van-thermal/config.json` | sensors + warn/crit thresholds + sample/log intervals |
| `van-thermal.service` | `/etc/systemd/system/van-thermal.service` | `Restart=always` |
| `van-nvme-watch` | `/usr/local/sbin/van-nvme-watch` | NVMe watchdog (Python): tails `journalctl -k` for I/O-timeout/reset events, Pushover alerts |
| `nvme-watch-config.json` | `/etc/van-nvme-watch/config.json` | grace period, cooldown, device paths |
| `van-nvme-watch.service` | `/etc/systemd/system/van-nvme-watch.service` | `Restart=always` |
| `van-battery` | `/usr/local/sbin/van-battery` | battery daemon (Python): Pushover low-charge alerts + safe shutdown |
| `battery-config.json` | `/etc/van-battery/config.json` | warn levels, shutdown level, poll interval, paths |
| `van-battery.service` | `/etc/systemd/system/van-battery.service` | `Restart=always` |
@@ -180,6 +184,15 @@ The `homeassistant` LAN name comes from `ap/van-ap-dnsmasq.conf` (`host-record`
- **Tuning**: edit `/etc/van-thermal/config.json` (thresholds, intervals, sensor list), then `systemctl restart van-thermal`. Defaults: CPU warn 80 / crit 95 °C (silicon crit is 100), NVMe warn 65 / crit 70 °C (drive crit ~71).
- Status: `systemctl status van-thermal` or `cat /run/van-thermal/state.json`.
### NVMe watchdog (`van-nvme-watch`)
- Event-driven, not polled: tails `journalctl -kf` for the `nvme nvmeN: I/O tag ... timeout, reset controller` signature that crashed and corrupted the root fs on 2026-08-02 (recurred 2026-08-04, self-healed — see the memory notes for that investigation). There's no sensor to sample, only a log line to catch.
- On a match it watches a `grace_period` (default 20 s) for either a clean re-init (self-healed) or a second timeout / an ext4 error following it (escalated) before alerting, so the Pushover message already says which outcome happened — no need to SSH in during a scare.
- The alert body includes a live `smartctl`/`tune2fs` snapshot (SMART health, critical-warning flag, media error count, filesystem state).
- **Live state**: `/run/van-nvme-watch/state.json` (last event, same convention as van-failover/van-thermal).
- Shares Pushover credentials with van-thermal/van-battery (`/etc/van-battery/pushover.json`).
- **Tuning**: edit `/etc/van-nvme-watch/config.json` (`grace_period`, `cooldown`, device paths), then `systemctl restart van-nvme-watch`.
- Status: `systemctl status van-nvme-watch`, `journalctl -u van-nvme-watch -f`, or `cat /run/van-nvme-watch/state.json`.
### Battery monitor (`van-battery`)
- Watches mains vs battery via `/sys/class/power_supply/AC0/online` (0 = on battery) and charge via `BAT0/capacity`. Both resolve by `type` (Mains/Battery) if those names ever differ.
- **Only while on battery**, it sends escalating **Pushover** alerts at **25 / 20 / 15 %**, and at **10 %** sends a final alert and runs `systemctl poweroff` (after `shutdown_grace`, default 8 s, so the alert flushes first).