power: add van-nvme-watch — pages on NVMe I/O-timeout/reset recurrence
Tails journalctl -kf for the "nvme nvmeN: I/O tag ... timeout, reset controller" signature that crashed and corrupted the root fs on 2026-08-02 (and recurred 2026-08-04, that time self-healing). Watches a short grace window to tell a clean self-heal from an escalation (repeated timeout or a following ext4 error) before paging via the existing Pushover credentials, with a live SMART/superblock snapshot in the alert body.
This commit is contained in:
@@ -48,6 +48,7 @@ This directory is the source of truth. The live system files live under `/etc`,
|
||||
| NAT + forwarding | **nftables** + sysctl |
|
||||
| WAN health + failover | **van-failover** daemon |
|
||||
| Temperature monitor / alert / log | **van-thermal** daemon |
|
||||
| NVMe I/O-timeout/reset watchdog + alert | **van-nvme-watch** daemon |
|
||||
| Battery monitor / low-charge alert + shutdown | **van-battery** daemon |
|
||||
| Auto-reboot on hang | **systemd hardware watchdog** (`intel_oc_wdt`) |
|
||||
| Liveness / dead-man's switch + metrics | **hbc** heartbeat client → hbd.wrede.pvt |
|
||||
@@ -121,6 +122,9 @@ This directory is the source of truth. The live system files live under `/etc`,
|
||||
| `van-thermal` | `/usr/local/sbin/van-thermal` | temperature daemon (Python): publishes state, alerts, logs history |
|
||||
| `thermal-config.json` | `/etc/van-thermal/config.json` | sensors + warn/crit thresholds + sample/log intervals |
|
||||
| `van-thermal.service` | `/etc/systemd/system/van-thermal.service` | `Restart=always` |
|
||||
| `van-nvme-watch` | `/usr/local/sbin/van-nvme-watch` | NVMe watchdog (Python): tails `journalctl -k` for I/O-timeout/reset events, Pushover alerts |
|
||||
| `nvme-watch-config.json` | `/etc/van-nvme-watch/config.json` | grace period, cooldown, device paths |
|
||||
| `van-nvme-watch.service` | `/etc/systemd/system/van-nvme-watch.service` | `Restart=always` |
|
||||
| `van-battery` | `/usr/local/sbin/van-battery` | battery daemon (Python): Pushover low-charge alerts + safe shutdown |
|
||||
| `battery-config.json` | `/etc/van-battery/config.json` | warn levels, shutdown level, poll interval, paths |
|
||||
| `van-battery.service` | `/etc/systemd/system/van-battery.service` | `Restart=always` |
|
||||
@@ -180,6 +184,15 @@ The `homeassistant` LAN name comes from `ap/van-ap-dnsmasq.conf` (`host-record`
|
||||
- **Tuning**: edit `/etc/van-thermal/config.json` (thresholds, intervals, sensor list), then `systemctl restart van-thermal`. Defaults: CPU warn 80 / crit 95 °C (silicon crit is 100), NVMe warn 65 / crit 70 °C (drive crit ~71).
|
||||
- Status: `systemctl status van-thermal` or `cat /run/van-thermal/state.json`.
|
||||
|
||||
### NVMe watchdog (`van-nvme-watch`)
|
||||
- Event-driven, not polled: tails `journalctl -kf` for the `nvme nvmeN: I/O tag ... timeout, reset controller` signature that crashed and corrupted the root fs on 2026-08-02 (recurred 2026-08-04, self-healed — see the memory notes for that investigation). There's no sensor to sample, only a log line to catch.
|
||||
- On a match it watches a `grace_period` (default 20 s) for either a clean re-init (self-healed) or a second timeout / an ext4 error following it (escalated) before alerting, so the Pushover message already says which outcome happened — no need to SSH in during a scare.
|
||||
- The alert body includes a live `smartctl`/`tune2fs` snapshot (SMART health, critical-warning flag, media error count, filesystem state).
|
||||
- **Live state**: `/run/van-nvme-watch/state.json` (last event, same convention as van-failover/van-thermal).
|
||||
- Shares Pushover credentials with van-thermal/van-battery (`/etc/van-battery/pushover.json`).
|
||||
- **Tuning**: edit `/etc/van-nvme-watch/config.json` (`grace_period`, `cooldown`, device paths), then `systemctl restart van-nvme-watch`.
|
||||
- Status: `systemctl status van-nvme-watch`, `journalctl -u van-nvme-watch -f`, or `cat /run/van-nvme-watch/state.json`.
|
||||
|
||||
### Battery monitor (`van-battery`)
|
||||
- Watches mains vs battery via `/sys/class/power_supply/AC0/online` (0 = on battery) and charge via `BAT0/capacity`. Both resolve by `type` (Mains/Battery) if those names ever differ.
|
||||
- **Only while on battery**, it sends escalating **Pushover** alerts at **25 / 20 / 15 %**, and at **10 %** sends a final alert and runs `systemctl poweroff` (after `shutdown_grace`, default 8 s, so the alert flushes first).
|
||||
|
||||
Reference in New Issue
Block a user