robustness: hardware watchdog + hbc heartbeat client

Two TODO Tier-1 items:

- Hardware watchdog: power/10-vanlink-watchdog.conf drop-in sets
  RuntimeWatchdogSec=20s + RebootWatchdogSec=5min; deploy.sh applies it via
  daemon-reexec. PID1 pets intel_oc_wdt; a >20s systemd hang hard-resets the box.

- Heartbeat / dead-man's switch: vendored hbc.yaml + hbc.service report to
  hbd.wrede.pvt (UDP 50003) every 15s as user andreas, shipping cpu/mem/disk/
  network/zfs metrics. The hbc binary is installed once via the heartbeat
  project's installer (README §4); deploy.sh starts the service when present.

Auto-power-on-when-mains-returns: documented as a BIOS-only option (not
OS-controllable on this ZenBook) in README §3 + TODO, per decision.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Andreas Wrede
2026-06-28 22:20:09 -04:00
co-authored by Claude Opus 4.8
parent 97309970ec
commit 8fcc959551
6 changed files with 95 additions and 12 deletions
+13 -11
View File
@@ -20,17 +20,19 @@ value-for-effort. Generated 2026-06-29 from a robustness review of the live box.
## Tier 1 — high value, pieces already on hand
- [ ] **systemd hardware watchdog**`/dev/watchdog` exists but isn't wired. A hung
kernel/driver currently needs someone to open the lid. Given the EC-latch and
USB-hub-hang history, auto-reboot-on-hang is the biggest unattended win.
Set `RuntimeWatchdogSec` (and `RebootWatchdogSec`) in `/etc/systemd/system.conf`.
- [ ] **Auto power-on when mains returns** — the 10% auto-shutdown is currently a
one-way trip; a laptop won't boot itself when solar/shore power returns. Needs the
BIOS "restore on AC loss → power on" setting (cannot be set from the OS). Without
it the safe-shutdown can strand the router. **Manual BIOS toggle.**
- [ ] **Heartbeat / dead-man's switch** — every existing alert can only fire if the box
is alive. A periodic ping to healthchecks.io (or similar) inverts it: silence =
something is dead (panic, no-boot, total power loss).
- [x] **systemd hardware watchdog**DONE. `power/10-vanlink-watchdog.conf` drop-in sets
`RuntimeWatchdogSec=20s` + `RebootWatchdogSec=5min`; PID1 pets `intel_oc_wdt`, chip
hard-resets the box if systemd hangs >20s. Deployed via `daemon-reexec`.
- [x] **Auto power-on when mains returns** — RESOLVED as "document only". Not
OS-controllable on this ZenBook: `/proc/acpi/wakeup` is wake-from-suspend (S3/S4) only,
and "restore on AC loss" is a BIOS/firmware feature this laptop likely doesn't expose.
The 10% auto-shutdown is therefore a one-way trip until manual power-on. **Action:**
check BIOS setup for an "AC power-on / restore on AC loss" option and enable it if
present (documented in README §3, Battery monitor).
- [x] **Heartbeat / dead-man's switch** — DONE. `hbc` (heartbeat client, from
git.wrede.ca/andreas/heartbeat) installed to `~/venvs/hbd` + `~/bin/hbc`; `hbc.service`
reports to **hbd.wrede.pvt** (UDP 50003) every 15s as user `andreas`, shipping cpu/mem/
disk/network/zfs metrics. Config `/etc/hbc.yaml`, vendored in `heartbeat/`.
- [ ] **Connectivity alerts → Pushover**`van-failover` already knows when it fails
over or when ALL WANs are down; it just logs it. Pipe those events to the Pushover
plumbing so internet loss is alerted, not just battery.