robustness: hardware watchdog + hbc heartbeat client
Two TODO Tier-1 items: - Hardware watchdog: power/10-vanlink-watchdog.conf drop-in sets RuntimeWatchdogSec=20s + RebootWatchdogSec=5min; deploy.sh applies it via daemon-reexec. PID1 pets intel_oc_wdt; a >20s systemd hang hard-resets the box. - Heartbeat / dead-man's switch: vendored hbc.yaml + hbc.service report to hbd.wrede.pvt (UDP 50003) every 15s as user andreas, shipping cpu/mem/disk/ network/zfs metrics. The hbc binary is installed once via the heartbeat project's installer (README §4); deploy.sh starts the service when present. Auto-power-on-when-mains-returns: documented as a BIOS-only option (not OS-controllable on this ZenBook) in README §3 + TODO, per decision. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
97309970ec
commit
8fcc959551
@@ -20,17 +20,19 @@ value-for-effort. Generated 2026-06-29 from a robustness review of the live box.
|
||||
|
||||
## Tier 1 — high value, pieces already on hand
|
||||
|
||||
- [ ] **systemd hardware watchdog** — `/dev/watchdog` exists but isn't wired. A hung
|
||||
kernel/driver currently needs someone to open the lid. Given the EC-latch and
|
||||
USB-hub-hang history, auto-reboot-on-hang is the biggest unattended win.
|
||||
Set `RuntimeWatchdogSec` (and `RebootWatchdogSec`) in `/etc/systemd/system.conf`.
|
||||
- [ ] **Auto power-on when mains returns** — the 10% auto-shutdown is currently a
|
||||
one-way trip; a laptop won't boot itself when solar/shore power returns. Needs the
|
||||
BIOS "restore on AC loss → power on" setting (cannot be set from the OS). Without
|
||||
it the safe-shutdown can strand the router. **Manual BIOS toggle.**
|
||||
- [ ] **Heartbeat / dead-man's switch** — every existing alert can only fire if the box
|
||||
is alive. A periodic ping to healthchecks.io (or similar) inverts it: silence =
|
||||
something is dead (panic, no-boot, total power loss).
|
||||
- [x] **systemd hardware watchdog** — DONE. `power/10-vanlink-watchdog.conf` drop-in sets
|
||||
`RuntimeWatchdogSec=20s` + `RebootWatchdogSec=5min`; PID1 pets `intel_oc_wdt`, chip
|
||||
hard-resets the box if systemd hangs >20s. Deployed via `daemon-reexec`.
|
||||
- [x] **Auto power-on when mains returns** — RESOLVED as "document only". Not
|
||||
OS-controllable on this ZenBook: `/proc/acpi/wakeup` is wake-from-suspend (S3/S4) only,
|
||||
and "restore on AC loss" is a BIOS/firmware feature this laptop likely doesn't expose.
|
||||
The 10% auto-shutdown is therefore a one-way trip until manual power-on. **Action:**
|
||||
check BIOS setup for an "AC power-on / restore on AC loss" option and enable it if
|
||||
present (documented in README §3, Battery monitor).
|
||||
- [x] **Heartbeat / dead-man's switch** — DONE. `hbc` (heartbeat client, from
|
||||
git.wrede.ca/andreas/heartbeat) installed to `~/venvs/hbd` + `~/bin/hbc`; `hbc.service`
|
||||
reports to **hbd.wrede.pvt** (UDP 50003) every 15s as user `andreas`, shipping cpu/mem/
|
||||
disk/network/zfs metrics. Config `/etc/hbc.yaml`, vendored in `heartbeat/`.
|
||||
- [ ] **Connectivity alerts → Pushover** — `van-failover` already knows when it fails
|
||||
over or when ALL WANs are down; it just logs it. Pipe those events to the Pushover
|
||||
plumbing so internet loss is alerted, not just battery.
|
||||
|
||||
Reference in New Issue
Block a user