robustness: hardware watchdog + hbc heartbeat client

Two TODO Tier-1 items:

- Hardware watchdog: power/10-vanlink-watchdog.conf drop-in sets
  RuntimeWatchdogSec=20s + RebootWatchdogSec=5min; deploy.sh applies it via
  daemon-reexec. PID1 pets intel_oc_wdt; a >20s systemd hang hard-resets the box.

- Heartbeat / dead-man's switch: vendored hbc.yaml + hbc.service report to
  hbd.wrede.pvt (UDP 50003) every 15s as user andreas, shipping cpu/mem/disk/
  network/zfs metrics. The hbc binary is installed once via the heartbeat
  project's installer (README §4); deploy.sh starts the service when present.

Auto-power-on-when-mains-returns: documented as a BIOS-only option (not
OS-controllable on this ZenBook) in README §3 + TODO, per decision.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

README.md

@@ -41,6 +41,8 @@ This directory is the source of truth. The live system files live under `/etc`,
 | WAN health + failover | **van-failover** daemon |
 | Temperature monitor / alert / log | **van-thermal** daemon |
 | Battery monitor / low-charge alert + shutdown | **van-battery** daemon |
+| Auto-reboot on hang | **systemd hardware watchdog** (`intel_oc_wdt`) |
+| Liveness / dead-man's switch + metrics | **hbc** heartbeat client → hbd.wrede.pvt |
 | ZeroTier DNS → resolved | **zerotier-systemd-manager** + systemd-networkd |
 | Web UI | **Cockpit** + `vanrouter` plugin |
 
@@ -103,10 +105,20 @@ This directory is the source of truth. The live system files live under `/etc`,
 | `battery-config.json` | `/etc/van-battery/config.json` | warn levels, shutdown level, poll interval, paths |
 | `van-battery.service` | `/etc/systemd/system/van-battery.service` | `Restart=always` |
 | `pushover.json.example` | → `/etc/van-battery/pushover.json` (seeded if absent) | Pushover token+user **template**; real file is 0600, **not** in the repo |
+| `10-vanlink-watchdog.conf` | `/etc/systemd/system.conf.d/10-vanlink-watchdog.conf` | hardware watchdog (`RuntimeWatchdogSec=20s`); applied via `daemon-reexec` |
 
 `deploy.sh` additionally **masks** `sleep.target suspend.target hibernate.target hybrid-sleep.target`
 (no repo file — symlinks to `/dev/null` under `/etc/systemd/system/`) so nothing else can suspend either.
 
+### `heartbeat/` — dead-man's switch
+| file | → installs to | purpose |
+|---|---|---|
+| `hbc.yaml` | `/etc/hbc.yaml` | heartbeat client config (interval + metric plugins) |
+| `hbc.service` | `/etc/systemd/system/hbc.service` | runs `~/bin/hbc` as `andreas` → hbd.wrede.pvt:50003 |
+
+The `hbc` binary itself (venv at `~/venvs/hbd`, symlink `~/bin/hbc`) is installed once via the
+heartbeat project's own installer — see §4. `deploy.sh` only starts the service once it exists.
+
 ---
 
 ## 3. Components & how to operate
@@ -147,6 +159,7 @@ This directory is the source of truth. The live system files live under `/etc`,
 - **Credentials** live in `/etc/van-battery/pushover.json` (mode 0600), seeded from `pushover.json.example` on first deploy and **never committed**. Missing/placeholder creds disable *sending* but **not** the shutdown — running flat must always power down safely (the skip is logged to the journal).
 - Tune in `/etc/van-battery/config.json` (`warn_levels`, `shutdown_level`, `poll_interval`), then `systemctl restart van-battery`.
 - Status: `systemctl status van-battery`, `journalctl -u van-battery -f`, or `cat /run/van-battery/state.json`.
+- **Caveat — the 10 % shutdown is one-way.** A laptop won't power itself back on when mains returns: "restore on AC loss" is a BIOS/firmware feature (not OS-controllable — `/proc/acpi/wakeup` only covers wake-from-suspend). If your BIOS exposes an "AC power-on / restore on AC loss" option, enable it so the router reboots itself once shore/solar power is back; this ZenBook likely doesn't have it, in which case a low-battery shutdown needs a manual power-on.
 
 ### Adding the 4G/5G modem
 1. Plug the USB modem in. ModemManager + the existing `Koodo` gsm NM connection (autoconnect) bring it up.
@@ -184,7 +197,7 @@ cd ~/vanlink
 sudo ./deploy.sh          # copies all files to their system locations, reloads + enables services
 ```
 
-Three things `deploy.sh` does **not** do (one-time, manual):
+Four things `deploy.sh` does **not** do (one-time, manual):
 
 1. **zerotier-systemd-manager binary** (v0.4.0, hand-installed — not in the ZeroTier apt repo):
    ```bash
@@ -194,6 +207,12 @@ Three things `deploy.sh` does **not** do (one-time, manual):
    ```
 2. **hostapd unmask** (Ubuntu ships it masked): `sudo systemctl unmask hostapd`.
 3. **Pushover credentials** for `van-battery` — deploy seeds `/etc/van-battery/pushover.json` (0600) with placeholders; fill in your app token + user key, then `sudo systemctl restart van-battery`. Until then low-battery alerts are skipped (logged), but the 10 % auto-shutdown still works.
+4. **Heartbeat client `hbc`** (dead-man's switch) — install the binary once into andreas' venv:
+   ```bash
+   git clone https://git.wrede.ca/andreas/heartbeat.git ~/git/heartbeat   # if not already cloned
+   sh ~/git/heartbeat/scripts/hb_install.sh client                        # -> ~/venvs/hbd + ~/bin/hbc
+   ```
+   `deploy.sh` installs `/etc/hbc.yaml` + the `hbc.service` unit and starts it once `~/bin/hbc` exists. The service reports to **hbd.wrede.pvt** (UDP 50003) as user `andreas`. Upgrade later with `~/bin/hb_install.sh client`.
 
 ---