Files
Andreas WredeandClaude Opus 4.8 8fcc959551 robustness: hardware watchdog + hbc heartbeat client
Two TODO Tier-1 items:

- Hardware watchdog: power/10-vanlink-watchdog.conf drop-in sets
  RuntimeWatchdogSec=20s + RebootWatchdogSec=5min; deploy.sh applies it via
  daemon-reexec. PID1 pets intel_oc_wdt; a >20s systemd hang hard-resets the box.

- Heartbeat / dead-man's switch: vendored hbc.yaml + hbc.service report to
  hbd.wrede.pvt (UDP 50003) every 15s as user andreas, shipping cpu/mem/disk/
  network/zfs metrics. The hbc binary is installed once via the heartbeat
  project's installer (README §4); deploy.sh starts the service when present.

Auto-power-on-when-mains-returns: documented as a BIOS-only option (not
OS-controllable on this ZenBook) in README §3 + TODO, per decision.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-28 22:20:09 -04:00

3.7 KiB

TODO — robustness backlog for the van router (wayback)

Candidate improvements toward a robust unattended mobile router, tiered by value-for-effort. Generated 2026-06-29 from a robustness review of the live box.

Already in place (for reference — don't redo)

  • Multi-WAN failover (wifi → Starlink → 4G) — van-failover
  • Thermal monitoring + alerts + history — van-thermal
  • Battery low-charge Pushover alerts + safe auto-shutdown — van-battery
  • Mains ↔ battery transition alerts
  • Cockpit dashboard (AP, temps, battery, failover, WAN)
  • Never-sleep / lid-closed operation
  • unattended-upgrades enabled
  • smartd active (SMART being read) — but see Tier 2 (alerts go nowhere)
  • Time sync healthy (NTP synced)
  • journald within sane ZFS limits

Tier 1 — high value, pieces already on hand

  • systemd hardware watchdog — DONE. power/10-vanlink-watchdog.conf drop-in sets RuntimeWatchdogSec=20s + RebootWatchdogSec=5min; PID1 pets intel_oc_wdt, chip hard-resets the box if systemd hangs >20s. Deployed via daemon-reexec.
  • Auto power-on when mains returns — RESOLVED as "document only". Not OS-controllable on this ZenBook: /proc/acpi/wakeup is wake-from-suspend (S3/S4) only, and "restore on AC loss" is a BIOS/firmware feature this laptop likely doesn't expose. The 10% auto-shutdown is therefore a one-way trip until manual power-on. Action: check BIOS setup for an "AC power-on / restore on AC loss" option and enable it if present (documented in README §3, Battery monitor).
  • Heartbeat / dead-man's switch — DONE. hbc (heartbeat client, from git.wrede.ca/andreas/heartbeat) installed to ~/venvs/hbd + ~/bin/hbc; hbc.service reports to hbd.wrede.pvt (UDP 50003) every 15s as user andreas, shipping cpu/mem/ disk/network/zfs metrics. Config /etc/hbc.yaml, vendored in heartbeat/.
  • Connectivity alerts → Pushovervan-failover already knows when it fails over or when ALL WANs are down; it just logs it. Pipe those events to the Pushover plumbing so internet loss is alerted, not just battery.

Tier 2 — worth doing soon

  • smartd → Pushover — smartd alerts default to emailing root (a black hole on a headless box). Wire -M exec to a Pushover script so NVMe SMART/wear warnings (reallocated sectors, wear-leveling) actually reach a phone.
  • ZFS hygiene (root is zroot) — schedule a periodic scrub (with zfs-zed alerting), and auto-snapshot before deploy.sh for one-command rollback. Cap ARC if RAM ever gets tight.
  • Security hardening:
    • Cockpit binds *:9090 (all interfaces incl. WANs unless nftables drops inbound — confirm). Restrict to LAN/ZeroTier.
    • Add fail2ban for Cockpit/SSH.
    • WPA3-transition on the AP.
    • Guest SSID isolated from trusted devices.
  • Cellular data-cap trackingvnstat or nftables counters on the gsm iface + a Pushover alert near the monthly cap, to avoid bill shock when parked on 4G.

Tier 3 — situational

  • Captive-portal handling for public-WiFi WANs (campsites / neighbour APs) — detection at minimum, auto-auth for frequently-used ones.
  • GPS — location logging / geofencing / offline time source.
  • Secrets backup — git covers configs, but /etc/van-battery/pushover.json and other secrets are intentionally out of git. Document a restore path so a disk swap doesn't lose them.

Suggested next: Tier 1. The watchdog and connectivity-alerts are quick and reuse existing plumbing; the AC-restore BIOS setting is what makes the battery auto-shutdown genuinely safe rather than a trap.