robustness: hardware watchdog + hbc heartbeat client

Two TODO Tier-1 items:

- Hardware watchdog: power/10-vanlink-watchdog.conf drop-in sets
  RuntimeWatchdogSec=20s + RebootWatchdogSec=5min; deploy.sh applies it via
  daemon-reexec. PID1 pets intel_oc_wdt; a >20s systemd hang hard-resets the box.

- Heartbeat / dead-man's switch: vendored hbc.yaml + hbc.service report to
  hbd.wrede.pvt (UDP 50003) every 15s as user andreas, shipping cpu/mem/disk/
  network/zfs metrics. The hbc binary is installed once via the heartbeat
  project's installer (README §4); deploy.sh starts the service when present.

Auto-power-on-when-mains-returns: documented as a BIOS-only option (not
OS-controllable on this ZenBook) in README §3 + TODO, per decision.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Andreas Wrede
2026-06-28 22:20:09 -04:00
co-authored by Claude Opus 4.8
parent 97309970ec
commit 8fcc959551
6 changed files with 95 additions and 12 deletions
+21
View File
@@ -54,9 +54,24 @@ install -D -m0644 power/10-vanlink-nolid.conf /etc/systemd/logind.conf.d/10-vanl
# Belt-and-suspenders: a router must never suspend from idle, GUI, or a stray `systemctl suspend`.
systemctl mask sleep.target suspend.target hibernate.target hybrid-sleep.target >/dev/null 2>&1 || true
echo "== hardware watchdog =="
install -D -m0644 power/10-vanlink-watchdog.conf /etc/systemd/system.conf.d/10-vanlink-watchdog.conf
echo "== heartbeat client (dead-man's switch) =="
install -D -m0644 heartbeat/hbc.yaml /etc/hbc.yaml
install -D -m0644 heartbeat/hbc.service /etc/systemd/system/hbc.service
# The hbc binary itself (~/bin/hbc + venv) is installed once via the heartbeat
# project's installer — see README §4. Only start the service if it's present.
if [ ! -x /home/andreas/bin/hbc ]; then
echo " -> /home/andreas/bin/hbc not found; run 'sh ~/git/heartbeat/scripts/hb_install.sh client' (README §4)"
fi
echo "== apply =="
sysctl --system >/dev/null
systemctl daemon-reload
# Re-exec PID1 so the system.conf.d watchdog drop-in takes effect (daemon-reload alone
# does NOT re-arm RuntimeWatchdogSec). Safe online: re-exec keeps all services running.
systemctl daemon-reexec
# networkd here owns only the AP + ZT overlay (neither a real uplink), so its wait-online
# can never satisfy "online" and just burns its 120s timeout, stalling network-online.target
# and ZeroTier by ~2min every boot. Real uplink readiness is covered by NetworkManager-wait-online.
@@ -66,6 +81,11 @@ systemctl restart systemd-logind >/dev/null 2>&1 || true
systemctl unmask hostapd >/dev/null 2>&1 || true
systemctl enable regdomain.service hostapd van-ap-dnsmasq nftables systemd-networkd van-failover van-thermal van-battery >/dev/null 2>&1 || true
systemctl restart van-thermal van-battery
# Heartbeat: only enable/start once the client binary is installed (README §4).
if [ -x /home/andreas/bin/hbc ]; then
systemctl enable hbc >/dev/null 2>&1 || true
systemctl restart hbc
fi
# restart in dependency order; AP iface IP first, then hostapd/dnsmasq, then NAT/failover
systemctl restart systemd-networkd
systemctl restart hostapd van-ap-dnsmasq nftables van-failover
@@ -77,4 +97,5 @@ echo " iw dev wlxc83a35a4ee55 info | grep -E 'ssid|channel|width'"
echo " cat /run/van-failover/state.json"
echo " cat /run/van-thermal/state.json # CPU + NVMe temps"
echo " cat /run/van-battery/state.json # mains/battery + charge %"
echo " systemctl status hbc # heartbeat client -> hbd.wrede.pvt"
echo "Manual one-time steps (see README §4): zerotier-systemd-manager binary + 'zerotier-cli set <nwid> allowDNS=1'."