diff --git a/README.md b/README.md index 79c91aa..16a820f 100644 --- a/README.md +++ b/README.md @@ -41,6 +41,8 @@ This directory is the source of truth. The live system files live under `/etc`, | WAN health + failover | **van-failover** daemon | | Temperature monitor / alert / log | **van-thermal** daemon | | Battery monitor / low-charge alert + shutdown | **van-battery** daemon | +| Auto-reboot on hang | **systemd hardware watchdog** (`intel_oc_wdt`) | +| Liveness / dead-man's switch + metrics | **hbc** heartbeat client → hbd.wrede.pvt | | ZeroTier DNS → resolved | **zerotier-systemd-manager** + systemd-networkd | | Web UI | **Cockpit** + `vanrouter` plugin | @@ -103,10 +105,20 @@ This directory is the source of truth. The live system files live under `/etc`, | `battery-config.json` | `/etc/van-battery/config.json` | warn levels, shutdown level, poll interval, paths | | `van-battery.service` | `/etc/systemd/system/van-battery.service` | `Restart=always` | | `pushover.json.example` | → `/etc/van-battery/pushover.json` (seeded if absent) | Pushover token+user **template**; real file is 0600, **not** in the repo | +| `10-vanlink-watchdog.conf` | `/etc/systemd/system.conf.d/10-vanlink-watchdog.conf` | hardware watchdog (`RuntimeWatchdogSec=20s`); applied via `daemon-reexec` | `deploy.sh` additionally **masks** `sleep.target suspend.target hibernate.target hybrid-sleep.target` (no repo file — symlinks to `/dev/null` under `/etc/systemd/system/`) so nothing else can suspend either. +### `heartbeat/` — dead-man's switch +| file | → installs to | purpose | +|---|---|---| +| `hbc.yaml` | `/etc/hbc.yaml` | heartbeat client config (interval + metric plugins) | +| `hbc.service` | `/etc/systemd/system/hbc.service` | runs `~/bin/hbc` as `andreas` → hbd.wrede.pvt:50003 | + +The `hbc` binary itself (venv at `~/venvs/hbd`, symlink `~/bin/hbc`) is installed once via the +heartbeat project's own installer — see §4. `deploy.sh` only starts the service once it exists. + --- ## 3. Components & how to operate @@ -147,6 +159,7 @@ This directory is the source of truth. The live system files live under `/etc`, - **Credentials** live in `/etc/van-battery/pushover.json` (mode 0600), seeded from `pushover.json.example` on first deploy and **never committed**. Missing/placeholder creds disable *sending* but **not** the shutdown — running flat must always power down safely (the skip is logged to the journal). - Tune in `/etc/van-battery/config.json` (`warn_levels`, `shutdown_level`, `poll_interval`), then `systemctl restart van-battery`. - Status: `systemctl status van-battery`, `journalctl -u van-battery -f`, or `cat /run/van-battery/state.json`. +- **Caveat — the 10 % shutdown is one-way.** A laptop won't power itself back on when mains returns: "restore on AC loss" is a BIOS/firmware feature (not OS-controllable — `/proc/acpi/wakeup` only covers wake-from-suspend). If your BIOS exposes an "AC power-on / restore on AC loss" option, enable it so the router reboots itself once shore/solar power is back; this ZenBook likely doesn't have it, in which case a low-battery shutdown needs a manual power-on. ### Adding the 4G/5G modem 1. Plug the USB modem in. ModemManager + the existing `Koodo` gsm NM connection (autoconnect) bring it up. @@ -184,7 +197,7 @@ cd ~/vanlink sudo ./deploy.sh # copies all files to their system locations, reloads + enables services ``` -Three things `deploy.sh` does **not** do (one-time, manual): +Four things `deploy.sh` does **not** do (one-time, manual): 1. **zerotier-systemd-manager binary** (v0.4.0, hand-installed — not in the ZeroTier apt repo): ```bash @@ -194,6 +207,12 @@ Three things `deploy.sh` does **not** do (one-time, manual): ``` 2. **hostapd unmask** (Ubuntu ships it masked): `sudo systemctl unmask hostapd`. 3. **Pushover credentials** for `van-battery` — deploy seeds `/etc/van-battery/pushover.json` (0600) with placeholders; fill in your app token + user key, then `sudo systemctl restart van-battery`. Until then low-battery alerts are skipped (logged), but the 10 % auto-shutdown still works. +4. **Heartbeat client `hbc`** (dead-man's switch) — install the binary once into andreas' venv: + ```bash + git clone https://git.wrede.ca/andreas/heartbeat.git ~/git/heartbeat # if not already cloned + sh ~/git/heartbeat/scripts/hb_install.sh client # -> ~/venvs/hbd + ~/bin/hbc + ``` + `deploy.sh` installs `/etc/hbc.yaml` + the `hbc.service` unit and starts it once `~/bin/hbc` exists. The service reports to **hbd.wrede.pvt** (UDP 50003) as user `andreas`. Upgrade later with `~/bin/hb_install.sh client`. --- diff --git a/TODO.md b/TODO.md index a695101..844bad3 100644 --- a/TODO.md +++ b/TODO.md @@ -20,17 +20,19 @@ value-for-effort. Generated 2026-06-29 from a robustness review of the live box. ## Tier 1 — high value, pieces already on hand -- [ ] **systemd hardware watchdog** — `/dev/watchdog` exists but isn't wired. A hung - kernel/driver currently needs someone to open the lid. Given the EC-latch and - USB-hub-hang history, auto-reboot-on-hang is the biggest unattended win. - Set `RuntimeWatchdogSec` (and `RebootWatchdogSec`) in `/etc/systemd/system.conf`. -- [ ] **Auto power-on when mains returns** — the 10% auto-shutdown is currently a - one-way trip; a laptop won't boot itself when solar/shore power returns. Needs the - BIOS "restore on AC loss → power on" setting (cannot be set from the OS). Without - it the safe-shutdown can strand the router. **Manual BIOS toggle.** -- [ ] **Heartbeat / dead-man's switch** — every existing alert can only fire if the box - is alive. A periodic ping to healthchecks.io (or similar) inverts it: silence = - something is dead (panic, no-boot, total power loss). +- [x] **systemd hardware watchdog** — DONE. `power/10-vanlink-watchdog.conf` drop-in sets + `RuntimeWatchdogSec=20s` + `RebootWatchdogSec=5min`; PID1 pets `intel_oc_wdt`, chip + hard-resets the box if systemd hangs >20s. Deployed via `daemon-reexec`. +- [x] **Auto power-on when mains returns** — RESOLVED as "document only". Not + OS-controllable on this ZenBook: `/proc/acpi/wakeup` is wake-from-suspend (S3/S4) only, + and "restore on AC loss" is a BIOS/firmware feature this laptop likely doesn't expose. + The 10% auto-shutdown is therefore a one-way trip until manual power-on. **Action:** + check BIOS setup for an "AC power-on / restore on AC loss" option and enable it if + present (documented in README §3, Battery monitor). +- [x] **Heartbeat / dead-man's switch** — DONE. `hbc` (heartbeat client, from + git.wrede.ca/andreas/heartbeat) installed to `~/venvs/hbd` + `~/bin/hbc`; `hbc.service` + reports to **hbd.wrede.pvt** (UDP 50003) every 15s as user `andreas`, shipping cpu/mem/ + disk/network/zfs metrics. Config `/etc/hbc.yaml`, vendored in `heartbeat/`. - [ ] **Connectivity alerts → Pushover** — `van-failover` already knows when it fails over or when ALL WANs are down; it just logs it. Pipe those events to the Pushover plumbing so internet loss is alerted, not just battery. diff --git a/deploy.sh b/deploy.sh index eb6217e..8a93a72 100755 --- a/deploy.sh +++ b/deploy.sh @@ -54,9 +54,24 @@ install -D -m0644 power/10-vanlink-nolid.conf /etc/systemd/logind.conf.d/10-vanl # Belt-and-suspenders: a router must never suspend from idle, GUI, or a stray `systemctl suspend`. systemctl mask sleep.target suspend.target hibernate.target hybrid-sleep.target >/dev/null 2>&1 || true +echo "== hardware watchdog ==" +install -D -m0644 power/10-vanlink-watchdog.conf /etc/systemd/system.conf.d/10-vanlink-watchdog.conf + +echo "== heartbeat client (dead-man's switch) ==" +install -D -m0644 heartbeat/hbc.yaml /etc/hbc.yaml +install -D -m0644 heartbeat/hbc.service /etc/systemd/system/hbc.service +# The hbc binary itself (~/bin/hbc + venv) is installed once via the heartbeat +# project's installer — see README §4. Only start the service if it's present. +if [ ! -x /home/andreas/bin/hbc ]; then + echo " -> /home/andreas/bin/hbc not found; run 'sh ~/git/heartbeat/scripts/hb_install.sh client' (README §4)" +fi + echo "== apply ==" sysctl --system >/dev/null systemctl daemon-reload +# Re-exec PID1 so the system.conf.d watchdog drop-in takes effect (daemon-reload alone +# does NOT re-arm RuntimeWatchdogSec). Safe online: re-exec keeps all services running. +systemctl daemon-reexec # networkd here owns only the AP + ZT overlay (neither a real uplink), so its wait-online # can never satisfy "online" and just burns its 120s timeout, stalling network-online.target # and ZeroTier by ~2min every boot. Real uplink readiness is covered by NetworkManager-wait-online. @@ -66,6 +81,11 @@ systemctl restart systemd-logind >/dev/null 2>&1 || true systemctl unmask hostapd >/dev/null 2>&1 || true systemctl enable regdomain.service hostapd van-ap-dnsmasq nftables systemd-networkd van-failover van-thermal van-battery >/dev/null 2>&1 || true systemctl restart van-thermal van-battery +# Heartbeat: only enable/start once the client binary is installed (README §4). +if [ -x /home/andreas/bin/hbc ]; then + systemctl enable hbc >/dev/null 2>&1 || true + systemctl restart hbc +fi # restart in dependency order; AP iface IP first, then hostapd/dnsmasq, then NAT/failover systemctl restart systemd-networkd systemctl restart hostapd van-ap-dnsmasq nftables van-failover @@ -77,4 +97,5 @@ echo " iw dev wlxc83a35a4ee55 info | grep -E 'ssid|channel|width'" echo " cat /run/van-failover/state.json" echo " cat /run/van-thermal/state.json # CPU + NVMe temps" echo " cat /run/van-battery/state.json # mains/battery + charge %" +echo " systemctl status hbc # heartbeat client -> hbd.wrede.pvt" echo "Manual one-time steps (see README §4): zerotier-systemd-manager binary + 'zerotier-cli set allowDNS=1'." diff --git a/heartbeat/hbc.service b/heartbeat/hbc.service new file mode 100644 index 0000000..11b0b87 --- /dev/null +++ b/heartbeat/hbc.service @@ -0,0 +1,16 @@ +[Unit] +Description=Heartbeat client (hbc) — dead-man's switch + metrics to hbd.wrede.pvt +After=network-online.target +Wants=network-online.target + +[Service] +Type=simple +# Least privilege: the hbd server can push CMD (run a shell command on the client), +# so this runs as the unprivileged user, not root. hbc lives in andreas' venv install. +User=andreas +ExecStart=/home/andreas/bin/hbc -b -c /etc/hbc.yaml hbd.wrede.pvt +Restart=always +RestartSec=10 + +[Install] +WantedBy=multi-user.target diff --git a/heartbeat/hbc.yaml b/heartbeat/hbc.yaml new file mode 100644 index 0000000..62474d4 --- /dev/null +++ b/heartbeat/hbc.yaml @@ -0,0 +1,15 @@ +# hbc (heartbeat client) config for wayback — the van router's dead-man's switch. +# Deployed to /etc/hbc.yaml by deploy.sh. Server host is passed on the command line +# (hbd.wrede.pvt, see hbc.service). No secrets here. +# +# The client (~/bin/hbc) is installed once via the heartbeat project's own installer — +# see README §4. This file only tunes intervals + which metric plugins to ship. + +interval: 15 # heartbeat every 15s (server flags overdue a few s after a miss) + +plugins: + cpu_monitor: { interval: 300 } + memory_monitor: { interval: 300 } # ZFS ARC-aware + disk_monitor: { interval: 300 } + network_monitor: { interval: 300 } + zfs_monitor: { interval: 300 } # zroot health/capacity (readable as the service user) diff --git a/power/10-vanlink-watchdog.conf b/power/10-vanlink-watchdog.conf new file mode 100644 index 0000000..110839b --- /dev/null +++ b/power/10-vanlink-watchdog.conf @@ -0,0 +1,10 @@ +# Hardware watchdog for unattended operation. +# +# PID1 pets /dev/watchdog0 (intel_oc_wdt) every RuntimeWatchdogSec/2. If systemd +# itself wedges for longer than RuntimeWatchdogSec, the chip hard-resets the box — +# the only way to recover a hung router with nobody there to open the lid. +# (See the EC-latch / USB-hub-hang history.) RebootWatchdogSec also guards against +# a reboot that hangs partway. +[Manager] +RuntimeWatchdogSec=20s +RebootWatchdogSec=5min