robustness: hardware watchdog + hbc heartbeat client
Two TODO Tier-1 items: - Hardware watchdog: power/10-vanlink-watchdog.conf drop-in sets RuntimeWatchdogSec=20s + RebootWatchdogSec=5min; deploy.sh applies it via daemon-reexec. PID1 pets intel_oc_wdt; a >20s systemd hang hard-resets the box. - Heartbeat / dead-man's switch: vendored hbc.yaml + hbc.service report to hbd.wrede.pvt (UDP 50003) every 15s as user andreas, shipping cpu/mem/disk/ network/zfs metrics. The hbc binary is installed once via the heartbeat project's installer (README §4); deploy.sh starts the service when present. Auto-power-on-when-mains-returns: documented as a BIOS-only option (not OS-controllable on this ZenBook) in README §3 + TODO, per decision. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Andreas Wrede
co-authored by
Claude Opus 4.8
Claude Opus 4.8
parent
97309970ec
commit
8fcc959551
@@ -41,6 +41,8 @@ This directory is the source of truth. The live system files live under `/etc`,
|
||||
| WAN health + failover | **van-failover** daemon |
|
||||
| Temperature monitor / alert / log | **van-thermal** daemon |
|
||||
| Battery monitor / low-charge alert + shutdown | **van-battery** daemon |
|
||||
| Auto-reboot on hang | **systemd hardware watchdog** (`intel_oc_wdt`) |
|
||||
| Liveness / dead-man's switch + metrics | **hbc** heartbeat client → hbd.wrede.pvt |
|
||||
| ZeroTier DNS → resolved | **zerotier-systemd-manager** + systemd-networkd |
|
||||
| Web UI | **Cockpit** + `vanrouter` plugin |
|
||||
|
||||
@@ -103,10 +105,20 @@ This directory is the source of truth. The live system files live under `/etc`,
|
||||
| `battery-config.json` | `/etc/van-battery/config.json` | warn levels, shutdown level, poll interval, paths |
|
||||
| `van-battery.service` | `/etc/systemd/system/van-battery.service` | `Restart=always` |
|
||||
| `pushover.json.example` | → `/etc/van-battery/pushover.json` (seeded if absent) | Pushover token+user **template**; real file is 0600, **not** in the repo |
|
||||
| `10-vanlink-watchdog.conf` | `/etc/systemd/system.conf.d/10-vanlink-watchdog.conf` | hardware watchdog (`RuntimeWatchdogSec=20s`); applied via `daemon-reexec` |
|
||||
|
||||
`deploy.sh` additionally **masks** `sleep.target suspend.target hibernate.target hybrid-sleep.target`
|
||||
(no repo file — symlinks to `/dev/null` under `/etc/systemd/system/`) so nothing else can suspend either.
|
||||
|
||||
### `heartbeat/` — dead-man's switch
|
||||
| file | → installs to | purpose |
|
||||
|---|---|---|
|
||||
| `hbc.yaml` | `/etc/hbc.yaml` | heartbeat client config (interval + metric plugins) |
|
||||
| `hbc.service` | `/etc/systemd/system/hbc.service` | runs `~/bin/hbc` as `andreas` → hbd.wrede.pvt:50003 |
|
||||
|
||||
The `hbc` binary itself (venv at `~/venvs/hbd`, symlink `~/bin/hbc`) is installed once via the
|
||||
heartbeat project's own installer — see §4. `deploy.sh` only starts the service once it exists.
|
||||
|
||||
---
|
||||
|
||||
## 3. Components & how to operate
|
||||
@@ -147,6 +159,7 @@ This directory is the source of truth. The live system files live under `/etc`,
|
||||
- **Credentials** live in `/etc/van-battery/pushover.json` (mode 0600), seeded from `pushover.json.example` on first deploy and **never committed**. Missing/placeholder creds disable *sending* but **not** the shutdown — running flat must always power down safely (the skip is logged to the journal).
|
||||
- Tune in `/etc/van-battery/config.json` (`warn_levels`, `shutdown_level`, `poll_interval`), then `systemctl restart van-battery`.
|
||||
- Status: `systemctl status van-battery`, `journalctl -u van-battery -f`, or `cat /run/van-battery/state.json`.
|
||||
- **Caveat — the 10 % shutdown is one-way.** A laptop won't power itself back on when mains returns: "restore on AC loss" is a BIOS/firmware feature (not OS-controllable — `/proc/acpi/wakeup` only covers wake-from-suspend). If your BIOS exposes an "AC power-on / restore on AC loss" option, enable it so the router reboots itself once shore/solar power is back; this ZenBook likely doesn't have it, in which case a low-battery shutdown needs a manual power-on.
|
||||
|
||||
### Adding the 4G/5G modem
|
||||
1. Plug the USB modem in. ModemManager + the existing `Koodo` gsm NM connection (autoconnect) bring it up.
|
||||
@@ -184,7 +197,7 @@ cd ~/vanlink
|
||||
sudo ./deploy.sh # copies all files to their system locations, reloads + enables services
|
||||
```
|
||||
|
||||
Three things `deploy.sh` does **not** do (one-time, manual):
|
||||
Four things `deploy.sh` does **not** do (one-time, manual):
|
||||
|
||||
1. **zerotier-systemd-manager binary** (v0.4.0, hand-installed — not in the ZeroTier apt repo):
|
||||
```bash
|
||||
@@ -194,6 +207,12 @@ Three things `deploy.sh` does **not** do (one-time, manual):
|
||||
```
|
||||
2. **hostapd unmask** (Ubuntu ships it masked): `sudo systemctl unmask hostapd`.
|
||||
3. **Pushover credentials** for `van-battery` — deploy seeds `/etc/van-battery/pushover.json` (0600) with placeholders; fill in your app token + user key, then `sudo systemctl restart van-battery`. Until then low-battery alerts are skipped (logged), but the 10 % auto-shutdown still works.
|
||||
4. **Heartbeat client `hbc`** (dead-man's switch) — install the binary once into andreas' venv:
|
||||
```bash
|
||||
git clone https://git.wrede.ca/andreas/heartbeat.git ~/git/heartbeat # if not already cloned
|
||||
sh ~/git/heartbeat/scripts/hb_install.sh client # -> ~/venvs/hbd + ~/bin/hbc
|
||||
```
|
||||
`deploy.sh` installs `/etc/hbc.yaml` + the `hbc.service` unit and starts it once `~/bin/hbc` exists. The service reports to **hbd.wrede.pvt** (UDP 50003) as user `andreas`. Upgrade later with `~/bin/hb_install.sh client`.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -20,17 +20,19 @@ value-for-effort. Generated 2026-06-29 from a robustness review of the live box.
|
||||
|
||||
## Tier 1 — high value, pieces already on hand
|
||||
|
||||
- [ ] **systemd hardware watchdog** — `/dev/watchdog` exists but isn't wired. A hung
|
||||
kernel/driver currently needs someone to open the lid. Given the EC-latch and
|
||||
USB-hub-hang history, auto-reboot-on-hang is the biggest unattended win.
|
||||
Set `RuntimeWatchdogSec` (and `RebootWatchdogSec`) in `/etc/systemd/system.conf`.
|
||||
- [ ] **Auto power-on when mains returns** — the 10% auto-shutdown is currently a
|
||||
one-way trip; a laptop won't boot itself when solar/shore power returns. Needs the
|
||||
BIOS "restore on AC loss → power on" setting (cannot be set from the OS). Without
|
||||
it the safe-shutdown can strand the router. **Manual BIOS toggle.**
|
||||
- [ ] **Heartbeat / dead-man's switch** — every existing alert can only fire if the box
|
||||
is alive. A periodic ping to healthchecks.io (or similar) inverts it: silence =
|
||||
something is dead (panic, no-boot, total power loss).
|
||||
- [x] **systemd hardware watchdog** — DONE. `power/10-vanlink-watchdog.conf` drop-in sets
|
||||
`RuntimeWatchdogSec=20s` + `RebootWatchdogSec=5min`; PID1 pets `intel_oc_wdt`, chip
|
||||
hard-resets the box if systemd hangs >20s. Deployed via `daemon-reexec`.
|
||||
- [x] **Auto power-on when mains returns** — RESOLVED as "document only". Not
|
||||
OS-controllable on this ZenBook: `/proc/acpi/wakeup` is wake-from-suspend (S3/S4) only,
|
||||
and "restore on AC loss" is a BIOS/firmware feature this laptop likely doesn't expose.
|
||||
The 10% auto-shutdown is therefore a one-way trip until manual power-on. **Action:**
|
||||
check BIOS setup for an "AC power-on / restore on AC loss" option and enable it if
|
||||
present (documented in README §3, Battery monitor).
|
||||
- [x] **Heartbeat / dead-man's switch** — DONE. `hbc` (heartbeat client, from
|
||||
git.wrede.ca/andreas/heartbeat) installed to `~/venvs/hbd` + `~/bin/hbc`; `hbc.service`
|
||||
reports to **hbd.wrede.pvt** (UDP 50003) every 15s as user `andreas`, shipping cpu/mem/
|
||||
disk/network/zfs metrics. Config `/etc/hbc.yaml`, vendored in `heartbeat/`.
|
||||
- [ ] **Connectivity alerts → Pushover** — `van-failover` already knows when it fails
|
||||
over or when ALL WANs are down; it just logs it. Pipe those events to the Pushover
|
||||
plumbing so internet loss is alerted, not just battery.
|
||||
|
||||
@@ -54,9 +54,24 @@ install -D -m0644 power/10-vanlink-nolid.conf /etc/systemd/logind.conf.d/10-vanl
|
||||
# Belt-and-suspenders: a router must never suspend from idle, GUI, or a stray `systemctl suspend`.
|
||||
systemctl mask sleep.target suspend.target hibernate.target hybrid-sleep.target >/dev/null 2>&1 || true
|
||||
|
||||
echo "== hardware watchdog =="
|
||||
install -D -m0644 power/10-vanlink-watchdog.conf /etc/systemd/system.conf.d/10-vanlink-watchdog.conf
|
||||
|
||||
echo "== heartbeat client (dead-man's switch) =="
|
||||
install -D -m0644 heartbeat/hbc.yaml /etc/hbc.yaml
|
||||
install -D -m0644 heartbeat/hbc.service /etc/systemd/system/hbc.service
|
||||
# The hbc binary itself (~/bin/hbc + venv) is installed once via the heartbeat
|
||||
# project's installer — see README §4. Only start the service if it's present.
|
||||
if [ ! -x /home/andreas/bin/hbc ]; then
|
||||
echo " -> /home/andreas/bin/hbc not found; run 'sh ~/git/heartbeat/scripts/hb_install.sh client' (README §4)"
|
||||
fi
|
||||
|
||||
echo "== apply =="
|
||||
sysctl --system >/dev/null
|
||||
systemctl daemon-reload
|
||||
# Re-exec PID1 so the system.conf.d watchdog drop-in takes effect (daemon-reload alone
|
||||
# does NOT re-arm RuntimeWatchdogSec). Safe online: re-exec keeps all services running.
|
||||
systemctl daemon-reexec
|
||||
# networkd here owns only the AP + ZT overlay (neither a real uplink), so its wait-online
|
||||
# can never satisfy "online" and just burns its 120s timeout, stalling network-online.target
|
||||
# and ZeroTier by ~2min every boot. Real uplink readiness is covered by NetworkManager-wait-online.
|
||||
@@ -66,6 +81,11 @@ systemctl restart systemd-logind >/dev/null 2>&1 || true
|
||||
systemctl unmask hostapd >/dev/null 2>&1 || true
|
||||
systemctl enable regdomain.service hostapd van-ap-dnsmasq nftables systemd-networkd van-failover van-thermal van-battery >/dev/null 2>&1 || true
|
||||
systemctl restart van-thermal van-battery
|
||||
# Heartbeat: only enable/start once the client binary is installed (README §4).
|
||||
if [ -x /home/andreas/bin/hbc ]; then
|
||||
systemctl enable hbc >/dev/null 2>&1 || true
|
||||
systemctl restart hbc
|
||||
fi
|
||||
# restart in dependency order; AP iface IP first, then hostapd/dnsmasq, then NAT/failover
|
||||
systemctl restart systemd-networkd
|
||||
systemctl restart hostapd van-ap-dnsmasq nftables van-failover
|
||||
@@ -77,4 +97,5 @@ echo " iw dev wlxc83a35a4ee55 info | grep -E 'ssid|channel|width'"
|
||||
echo " cat /run/van-failover/state.json"
|
||||
echo " cat /run/van-thermal/state.json # CPU + NVMe temps"
|
||||
echo " cat /run/van-battery/state.json # mains/battery + charge %"
|
||||
echo " systemctl status hbc # heartbeat client -> hbd.wrede.pvt"
|
||||
echo "Manual one-time steps (see README §4): zerotier-systemd-manager binary + 'zerotier-cli set <nwid> allowDNS=1'."
|
||||
|
||||
@@ -0,0 +1,16 @@
|
||||
[Unit]
|
||||
Description=Heartbeat client (hbc) — dead-man's switch + metrics to hbd.wrede.pvt
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
# Least privilege: the hbd server can push CMD (run a shell command on the client),
|
||||
# so this runs as the unprivileged user, not root. hbc lives in andreas' venv install.
|
||||
User=andreas
|
||||
ExecStart=/home/andreas/bin/hbc -b -c /etc/hbc.yaml hbd.wrede.pvt
|
||||
Restart=always
|
||||
RestartSec=10
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
@@ -0,0 +1,15 @@
|
||||
# hbc (heartbeat client) config for wayback — the van router's dead-man's switch.
|
||||
# Deployed to /etc/hbc.yaml by deploy.sh. Server host is passed on the command line
|
||||
# (hbd.wrede.pvt, see hbc.service). No secrets here.
|
||||
#
|
||||
# The client (~/bin/hbc) is installed once via the heartbeat project's own installer —
|
||||
# see README §4. This file only tunes intervals + which metric plugins to ship.
|
||||
|
||||
interval: 15 # heartbeat every 15s (server flags overdue a few s after a miss)
|
||||
|
||||
plugins:
|
||||
cpu_monitor: { interval: 300 }
|
||||
memory_monitor: { interval: 300 } # ZFS ARC-aware
|
||||
disk_monitor: { interval: 300 }
|
||||
network_monitor: { interval: 300 }
|
||||
zfs_monitor: { interval: 300 } # zroot health/capacity (readable as the service user)
|
||||
@@ -0,0 +1,10 @@
|
||||
# Hardware watchdog for unattended operation.
|
||||
#
|
||||
# PID1 pets /dev/watchdog0 (intel_oc_wdt) every RuntimeWatchdogSec/2. If systemd
|
||||
# itself wedges for longer than RuntimeWatchdogSec, the chip hard-resets the box —
|
||||
# the only way to recover a hung router with nobody there to open the lid.
|
||||
# (See the EC-latch / USB-hub-hang history.) RebootWatchdogSec also guards against
|
||||
# a reboot that hangs partway.
|
||||
[Manager]
|
||||
RuntimeWatchdogSec=20s
|
||||
RebootWatchdogSec=5min
|
||||
Reference in New Issue
Block a user