From 4b7c3600953953335d52ccbb2896e67c6b92c09d Mon Sep 17 00:00:00 2001
From: Andreas Wrede <andreas@wrede.ca>
Date: Sat, 19 Sep 2026 15:37:25 -0400
Subject: [PATCH] Document R5C boot failure and recovery plan

SD card filesystem corrupted: ext4 journal version incompatible with
standard e2fsck (1.47.2 from Linux host). Filesystem inaccessible.

Root cause: Podman homeassistant container startup during network
instability and system reconfiguration caused unclean shutdown and
filesystem corruption.

Recovery: Flash fresh Armbian 6.18, run deploy.sh with fixes:
- DNS resolution fix added to HOMEASSISTANT.md
- resolv.conf fix should be added to deploy.sh post-processing
- Deploy containers after network is stable

All scripts and configs are ready and committed. Only need fresh SD card.
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
---
 RECOVERY.md | 88 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 88 insertions(+)
 create mode 100644 RECOVERY.md

diff --git a/RECOVERY.md b/RECOVERY.md
new file mode 100644
index 0000000..ef1b9c2
--- /dev/null
+++ b/RECOVERY.md
@@ -0,0 +1,88 @@
+# R5C Boot Failure - Filesystem Corruption
+
+## Status
+❌ **Boot Failed** - SD card filesystem corrupted
+- Journal superblock corrupt
+- ext4 journal version incompatible with standard e2fsck (1.47.2)
+- Filesystem inaccessible from Linux host
+
+## Root Cause Analysis
+
+**Likely trigger:** Podman homeassistant container startup during resolv.conf changes
+- Heavy I/O: container image pull (~1.5GB)
+- System unstable: DNS changes, network issues, package installation in progress
+- Result: Filesystem corruption, unclean shutdown
+
+## Recovery Options
+
+### Option 1: Fresh Armbian 6.18 Install (Recommended)
+1. Download latest Armbian 6.18 for R5C from armbian.com
+2. Flash to new SD card
+3. Re-run deployment:
+   ```bash
+   cd /home/andreas/vanlink
+   sudo ./deploy.sh
+   ```
+4. Restore configs:
+   - `/srv/homeassistant` (if still on USB/external)
+   - DNS fix: `/etc/resolv.conf` (see HOMEASSISTANT.md)
+
+### Option 2: Boot Armbian on R5C to Repair
+1. Need working Armbian installation to use Armbian's e2fsck
+2. Mount corrupted SD as secondary drive
+3. Run Armbian's fsck to repair journal
+
+### Option 3: Data Recovery
+- Mount with `noload` flag (skip journal):
+  ```bash
+  sudo mount -t ext4 -o noload /dev/sda1 /mnt/r5c
+  ```
+- May allow readonly access to copy critical files
+
+## Lessons Learned
+
+1. **Don't install heavy services during initial boot**
+   - Podman pull during unstable network = corruption risk
+   - Should have stabilized DNS first, then tested, then deployed containers
+
+2. **systemd-resolved disable needs better handling**
+   - Created broken symlink situation
+   - Should have been handled in deploy.sh post-processing
+
+3. **Network instability is critical**
+   - 1000+ ms latency on Starlink caused cascading failures
+   - Package downloads failing during container startup = corruption
+
+## Prevention
+
+For future R5C deployments:
+
+1. **Boot → Verify → Stabilize → Deploy**
+   - Boot fresh Armbian
+   - Verify network (DNS, latency)
+   - Run deploy.sh
+   - Test services individually
+
+2. **Add resolv.conf fix to deploy.sh:**
+   ```bash
+   # Remove broken systemd-resolved symlink if it exists
+   rm -f /etc/resolv.conf
+   cat > /etc/resolv.conf << EOF
+   nameserver 10.42.1.1
+   nameserver 1.1.1.1
+   nameserver 8.8.8.8
+   EOF
+   ```
+
+3. **Delay container startup:**
+   - Deploy podman/containers separately
+   - Test WiFi/network/DHCP first
+   - Then start resource-heavy services
+
+## Current State
+
+- All config/scripts committed to git
+- R5C SD card: corrupted, unrecoverable
+- Homeassistant setup: scripts ready, needs fresh SD card
+- DNS fix: documented, ready to deploy
+- **Recommendation: Flash fresh Armbian 6.18, run deploy.sh**