vanlink/RECOVERY.md

R5C Boot Failure - Filesystem Corruption

Status

❌ Boot Failed - SD card filesystem corrupted

• Journal superblock corrupt
• ext4 journal version incompatible with standard e2fsck (1.47.2)
• Filesystem inaccessible from Linux host

Root Cause Analysis

Likely trigger: Podman homeassistant container startup during resolv.conf changes

• Heavy I/O: container image pull (~1.5GB)
• System unstable: DNS changes, network issues, package installation in progress
• Result: Filesystem corruption, unclean shutdown

Recovery Options

Option 1: Fresh Armbian 6.18 Install (Recommended)

1. Download latest Armbian 6.18 for R5C from armbian.com
2. Flash to new SD card
3. Re-run deployment:

   cd /home/andreas/vanlink
   sudo ./deploy.sh
   

4. Restore configs:
   • /srv/homeassistant (if still on USB/external)
   • DNS fix: /etc/resolv.conf (see HOMEASSISTANT.md)

Option 2: Boot Armbian on R5C to Repair

1. Need working Armbian installation to use Armbian's e2fsck
2. Mount corrupted SD as secondary drive
3. Run Armbian's fsck to repair journal

Option 3: Data Recovery

• Mount with noload flag (skip journal):

  sudo mount -t ext4 -o noload /dev/sda1 /mnt/r5c
  

• May allow readonly access to copy critical files

Lessons Learned

1. Don't install heavy services during initial boot

   • Podman pull during unstable network = corruption risk
   • Should have stabilized DNS first, then tested, then deployed containers

2. systemd-resolved disable needs better handling

   • Created broken symlink situation
   • Should have been handled in deploy.sh post-processing

3. Network instability is critical

   • 1000+ ms latency on Starlink caused cascading failures
   • Package downloads failing during container startup = corruption

Prevention

For future R5C deployments:

1. Boot → Verify → Stabilize → Deploy

   • Boot fresh Armbian
   • Verify network (DNS, latency)
   • Run deploy.sh
   • Test services individually

2. Add resolv.conf fix to deploy.sh:

   # Remove broken systemd-resolved symlink if it exists
   rm -f /etc/resolv.conf
   cat > /etc/resolv.conf << EOF
   nameserver 10.42.1.1
   nameserver 1.1.1.1
   nameserver 8.8.8.8
   EOF
   

3. Delay container startup:

   • Deploy podman/containers separately
   • Test WiFi/network/DHCP first
   • Then start resource-heavy services

Current State

• All config/scripts committed to git
• R5C SD card: corrupted, unrecoverable
• Homeassistant setup: scripts ready, needs fresh SD card
• DNS fix: documented, ready to deploy
• Recommendation: Flash fresh Armbian 6.18, run deploy.sh