Files

T

andreas 4b7c360095 Document R5C boot failure and recovery plan

SD card filesystem corrupted: ext4 journal version incompatible with
standard e2fsck (1.47.2 from Linux host). Filesystem inaccessible.

Root cause: Podman homeassistant container startup during network
instability and system reconfiguration caused unclean shutdown and
filesystem corruption.

Recovery: Flash fresh Armbian 6.18, run deploy.sh with fixes:
- DNS resolution fix added to HOMEASSISTANT.md
- resolv.conf fix should be added to deploy.sh post-processing
- Deploy containers after network is stable

All scripts and configs are ready and committed. Only need fresh SD card.
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>

2026-09-19 15:37:25 -04:00

2.6 KiB

Raw Permalink Blame History

R5C Boot Failure - Filesystem Corruption

Status

❌ Boot Failed - SD card filesystem corrupted

Journal superblock corrupt
ext4 journal version incompatible with standard e2fsck (1.47.2)
Filesystem inaccessible from Linux host

Root Cause Analysis

Likely trigger: Podman homeassistant container startup during resolv.conf changes

Heavy I/O: container image pull (~1.5GB)
System unstable: DNS changes, network issues, package installation in progress
Result: Filesystem corruption, unclean shutdown

Recovery Options

Option 1: Fresh Armbian 6.18 Install (Recommended)

Download latest Armbian 6.18 for R5C from armbian.com
Flash to new SD card

Re-run deployment:

cd /home/andreas/vanlink
sudo ./deploy.sh

Restore configs:
- /srv/homeassistant (if still on USB/external)
- DNS fix: /etc/resolv.conf (see HOMEASSISTANT.md)

Option 2: Boot Armbian on R5C to Repair

Need working Armbian installation to use Armbian's e2fsck
Mount corrupted SD as secondary drive
Run Armbian's fsck to repair journal

Option 3: Data Recovery

Mount with noload flag (skip journal):

sudo mount -t ext4 -o noload /dev/sda1 /mnt/r5c

May allow readonly access to copy critical files

Lessons Learned

Don't install heavy services during initial boot
- Podman pull during unstable network = corruption risk
- Should have stabilized DNS first, then tested, then deployed containers
systemd-resolved disable needs better handling
- Created broken symlink situation
- Should have been handled in deploy.sh post-processing
Network instability is critical
- 1000+ ms latency on Starlink caused cascading failures
- Package downloads failing during container startup = corruption

Prevention

For future R5C deployments:

Boot → Verify → Stabilize → Deploy
- Boot fresh Armbian
- Verify network (DNS, latency)
- Run deploy.sh
- Test services individually

Add resolv.conf fix to deploy.sh:

# Remove broken systemd-resolved symlink if it exists
rm -f /etc/resolv.conf
cat > /etc/resolv.conf << EOF
nameserver 10.42.1.1
nameserver 1.1.1.1
nameserver 8.8.8.8
EOF

Delay container startup:
- Deploy podman/containers separately
- Test WiFi/network/DHCP first
- Then start resource-heavy services

Current State

All config/scripts committed to git
R5C SD card: corrupted, unrecoverable
Homeassistant setup: scripts ready, needs fresh SD card
DNS fix: documented, ready to deploy
Recommendation: Flash fresh Armbian 6.18, run deploy.sh

2.6 KiB Raw Permalink Blame History