andreas
4b7c360095
SD card filesystem corrupted: ext4 journal version incompatible with standard e2fsck (1.47.2 from Linux host). Filesystem inaccessible. Root cause: Podman homeassistant container startup during network instability and system reconfiguration caused unclean shutdown and filesystem corruption. Recovery: Flash fresh Armbian 6.18, run deploy.sh with fixes: - DNS resolution fix added to HOMEASSISTANT.md - resolv.conf fix should be added to deploy.sh post-processing - Deploy containers after network is stable All scripts and configs are ready and committed. Only need fresh SD card. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
89 lines
2.6 KiB
Markdown
89 lines
2.6 KiB
Markdown
# R5C Boot Failure - Filesystem Corruption
|
|
|
|
## Status
|
|
❌ **Boot Failed** - SD card filesystem corrupted
|
|
- Journal superblock corrupt
|
|
- ext4 journal version incompatible with standard e2fsck (1.47.2)
|
|
- Filesystem inaccessible from Linux host
|
|
|
|
## Root Cause Analysis
|
|
|
|
**Likely trigger:** Podman homeassistant container startup during resolv.conf changes
|
|
- Heavy I/O: container image pull (~1.5GB)
|
|
- System unstable: DNS changes, network issues, package installation in progress
|
|
- Result: Filesystem corruption, unclean shutdown
|
|
|
|
## Recovery Options
|
|
|
|
### Option 1: Fresh Armbian 6.18 Install (Recommended)
|
|
1. Download latest Armbian 6.18 for R5C from armbian.com
|
|
2. Flash to new SD card
|
|
3. Re-run deployment:
|
|
```bash
|
|
cd /home/andreas/vanlink
|
|
sudo ./deploy.sh
|
|
```
|
|
4. Restore configs:
|
|
- `/srv/homeassistant` (if still on USB/external)
|
|
- DNS fix: `/etc/resolv.conf` (see HOMEASSISTANT.md)
|
|
|
|
### Option 2: Boot Armbian on R5C to Repair
|
|
1. Need working Armbian installation to use Armbian's e2fsck
|
|
2. Mount corrupted SD as secondary drive
|
|
3. Run Armbian's fsck to repair journal
|
|
|
|
### Option 3: Data Recovery
|
|
- Mount with `noload` flag (skip journal):
|
|
```bash
|
|
sudo mount -t ext4 -o noload /dev/sda1 /mnt/r5c
|
|
```
|
|
- May allow readonly access to copy critical files
|
|
|
|
## Lessons Learned
|
|
|
|
1. **Don't install heavy services during initial boot**
|
|
- Podman pull during unstable network = corruption risk
|
|
- Should have stabilized DNS first, then tested, then deployed containers
|
|
|
|
2. **systemd-resolved disable needs better handling**
|
|
- Created broken symlink situation
|
|
- Should have been handled in deploy.sh post-processing
|
|
|
|
3. **Network instability is critical**
|
|
- 1000+ ms latency on Starlink caused cascading failures
|
|
- Package downloads failing during container startup = corruption
|
|
|
|
## Prevention
|
|
|
|
For future R5C deployments:
|
|
|
|
1. **Boot → Verify → Stabilize → Deploy**
|
|
- Boot fresh Armbian
|
|
- Verify network (DNS, latency)
|
|
- Run deploy.sh
|
|
- Test services individually
|
|
|
|
2. **Add resolv.conf fix to deploy.sh:**
|
|
```bash
|
|
# Remove broken systemd-resolved symlink if it exists
|
|
rm -f /etc/resolv.conf
|
|
cat > /etc/resolv.conf << EOF
|
|
nameserver 10.42.1.1
|
|
nameserver 1.1.1.1
|
|
nameserver 8.8.8.8
|
|
EOF
|
|
```
|
|
|
|
3. **Delay container startup:**
|
|
- Deploy podman/containers separately
|
|
- Test WiFi/network/DHCP first
|
|
- Then start resource-heavy services
|
|
|
|
## Current State
|
|
|
|
- All config/scripts committed to git
|
|
- R5C SD card: corrupted, unrecoverable
|
|
- Homeassistant setup: scripts ready, needs fresh SD card
|
|
- DNS fix: documented, ready to deploy
|
|
- **Recommendation: Flash fresh Armbian 6.18, run deploy.sh**
|