Disaster Recovery
Procedures for recovering MEGATRON from major failures.
Severity Levels
| Level | Scenario | Recovery Time |
|---|---|---|
| 1 | Single pod crash | Minutes (auto-restart) |
| 2 | Node failure (worker) | 10-30 minutes (drain + rejoin) |
| 3 | Control plane failure | 30-60 minutes (K3s restart or etcd restore) |
| 4 | Full system failure | Hours (OS reinstall + K3s restore) |
| 5 | Hardware failure | Days (replacement + full restore) |
Emergency Access
If normal SSH is unresponsive:
# Rescue SSH (high priority, OOM protected)
ssh -p 2222 [email protected]
# Or via SSH config
ssh megatron-rescue
Level 3: Control Plane Recovery
# 1. SSH via rescue port
ssh megatron-rescue
# 2. Check K3s status
sudo systemctl status k3s
# 3. Restart K3s
sudo systemctl restart k3s
# 4. Wait for nodes to rejoin
kubectl get nodes -w
# 5. If etcd is corrupted, restore from snapshot
k3s server --cluster-reset --cluster-reset-restore-path=/var/lib/rancher/k3s/server/db/snapshots/<snapshot>
Level 4: Full System Recovery
- Reinstall Ubuntu 20.04 LTS
- Restore SSH keys from backup
- Install K3s:
curl -sfL https://get.k3s.io | sh - - Restore etcd from snapshot
- Rejoin worker nodes
- Restore Cloudflare tunnel config
- Verify all services
Critical Files to Preserve
| File/Directory | Purpose |
|---|---|
| /etc/cloudflared/ | Tunnel config and credentials |
| /var/lib/rancher/k3s/server/db/snapshots/ | K3s etcd snapshots |
| ~/.ssh/ | SSH keys |
| /etc/ssh/sshd_config_rescue | Rescue SSH config |
| ~/Papers/ | Research data (234 GB) |
TODO: expand with automated DR testing, RTO/RPO targets, and runbook for each severity level.