Node Recovery

Procedures for recovering K3s cluster nodes that are NotReady, unreachable, or require re-joining.

Diagnosing Node Issues

# Check node status
kubectl get nodes -o wide

# Describe the problematic node
kubectl describe node <node-name>

# Check events
kubectl get events --field-selector involvedObject.name=<node-name> --sort-by='.lastTimestamp'

Common Scenarios

Node Shows NotReady

  1. SSH to the node and check the K3s agent:
ssh <node> 'systemctl status k3s-agent'
ssh <node> 'journalctl -u k3s-agent --since "10 minutes ago"'
  1. Restart the agent:
ssh <node> 'sudo systemctl restart k3s-agent'

Node Unreachable

  1. Check network connectivity:
ping <node-ip>
ssh <node> 'echo reachable'
  1. If the node is physically down, drain it:
kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data --timeout=60s

Re-joining a Node

# On the worker node, get the join token from megatron:
K3S_TOKEN=$(ssh megatron 'sudo cat /var/lib/rancher/k3s/server/node-token')

# Install/rejoin K3s agent
curl -sfL https://get.k3s.io | K3S_URL=https://megatron:6443 K3S_TOKEN=$K3S_TOKEN sh -

Post-Recovery

# Verify node is Ready
kubectl get nodes

# Uncordon if drained
kubectl uncordon <node-name>

# Check Longhorn replica health
kubectl get volumes.longhorn.io -n longhorn-system

TODO: expand with automated recovery scripts and monitoring alerts for node health.