Daily Health Check

Step-by-step procedure for verifying MEGATRON system health. Run these checks daily or after any maintenance event.

Quick Check (5 minutes)

Run this for a rapid overview:

# 1. System basics
echo "=== UPTIME ===" && uptime
echo "=== MEMORY ===" && free -h
echo "=== DISK ===" && df -h / | tail -1
echo "=== GPU ===" && nvidia-smi --query-gpu=index,memory.used,temperature.gpu,power.draw --format=csv,noheader

# 2. Kubernetes nodes
echo "=== K3s NODES ===" && kubectl get nodes

# 3. Problem pods
echo "=== NON-RUNNING PODS ===" && kubectl get pods -A --field-selector=status.phase!=Running,status.phase!=Succeeded

# 4. Certificate issues
echo "=== CERT STATUS ===" && kubectl get certificates -A | grep -v "True"

# 5. Docker containers
echo "=== DOCKER ISSUES ===" && docker ps -a --filter "status=restarting" --filter "status=exited"

Full Health Check Procedure

Step 1: System Resources

# Check load averages and uptime
uptime

# Expected: load < 40 (for 128 threads), uptime consistent

Check memory:

free -h

Check disk:

df -h /
df -h /path/to/smb/mount  # if mounted

Step 2: GPU Status

nvidia-smi

Check for stuck GPU processes:

nvidia-smi --query-compute-apps=pid,name,used_memory --format=csv

Step 3: Kubernetes Nodes

kubectl get nodes -o wide

All three nodes (megatron, thing1, thing2) should show Ready. If any show NotReady:

kubectl describe node <node-name>
# Check for disk pressure, memory pressure, PID pressure, or network issues

Step 4: Kubernetes Pods

# All non-running pods (excluding completed jobs)
kubectl get pods -A --field-selector=status.phase!=Running,status.phase!=Succeeded

# Pods with restarts
kubectl get pods -A -o wide | awk '$5 > 5 {print}'

# Recently crashed pods
kubectl get events -A --sort-by='.lastTimestamp' | grep -E "BackOff|Failed|Error|OOM" | tail -20

Step 5: TLS Certificates

kubectl get certificates -A

All certificates should show True in the READY column. If any show False:

kubectl describe certificate <name> -n <namespace>
kubectl get challenges -A  # Check for pending ACME challenges

Step 6: Key Services

Authentik (SSO):

kubectl get pods -n authentik
# Verify: auth.m3gatr0n.com responds
curl -s -o /dev/null -w "%{http_code}" https://auth.m3gatr0n.com/

Vault (Secrets):

kubectl get pods -n vault
kubectl exec -n vault vault-0 -- vault status 2>/dev/null | grep "Sealed"
# Should show: Sealed false

Monitoring:

kubectl get pods -n monitoring
# Check AlertManager CPU
kubectl top pod -n monitoring -l app.kubernetes.io/name=alertmanager

Cloudflare Tunnel:

systemctl status cloudflared
# Verify: external domain responds
curl -s -o /dev/null -w "%{http_code}" https://stiq.us/

Step 7: Docker Containers

# Check for problems
docker ps -a --format "table {{.Names}}\t{{.Status}}" | grep -E "Restarting|Exited"

# Check Bibliotheca stack
docker ps --filter "name=bibliotheca" --format "table {{.Names}}\t{{.Status}}"

Step 8: Network Mounts

# Check SMB mount
mount | grep cifs
ls /path/to/smb/mount > /dev/null 2>&1 && echo "SMB: OK" || echo "SMB: FAILED"

# Check NFS (expect failure currently)
mount | grep nfs
ls /mnt/cloudy > /dev/null 2>&1 && echo "NFS: OK" || echo "NFS: DOWN"

Step 9: Tailscale

tailscale status

Verify all four machines are connected:

• megatron (100.70.25.47)
• lucass-mac-studio (100.87.201.64)
• macbook-pro-m4 (100.91.253.88)
• tensorbook (100.108.49.109)

Health Check Summary Template

Date: YYYY-MM-DD
Operator: ___________

System Resources:
  [ ] Load average < 40
  [ ] RAM usage < 80%
  [ ] Swap usage < 50%
  [ ] Root disk < 75%
  [ ] SMB share < 85%

GPU:
  [ ] Both GPUs detected
  [ ] Temperatures < 75C
  [ ] No stuck processes

Kubernetes:
  [ ] All 3 nodes Ready
  [ ] No CrashLoopBackOff pods
  [ ] All certificates Ready (or known exceptions)

Services:
  [ ] Authentik responding
  [ ] Vault unsealed
  [ ] AlertManager CPU normal
  [ ] Cloudflare tunnel active
  [ ] Tailscale mesh connected

Docker:
  [ ] No unexpected container restarts

Network:
  [ ] SMB mount accessible
  [ ] NFS status: [expected down / restored]

Known Issues Unchanged:
  [ ] VNC (disabled / still running)
  [ ] AlertManager CPU (restarted / still high)
  [ ] Harbor TLS (fixed / still pending)

Notes:
___________________________________________