Storage

Disk Layout

Local Storage

The root filesystem is on an NVMe drive providing high-speed I/O for the OS, K3s cluster data, Docker images, and research workloads.

Network Storage

Warning: The SMB share is at 90% capacity (79T/88T). Only ~8.2T remains. Monitor closely and plan cleanup or expansion. See Current Issues.

Warning: NFS mounts to /mnt/cloudy and /mnt/cloudy-rw are unreachable -- the NFS host is down.

Storage Architecture

┌──────────────────────────────────────────────────────┐
│                     MEGATRON                          │
│                                                      │
│  ┌─────────────────────────┐  ┌───────────────────┐  │
│  │  NVMe (nvme0n1p5)       │  │  Longhorn PVs     │  │
│  │  3.1T root filesystem   │  │  Distributed K3s  │  │
│  │  ├── /                  │  │  storage across    │  │
│  │  ├── /home/lucas/       │  │  megatron, thing1, │  │
│  │  │   ├── Papers/ 234GB  │  │  thing2            │  │
│  │  │   ├── juke/ 60GB     │  └───────────────────┘  │
│  │  │   └── miniconda3/44GB│                         │
│  │  ├── /var/lib/rancher/  │  ┌───────────────────┐  │
│  │  │   (K3s data)         │  │  SMB //10.0.0.41  │  │
│  │  └── /var/lib/docker/   │  │  88T NAS share    │  │
│  └─────────────────────────┘  │  90% full         │  │
│                               └───────────────────┘  │
│  ┌─────────────────────────┐                         │
│  │  NFS (DOWN)             │                         │
│  │  /mnt/cloudy            │                         │
│  │  /mnt/cloudy-rw         │                         │
│  │  Host unreachable       │                         │
│  └─────────────────────────┘                         │
└──────────────────────────────────────────────────────┘

Top Directories by Size

Shared Storage Layout (Research)

The ULTRATHINK research projects use a shared storage structure:

/mnt/shared/
├── datasets/         # Shared datasets (ImageNet, CIFAR, etc.)
├── checkpoints/      # Model checkpoints per project
├── pretrained/       # Pretrained models (TORCH_HOME)
├── logs/             # TensorBoard, MLflow logs
└── results/          # Experiment results

Environment variables for each project:

export {PROJECT}_DATA=/mnt/shared/datasets
export {PROJECT}_CHECKPOINTS=/mnt/shared/checkpoints/{project}
export {PROJECT}_LOGS=/mnt/shared/logs/{project}
export TORCH_HOME=/mnt/shared/pretrained

Kubernetes Persistent Storage

K3s uses Longhorn as the distributed storage backend for persistent volumes. Longhorn replicates data across the three cluster nodes (megatron, thing1, thing2) for redundancy.

See Longhorn and Storage Classes for details.

Disk Usage Quick Check

# Check all mount points
df -h

# Check root filesystem usage
df -h /

# Top-level directory sizes
du -sh /home/lucas/*/ 2>/dev/null | sort -rh | head -20

# Check inode usage
df -i /

# Check NVMe health
sudo smartctl -a /dev/nvme0n1

# Check SMB share status
mount | grep cifs

Maintenance Recommendations

1. SMB share at 90% -- Audit large files, archive old data, or expand the NAS
2. NFS mounts down -- Investigate and restore the NFS host or remove stale mount entries from /etc/fstab
3. Root disk at 54% -- Healthy but growing; monitor ~/Papers/ (234 GB) and conda envs (44 GB)
4. Swap usage (21/31 GiB) -- Consider swapoff -a && swapon -a during a maintenance window to reclaim
5. Longhorn health -- Verify replica counts and volume health via the Longhorn UI