GPUs
MEGATRON has two NVIDIA GeForce RTX 3090 GPUs, primarily used for deep learning training (ULTRATHINK research) and inference workloads (Ollama LLM serving).
GPU Specifications
| Specification | Value |
|---|---|
| Model | NVIDIA GeForce RTX 3090 |
| Count | 2 |
| Architecture | Ampere (GA102) |
| CUDA Cores | 10,496 per GPU |
| Tensor Cores | 328 per GPU (3rd gen) |
| RT Cores | 82 per GPU (2nd gen) |
| VRAM | 24 GB GDDR6X per GPU |
| Memory Bus | 384-bit |
| Memory Bandwidth | 936.2 GB/s |
| TDP | 350W per GPU |
| Compute Capability | 8.6 |
| NVLink | Not available (consumer card) |
Current Status
| Metric | GPU 0 | GPU 1 |
|---|---|---|
| VRAM Used | 2.7 GB / 24 GB | 14 MB / 24 GB |
| VRAM Utilization | 11.3% | 0.06% |
| Temperature | 53C | 46C |
| Power Draw | 29W / 350W | 17W / 350W |
| Fan Speed | Auto | Auto |
| Performance State | P8 (idle) | P8 (idle) |
GPU 0: VRAM [##..................] 11% Temp: 53C Power: 29W / 350W
GPU 1: VRAM [....................] 0% Temp: 46C Power: 17W / 350W
GPU 0 has a 2.7 GB allocation, likely from an Ollama model loaded in VRAM or a persistent CUDA context. GPU 1 is essentially idle.
Quick Commands
# Full GPU status
nvidia-smi
# Compact CSV output
nvidia-smi --query-gpu=index,name,memory.used,memory.total,temperature.gpu,power.draw \
--format=csv
# Real-time monitoring (updates every second)
watch -n 1 nvidia-smi
# List GPU compute processes
nvidia-smi --query-compute-apps=pid,name,gpu_uuid,used_memory --format=csv
# Check driver and CUDA version
nvidia-smi | head -4
GPU Usage by Workload
| Workload | GPU(s) | Typical VRAM | Notes |
|---|---|---|---|
| ULTRATHINK Training | 0 + 1 | 20-48 GB total | Distributed training across both GPUs |
| Ollama (LLM inference) | 0 | 2-20 GB | Model-dependent; 7B ~ 4GB, 70B ~ 40GB |
| Bibliotheca TTS | 0 or 1 | 1-4 GB | GPU-accelerated text-to-speech |
| JupyterHub notebooks | Any | Variable | Interactive research |
| MLflow experiments | 0 or 1 | Variable | Tracked experiment runs |
Multi-GPU Training
For ULTRATHINK research, both GPUs can be used together:
# PyTorch DataParallel (simpler, less efficient)
CUDA_VISIBLE_DEVICES=0,1 python train.py
# PyTorch DistributedDataParallel (recommended for performance)
torchrun --nproc_per_node=2 train.py
# Use only a specific GPU
CUDA_VISIBLE_DEVICES=1 python inference.py
Kubernetes GPU Access
The cluster has a gpu-operator namespace, indicating the NVIDIA GPU Operator is deployed for Kubernetes GPU scheduling. Pods can request GPU resources:
resources:
limits:
nvidia.com/gpu: 1
The GPU Operator manages:
- NVIDIA driver lifecycle
- NVIDIA Container Toolkit
- Device plugin for Kubernetes
- GPU Feature Discovery
- DCGM Exporter (monitoring)
Thermal Notes
Both GPUs are running cool (46-53C) at near-idle power draw. Under full training load, expect:
| Condition | GPU Temp | Power Draw | System Power |
|---|---|---|---|
| Idle | 45-55C | 17-30W | ~100W GPU total |
| Single GPU training | 75-83C | 300-350W | ~350W GPU total |
| Dual GPU training | 78-85C | 300-350W each | ~700W GPU total |
Ensure adequate cooling and a sufficient PSU (recommended 1200W+ for dual 3090 under full load, accounting for the 280W TDP CPU).
Troubleshooting
# GPU not detected
lspci | grep -i nvidia
dmesg | grep -i nvidia
# Reset GPU (use with caution)
sudo nvidia-smi --gpu-reset -i 0
# Check for stuck processes
nvidia-smi --query-compute-apps=pid,name,used_memory --format=csv
kill -9 <stuck_pid>
# Verify CUDA installation
python -c "import torch; print(torch.cuda.is_available()); print(torch.cuda.device_count())"