MEGATRON Infrastructure Documentation

MEGATRON is a high-performance workstation serving as the central hub for software development, Kubernetes cluster management, machine learning research, and production SaaS hosting. Built around an AMD Threadripper 3990X with dual RTX 3090 GPUs and 251 GiB of RAM, it runs a K3s Kubernetes cluster spanning three nodes and hosts over a dozen production applications.

Quick Stats

What Runs on MEGATRON

Production Applications

• STIQ (stiq.us) -- SaaS platform with prod/dev/staging environments
• GlamHub (glamhub.m3gatr0n.com) -- Content platform with Astro, Directus, Umami analytics
• CoreStation (corestation.io) -- Fire department SaaS with 3 environments
• IDP (idp.pro) -- Identity provider platform
• Kelsey Farm Equipment (kelseyfarmequipment.com) -- E-commerce
• Aegis VPN (aegis.m3gatr0n.com) -- VPN service
• Bibliotheca -- Docker-based book management with TTS and search

Infrastructure Services

Authentik (SSO), Vault (secrets), Harbor (container registry), Devtron (CI/CD), Prometheus/Grafana/Loki (monitoring), Longhorn (distributed storage), Linkerd (service mesh), cert-manager (TLS), and more.

Research

The ULTRATHINK project -- 130+ spiking neural network and efficient attention research papers organized across 8 waves, with dedicated Claude Code agents per project.

Architecture at a Glance

Internet
  |
  v
Cloudflare (DNS + WAF)
  |
  v
cloudflared tunnel ──> Traefik Ingress (NodePort :32672)
                            |
                    ┌───────┼───────┐
                    v       v       v
                K3s Pods  K3s Pods  K3s Pods
              (megatron) (thing1)  (thing2)
                    |
                    v
              Docker Containers (Bibliotheca, Coolify)
              GPU Workloads (ULTRATHINK research)
              Tailscale Mesh (Mac Studio, MacBook, Lambda)

Documentation Sections

Audit Date

This documentation was generated from a comprehensive system audit performed on 2026-03-24. Resource usage figures (RAM, disk, GPU) reflect point-in-time snapshots and will drift over time.

Hardware

MEGATRON is a workstation-class server built for parallel compute workloads -- deep learning training, Kubernetes orchestration, and multi-tenant application hosting.

CPU

Memory

Note: 21 GiB of swap in use despite 206 GiB available RAM suggests processes were swapped out over the 37-day uptime and not yet paged back in. Consider a targeted swapoff -a && swapon -a cycle during a maintenance window if swap pressure becomes a concern.

GPUs

See GPUs for detailed GPU information including driver version and active processes.

System Load

With 128 logical threads, a load average of ~20 represents approximately 16% CPU utilization. The steady load across all three windows indicates consistent background workload, likely from K3s cluster operations and running pods.

Resource Usage Summary

CPU:  [####................] ~16% (load 20 / 128 threads)
RAM:  [###.................] ~17% (42 / 251 GiB)
SWAP: [##############......] ~68% (21 / 31 GiB)
GPU0: [##..................] ~11% VRAM (2.7 / 24 GB)
GPU1: [....................] ~0%  VRAM (14 MB / 24 GB)
ROOT: [###########.........] 54% (1.6T / 3.1T)
SMB:  [##################..] 90% (79T / 88T)

Top Processes by Resource Usage

At the time of audit, the primary consumers are:

• K3s server and agent processes -- Kubernetes control plane and kubelet
• containerd -- Container runtime for all K3s pods
• cloudflared -- Cloudflare tunnel daemon
• AlertManager -- At 203% CPU (known issue, see Current Issues)
• VNC server -- In restart loop consuming resources (known issue)

Peripheral Notes

• Storage: NVMe primary (3.1T), SMB network share (88T), NFS mounts (currently down)
• Network: Dual-stack networking with Cloudflare tunnel, Tailscale mesh, and direct SSH
• The system runs in a DEGRADED systemd state due to failed motd-news and wol-enable services (non-critical)

OS & Kernel

Operating System

Kernel Details

The system runs kernel 5.15, the HWE (Hardware Enablement) kernel for Ubuntu 20.04. This is a long-term support kernel with backported hardware support.

$ uname -a
Linux megatron 5.15.0-139-generic #149~20.04.1-Ubuntu SMP x86_64 GNU/Linux

Key kernel features in use:

• cgroups v2 -- Used by K3s for container resource isolation
• NVIDIA kernel modules -- GPU passthrough to containers and bare-metal workloads
• NVMe support -- Primary storage driver
• CIFS/SMB -- Network filesystem for the 88T share
• NFS -- Network filesystem (currently degraded, host unreachable)
• WireGuard -- Used by Tailscale for mesh networking

Uptime & Load

The load averages are stable across all three windows, indicating a consistent baseline workload. With 128 logical threads, this represents roughly 16% sustained utilization.

Systemd Service Summary

The system is in a DEGRADED state due to failed non-critical services.

Failed Services

Key Running Services

Service Counts (Approximate)

Timezone & Locale

Notable Configuration

• Rescue SSH runs on port 2222 with Nice -20 priority and OOM protection, ensuring emergency access even under extreme load. See Rescue SSH.
• Forgejo SSH runs on port 2223 for Git operations.
• The system has been running for 37 days without reboot, accumulating 21 GiB of swap usage despite ample free RAM.

Upgrade Considerations

Ubuntu 20.04 LTS reaches end of standard support in April 2025 (extended security maintenance available through 2030). Consider planning an upgrade to Ubuntu 22.04 LTS or 24.04 LTS during a maintenance window.

Storage

Disk Layout

Local Storage

The root filesystem is on an NVMe drive providing high-speed I/O for the OS, K3s cluster data, Docker images, and research workloads.

Network Storage

Warning: The SMB share is at 90% capacity (79T/88T). Only ~8.2T remains. Monitor closely and plan cleanup or expansion. See Current Issues.

Warning: NFS mounts to /mnt/cloudy and /mnt/cloudy-rw are unreachable -- the NFS host is down.

Storage Architecture

┌──────────────────────────────────────────────────────┐
│                     MEGATRON                          │
│                                                      │
│  ┌─────────────────────────┐  ┌───────────────────┐  │
│  │  NVMe (nvme0n1p5)       │  │  Longhorn PVs     │  │
│  │  3.1T root filesystem   │  │  Distributed K3s  │  │
│  │  ├── /                  │  │  storage across    │  │
│  │  ├── /home/lucas/       │  │  megatron, thing1, │  │
│  │  │   ├── Papers/ 234GB  │  │  thing2            │  │
│  │  │   ├── juke/ 60GB     │  └───────────────────┘  │
│  │  │   └── miniconda3/44GB│                         │
│  │  ├── /var/lib/rancher/  │  ┌───────────────────┐  │
│  │  │   (K3s data)         │  │  SMB //10.0.0.41  │  │
│  │  └── /var/lib/docker/   │  │  88T NAS share    │  │
│  └─────────────────────────┘  │  90% full         │  │
│                               └───────────────────┘  │
│  ┌─────────────────────────┐                         │
│  │  NFS (DOWN)             │                         │
│  │  /mnt/cloudy            │                         │
│  │  /mnt/cloudy-rw         │                         │
│  │  Host unreachable       │                         │
│  └─────────────────────────┘                         │
└──────────────────────────────────────────────────────┘

Top Directories by Size

Shared Storage Layout (Research)

The ULTRATHINK research projects use a shared storage structure:

/mnt/shared/
├── datasets/         # Shared datasets (ImageNet, CIFAR, etc.)
├── checkpoints/      # Model checkpoints per project
├── pretrained/       # Pretrained models (TORCH_HOME)
├── logs/             # TensorBoard, MLflow logs
└── results/          # Experiment results

Environment variables for each project:

export {PROJECT}_DATA=/mnt/shared/datasets
export {PROJECT}_CHECKPOINTS=/mnt/shared/checkpoints/{project}
export {PROJECT}_LOGS=/mnt/shared/logs/{project}
export TORCH_HOME=/mnt/shared/pretrained

Kubernetes Persistent Storage

K3s uses Longhorn as the distributed storage backend for persistent volumes. Longhorn replicates data across the three cluster nodes (megatron, thing1, thing2) for redundancy.

See Longhorn and Storage Classes for details.

Disk Usage Quick Check

# Check all mount points
df -h

# Check root filesystem usage
df -h /

# Top-level directory sizes
du -sh /home/lucas/*/ 2>/dev/null | sort -rh | head -20

# Check inode usage
df -i /

# Check NVMe health
sudo smartctl -a /dev/nvme0n1

# Check SMB share status
mount | grep cifs

Maintenance Recommendations

1. SMB share at 90% -- Audit large files, archive old data, or expand the NAS
2. NFS mounts down -- Investigate and restore the NFS host or remove stale mount entries from /etc/fstab
3. Root disk at 54% -- Healthy but growing; monitor ~/Papers/ (234 GB) and conda envs (44 GB)
4. Swap usage (21/31 GiB) -- Consider swapoff -a && swapon -a during a maintenance window to reclaim
5. Longhorn health -- Verify replica counts and volume health via the Longhorn UI

GPUs

MEGATRON has two NVIDIA GeForce RTX 3090 GPUs, primarily used for deep learning training (ULTRATHINK research) and inference workloads (Ollama LLM serving).

GPU Specifications

Current Status

GPU 0:  VRAM [##..................] 11%   Temp: 53C   Power:  29W / 350W
GPU 1:  VRAM [....................]  0%   Temp: 46C   Power:  17W / 350W

GPU 0 has a 2.7 GB allocation, likely from an Ollama model loaded in VRAM or a persistent CUDA context. GPU 1 is essentially idle.

Quick Commands

# Full GPU status
nvidia-smi

# Compact CSV output
nvidia-smi --query-gpu=index,name,memory.used,memory.total,temperature.gpu,power.draw \
  --format=csv

# Real-time monitoring (updates every second)
watch -n 1 nvidia-smi

# List GPU compute processes
nvidia-smi --query-compute-apps=pid,name,gpu_uuid,used_memory --format=csv

# Check driver and CUDA version
nvidia-smi | head -4

GPU Usage by Workload

Multi-GPU Training

For ULTRATHINK research, both GPUs can be used together:

# PyTorch DataParallel (simpler, less efficient)
CUDA_VISIBLE_DEVICES=0,1 python train.py

# PyTorch DistributedDataParallel (recommended for performance)
torchrun --nproc_per_node=2 train.py

# Use only a specific GPU
CUDA_VISIBLE_DEVICES=1 python inference.py

Kubernetes GPU Access

The cluster has a gpu-operator namespace, indicating the NVIDIA GPU Operator is deployed for Kubernetes GPU scheduling. Pods can request GPU resources:

resources:
  limits:
    nvidia.com/gpu: 1

The GPU Operator manages:

• NVIDIA driver lifecycle
• NVIDIA Container Toolkit
• Device plugin for Kubernetes
• GPU Feature Discovery
• DCGM Exporter (monitoring)

Thermal Notes

Both GPUs are running cool (46-53C) at near-idle power draw. Under full training load, expect:

Ensure adequate cooling and a sufficient PSU (recommended 1200W+ for dual 3090 under full load, accounting for the 280W TDP CPU).

Troubleshooting

# GPU not detected
lspci | grep -i nvidia
dmesg | grep -i nvidia

# Reset GPU (use with caution)
sudo nvidia-smi --gpu-reset -i 0

# Check for stuck processes
nvidia-smi --query-compute-apps=pid,name,used_memory --format=csv
kill -9 <stuck_pid>

# Verify CUDA installation
python -c "import torch; print(torch.cuda.is_available()); print(torch.cuda.device_count())"

Cluster Overview

MEGATRON runs a K3s Kubernetes cluster -- a lightweight, production-ready distribution of Kubernetes designed for resource-constrained and edge environments.

Cluster Summary

Architecture

                    ┌─────────────────────────────────────────────┐
                    │              K3s Cluster                    │
                    │                                             │
  Internet ───>     │  ┌─────────────────────────────┐           │
  Cloudflare ───>   │  │     megatron (control-plane) │           │
  cloudflared ───>  │  │     ─────────────────────────│           │
                    │  │  K3s Server (API, scheduler,  │           │
  :32672 ──────>    │  │    controller-manager, etcd)  │           │
  Traefik           │  │  Traefik Ingress Controller   │           │
                    │  │  cert-manager                 │           │
                    │  │  GPU Operator (2x RTX 3090)   │           │
                    │  │  Longhorn Manager              │           │
                    │  │  Application Pods              │           │
                    │  │  Age: 88 days                  │           │
                    │  └─────────────────────────────┘           │
                    │                                             │
                    │  ┌──────────────────┐  ┌──────────────────┐│
                    │  │  thing1 (worker)  │  │  thing2 (worker)  ││
                    │  │  ────────────────  │  │  ────────────────  ││
                    │  │  K3s Agent        │  │  K3s Agent        ││
                    │  │  Application Pods │  │  Application Pods ││
                    │  │  Longhorn Replica │  │  Longhorn Replica ││
                    │  │  Age: 11 hours    │  │  Age: 11 hours    ││
                    │  └──────────────────┘  └──────────────────┘│
                    └─────────────────────────────────────────────┘

Traffic Flow

Internet
    │
    v
Cloudflare DNS + WAF
    │
    v
cloudflared (tunnel 6d8885c4-...)
    │
    v
https://localhost:32672 (noTLSVerify)
    │
    v
Traefik Ingress Controller (kube-system)
    │
    ├──> IngressRoute (hostname-based routing)
    │        │
    │        ├──> stiq.us ──────────> stiq namespace pods
    │        ├──> glamhub.m3gatr0n ─> glamhub namespace pods
    │        ├──> corestation.io ───> corestation namespace pods
    │        ├──> idp.pro ──────────> idp namespace pods
    │        ├──> kelsey... ────────> kelsey namespace pods
    │        ├──> aegis.m3gatr0n ──> aegis namespace pods
    │        ├──> auth.m3gatr0n ───> authentik namespace pods
    │        ├──> registry.m3gatr0n > registry namespace pods
    │        ├──> grafana.m3gatr0n ─> monitoring namespace pods
    │        └──> platform.m3gatr0n > devtroncd namespace pods
    │
    v
Linkerd Service Mesh (mTLS between pods)

Key Components

Ingress: Traefik

Traefik runs as the default K3s ingress controller in kube-system. It terminates TLS (via cert-manager certificates) and routes traffic to backend services based on hostname matching. All external traffic arrives through the Cloudflare tunnel on NodePort 32672.

Storage: Longhorn

Longhorn provides distributed block storage with replication across nodes. It manages persistent volumes for stateful workloads like databases, registries, and secret stores.

Mesh: Linkerd

Linkerd provides mutual TLS between pods, traffic splitting, and observability. Recently deployed (11 hours old at audit time).

Certificates: cert-manager

cert-manager automates TLS certificate provisioning via Let's Encrypt. Currently configured with HTTP-01 challenge, which is broken behind the Cloudflare tunnel. Migration to DNS-01 is required. See TLS Certificates.

GPU: NVIDIA GPU Operator

The GPU Operator in the gpu-operator namespace manages NVIDIA drivers, container toolkit, and device plugin for GPU-accelerated pods.

Quick Commands

# Cluster status
kubectl get nodes -o wide

# All pods across all namespaces
kubectl get pods -A

# Namespace list with pod counts
kubectl get ns

# Cluster info
kubectl cluster-info

# K3s specific
k3s kubectl get nodes
systemctl status k3s

Nodes

The K3s cluster consists of three nodes: one control-plane/master node (megatron) and two worker nodes (thing1, thing2).

Node Summary

Note: thing1 and thing2 were recently joined or re-joined to the cluster (11 hours old at audit time). This is significantly newer than megatron's 88-day age, suggesting the workers were recently provisioned, re-imaged, or reconnected.

Node Details

megatron (Control Plane)

This node runs:

• K3s server (API server, scheduler, controller-manager, embedded etcd)
• Traefik ingress controller
• cert-manager
• Most application pods (as the primary node)
• GPU-accelerated workloads
• Docker daemon (for non-K3s containers)

thing1 (Worker)

thing2 (Worker)

Hardware details for thing1 and thing2 are not captured in the audit. Run kubectl describe node thing1 and kubectl describe node thing2 for full specs.

Node Architecture

┌──────────────────────────────────────────────────┐
│  megatron (control-plane)                        │
│  AMD Threadripper 3990X | 251GB RAM | 2x 3090    │
│  ┌──────────────────────────────────────────────┐│
│  │ K3s Server: API, scheduler, etcd, controller ││
│  │ Traefik, cert-manager, Longhorn mgr          ││
│  │ Application pods (stiq, glamhub, core, ...)  ││
│  │ GPU workloads, Docker containers             ││
│  └──────────────────────────────────────────────┘│
└──────────────────────────────────────────────────┘
         │                    │
         v                    v
┌────────────────┐   ┌────────────────┐
│ thing1 (worker)│   │ thing2 (worker)│
│ K3s Agent      │   │ K3s Agent      │
│ App pods       │   │ App pods       │
│ Longhorn       │   │ Longhorn       │
│ replicas       │   │ replicas       │
└────────────────┘   └────────────────┘

Node Management

# List nodes with details
kubectl get nodes -o wide

# Describe a specific node
kubectl describe node megatron

# Check node resource usage
kubectl top nodes

# Drain a node for maintenance
kubectl drain thing1 --ignore-daemonsets --delete-emptydir-data

# Uncordon after maintenance
kubectl uncordon thing1

# Check K3s agent status on worker
ssh thing1 'systemctl status k3s-agent'

# Join a new worker node
# On the new node:
curl -sfL https://get.k3s.io | K3S_URL=https://megatron:6443 \
  K3S_TOKEN=<token> sh -

Labels and Taints

To schedule GPU workloads only on megatron:

# Label GPU node
kubectl label node megatron gpu=true

# Taint non-GPU nodes
kubectl taint nodes thing1 gpu=false:NoSchedule
kubectl taint nodes thing2 gpu=false:NoSchedule

Pods requesting nvidia.com/gpu resources will automatically be scheduled on nodes with available GPUs through the NVIDIA device plugin.

Namespaces

The cluster has 31 namespaces spanning production applications, development environments, infrastructure services, and specialized workloads.

Namespace Inventory

Application Namespaces

Infrastructure Namespaces

Specialized Namespaces

Namespace Map

K3s Cluster (31 namespaces)
│
├── System (4)
│   ├── kube-system          (Traefik, CoreDNS, metrics)
│   ├── kube-node-lease      (node heartbeats)
│   ├── kube-public          (public config)
│   └── default              (unused)
│
├── Applications (13)
│   ├── stiq / stiq-dev / stiq-staging
│   ├── glamhub / glamhub-dev
│   ├── corestation / corestation-dev / corestation-staging
│   ├── idp / idp-dev
│   ├── kelsey
│   └── aegis / aegis-dev
│
├── Infrastructure (7)
│   ├── authentik            (SSO)
│   ├── vault                (secrets)
│   ├── registry             (Harbor)
│   ├── devtroncd            (CI/CD)
│   ├── monitoring           (Prometheus, Grafana, Loki)
│   ├── cert-manager         (TLS certs)
│   └── longhorn-system      (storage)
│
├── Platform Services (4)
│   ├── linkerd              (service mesh)
│   ├── argo                 (workflows)
│   ├── openfaas             (serverless)
│   └── sonarqube            (code quality)
│
└── Specialized (3)
    ├── ai-ml                (MLflow, Ollama)
    ├── gpu-operator         (NVIDIA)
    ├── jupyterhub           (notebooks)
    ├── data                 (data processing)
    └── docs                 (documentation)

Quick Commands

# List all namespaces
kubectl get namespaces

# Pods in a specific namespace
kubectl get pods -n stiq

# All resources in a namespace
kubectl get all -n glamhub

# Resource usage by namespace
kubectl top pods -n monitoring

# Create a new namespace
kubectl create namespace my-new-ns

# Set default namespace for kubectl
kubectl config set-context --current --namespace=stiq

Multi-Environment Pattern

Most applications follow a consistent multi-environment pattern:

Example for STIQ:

• stiq -> stiq.us, api.stiq.us, app.stiq.us, admin.stiq.us
• stiq-dev -> dev.stiq.us, dev-api.stiq.us, dev-app.stiq.us, dev-admin.stiq.us
• stiq-staging -> staging.stiq.us, staging-api.stiq.us, staging-app.stiq.us, staging-admin.stiq.us

Pods & Services

Auto-generated: 2026-03-24 14:19:42 CDT

This page is automatically generated from live cluster state.

Nodes

NAME       STATUS   ROLES                  AGE   VERSION        INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION       CONTAINER-RUNTIME
megatron   Ready    control-plane,master   88d   v1.33.6+k3s1   10.0.0.85     <none>        Ubuntu 20.04.6 LTS   5.15.0-139-generic   containerd://2.1.5-k3s1.33
thing1     Ready    <none>                 12h   v1.33.6+k3s1   10.0.0.92     <none>        Ubuntu 24.04.4 LTS   6.17.0-19-generic    containerd://2.1.5-k3s1.33
thing2     Ready    <none>                 12h   v1.33.6+k3s1   10.0.0.91     <none>        Ubuntu 24.04.4 LTS   6.17.0-19-generic    containerd://2.1.5-k3s1.33

Pods by Namespace

Services

Ingresses

Certificates

Storage Classes

Kubernetes storage classes define how persistent volumes are provisioned in the K3s cluster.

Available Storage Classes

local-path (K3s Default)

The K3s default storage class uses the local-path provisioner, which creates PersistentVolumes on the local filesystem of the node where the pod is scheduled.

• Pros: Simple, fast, no additional infrastructure
• Cons: No replication, data lost if node fails, pod-to-node affinity required
• Best for: Non-critical workloads, ephemeral data, development environments

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-pvc
spec:
  storageClassName: local-path
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi

Longhorn (Distributed)

Longhorn provides distributed block storage with replication across the three cluster nodes (megatron, thing1, thing2).

• Pros: Data replication (configurable replicas), snapshots, backups, web UI
• Cons: Higher resource overhead, network-dependent I/O, more complex
• Best for: Databases, stateful applications, anything requiring data durability

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-durable-pvc
spec:
  storageClassName: longhorn
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi

Quick Commands

# List storage classes
kubectl get storageclass

# List persistent volumes
kubectl get pv

# List persistent volume claims across all namespaces
kubectl get pvc -A

# Check Longhorn volumes
kubectl get volumes.longhorn.io -n longhorn-system

# Check Longhorn replicas
kubectl get replicas.longhorn.io -n longhorn-system

Choosing a Storage Class

TODO: expand with Longhorn backup configuration, snapshot policies, and volume health monitoring.

Application Index

All applications hosted on MEGATRON, including Kubernetes-based services and Docker containers.

Production Applications

Infrastructure Services

Docker-Only Containers

Domain Map

stiq.us ─────────────────────> stiq namespace
  ├── api.stiq.us
  ├── app.stiq.us
  ├── admin.stiq.us
  ├── dev*.stiq.us ──────────> stiq-dev namespace
  └── staging*.stiq.us ─────> stiq-staging namespace

corestation.io ──────────────> corestation namespace
  ├── api.corestation.io
  ├── app.corestation.io
  ├── admin.corestation.io
  ├── dev*.corestation.io ──> corestation-dev namespace
  └── staging*.corestation ─> corestation-staging namespace

idp.pro ─────────────────────> idp namespace
  ├── api.idp.pro
  ├── app.idp.pro
  └── dev*.idp.pro ─────────> idp-dev namespace

kelseyfarmequipment.com ─────> kelsey namespace
  ├── www.kelseyfarmequipment.com
  └── api.kelseyfarmequipment.com

m3gatr0n.com (infrastructure)
  ├── glamhub.m3gatr0n.com ─> glamhub namespace
  ├── auth.m3gatr0n.com ────> authentik namespace
  ├── aegis.m3gatr0n.com ───> aegis namespace
  ├── registry.m3gatr0n.com > registry namespace
  ├── platform.m3gatr0n.com > devtroncd namespace
  ├── grafana.m3gatr0n.com ─> monitoring namespace
  ├── prometheus.m3gatr0n.com > monitoring namespace
  └── alertmanager.m3gatr0n.com > monitoring namespace

Application Details

• STIQ -- SaaS platform with 3 environments
• GlamHub -- Content platform with Astro, Directus, and AI components
• CoreStation -- Fire department SaaS
• IDP -- Identity provider platform
• Kelsey Farm Equipment -- E-commerce
• Aegis VPN -- VPN service
• Bibliotheca -- Docker-based book management system

GlamHub

GlamHub is a content management and publishing platform running on the MEGATRON cluster. It features a modern frontend, headless CMS, analytics, workflow automation, and AI-powered features.

Overview

URLs

Architecture

                    ┌─────────────────────────────────────┐
                    │          glamhub namespace           │
                    │                                     │
  glamhub.m3gatr0n  │  ┌──────────┐    ┌──────────────┐  │
  ─────────────────>│  │ Astro 6  │───>│ Directus 11  │  │
                    │  │ Frontend │    │ Headless CMS │  │
                    │  └──────────┘    └──────┬───────┘  │
                    │                         │          │
  admin-glamhub     │  ┌──────────────────────┘          │
  ─────────────────>│  │                                 │
                    │  v                                  │
                    │  ┌──────────┐    ┌──────────────┐  │
                    │  │ Database │    │   Umami      │  │
                    │  │ (Postgres│    │  Analytics   │  │
                    │  │  or SQL) │    └──────────────┘  │
                    │  └──────────┘                      │
                    │                                     │
  n8n-glamhub       │  ┌──────────┐    ┌──────────────┐  │
  ─────────────────>│  │   N8N    │───>│   Ollama     │  │
                    │  │ Workflows│    │  LLM Engine  │  │
                    │  └──────────┘    └──────────────┘  │
                    │                                     │
                    │  ┌──────────────┐                   │
                    │  │   Qdrant     │                   │
                    │  │ Vector Store │                   │
                    │  └──────────────┘                   │
                    └─────────────────────────────────────┘

Component Details

Astro 6 (Frontend)

• Static site generation with server-side rendering
• Modern web framework for content-driven sites
• Connected to Directus for dynamic content

Directus 11 (Headless CMS)

• API-first content management
• Admin dashboard at admin-glamhub.m3gatr0n.com
• RESTful and GraphQL APIs
• Role-based access control

Umami (Analytics)

• Privacy-focused web analytics
• Lightweight alternative to Google Analytics
• Tracks page views, events, and user sessions

N8N (Workflow Automation)

• Visual workflow builder
• Connects to Ollama for AI-powered content workflows
• Scheduled tasks and event-driven automation
• Accessible at n8n-glamhub.m3gatr0n.com

Ollama (LLM Inference)

• Local LLM serving on port 11434
• Used by N8N for content generation, summarization, and processing
• Also available in the ai-ml namespace for other workloads
• GPU-accelerated inference using RTX 3090

Qdrant (Vector Database)

• Vector similarity search engine
• Stores embeddings for semantic search over content
• Supports RAG (Retrieval-Augmented Generation) workflows with Ollama

Environments

Quick Commands

# Check pods in glamhub namespace
kubectl get pods -n glamhub

# Check services
kubectl get svc -n glamhub

# View logs for Directus
kubectl logs -n glamhub -l app=directus --tail=100 -f

# Restart a component
kubectl rollout restart deployment/astro -n glamhub

STIQ

STIQ is a SaaS platform hosted on the MEGATRON cluster with full multi-environment support across production, development, and staging.

Overview

URLs

Production (stiq namespace)

Development (stiq-dev namespace)

Staging (stiq-staging namespace)

Architecture

                        stiq.us
                           │
                           v
              ┌────────────────────────┐
              │    Cloudflare Tunnel    │
              └───────────┬────────────┘
                          │
                          v
              ┌────────────────────────┐
              │   Traefik Ingress      │
              │   Host-based routing   │
              └───────────┬────────────┘
                          │
          ┌───────────────┼───────────────┐
          v               v               v
 ┌──────────────┐ ┌──────────────┐ ┌──────────────┐
 │  stiq (prod) │ │ stiq-dev     │ │ stiq-staging │
 │              │ │              │ │              │
 │ ┌──────────┐ │ │ ┌──────────┐ │ │ ┌──────────┐ │
 │ │   App    │ │ │ │   App    │ │ │ │   App    │ │
 │ │ Frontend │ │ │ │ Frontend │ │ │ │ Frontend │ │
 │ └──────────┘ │ │ └──────────┘ │ │ └──────────┘ │
 │ ┌──────────┐ │ │ ┌──────────┐ │ │ ┌──────────┐ │
 │ │   API    │ │ │ │   API    │ │ │ │   API    │ │
 │ │ Backend  │ │ │ │ Backend  │ │ │ │ Backend  │ │
 │ └──────────┘ │ │ └──────────┘ │ │ └──────────┘ │
 │ ┌──────────┐ │ │ ┌──────────┐ │ │ ┌──────────┐ │
 │ │  Admin   │ │ │ │  Admin   │ │ │ │  Admin   │ │
 │ │  Panel   │ │ │ │  Panel   │ │ │ │  Panel   │ │
 │ └──────────┘ │ │ └──────────┘ │ │ └──────────┘ │
 │ ┌──────────┐ │ │ ┌──────────┐ │ │ ┌──────────┐ │
 │ │ Database │ │ │ │ Database │ │ │ │ Database │ │
 │ └──────────┘ │ │ └──────────┘ │ │ └──────────┘ │
 └──────────────┘ └──────────────┘ └──────────────┘

Environment Promotion

Code changes flow through the environments:

Development (stiq-dev)
    │
    │  Manual promotion / PR merge
    v
Staging (stiq-staging)
    │
    │  Validation + approval
    v
Production (stiq)

Networking

All STIQ subdomains are routed through the Cloudflare tunnel using Pattern B:

• Hostname matched at Cloudflare
• Proxied to https://localhost:32672 (Traefik NodePort) with noTLSVerify
• Traefik routes to the correct namespace based on hostname

Quick Commands

# Check all STIQ pods across environments
kubectl get pods -n stiq
kubectl get pods -n stiq-dev
kubectl get pods -n stiq-staging

# View services
kubectl get svc -n stiq

# View ingress routes
kubectl get ingressroute -n stiq

# Tail API logs
kubectl logs -n stiq -l app=api --tail=100 -f

# Scale API replicas
kubectl scale deployment/api -n stiq --replicas=3

CoreStation

CoreStation is a SaaS platform designed for fire departments, running on the MEGATRON cluster with three environments.

Overview

URLs

Production (corestation namespace)

Development (corestation-dev namespace)

Staging (corestation-staging namespace)

Architecture

CoreStation follows the same multi-environment pattern as STIQ:

corestation.io
      │
      v
Cloudflare Tunnel ──> Traefik ──> corestation namespace
                                  ├── app (frontend)
                                  ├── api (backend)
                                  ├── admin (admin panel)
                                  └── database

Deployment

CoreStation has Kustomize-based deployment manifests:

/home/lucas/fire-department-saas/deploy/k8s/
├── base/           # Base manifests
├── overlays/
│   ├── dev/        # Development overrides
│   └── prod/       # Production overrides

Apply with:

# Development
kubectl apply -k /home/lucas/fire-department-saas/deploy/k8s/overlays/dev/

# Production
kubectl apply -k /home/lucas/fire-department-saas/deploy/k8s/overlays/prod/

Quick Commands

# Check pods across all environments
for ns in corestation corestation-dev corestation-staging; do
  echo "=== $ns ===" && kubectl get pods -n $ns
done

# View logs
kubectl logs -n corestation -l app=api --tail=100 -f

# Restart deployment
kubectl rollout restart deployment/api -n corestation

IDP

IDP is an identity provider platform hosted on MEGATRON with production and development environments.

Overview

URLs

Production (idp namespace)

Development (idp-dev namespace)

Helm Charts

IDP has a comprehensive set of Helm charts at /home/lucas/idp/infra/charts/:

Quick Commands

kubectl get pods -n idp
kubectl get pods -n idp-dev
kubectl get svc -n idp
helm list -n idp

TODO: expand with architecture diagram, deployment procedures, and database details.

Kelsey Farm Equipment

Kelsey Farm Equipment is an e-commerce platform for farm and irrigation equipment.

Overview

URLs

Deployment

Kelsey has both Helm and Kustomize deployment options:

Kustomize Overlays

/home/lucas/irrigation-parts/deploy/k8s/
├── base/
├── overlays/
│   ├── demo/
│   ├── dev/
│   ├── qa/
│   └── prod/

# Deploy to dev
kubectl apply -k /home/lucas/irrigation-parts/deploy/k8s/overlays/dev/

# Deploy to prod
kubectl apply -k /home/lucas/irrigation-parts/deploy/k8s/overlays/prod/

Quick Commands

kubectl get pods -n kelsey
kubectl get svc -n kelsey

TODO: expand with architecture diagram, product catalog details, and payment integration.

Bibliotheca

Bibliotheca is a Docker-based book management system running outside of Kubernetes. It provides book discovery, download, categorization, text-to-speech conversion, and full-text search.

Overview

Container Inventory

Known Issue: bibliotheca-categorizer is continuously restarting. See Current Issues.

Architecture

┌──────────────────────────────────────────────────────┐
│                   Bibliotheca Stack                    │
│                  (Docker Compose)                      │
│                                                      │
│  ┌──────────────┐     ┌────────────────────────────┐ │
│  │  nginx :8088 │────>│  bibliotheca-web           │ │
│  │  (reverse    │     │  Book management UI        │ │
│  │   proxy)     │     └────────────────────────────┘ │
│  └──────────────┘                                    │
│                                                      │
│  ┌──────────────────────────────────────────────────┐│
│  │  bibliotheca-discovery                           ││
│  │  Finds books, fetches metadata                   ││
│  └───────────────────────┬──────────────────────────┘│
│                          │                           │
│                          v                           │
│  ┌──────────────────────────────────────────────────┐│
│  │  bibliotheca-vpn (Gluetun)                       ││
│  │  ┌────────────────────────────────────────────┐  ││
│  │  │  bibliotheca-torrent (qBittorrent) :8080   │  ││
│  │  │  Downloads via VPN tunnel                   │  ││
│  │  └────────────────────────────────────────────┘  ││
│  └──────────────────────────────────────────────────┘│
│                                                      │
│  ┌──────────────────┐  ┌──────────────────────────┐  │
│  │ bibliotheca-tts  │  │ bibliotheca-categorizer  │  │
│  │ Text-to-Speech   │  │ Book categorization      │  │
│  │ (NVIDIA GPU)     │  │ STATUS: RESTARTING       │  │
│  └──────────────────┘  └──────────────────────────┘  │
│                                                      │
│  External: Meilisearch :7700 (full-text search)      │
└──────────────────────────────────────────────────────┘

Network Topology

The torrent container runs inside the Gluetun VPN container's network namespace, ensuring all download traffic is routed through the VPN:

Internet <──> Gluetun VPN <──> qBittorrent
                  │
                  └── Port 6881 (torrent)
                  └── Port 8080 (qBittorrent web UI)

Services

qBittorrent (Torrent Client)

• Web UI: http://localhost:8080
• Image: linuxserver/qbittorrent
• Runs inside Gluetun VPN network for privacy
• Port 6881 for BitTorrent protocol

Gluetun (VPN)

• Image: qmcgaw/gluetun
• Health-checked container
• Provides VPN network namespace for qBittorrent
• Status: Healthy

Text-to-Speech (TTS)

• Custom image with NVIDIA GPU support
• Converts ebooks to audio format
• Uses RTX 3090 for accelerated inference
• Running for 2+ weeks

Meilisearch (Search)

• Port: 7700
• Image: getmeili/meilisearch
• Full-text search engine for book metadata and content
• Running for 4 weeks

Discovery

• Custom service for finding and fetching book metadata
• Running for 2+ weeks

Categorizer

• Automatically categorizes books by genre/topic
• Currently broken -- continuously restarting

Management

# Check container status
docker ps --filter "name=bibliotheca"

# View logs for the failing categorizer
docker logs bibliotheca-categorizer --tail=50

# Restart the entire stack
cd /home/lucas/bibliotheca/ && docker compose restart

# Restart a specific container
docker restart bibliotheca-torrent

# Access qBittorrent UI
# Open http://localhost:8080 in browser

# Check Gluetun VPN health
docker inspect bibliotheca-vpn | jq '.[0].State.Health'

Aegis VPN

Aegis is a VPN service hosted on MEGATRON with multi-cloud infrastructure managed by Terraform.

Overview

URLs

Multi-Cloud Infrastructure

Aegis VPN has Terraform configurations for deploying VPN endpoints across multiple cloud providers:

Kustomize Overlays

/home/lucas/vpn/k8s/
├── base/
├── overlays/
│   ├── megatron/
│   ├── dev/
│   └── prod/

Quick Commands

kubectl get pods -n aegis
kubectl get pods -n aegis-dev

TODO: expand with VPN architecture, client configuration, and cloud deployment procedures.

Other Applications

Additional applications and services not covered by individual pages.

Coolify

Coolify is a self-hosted PaaS (Platform as a Service) for deploying applications. It runs as a Docker container alongside its MariaDB database and Redis cache.

Forgejo (Git)

Self-hosted Git forge for source code management.

# Git clone via SSH
git clone ssh://git@megatron:2223/user/repo.git

Meilisearch

Full-text search engine used by the Bibliotheca stack for book search functionality.

SonarQube

TODO: expand with SonarQube configuration, project setup, and quality gate definitions.

MinIO / S3

TODO: expand with bucket configuration and access credentials management.

Authentik (Auth)

Authentik is the SSO/OIDC identity provider for the MEGATRON cluster, providing centralized authentication for all services.

Overview

Components

Access

• Admin UI: https://auth.m3gatr0n.com/if/admin/
• User Portal: https://auth.m3gatr0n.com/

Capabilities

• OIDC/OAuth2 provider for application SSO
• SAML 2.0 for enterprise integrations
• LDAP outpost for legacy applications
• Flows and stages for customizable authentication
• MFA support (TOTP, WebAuthn, SMS)

kubectl get pods -n authentik
kubectl logs -n authentik -l app.kubernetes.io/name=authentik-server --tail=50

TODO: expand with OIDC client configurations, connected applications, and flow details.

HashiCorp Vault

Vault provides centralized secrets management for the MEGATRON cluster.

Overview

Key Concerns

• Vault was recently restarted (only 3h51m uptime at audit). After a restart, Vault needs to be unsealed before it can serve secrets.
• Verify seal status: kubectl exec -n vault vault-0 -- vault status

Quick Commands

kubectl get pods -n vault
kubectl exec -n vault vault-0 -- vault status
kubectl exec -n vault vault-0 -- vault secrets list

TODO: expand with secret engine configuration, policies, Kubernetes auth method, and backup procedures.

Harbor (Registry)

Harbor is a container image registry hosted on the MEGATRON cluster.

Overview

Known Issue

The harbor-tls certificate has been NOT READY for 20 days due to the HTTP-01 ACME challenge failure behind the Cloudflare tunnel. See TLS Certificates and Certificate Renewal.

Usage

# Login (may fail due to TLS issue)
docker login registry.m3gatr0n.com

# Push an image
docker tag myapp:latest registry.m3gatr0n.com/project/myapp:latest
docker push registry.m3gatr0n.com/project/myapp:latest

# Check pods
kubectl get pods -n registry

TODO: expand with project configuration, robot accounts, replication rules, and vulnerability scanning setup.

Devtron (CI/CD)

Devtron is a CI/CD and GitOps platform running on the MEGATRON cluster.

Overview

Known Issue

The devtron-tls certificate has been NOT READY for 20 days. See TLS Certificates.

Components

kubectl get pods -n devtroncd

TODO: expand with pipeline configurations, connected repositories, and deployment workflows.

Prometheus & Grafana

The monitoring stack provides metrics collection, visualization, and alerting for the MEGATRON cluster.

Overview

Prometheus

Prometheus scrapes metrics from Kubernetes pods, nodes, and services. Key metrics include:

• Node CPU, memory, disk, network
• Pod resource usage
• K3s API server latency
• Application-specific metrics

Grafana

Grafana provides dashboards for visualizing Prometheus metrics. Access at https://grafana.m3gatr0n.com.

AlertManager

AlertManager routes alerts from Prometheus to notification channels.

Known Issue: AlertManager is consuming 203% CPU (24K+ hours runtime). See Current Issues.

kubectl get pods -n monitoring
kubectl top pods -n monitoring

TODO: expand with dashboard inventory, alert rules, notification channels, and retention configuration.

Loki (Logs)

Loki is a log aggregation system in the monitoring namespace, designed for efficient log querying with Grafana.

Overview

How It Works

Loki collects logs from all pods via Promtail (or Grafana Agent) and indexes them by labels rather than full-text. Logs can be queried through Grafana using LogQL.

kubectl get pods -n monitoring -l app=loki

TODO: expand with retention policies, storage backend, LogQL query examples, and Promtail/Agent configuration.

AlertManager

AlertManager handles alert routing, deduplication, and notification from Prometheus.

Overview

Known Issue

AlertManager is consuming 203% CPU with 24,000+ accumulated CPU hours. This indicates a memory leak, alert routing loop, or excessive alert volume. See Current Issues.

Immediate remediation:

kubectl rollout restart statefulset/alertmanager -n monitoring

Quick Commands

kubectl get pods -n monitoring -l app.kubernetes.io/name=alertmanager
kubectl top pod -n monitoring -l app.kubernetes.io/name=alertmanager
kubectl logs -n monitoring -l app.kubernetes.io/name=alertmanager --tail=100

TODO: expand with alert routing configuration, receiver setup, silence management, and integration with Slack (#ultrathink-agents channel).

Longhorn (Storage)

Longhorn provides distributed block storage for the K3s cluster, replicating data across all three nodes for redundancy.

Overview

Features

• Automatic replication across nodes
• Snapshot and backup support
• Volume expansion
• Web UI for management
• CSI driver for Kubernetes PV provisioning

Quick Commands

kubectl get pods -n longhorn-system
kubectl get volumes.longhorn.io -n longhorn-system
kubectl get replicas.longhorn.io -n longhorn-system

TODO: expand with volume inventory, backup configuration, snapshot policies, and disaster recovery procedures.

Linkerd (Mesh)

Linkerd is a lightweight service mesh providing mutual TLS, traffic metrics, and reliability features for the K3s cluster.

Overview

Features

• Automatic mutual TLS (mTLS) between meshed pods
• Per-route metrics and latency tracking
• Traffic splitting for canary deployments
• Retries and timeouts

Status

Linkerd was recently deployed/redeployed (11 hours old at audit time). Monitor for stability issues.

kubectl get pods -n linkerd
# If linkerd CLI is installed:
linkerd check
linkerd stat deployment -A

TODO: expand with injection status per namespace, traffic policies, and dashboard access.

MLflow

MLflow provides experiment tracking for ULTRATHINK research projects on the MEGATRON cluster.

Overview

Usage with ULTRATHINK

Each research project logs experiments to MLflow:

import mlflow

mlflow.set_tracking_uri("http://mlflow.ai-ml.svc.cluster.local:5000")
mlflow.set_experiment("rosa-vit")

with mlflow.start_run():
    mlflow.log_param("learning_rate", 0.001)
    mlflow.log_metric("accuracy", 0.95)
    mlflow.log_artifact("model.pt")

kubectl get pods -n ai-ml -l app=mlflow

TODO: expand with experiment inventory, model registry contents, and artifact storage configuration.

Ollama

Ollama provides local LLM inference on the MEGATRON cluster, using GPU acceleration for fast generation.

Overview

Deployments

Ollama runs in two namespaces:

Usage

# From within the cluster
curl http://ollama.ai-ml.svc.cluster.local:11434/api/generate \
  -d '{"model": "llama3", "prompt": "Hello"}'

# List loaded models
curl http://ollama.ai-ml.svc.cluster.local:11434/api/tags

kubectl get pods -n ai-ml -l app=ollama

TODO: expand with model inventory, VRAM requirements per model, and performance benchmarks.

JupyterHub

JupyterHub provides multi-user Jupyter notebook server for interactive research on the MEGATRON cluster.

Overview

Features

• Multi-user notebook environment
• GPU access for research notebooks
• Persistent storage per user
• Support for Python, R, and Julia kernels

kubectl get pods -n jupyterhub

TODO: expand with user management, GPU allocation, storage configuration, and custom image details.

OpenFaaS

OpenFaaS provides serverless function execution on the K3s cluster.

Overview

Features

• Deploy functions as containers
• Auto-scaling based on demand
• Multiple language templates
• REST API for invocation

kubectl get pods -n openfaas
# If faas-cli installed:
faas-cli list

TODO: expand with deployed functions, scaling configuration, and integration points.

Argo Workflows

Argo Workflows provides Kubernetes-native workflow orchestration for data pipelines and ML workflows.

Overview

Use Cases

• ML training pipelines
• Data processing workflows
• CI/CD pipeline steps
• Batch job orchestration

kubectl get pods -n argo
# If argo CLI installed:
argo list -n argo

TODO: expand with workflow templates, cron workflows, artifact configuration, and integration with MLflow.

cert-manager

cert-manager automates TLS certificate provisioning and renewal in the K3s cluster.

Overview

Known Issue

The HTTP-01 ACME challenge solver does not work behind the Cloudflare tunnel. This is the root cause of multiple NOT READY certificates. See TLS Certificates and Certificate Renewal for the DNS-01 migration plan.

Components

The high restart counts (22-28) are accumulated over 37 days of uptime.

Issuers

Quick Commands

kubectl get pods -n cert-manager
kubectl get clusterissuer
kubectl get certificates -A
kubectl get challenges -A
kubectl logs -n cert-manager deployment/cert-manager --tail=100

TODO: expand with full issuer YAML, wildcard certificate setup, and DNS-01 configuration after migration.

Coolify

Coolify is a self-hosted PaaS (Platform as a Service) running as a Docker container on MEGATRON.

Overview

Containers

Access

Coolify is available at http://localhost:8000 from MEGATRON.

docker ps --filter "name=coolify"
docker logs coolify --tail=50

TODO: expand with deployed applications, team configuration, and Git integration.

Docker Containers Overview

In addition to the K3s Kubernetes cluster, MEGATRON runs several workloads directly on Docker. These are services that either predate the K3s setup, require direct hardware access (GPU), or are managed by external tools (Coolify).

Docker vs K3s

Container Groups

Bibliotheca Stack (7 containers)

Book management system with torrent client, VPN, TTS, and search. See Bibliotheca.

Coolify Stack (3 containers)

Self-hosted PaaS platform with MariaDB and Redis. See Coolify.

Meilisearch (1 container)

Full-text search engine used by Bibliotheca.

Quick Commands

# List all containers
docker ps -a

# List running containers with resource usage
docker stats --no-stream

# Check for problems
docker ps -a --filter "status=restarting"
docker ps -a --filter "status=exited"

# View Docker disk usage
docker system df

Container Inventory

Auto-generated: 2026-03-24 14:19:43 CDT

Running Containers

Stopped/Failed Containers

Docker Networks

NETWORK ID     NAME                  DRIVER    SCOPE
27f8a63320e1   bibliotheca_default   bridge    local
6bfbd9cb7abd   bridge                bridge    local
b661cb5621cd   coolify               bridge    local
5f611afa1878   host                  host      local
0c7471e13bbb   none                  null      local

Docker Volumes

DRIVER    VOLUME NAME
local     7a5fe936b9a7dea50b570ed2d98e3aeb059d4a37e48a7f78ce6ecf7a2c003721
local     171145ba1b89149e3b254bb9480fb38a7de1135d9ee6c08689bd1c8542233b2c
local     bibliotheca_converter_cache
local     bibliotheca_db_data
local     bibliotheca_forgejo_data
local     bibliotheca_gluetun_data
local     bibliotheca_redis_data
local     bibliotheca_search_data
local     bibliotheca_torrent_config
local     coolify-db
local     coolify-redis

Network Architecture

MEGATRON uses a layered networking architecture combining Cloudflare tunnels for public ingress, Tailscale for private mesh networking, and Traefik for Kubernetes-level routing.

Architecture Overview

┌─────────────────────────────────────────────────────────────────┐
│                        INTERNET                                  │
└──────────────────────────┬──────────────────────────────────────┘
                           │
                           v
┌─────────────────────────────────────────────────────────────────┐
│                   CLOUDFLARE (DNS + WAF)                         │
│                                                                  │
│  Domains: stiq.us, corestation.io, idp.pro,                     │
│           kelseyfarmequipment.com, m3gatr0n.com                  │
│                                                                  │
│  Tunnel: 6d8885c4-97d9-43fd-8426-78a00c2834f1                   │
│  Routes: 49 hostname-based rules                                 │
│  Pattern: hostname -> https://localhost:32672 (noTLSVerify)      │
└──────────────────────────┬──────────────────────────────────────┘
                           │
                           v
┌─────────────────────────────────────────────────────────────────┐
│                   MEGATRON (Host)                                 │
│                                                                  │
│  cloudflared ──────> localhost:32672                              │
│                           │                                      │
│                           v                                      │
│  ┌────────────────────────────────────────────────────────────┐  │
│  │              Traefik Ingress Controller                     │  │
│  │              (kube-system, NodePort :32672)                 │  │
│  │                                                            │  │
│  │  Host-based routing:                                       │  │
│  │  *.stiq.us ──────────────> stiq / stiq-dev / stiq-staging  │  │
│  │  *.corestation.io ───────> corestation / *-dev / *-staging │  │
│  │  *.idp.pro ──────────────> idp / idp-dev                   │  │
│  │  *.m3gatr0n.com ─────────> glamhub, auth, monitoring, etc  │  │
│  │  kelseyfarmequipment.com > kelsey                          │  │
│  └────────────────────────────────────────────────────────────┘  │
│                           │                                      │
│                           v                                      │
│  ┌────────────────────────────────────────────────────────────┐  │
│  │              K3s Pod Network (ClusterIP services)           │  │
│  │              Linkerd Service Mesh (mTLS)                    │  │
│  └────────────────────────────────────────────────────────────┘  │
│                                                                  │
│  ┌────────────────────────────────────────────────────────────┐  │
│  │              Tailscale Mesh (WireGuard)                     │  │
│  │  megatron: 100.70.25.47                                    │  │
│  │  mac-studio: 100.87.201.64                                 │  │
│  │  macbook-m4: 100.91.253.88                                 │  │
│  │  lambda: 100.108.49.109                                    │  │
│  └────────────────────────────────────────────────────────────┘  │
│                                                                  │
│  ┌────────────────────────────────────────────────────────────┐  │
│  │              Docker Bridge Network                          │  │
│  │  Bibliotheca containers, Coolify                            │  │
│  │  localhost:8080 (qBittorrent), :8088 (nginx), :8000        │  │
│  └────────────────────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────────────┘

Port Map

External Ports (Internet-facing via Cloudflare)

All public traffic enters through the Cloudflare tunnel -- no ports are directly exposed to the internet.

Host Ports

Kubernetes Service Ports (ClusterIP, internal only)

Network Layers

Layer 1: Cloudflare (Public Ingress)

All public traffic flows through Cloudflare, which provides DNS, DDoS protection, WAF, and the tunnel endpoint. See Cloudflare Tunnels.

Layer 2: Traefik (Kubernetes Ingress)

Traefik receives tunnel traffic on NodePort 32672 and routes based on hostname to the appropriate Kubernetes service. See Traefik Ingress.

Layer 3: Linkerd (Service Mesh)

Linkerd provides mutual TLS between pods, traffic metrics, and reliability features. See Linkerd.

Layer 4: Tailscale (Private Mesh)

Tailscale connects MEGATRON to other machines (Mac Studio, MacBook, Lambda) via WireGuard-based mesh VPN. See Tailscale.

Layer 5: Docker Bridge (Container Networking)

Non-K3s containers (Bibliotheca, Coolify) use Docker bridge networking with port mappings to localhost.

DNS Flow

User browser
    │
    v
DNS query: app.stiq.us
    │
    v
Cloudflare DNS ──> resolves to Cloudflare edge
    │
    v
Cloudflare tunnel routes to MEGATRON
    │
    v
cloudflared ──> https://localhost:32672
    │
    v
Traefik matches Host header ──> routes to K3s service
    │
    v
K3s service ──> Pod (via kube-proxy/iptables)

Key Networking Issues

1. cert-manager HTTP-01 broken -- Cannot complete ACME challenges behind Cloudflare tunnel. Must migrate to DNS-01 solver.
2. Harbor TLS NOT READY (20 days) -- Certificate pending due to HTTP-01 failure.
3. Devtron TLS NOT READY (20 days) -- Same root cause.
4. Monitoring wildcard cert NOT READY -- Same root cause.

See TLS Certificates for full details.

Cloudflare Tunnels

Auto-generated: 2026-03-24 14:19:43 CDT

Tunnel Configuration

• Tunnel ID: 6d8885c4-97d9-43fd-8426-78a00c2834f1
• Config: /etc/cloudflared/config.yml
• Credentials: /etc/cloudflared/6d8885c4-97d9-43fd-8426-78a00c2834f1.json
• Service: cloudflared.service (systemd)
• Target: https://localhost:32672 (Traefik NodePort)
• Management: mgt route CLI at /home/lucas/paas/bin/

Traffic Flow

Internet → Cloudflare Edge → cloudflared tunnel → https://localhost:32672 → Traefik → k8s Service

Route Table

Routes by Domain

m3gatr0n.com

• dev-aegis.m3gatr0n.com
• aegis.m3gatr0n.com
• api.aegis.m3gatr0n.com
• vpn.m3gatr0n.com
• alertmanager.m3gatr0n.com
• auth.m3gatr0n.com
• grafana.m3gatr0n.com
• s3.m3gatr0n.com
• minio.m3gatr0n.com
• platform.m3gatr0n.com
• prometheus.m3gatr0n.com
• registry.m3gatr0n.com
• glamhub.m3gatr0n.com
• admin-glamhub.m3gatr0n.com
• cms-glamhub.m3gatr0n.com
• n8n-glamhub.m3gatr0n.com
• docs.m3gatr0n.com
• wiki.m3gatr0n.com
• git.m3gatr0n.com

stiq.us

• dev.stiq.us
• dev-api.stiq.us
• dev-app.stiq.us
• dev-admin.stiq.us
• grafana.stiq.us
• api.stiq.us
• app.stiq.us
• admin.stiq.us
• staging.stiq.us
• staging-api.stiq.us
• staging-app.stiq.us
• staging-admin.stiq.us

corestation.io

• dev.corestation.io
• dev-api.corestation.io
• api.corestation.io
• app.corestation.io
• admin.corestation.io
• staging.corestation.io
• staging-api.corestation.io

idp.pro

• dev.idp.pro
• api.idp.pro
• app.idp.pro

kelseyfarmequipment.com

• dev.kelseyfarmequipment.com
• www.kelseyfarmequipment.com
• api.kelseyfarmequipment.com

Catch-All Rule

Unmatched requests return HTTP 404.

Tailscale

Tailscale provides a WireGuard-based mesh VPN connecting MEGATRON to other machines in the network. It enables secure, direct SSH access and inter-machine communication without exposing ports to the internet.

Tailscale Mesh

Network Diagram

                    ┌─────────────────────┐
                    │     Tailscale       │
                    │   Coordination      │
                    │     Server          │
                    └─────────┬───────────┘
                              │ (control plane only,
                              │  data flows peer-to-peer)
         ┌────────────────────┼────────────────────┐
         │                    │                    │
         v                    v                    v
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│    MEGATRON     │ │   Mac Studio    │ │  MacBook M4     │
│  100.70.25.47   │ │ 100.87.201.64   │ │ 100.91.253.88   │
│                 │ │                 │ │                 │
│  K3s cluster    │<────────────────>│  Development    │
│  GPU training   │ │  Inference      │ │  Quick tests    │
│  Prod hosting   │ │  Visualization  │ │                 │
└────────┬────────┘ │  Large models   │ └─────────────────┘
         │          └─────────────────┘
         │
         v
┌─────────────────┐
│  Lambda Laptop  │
│ 100.108.49.109  │
│                 │
│  Ablations      │
│  Single-GPU     │
└─────────────────┘

SSH via Tailscale

SSH config entries for Tailscale-based access (from ~/.ssh/config):

# SSH to Mac Studio via Tailscale
ssh mac-studio

# SSH to MacBook
ssh macbook-m4

# SSH to Lambda
ssh tensorbook

Use Cases

Research Workload Distribution

Data Transfer

# Copy dataset from MEGATRON to Mac Studio
scp -r /mnt/shared/datasets/cifar10 mac-studio:~/datasets/

# Rsync checkpoints
rsync -avz /mnt/shared/checkpoints/rosa/ mac-studio:~/checkpoints/rosa/

# Copy results back
scp mac-studio:~/results/inference.json /mnt/shared/results/

Tailscale Management

# Check Tailscale status
tailscale status

# Check connection to a specific peer
tailscale ping mac-studio

# View Tailscale IP
tailscale ip

# Check network diagnostics
tailscale netcheck

# Enable/disable Tailscale
sudo tailscale up
sudo tailscale down

Tailscale on K3s

Tailscale runs as a system service (tailscaled.service) and provides an overlay network independent of the K3s cluster network. Pods within K3s do not use Tailscale directly -- Tailscale is used for machine-to-machine communication and SSH access.

Traefik Ingress

Traefik is the default ingress controller for K3s, handling all HTTP/HTTPS traffic routing to Kubernetes services.

Overview

How Traffic Reaches Traefik

cloudflared (Cloudflare tunnel)
    |
    v
https://localhost:32672 (noTLSVerify)
    |
    v
Traefik (kube-system)
    |
    v
IngressRoute rules match Host header
    |
    v
Backend K8s Service -> Pod

Routing

Traefik uses IngressRoute CRDs (Traefik's native routing) for hostname-based routing. Each application defines its own IngressRoute in its namespace.

# List all IngressRoutes
kubectl get ingressroute -A

# Describe a specific route
kubectl describe ingressroute <name> -n <namespace>

# Check Traefik pods
kubectl get pods -n kube-system -l app.kubernetes.io/name=traefik

# Traefik dashboard (if enabled)
kubectl port-forward -n kube-system svc/traefik-dashboard 9000:9000

TODO: expand with middleware configuration, rate limiting, TLS configuration, and dashboard access.

DNS & Domains

All external domains are managed through Cloudflare DNS, with traffic routed through the Cloudflare tunnel.

Domain Inventory

Subdomain Usage

m3gatr0n.com (Infrastructure)

Internal DNS

Kubernetes internal DNS is handled by CoreDNS in the kube-system (or dns) namespace. Services are reachable at:

<service>.<namespace>.svc.cluster.local

TODO: expand with DNS record details, TTL settings, and external-dns configuration.

TLS Certificates

TLS certificates are managed by cert-manager in the Kubernetes cluster. There are currently several certificates in a broken state due to the HTTP-01 challenge solver being incompatible with the Cloudflare tunnel architecture.

Certificate Inventory

Issuers

The HTTP-01 Problem

The current cert-manager configuration uses HTTP-01 ACME challenges. This solver requires Let's Encrypt to make an HTTP request to http://<domain>/.well-known/acme-challenge/<token> to verify domain ownership.

This does not work behind a Cloudflare tunnel because:

Let's Encrypt ──> Cloudflare ──> tunnel ──> Traefik ──> cert-manager solver pod
                       │
                       └── Cloudflare may cache, redirect to HTTPS,
                           or otherwise interfere with the challenge

The Cloudflare tunnel and its "Always Use HTTPS" setting redirects HTTP challenges to HTTPS, causing validation to fail.

Required Fix: DNS-01 Solver

The solution is to migrate from HTTP-01 to DNS-01 challenge solver. DNS-01 verifies domain ownership by creating a TXT record in DNS, which works regardless of tunnel configuration.

Migration Steps

1. Create a Cloudflare API token with DNS edit permissions
2. Store the token as a Kubernetes secret
3. Update the ClusterIssuer to use DNS-01 with Cloudflare
4. Re-request failed certificates

See Certificate Renewal Runbook for the full procedure.

Target Configuration

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: [email protected]
    privateKeySecretRef:
      name: letsencrypt-prod-key
    solvers:
    - dns01:
        cloudflare:
          apiTokenSecretRef:
            name: cloudflare-api-token
            key: api-token

Affected Services

The following services have broken TLS due to the HTTP-01 issue:

Working Certificates

These certificates were issued before the problem manifested or use a different mechanism:

Quick Commands

# List all certificates
kubectl get certificates -A

# Check certificate details
kubectl describe certificate harbor-tls -n registry

# Check cert-manager logs
kubectl logs -n cert-manager -l app=cert-manager --tail=100

# Check issuer status
kubectl get clusterissuer

# Force certificate re-request (delete and recreate)
kubectl delete certificate harbor-tls -n registry
# Then re-apply the Certificate resource

# Check ACME challenge status
kubectl get challenges -A
kubectl describe challenge -A

IaC Overview

MEGATRON's infrastructure is managed through a combination of Infrastructure as Code tools: Terraform for cloud resources, Helm for Kubernetes package management, Kustomize for environment-specific overlays, and Docker Compose for non-K3s workloads.

Tool Inventory

Coverage Map

┌─────────────────────────────────────────────────────────┐
│                   IaC Coverage                           │
│                                                         │
│  ┌──────────────────────┐  ┌─────────────────────────┐  │
│  │  TERRAFORM            │  │  HELM                    │  │
│  │                      │  │                         │  │
│  │  Aegis VPN infra:    │  │  IDP platform:           │  │
│  │  - AWS               │  │  - core-platform        │  │
│  │  - Azure             │  │  - argocd               │  │
│  │  - GCP               │  │  - external-dns         │  │
│  │  - DigitalOcean      │  │  - cnpg                 │  │
│  │  - Hetzner           │  │  - weaviate             │  │
│  │  - Vultr             │  │  - grafana              │  │
│  │  - Oracle            │  │                         │  │
│  └──────────────────────┘  │  Aegis VPN:             │  │
│                            │  - aegis chart          │  │
│  ┌──────────────────────┐  │                         │  │
│  │  KUSTOMIZE            │  │  Kelsey Equipment:      │  │
│  │                      │  │  - irrigation-parts     │  │
│  │  CoreStation:        │  └─────────────────────────┘  │
│  │  - dev overlay       │                               │
│  │  - prod overlay      │  ┌─────────────────────────┐  │
│  │                      │  │  DOCKER COMPOSE          │  │
│  │  Aegis VPN:          │  │                         │  │
│  │  - base              │  │  ULTRATHINK infra:      │  │
│  │  - megatron overlay  │  │  - JupyterLab           │  │
│  │  - dev overlay       │  │  - MLflow               │  │
│  │  - prod overlay      │  │  - TensorBoard          │  │
│  │                      │  │                         │  │
│  │  Kelsey Equipment:   │  │  Bibliotheca stack      │  │
│  │  - demo              │  │  Gene editing env       │  │
│  │  - dev               │  └─────────────────────────┘  │
│  │  - qa                │                               │
│  │  - prod              │                               │
│  └──────────────────────┘                               │
└─────────────────────────────────────────────────────────┘

Coverage Gaps

Several deployed services lack IaC definitions and were likely deployed via manual kubectl apply or UI-based tools:

Links

• Terraform -- Cloud infrastructure definitions
• Helm Charts -- Kubernetes package management
• Kustomize -- Environment overlay system
• Docker Compose Files -- Local container orchestration

Terraform

Terraform manages cloud infrastructure for the Aegis VPN service, provisioning VPN endpoints across seven cloud providers.

Location

/home/lucas/vpn/infrastructure/terraform/

Provider Inventory

Usage

cd /home/lucas/vpn/infrastructure/terraform/<provider>/

# Initialize
terraform init

# Plan changes
terraform plan

# Apply changes
terraform apply

# Destroy resources
terraform destroy

TODO: expand with state backend configuration, variable definitions, module structure, and cost estimates per provider.

Helm Charts

Helm charts are used for deploying complex Kubernetes applications with templated manifests and configurable values.

Chart Inventory

IDP Platform Charts

Location: /home/lucas/idp/infra/charts/

Aegis VPN Chart

Location: /home/lucas/vpn/charts/aegis/

Kelsey Farm Equipment Chart

Location: /home/lucas/irrigation-parts/deploy/helm/

Usage

# Install a chart
helm install <release-name> <chart-path> -n <namespace> -f values.yaml

# Upgrade a release
helm upgrade <release-name> <chart-path> -n <namespace> -f values.yaml

# List releases
helm list -A

# Check release status
helm status <release-name> -n <namespace>

# View rendered templates (dry run)
helm template <release-name> <chart-path> -f values.yaml

# Rollback to previous revision
helm rollback <release-name> <revision> -n <namespace>

Values Files

Each chart uses values files for environment-specific configuration:

charts/core-platform/
├── Chart.yaml
├── templates/
│   ├── deployment.yaml
│   ├── service.yaml
│   ├── ingress.yaml
│   └── ...
├── values.yaml              # Default values
├── values-dev.yaml          # Development overrides
├── values-staging.yaml      # Staging overrides
└── values-prod.yaml         # Production overrides

Deploy with environment-specific values:

# Development
helm upgrade --install idp-dev ./charts/core-platform \
  -n idp-dev -f values.yaml -f values-dev.yaml

# Production
helm upgrade --install idp ./charts/core-platform \
  -n idp -f values.yaml -f values-prod.yaml

Third-Party Helm Charts

In addition to custom charts, several third-party Helm charts are likely in use for infrastructure services (installed via helm install or ArgoCD):

Note: Verify actual installations with helm list -A. The above are inferred from deployed services.

Kustomize

Kustomize provides environment-specific Kubernetes manifest overlays without modifying base resources.

Projects Using Kustomize

CoreStation (Fire Department SaaS)

/home/lucas/fire-department-saas/deploy/k8s/
├── base/           # Base manifests (deployments, services, configmaps)
├── overlays/
│   ├── dev/        # Development environment overrides
│   └── prod/       # Production environment overrides

# Deploy to dev
kubectl apply -k /home/lucas/fire-department-saas/deploy/k8s/overlays/dev/

# Deploy to prod
kubectl apply -k /home/lucas/fire-department-saas/deploy/k8s/overlays/prod/

Aegis VPN

/home/lucas/vpn/k8s/
├── base/           # Base VPN manifests
├── overlays/
│   ├── megatron/   # MEGATRON-specific config
│   ├── dev/        # Development overrides
│   └── prod/       # Production overrides

Kelsey Farm Equipment

/home/lucas/irrigation-parts/deploy/k8s/
├── base/
├── overlays/
│   ├── demo/       # Demo environment
│   ├── dev/        # Development
│   ├── qa/         # QA testing
│   └── prod/       # Production

Usage

# Preview rendered manifests
kubectl kustomize overlays/dev/

# Apply an overlay
kubectl apply -k overlays/dev/

# Diff against live state
kubectl diff -k overlays/prod/

TODO: expand with common patches, strategic merge patches, and shared component references.

Docker Compose Files

Docker Compose is used for multi-container workloads that run outside the K3s cluster.

Compose Files

ULTRATHINK Research Infrastructure

Location: /home/lucas/ultrathink-infra/docker-compose.yml

Bibliotheca (Book Management)

Location: /home/lucas/bibliotheca/

Full book management stack with 15+ containers. See Bibliotheca and Container Inventory.

Gene Editing Environment

Location: /home/lucas/gene-editing/docker-compose.yml

Development environment for gene editing research tools.

Usage

# Start a stack
cd /path/to/project && docker compose up -d

# View running services
docker compose ps

# View logs
docker compose logs -f

# Stop a stack
docker compose down

# Rebuild and restart
docker compose up -d --build

TODO: expand with environment variable configuration, volume mounts, and network settings per compose file.

SSH Configuration

MEGATRON provides multiple SSH access methods for different use cases: normal SSH, rescue SSH for emergencies, and Forgejo SSH for Git operations.

SSH Daemons

SSH Host Configuration

The following entries are configured in ~/.ssh/config:

Connecting to MEGATRON

From Local Network

# Normal SSH
ssh lucas@megatron

# Or by IP
ssh [email protected]

Via Tailscale (from any machine in the mesh)

ssh [email protected]

Rescue SSH (Emergency)

# From local
ssh -p 2222 lucas@localhost

# Using SSH config alias
ssh megatron-rescue

# From Mac Studio
ssh -p 2222 [email protected]

To Other Machines

# Mac Studio
ssh mac-studio

# MacBook M4
ssh macbook-m4

# Lambda Laptop
ssh tensorbook

Rescue SSH Details

The rescue SSH daemon provides guaranteed access even when the system is under extreme load:

This is critical for recovery scenarios where the main SSH daemon is unresponsive due to:

• High CPU load (e.g., runaway processes)
• Memory exhaustion (OOM conditions)
• Disk I/O saturation
• Kubernetes pod storms

See Rescue SSH for full details.

Forgejo SSH (Port 2223)

Port 2223 is used by Forgejo (self-hosted Git) for SSH-based Git operations:

# Clone a repo via Forgejo SSH
git clone ssh://git@megatron:2223/user/repo.git

SSH Key Management

# List authorized keys
cat ~/.ssh/authorized_keys

# Add a new key
echo "ssh-ed25519 AAAA... user@machine" >> ~/.ssh/authorized_keys

# Test connection
ssh -T git@megatron -p 2223

Kubernetes Access via SSH

After SSH-ing to MEGATRON, kubectl is available for cluster management:

# SSH to MEGATRON, then use kubectl
ssh [email protected]
kubectl get nodes
kubectl get pods -A

For remote kubectl access without SSH, see Kubeconfig.

Security Notes

• Password authentication should be disabled; use key-based auth only
• Root login should be disabled
• The rescue SSH daemon uses the same authentication as the primary daemon
• Forgejo SSH runs as a separate user/process for Git operations
• Tailscale provides an encrypted tunnel; SSH over Tailscale is double-encrypted

Rescue SSH (Port 2222)

The rescue SSH daemon provides guaranteed emergency access to MEGATRON even under extreme system load, OOM conditions, or process storms.

Configuration

When to Use

Use rescue SSH when:

• Normal SSH (port 22) is unresponsive
• System is under extreme CPU load
• OOM killer is active
• K3s pod storms are consuming resources
• You need guaranteed access for emergency maintenance

Connecting

# From local machine
ssh -p 2222 lucas@megatron

# Using SSH config alias
ssh megatron-rescue

# From Mac Studio via Tailscale
ssh -p 2222 [email protected]

Service Management

# Check rescue SSH status
sudo systemctl status sshd-rescue

# Restart rescue SSH
sudo systemctl restart sshd-rescue

# View rescue SSH config
cat /etc/ssh/sshd_config_rescue

How It Works

The rescue SSH daemon runs as a separate sshd process with elevated scheduling priority:

1. Nice -20 ensures it gets CPU time before any normal process
2. OOMScoreAdjust=-1000 prevents the OOM killer from targeting it
3. Realtime I/O scheduling ensures disk access even under I/O saturation
4. Separate port (2222) avoids contention with the primary sshd on port 22

Tailscale Machines

All machines in the Tailscale mesh network. See also Tailscale Networking.

Machine Inventory

Connectivity Check

tailscale status
tailscale ping mac-studio
tailscale ping macbook-m4
tailscale ping tensorbook

Kubeconfig

Kubeconfig provides remote kubectl access to the K3s cluster without SSH-ing to MEGATRON.

Default Location

On MEGATRON, the kubeconfig is automatically available:

# K3s stores it at:
/etc/rancher/k3s/k3s.yaml

# Usually symlinked or copied to:
~/.kube/config

Remote Access

To use kubectl from another machine (e.g., MacBook M4):

1. Copy the kubeconfig from MEGATRON:

scp megatron:/etc/rancher/k3s/k3s.yaml ~/.kube/megatron-config

2. Update the server address to the Tailscale IP:

sed -i 's/127.0.0.1/100.70.25.47/' ~/.kube/megatron-config

3. Use it:

export KUBECONFIG=~/.kube/megatron-config
kubectl get nodes

K3s API Server

TODO: expand with multi-context setup, RBAC configuration, and service account tokens.

Research Overview (ULTRATHINK)

MEGATRON hosts the ULTRATHINK research system -- a large-scale collection of spiking neural network (SNN) and efficient attention research papers, each with its own codebase, experiments, and dedicated Claude Code agent.

Summary

Project Waves

Core Projects (Original)

Wave 2: Extended Architectures (paper26-35)

Wave 3: Advanced Applications (paper36-45)

Later Waves (4-8)

See individual wave pages for full project listings:

• Core Projects
• Wave 2
• Wave 3
• Wave 4: Audio/Signal
• Wave 5: Thorpe TB
• Wave 6: Tree Machines
• Wave 7: DeepSeek/RNN
• Wave 8: Domain Apps

Agent Architecture

Each project has a dedicated Claude Code agent (Sonnet-class) coordinated by a master orchestrator (Opus-class):

orchestrator-agent (Opus)
    |
    +-- rosa-agent (paper/)
    +-- grandmother-agent (paper3/)
    +-- spikegpt-agent (paper4/)
    +-- memory-agent (paper6/)
    +-- imagination-agent (paper7/)
    +-- audio-agent (paper8/)
    |
    +-- spikemoe-agent (paper26/)
    +-- neuromamba-agent (paper27/)
    +-- ... (one agent per paper26-45)
    |
    +-- audio-snn-agent (Wave 4 coordinator, paper46-60)
    +-- thorpe-tb-agent (Wave 5 coordinator, paper61-80)
    +-- tree-snn-agent (Wave 6 coordinator, paper81-95)
    +-- deepseek-snn-agent (Wave 7 coordinator, paper96-110)
    +-- domain-snn-agent (Wave 8 coordinator, paper111-130)

See Agent Architecture for details.

Computing Resources for Research

Review System

Each project undergoes systematic review with scoring across four dimensions:

Review artifacts per project: SUMMARY.md, CLAUDE.md, GAPS.md, PROPOSALS.md

Quick Commands

# Check all project status
for dir in paper paper3 paper4 paper6 paper7 paper8 paper{26..45}; do
  echo "=== $dir ===" && head -5 /home/lucas/Papers/$dir/REVIEW.md 2>/dev/null
done

# Run orchestrator
python /home/lucas/ultrathink_orchestrator.py status
python /home/lucas/ultrathink_orchestrator.py launch
python /home/lucas/ultrathink_orchestrator.py collect

# Run reviews
python /home/lucas/Papers/ultrathink_review_orchestrator.py status
python /home/lucas/Papers/ultrathink_review_orchestrator.py launch

# GPU status for training
nvidia-smi --query-gpu=index,memory.used,utilization.gpu --format=csv

Core Projects

The original ULTRATHINK research projects, focused on foundational spiking neural network and rank-order coding techniques.

Project Summary

ROSA ViT (paper/)

Rank-Order Spiking Attention applied to Vision Transformers. The core technique encodes input features as spike timing patterns (rank-order coding) and computes attention using temporal coincidence detection rather than dot-product similarity.

Grandmother (paper3/)

An invertible autoencoder using rank-order coding, inspired by the "grandmother cell" hypothesis in neuroscience. Features working code implementation.

SpikeGPT (paper4/)

Adapts the GPT architecture to use spike-timing-based attention, achieving O(N) complexity instead of O(N^2) by leveraging temporal sparsity in spike trains.

Memory Engine (paper6/)

Implements Hebbian learning rules for continual learning in spiking networks, enabling memory consolidation without catastrophic forgetting.

Imagination (paper7/)

SNN-based generative model for video, producing temporally coherent frame sequences through spike-based dynamics.

Audio Engine (paper8/)

Applies spiking neural networks to audio processing tasks, leveraging the natural temporal structure of audio signals.

Wave 2: Extended (paper26-35)

Extended SNN architectures exploring new model families (MoE, SSM, BERT, Normalizing Flows, Point Clouds, World Models, Retrieval, NeRF, Codecs, Navigation).

Projects