Current Issues

Known issues on MEGATRON, combining auto-detected issues and audit findings.

Partially auto-generated: 2026-03-24

Issue Summary

Issue 1: VNC Server Restart Loop (CRITICAL)

Description: The VNC server is in an infinite restart loop with 617,000+ restart attempts. It appears to have no authentication configured, making it a security vulnerability if accessible on any network interface.

Impact:

• Security risk: Unauthenticated remote desktop access
• Resource waste: Continuous process spawning
• Log spam: Filling system logs with restart attempts

Suggested Fix:

# Immediately disable the VNC service
sudo systemctl stop 'vncserver@*.service'
sudo systemctl disable 'vncserver@*.service'

# Find and kill any running VNC processes
ps aux | grep -i vnc
sudo kill <pid>

# Verify no VNC ports are listening
ss -tlnp | grep 590

# If VNC is needed, reconfigure WITH authentication before re-enabling

Issue 2: AlertManager at 203% CPU (CRITICAL)

Description: AlertManager has accumulated 24,000+ hours of CPU time and is consuming 203% CPU (2+ cores). This suggests a memory leak, alert routing loop, or excessive alert volume.

Impact:

• Wasting 2+ CPU cores continuously
• Monitoring and alerting may be unreliable
• Could contribute to overall system load

Suggested Fix:

# Check AlertManager pod
kubectl get pods -n monitoring -l app.kubernetes.io/name=alertmanager

# Check resource usage
kubectl top pod -n monitoring

# Restart AlertManager
kubectl rollout restart statefulset/alertmanager -n monitoring

# If issue persists, check for alert routing loops
kubectl get configmap -n monitoring -o yaml | grep -A 20 alertmanager

Issue 3: TLS Certificates NOT READY (HIGH)

Four certificates are in NOT READY state:

Root Cause: ClusterIssuer uses HTTP-01 ACME challenges which cannot complete behind the Cloudflare tunnel.

Fix: Migrate to DNS-01 solver. See Certificate Renewal.

Issue 4: cert-manager HTTP-01 Broken Behind Tunnel (HIGH)

Description: The letsencrypt-prod ClusterIssuer uses HTTP-01 challenges. Cloudflare's "Always Use HTTPS" redirects the challenge request, causing ACME validation to fail.

Impact:

• No new certificates can be issued
• Existing certificates will fail to auto-renew at expiry
• All four NOT READY certs stem from this root cause

Suggested Fix: Switch to DNS-01 solver with Cloudflare API token. Detailed steps in Certificate Renewal.

Issue 5: SMB Share 90% Full (MEDIUM)

Description: The network SMB share at //10.0.0.41/Main (mounted as /mnt/sunny) is 90% full (79T/88T), with ~8.2T remaining.

Impact: Risk of share filling completely, causing write failures for any workloads using the share.

Suggested Fix:

# Audit large files
du -sh /mnt/sunny/*/ | sort -rh | head -20

# Find old, large files
find /mnt/sunny -atime +180 -size +1G -ls 2>/dev/null | head -30

# Consider archiving or expanding the NAS

Issue 6: NFS Mounts Down (MEDIUM)

Description: NFS mounts at /mnt/cloudy and /mnt/cloudy-rw are unreachable -- the NFS host is down.

Impact: Stale NFS mounts can cause process hangs when accessed (e.g., ls, df, or du commands may block indefinitely).

Suggested Fix:

# Lazy unmount to prevent hangs
sudo umount -l /mnt/cloudy
sudo umount -l /mnt/cloudy-rw

# If NFS is no longer needed, comment out in /etc/fstab
# Otherwise, restore the NFS host and remount

Issue 7: High Swap Usage (MEDIUM)

Description: Swap is at 62-68% (approximately 20-21 GiB of 31 GiB used), despite ~206 GiB of RAM being available. This is a result of 37 days of uptime.

Suggested Fix:

# Reclaim swap during a maintenance window
sudo swapoff -a && sudo swapon -a

# Monitor afterwards
free -h

Issue 8: Failed Systemd Services (MEDIUM)

motd-news.service  loaded failed failed Message of the Day
wol-enable.service loaded failed failed Enable Wake-on-LAN on all Ethernet interfaces

Impact: None functional. Causes systemctl status to report DEGRADED.

Fix:

sudo systemctl reset-failed motd-news.service
sudo systemctl reset-failed wol-enable.service

Issue 9: megatron-docs Pod CrashLoopBackOff (MEDIUM)

Impact: Documentation site is unavailable.

Suggested Fix:

# Check logs for root cause
kubectl logs -n docs megatron-docs-5dd4d46dbf-x6gbj --previous

# Check events
kubectl describe pod -n docs megatron-docs-5dd4d46dbf-x6gbj

# Fix the issue and restart
kubectl rollout restart deployment/megatron-docs -n docs

Issue 10: bibliotheca-categorizer Restarting (LOW)

bibliotheca-categorizer   Restarting (exit 0)

Impact: Book categorization functionality is broken.

Suggested Fix:

docker logs bibliotheca-categorizer --tail=100
docker inspect bibliotheca-categorizer --format='{{.State.ExitCode}}'
# Fix underlying issue, then: docker restart bibliotheca-categorizer

Issue 11: Let's Encrypt Staging Issuer FALSE (LOW)

Impact: Cannot test certificate issuance with the staging CA. Will be resolved as part of the DNS-01 migration.