TLS Certificates

TLS certificates are managed by cert-manager in the Kubernetes cluster. There are currently several certificates in a broken state due to the HTTP-01 challenge solver being incompatible with the Cloudflare tunnel architecture.

Certificate Inventory

Certificate	Namespace	Issuer	Status	Age	Notes
kuber-ca	default	Self-signed	Ready	-	Internal CA
traefik-dashboard-tls	kube-system	letsencrypt-prod	Ready	-	Traefik admin
wildcard-sslip	default	letsencrypt-prod	Ready	-	sslip.io wildcard
harbor-tls	registry	letsencrypt-prod	NOT READY	20 days	HTTP-01 failure
devtron-tls	devtroncd	letsencrypt-prod	NOT READY	20 days	HTTP-01 failure
wildcard-sslip-tls	monitoring	letsencrypt-prod	NOT READY	-	HTTP-01 failure

Issuers

Issuer	Type	Status	Notes
letsencrypt-prod	ClusterIssuer, ACME HTTP-01	Active	BROKEN behind tunnel
letsencrypt-staging	ClusterIssuer, ACME HTTP-01	FALSE	Staging issuer in error state

The HTTP-01 Problem

The current cert-manager configuration uses HTTP-01 ACME challenges. This solver requires Let's Encrypt to make an HTTP request to http://<domain>/.well-known/acme-challenge/<token> to verify domain ownership.

This does not work behind a Cloudflare tunnel because:

Let's Encrypt ──> Cloudflare ──> tunnel ──> Traefik ──> cert-manager solver pod
                       │
                       └── Cloudflare may cache, redirect to HTTPS,
                           or otherwise interfere with the challenge

The Cloudflare tunnel and its "Always Use HTTPS" setting redirects HTTP challenges to HTTPS, causing validation to fail.

Required Fix: DNS-01 Solver

The solution is to migrate from HTTP-01 to DNS-01 challenge solver. DNS-01 verifies domain ownership by creating a TXT record in DNS, which works regardless of tunnel configuration.

Migration Steps

Create a Cloudflare API token with DNS edit permissions
Store the token as a Kubernetes secret
Update the ClusterIssuer to use DNS-01 with Cloudflare
Re-request failed certificates

See Certificate Renewal Runbook for the full procedure.

Target Configuration

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: [email protected]
    privateKeySecretRef:
      name: letsencrypt-prod-key
    solvers:
    - dns01:
        cloudflare:
          apiTokenSecretRef:
            name: cloudflare-api-token
            key: api-token

Affected Services

The following services have broken TLS due to the HTTP-01 issue:

Service	Certificate	Impact
Harbor	harbor-tls	Registry unreachable via HTTPS from external
Devtron	devtron-tls	CI/CD dashboard TLS broken
Monitoring	wildcard-sslip-tls	Monitoring dashboards may show cert warnings

Working Certificates

These certificates were issued before the problem manifested or use a different mechanism:

Certificate	Why It Works
kuber-ca	Self-signed, no ACME needed
traefik-dashboard-tls	Issued when HTTP-01 was working, auto-renewed before expiry
wildcard-sslip	sslip.io domain may resolve differently

Quick Commands

# List all certificates
kubectl get certificates -A

# Check certificate details
kubectl describe certificate harbor-tls -n registry

# Check cert-manager logs
kubectl logs -n cert-manager -l app=cert-manager --tail=100

# Check issuer status
kubectl get clusterissuer

# Force certificate re-request (delete and recreate)
kubectl delete certificate harbor-tls -n registry
# Then re-apply the Certificate resource

# Check ACME challenge status
kubectl get challenges -A
kubectl describe challenge -A

MEGATRON Infrastructure Documentation